Imposter Syndrome is taking hold as the deadline approaches. December 6th is judgement day. http://fdg2014.org/
I have interviewed a decent number of people now, and the graphs have not converged as expected. Half of the people I spoke to didn’t have graphs to give me. The following is a summarization of what I learned from each interviewed person and how it’s shaped the overall paper. My goal by the end of this post, or at least in the very near future, is to find some commonality between all those I’ve talked to which can be ground out in a concrete suggestion for improvement. That suggestion must then be tested by different teams and results given.
Larry (informally) and Karen: ABL
Some of this was apparent in the last post or two. I got a graph and a discussion from Karen that detailed the scenario-authoring pipeline where the largest and clearest time sinks came from the graphics side of authoring. Having all assets, it would take roughly a day of tweaking animation and code to get a specific low-level behavior, such as picking up an object on a table, to work and look well. Larry’s feedback was more on the ABL-level authoring: during that day of time, there would be lots of weird debugging tactics that would go into sucking up attention. Besides the process from writing code to seeing characters performing the action is so long, there is lots of room for errors to halt the process. Lastly, as Karen put it, there needs to be interfacing with higher-ups to make sure the behavior performs up to spec. It takes a lot of back-and-forth to figure out just what the behavior should do, how it should look, and (once it is made to do that) was it really what we wanted in the first place?
In summary, improvements can be made on all levels of the authoring process identified through delays between other team members and in performing individual tasks.
Martin Van Velsen: Authoring By Demonstration
Martin introduced me to a field of authoring I was unfamiliar with: authoring tutors. Human A authors a tutor to teach Human B some subject matter, be it math or some other easily verifiable and highly computational task. He spoke of what he’s working on now: authoring by doing! Human A sits at a tool and solves a math problem while the program tracks these “correct” paths to solving the problem. Human A then explores alternative paths of easy misunderstandings, such as adding fractions by adding numerators and denominators at the same time (1/2 + 1/2 = 2/4). The tool is gathering data of correct and incorrect methods, and maybe would then be able to extrapolate the methods to other fractions or data patterns.
An interesting approach that would require more thought to apply to our domain. However, his remarks on tools and visualizations therein were very useful. For example, accepting that very few (probably no one, other than you) will become experts at your system. (I see this as Andrew being the expert at ABL, and everyone else putters around doing the best they can). Deep visualizations that show a lot, or all, of some data will be indecipherable and thus useless to most users. “Wizards” to help author certain things over and over get repetitive and insulting: once people are trained to do something, they should be able to do that something with a minimal amount of clicks.
An aside: tools like Narratoria were similar to ours, but they authored for cut scenes instead of richly interactive experiences. As such, their script + movie + brief dialogue-tree-like interaction authoring tools were sufficient.
Samuel and Paulo: FAtiMa
FAtiMA’s strengths lie primarily in its focus on emotion, grounding in research, modularity, and its complete isolation from the enactment side of the experience. I wasn’t able to extract a specific graph from Samuel, but I did get references to other projects and an overview of some of the bigger projects that FAtiMA has been used in. Most recently, the focus of his research has involved a new module about “Social Importance” and makes FAtiMA’s goals feel less utilitarian. Every action considers the social context in which it exists.
One of the more amusing observations is that FAtiMA’s strengths are also its weaknesses. It is difficult to conceptualize a model that doesn’t include emotional responses (see Paulo’s thing later), so it feels like you’re strong-arming the system into doing something it wasn’t meant to do. Some of the research that FAtiMA is grounded in feels inflexible (the plans that Samuel feel do not always capture the decision-making process well if you do not have a concrete end state). And being isolated from the enactment means that physical properties related to behaviors outside of FAtiMA are hard to handle (ex: FAtiMA has no model of eye gaze). The biggest authoring challenge by far, however, is number tweaking. For example, assume every action that an agent can take has two forms: formal and informal ways of expressing the action. “Number tweaking” refers to testing and discovering the threshold of some numeric variable that determines at which point each agent does one or another.
Paulo and I walked through two FAtiMA scenarios that reinforced the strengths and weaknesses talked about above. ABL would likely become easier to author for if we had libraries of pre-programmed actions that the user grounded out in performance. Likewise, FAtiMA would likely be much easier to use with loads of pre-calibrated modules representing different social considerations and theories (OCC for emotions, social considerations, eye gaze, etc). But those are pie-in-the-sky ideas.
Swen: BOD / POSH
Simplicity is key. Swen and a different Czech team have been using that as their motto to make a decision-making system. Swen and his adviser have a simple multi-step process Behavior-Oriented Design (that I’ll describe in detail shortly) between the programmer and designer to craft the tree. Their tree structure is very controlled, and they have a visual editor for designers to use which helps in making the structure.
BOD involves the following steps: Designer (D) + Programmer (P) work out every action they want the agent to perform in an abstract way. D + P then work together to distill those abstract ideas into Actions and Sensors, what will ultimately become function names. The P then creates the list of functions in code, and D then uses an editor to make the behavior tree. Any hiccoughs that occur in the process (ex: forgetting an action or sensor) have the D + P go back and make revisions as necessary in an iterative process. If any of the behaviors were left as stubs or disconnected from the final system, the P will fill out the functions as necessary once the graph is complete.
The work pipeline is smooth, explicit, flexible, beautiful…! It’s as complicated as it needs to be for the system. But how can we abstract the process for more complicated systems? Is it even possible?
Ugh, took way too long to write this and to share my notes. Sorry! Follow-up posts to come shortly.