Embodied Conversational Agents
Embodied Conversational Agents
After having spent ten years studying verbal and non-verbal aspects of human communication through microanalysis of videotaped data (starting as a graduate student) I began to bring my knowledge of human conversation to the design of computational systems. I built the very first embodied conversational agent as NSF visiting faculty at the University of Pennsylvania, in the Center for Human Modeling and Simulation, working with their faculty and graduate students. Previously professional animators manually synthesized conversational behaviors for animated figures based on their intuitions, and they "hard-wired" facial expressions and gestures. Although the intuitions of such animation artists are excellent, and hard-wiring is a satisfactory approach to regular animation, their approach cannot be extended to the generation of these behaviors in systems running independently of a human designer. My work introduced the first rule-governed, autonomous generation of verbal and non-verbal conversational behaviors in animated characters. Secondly, previous conversational interfaces or dialogue systems concentrated on the content of the conversation -- the statements and questions that advance the discourse. My work introduced for the first time a conversational agent capable of generating and understanding both those propositional components and synchronized interactional components such as back-channel speech, gestures and facial expressions. These interactional components are crucial to the construction of what I have called the 'conversational envelope'.
In the newest Embodied Conversational Agent project, Rea, we are working with Matthew Stone to integrate a natural language generation engine (SPUD) and are now generating hand gestures and sentences as one single process, in real time. This work addresses the challenges of specifying an underlying representation of discourse that is capable of driving generation of several modalities. We have also integrated into Rea the ability to engage in social chit-chat as a way of reducing interpersonal distance and increasing trust between the user and the system.
We have also integrated the foundations of the Embodied Conversational Agent work into the design of a 3D graphical online world (BodyChat), an interactive kiosk (MACK), and an animator’s tool, BEAT, which allows animators to input typed text that they wish to be spoken by an animated human figure, and to obtain as output appropriate and synchronized nonverbal behaviors and synthesized speech in a form that can be sent to a number of different animation systems.
In studies of these systems at the Gesture and Narrative Language Group, we have shown that these autonomously-generated conversational signals are more important to user satisfaction and efficiency than some simple facial displays of emotion (Cassell & Thorisson 1998), and than the ability to directly manipulate the interface (Cassell & Vilhjalmsson, 1998). A recent experiment demonstrated that Rea’s social chit-chat significantly increases the trust that people have in the system, particularly for extroverted users.
For publications about the Embodied Conversational Agents, see publications.
Press clippings about this work may be seen here. By the way,
Rea has now made her debut on the world stage as a permanent exhibit at the
Deutsche Telekom Future Labs.
Children and Technology: Story-Listening Systems
Technology for Empowerment and Voice
What is an Embodied Conversational Agent? It is a lifesize virtual human capable of carrying on conversations with humans by both understanding and producing speech, hand gesture and facial expressions. Embodied Conversational Agents are a type of multimodal interface where the modalities are the natural modalities of human conversation: speech, facial displays, hand gestures, body stance. They are a type of software agent insofar as they exist to do the bidding of their human users, or to represent their human users in a computational environment. They are a type of dialogue system where both verbal and non-verbal devices advance and regulate the dialogue between the user and the computer. In the Embodied Conversational Agent, the visual dimension of interacting with a cartoon character on a screen (rather than a keyboard) is intrinsic to its function. The graphics are not just pretty pictures, but visual displays of conversation, in the same way that the face and hands serve that function in face-to-face conversation among humans.
With this goal, my students and I build Story Listening Systems that listen to and respond appropriately to children's stories. What sets this work apart from previous Eliza-like systems that respond to users, or current CD-Roms that tell stories to children, is the fact that our systems encourage childen’s active exploration of narrative, linguistic creativity and verbal play. In this sense, the work fits into the long tradition of constructionist research at the Media Lab. Our contribution is to extend the notions of child as technology designer to systems that explore story, self-concept, and linguistic creativity. In addition, the majority of our research is embedded into electronic toys, and not desktop computers, supporting children's full-bodied, collaborative, social play-based learning.
In some of our Story Listening Systems children tell a story in order to elicit another child’s story on a similar topic (Victorian Laptop, Storymat). In other systems, children are invited to build their own storytellers, so that they may play the role both of storyteller and storylistener (SAGE, WISE, Sabio). In still others, children are invited to edit and revise their stories and to make up entire story books with the story-listening system as ally (Rosebud, Eddie Edit, TellTale). Finally, more recently we have actually built a virtual playmate for children who is able to attend to children’s stories, and tell back relevant stories in return. In this project, called Sam, the Castlemate, children can even pass figurines back and forth from the real to the virtual world.
We are currently gearing our research to the role that these Story Listening systems can play in promoting emergent literacy skills - children’s first steps into reading and writing. Additional features that distinguish this research are the importance placed on communities of story tellers and collaborative play among children, and the use of stuffed animals as interfaces to the system. With respect to this latter point, my students and I pioneered the use of small stuffed animals as interface, soft interactive devices with which the user is already familiar, to serve as ally or interlocutor.
Our storytelling systems have been used by children around the world. Renga is a permanent exhibit in the science museum of Singapore, and many of the other systems have been used by schools around the world and in several industry research labs. WISE has taken on a new function, teaching industry executives about the power and uses of storytelling. Check out some of the storytellers created in those workshops here.
Publications about our storytelling systems work maybe found here. And here are some press clippings about the work.
Children and Technology: Story-Listening Systems
The discussion of the role of computational technology in children’s development has become increasingly polarized over the last year or so. On the one hand we find a frantic push to place computers and internet access into all U.S. schools, and on the other hand, a frantic push-back to place a “moratorium” on children’s access to computers. Clearly the answer lies at neither end of this long spectrum, and a careful review of existent studies shows a number of benefits, a palmful of harmful effects, and a plethora of unknowns. Based on my earlier (read, Developmental Psychology days) investigations into children's developing competence in narrative structures, and the important role that competence plays in their cognitive and social development, my response is that the answer lies in responsible and developmentally-informed design and evaluation of technology that specifically targets the needs and unique abilities of young children. “Computer technology” need not be incompatible with play-based learning, physical activity, active engagement, and social interaction, those features of childhood whose loss computer phobists decry.
people
research courses @ctivism publications
press
MIT Media Laboratory E15-315 20 Ames St Cambridge MA 617.253.4899
617.258.6264 [fax] justine@media.mit.edu