Henry Lieberman
MIT Media Lab
I believe that an important direction for the future is to explore how we could build human-computer interaction around the notion of advice as the primary means of communication. The computer can both give advice to the user and take advice from the user.
The idea of the computer relating to people as both advisor and advisee is becoming increasingly important as applications get more and more advanced. Sophisticated media interfaces are accruing more and more functionality, and soon the bottleneck will not be how to get our computers to do more for us, but how to let them know what we would like them to do, and how we would like them to do it. Intelligent software will have the capability to do more and more problem solving for us, but there is also the danger that the machine may take unwanted actions or we may distance ourselves too far from the problem solving process. The best solution may sometimes be to set up the computer in an advisory capacity, leaving critical decisions and top-level control to the human.
Too much of today's human-computer interaction is couched in terms of commands. This is evidenced by the fact that the word "command" has almost come to mean any input whatsoever to a computer. The human "commands" the computer through icons and menu operations. Traditional computer interfaces tend to leave users feeling "commanded" by their rigid and inflexible nature, which accounts for much of the hostility the general public feels towards computers.
Advice is softer. It needn't be as exact as commands. It need not specify everything the advisee must do, it need only address some aspect or part of the advisee's behavior. It can suggest action or modify action, rather than specify action. It need not solve the whole problem, only part. It does not need to be given or received in a strict sequential order. It can critique past behavior or serve as guidelines for future behavior. In these qualities advice differs from what we ordinarily think of as "programming" computers.
Advice is ongoing. Although advice may be in the form of a conversation between the advisor and advisee, it assumes that both the advisor and advisee are active simultaneously. This is quite unlike traditional query-response computer interfaces, that assume that either the human is working or the computer is working, but not both at the same time. Advice assumes that both the advisor and the advisee will continue to operate even in the absence of the advice, and the goal of the advice is improvement of the advisee's behavior.
The necessity of advice in computer interfaces was recognized as early as 1959, when John McCarthy proposed as a goal a machine which could act as an Advice Taker [McCarthy 59]. He defined "common sense" as the knowledge that would enable a human or machine to meaningfully accept advice. McCarthy viewed advice-taking as a stepping-stone toward a fully autonomous artificial intelligence, but advice-taking interfaces have value of their own that is yet to be fully appreciated.
Intelligent software agents have recently become a popular movement for extending the human interface beyond traditional direct manipulation interfaces. Not all agents are advice-givers or advice-takers, but the role of advice as a means of communication between an agent and a user is an central component of agent competence.
In many agent projects, the user is encouraged to think of the agent as serving the role of a butler or secretary [Negroponte 95]. Someone in such a role can often provide services that are personalized to an individual's needs and desires, and they are often most appreciated when the agent can provide them without even being asked. When the agent is simply providing a service to the user, the issue of advice may not come into play at all.
But the user's relationship with the agent consists of more than having the agent do things for the user. First of all, if the agent is to act according to the user's desires and preferences, the agent must first learn what it is that the user wants, likes and dislikes. The most natural way for the user to express desires and preferences to the agent is in the form of advice to the agent. As the agent performs services [or fails to in a situation where it really should], the user can critique the performance of the agent, acting as advice for future behavior. Finally the user may turn to the agent for advice in deciding what action to take him or herself, which is one of the ways in which agents can be of the most helpful to the user.
Advice-giving and advice-taking agents can be a way of responding to critics who oppose the whole idea of agents. They argue that humans are abdicating responsibility to software agents, and that there is a danger that the agent will autonomously take harmful actions. Advice gives the user an avenue of control without being burdensome, and enables the user to benefit from the agent's work while retaining responsibility for decisions.
What would advice really look like in a computer interface?
I'll illustrate two examples from my own work, one in which the computer gives advice, another where it takes advice from the user. In both cases, the original goal of the project was not solely to explore the notion of advice, but as these projects developed, they convinced me of the importance of the advisory nature of the interaction.
One way that an agent can advise the user is for the agent to try to accomplish the same task as it perceives the user to be doing. It can make its attempt while the user is idle, working on the same task, or perhaps another task. When the agent feels it has accomplished something that might interest the user, it gives the benefit of its efforts to him or her as advice. It is up to the user to accept, reject, or ignore the advice.
Letizia is a user interface agent that assists a user browsing the World Wide Web [Lieberman 97]. It treats browsing as a cooperative venture between the human user and the agent. As the human is surfing [in the left window in the illustration], the agent also surfs, and continuously shows its advice to the user in the windows on the right.
Letizia's advice is
* Personalized. It watches the user's actions and compiles an interest
profile, which it uses to filter its search.
* Ongoing. Letizia operates continuously, making use of the user's
idle time, and displays a continuous, "channel surfing" display
of its recommendations. It reacts immediately to changes in the user's
browsing behavior.
* Non-coercive. The user may, at any moment, choose to browse the
pages that Letizia recommends, or simply ignore it.
Right now, Letizia doesn't really accept explicit advice from the user, except in the form of observing the user's actions and making decisions based on them. However, explicit advice could be incorporated in a number of ways.
Users could give both feedback on the specific choices, or criteria for determining those choices, all in real time as the program is running. Speech [see below] would also be a particularly useful modality for accepting advice, since the user is otherwise occupied by the browsing process. Users could supply commentary on both their own behavior and that of the agent, and the relationship between them. Users could also explicitly advise and modify the profile that Letizia forms of them.
Mondrian [Lieberman 95] is a learning agent for an object-oriented graphical editor, that learns new procedures demonstrated by the user by example. Mondrian takes advice from the user about how to learn new graphical procedures. Since the domain is graphical, often the advice comes in contextual forms.
Advice can be demonstrative. In the beginning of the demonstration, specific objects can be designated as being examples, and that action advises Mondrian that any further operations involving these examples may be generalized in the future.
Advice can be graphical. The user draws a diagram that shows Mondrian about part-whole relationships. In my experiments in a technical procedure domain, I allow the user to annotate selected frames of video by outlining images of parts, and grouping them into part-whole relationships. This means that when Mondrian records an action involving that part, it is described in terms of its relationship in the object's structure.
Advice can be verbal. A speech recognition system is trained to recognize a set of visual relations, constraints, and predicates that modify the actions the user is taking or objects that the user is manipulating. When the user gives verbal advice, that changes how Mondrian is interpreting the user's actions [Stoehr and Lieberman 95].
This gives the interaction a natural "show and tell" feel. In many speech applications where spoken commands simply replace typed or mouse commands, speech is as a medium for giving advice, since the hands and eyes are already occupied with interacting with the graphical interface.
Mondrian represents a class of interfaces where advice is given by example. The user uses concrete, visual examples to explain what is expected, together with actions that tell the system how to interpret the examples. Using examples is often easier for people than preparing advice in the abstract, and less prone to error. An outstanding problem is achieving some sort of integration between the kind of advice-giving interfaces represented by Letizia and the kind of advice-taking interfaces represented by Mondrian.
Letizia and Mondrian are two examples of the current state of the art in advice-giving and advice-taking interfaces. Current trends in interface development point towards the increasing need for such interfaces and some of the directions in which they might develop.
The vastness and lack of organization of the Internet cries out for interfaces that advise users on how to look for, provide and use information. Letizia is just one example of a wide class of such advisors. In the future, people will not be content with simply browsing and consuming information as in today's "surfing". Net-based information will be integrated in with other software applications that process information, use it for problem solving, transcode it, and redistribute it. All these processes can be supported by advisory agents, and benefit from the ability to accept advice.
Many people currently look to the net for advice, by searching for information "content" to advise their activities, either for work or play. People also use the net to make connections with people who may be able to serve as advisors, or to offer advice to others. Advice-based interfaces could be developed to assist this process.
A primary function of personal information devices such as PDAs, wearable computers, and computers embedded in household and office devices is that they will serve as reminders to people in their daily activities. In this role they are acting as advice givers, and care must be taken in designing these interfaces so that the user perceives them as helpful rather than annoying. Many of these devices will be so small that they will be unable to provide traditional display and mouse interfaces, and have to accept advice from the user primarily by speech recognition, and automatic observation and sensing of the user's activities. Understanding interfaces that give and take advice will be essential in integrating these devices into our daily lives.
A large part of making users feel that their computers are acting intelligently is to make the communication between people and computers more like the interaction between people themselves. This involves not only direct commands given via speech recognition, but also indirect advice such as that accepted by Mondrian. Clues to the user's desires can be given by tone of voice, and gestures or facial expressions recognized by a vision system. These secondary inputs act to advise the primary interaction specified by the surface commands. Theories of discourse, speech acts, and multi-agent communication can inform the advice-taking process so as to make use of this implicit advice in the human-computer dialog. This can streamline the communication process, requiring less explicit communication from the user, and make the computer seem more responsive and personalized.
An important component of visual design in interactive interfaces, publishing on the Web, and other contexts is the relation between visual and textual information. Often, the visual information serves as advice to the reader in how to interpret textual information, or vice versa. Problem solving expertise in visual design is often communicated through the use of concrete visual examples that serve as advice about how to visually represent information [Lieberman 95b, 96].
Seymour Papert has long advocated "teaching the computer" through programming as a path for people, especially children, to become better problem solvers themselves. With more intelligent learning and programming environments, this can be extended to "advising the computer". Advising has the advantage that it is less dependent upon the precision of instruction, and the exact of order of sequencing of events -- precisely those aspects of learning programming that are most troublesome to beginners.
While there have been many specific examples of computer applications acting as advisors and advisees, the central role of advice in human-computer interaction has not yet been fully appreciated. Artificial intelligence may contribute common-sense knowledge and learning algorithms, but traditional AI has been more focused on the idea of a computer as a stand-alone problem solver. Human interface research offers more sensitivity to user needs, but it currently seems stuck on the direct-manipulation paradigm, which needs to be transformed into an advisor-advisee relationship. It is this relationship that we need to build the foundation of the digital future.
Take my word for it.
Henry Lieberman, A Demonstrational Interface for Recording Technical Procedures by Annotation of Videotaped Examples , International Journal of Human-Computer Studies, Vol. 43, pp. 383-417, 1995.
Henry Lieberman, The Visual Language of Experts in Graphic Design , IEEE Symposium on Visual Languages, Darmstadt, Germany, September 1995.
Henry Lieberman, Intelligent Graphics : A New Paradigm, Communications of the ACM, August 1996, [Special Issue on New Paradigms for Using Computers].
Henry Lieberman, Autonomous Interface Agents , CHI 97, Atlanta, April 1997.
John McCarthy, Programs with Common Sense , in Formalization of Common Sense, Papers by John McCarthy, V. Lifschitz, ed., Ablex, 1990.
Nicholas Negroponte,
Henry Lieberman, Hearing Aid : Adding Verbal Hints to a Learning Interface, ACM Multimedia Conference, San Francisco, November 1995.