4.889 Intelligent Interface Software Design Workshop

Project 1: Learning in Graphical Editors

About the project descriptions

These projects are intended to be open-ended. The project description, tasks, and hints for their solution are intended to be suggestions to help you get started exploring the issues we are concerned about rather than ironclad marching orders. Feel free to modify the task statement, think of new features, or write your own systems instead of using the ones provided.

You may work alone, or in groups of 2 or 3. If you do team up, try to find someone who complements your skills -- if you are not so experienced in Lisp, team up with someone who is, and vice versa.

Try to be a reflective practicioner while you are working on the problem. Keep a "design notebook" [either electronic or hardcopy] while you are working on the problem and jot down:

* Problems you have while you are working on it, even if you can't deal with these problems immediately [or ever]. Note especially interesting bugs that arise.

* Suggestions for neat features, interesting variants of the problem, etc. even if you don't have time to work on them immediately.

* Interesting design or philosophical issues raised.

These can all serve as fuel for class discussion.

Objective

The object of this project is to give you experience with programming direct-manipulation graphical editors, and to experiment with machine learning techniques in the context of interactive interfaces.. You will be making extensions to Henry Lieberman's Mondian system.

We will start with a relatively simple exercise to make sure you understand the basics of interface programming, and the details of the code provided, and to give you some experience making an extension to an interactive graphical interface. This will be followed by more substantive extensions to the interface. The last exercise implements a significant extension to Mondrian's learning capability, to give you experience with learning techniques.

Task: Add a new graphical operation to Mondrian's palette

The task will be to add a new graphical operation to Mondrian's repertoire. Look to your favorite drawing program, such as Canvas or Illustrator for suggestions of features to implement, or think of your own.

A suggestion is to add a new graphical primitive, say an operation that draws an oval. An important heuristic in any programming, and especially interface programming is to find some code that "almost kinda" does what you want and use it as a model for the functionality you want to implement. Mondrian's existing Rectangle operation can serve as a model. When you need to do something, try to look at how Mondrian's Rectangle operation accomplishes a similar task.

Steps

To implement your new operation, you will need to

* Make a domino icon for your new operation

* Make a command object for your new operation

* Define an action routine for your new operation

The action routine

Start with defining the action routine. You can test it out from the Lisp Listener first, then "hook it up" to the interface.

Define an oval object like Mondrian's rectangle object. The object definition is in the file Suit:Suit-Defs, and the methods in the file Suit:Graphic. Define filled-oval. The graphics function for drawing an oval is called paint-oval and works like paint-rect.

Look at how the rectangle-action function in the file Mondrian: Mondrian-Editor works.

Test out the action routine from the Listener. To do this, you need a canvas object. The variable ma is bound to the application object for Mondrian when it starts up. (canvas ma) will give you the current canvas.

The tool object

In Mondrian:Mondrian-Defs, the object rectangle-tool is defined. This object represents the tool in Mondrian's palette. Define an analogous one for your oval operation.

The resource-file and resource-name refer to a resource file where you will keep the domino icon.

You must install the tool on Mondrian's palette. Call the function install-new-tool on your tool object and (palette ma).

The domino icon

Using your favorite draw ing program, such as Canvas or MacDraw, draw an icon for your operation. Copy it from the drawing program, and paste it into a resource file using the Resource Editor. Consult a Mac expert if you do not know how to use the Resource Editor.

Task: Implement a more challenging graphical operation

Choose another graphical operation to implement. The Oval operation was easy because it took the same arguments as the rectangle operation. This time, choose one that takes different kinds of arguments. This is more challenging; if you have too much trouble, choose something simpler, but try to choose something that does not rely on the built-in argument routines.

Suggestion: Implement the Resize operation

Here's a suggestion: Implement the Resize operation, for which you will see an icon on the palette [but that presently does not do anything]. \What arguments should resize take?

You will need to learn about Argument objects and about Mouse-Tracker objects. The definitions are in Suit:Defs. You will need to implement a new kind of Mouse-Tracker object.

The drag loop

In the file Suit:Track, you can see how the mouse tracker objects work. The drag function implements the loop that handles the change of mouse position when the mouse button is held down.

It is defined in terms of three functions,

* start-tracking

* move tracker, called each time the mouse moves

* stop-tracking

You can make a tracker that inherits from Mouse-Tracker and redefines any one or all of these without redefining others.

Look in the file Suit:Easel to learn how the system collects arguments and invokes commands, if necessary.

Task: Improve Mondrian's user feedback

The object of this task is to improve the interactive feedback that Mondrian gives to the user. An important interface principle is to show feedback to the user as soon as possible.

Anticipation feedback

Right now, when you start defining a new command, Mondrian creates a domino icon for the new operation . This is a kind of anticipation feedback because it shows you "what things will look like" when you complete the operation [you will have a new operation on the palette]. The left panel of the domino shows the state at the outset of the command definition process. The right panel initially shows the question mark, and when the command is completed, the final state will be filled in.

Why wait to fill in the final state? Instead, each tentative final state could be shown after each command. This willanticipate what the domino icon will look like if the user decides to terminate the definition at that moment.

Change Mondrian so that after each command, it fills the right panel of the domino with a miniature copy of the current state immediately after each command.

Steps

You will need to

* Remove the "?" that is currently drawn in the panel.

* Figure out how to insert some code after each command is executed.

* Figure out how to draw a miniature copy of thescreen state.

What code in Mondrian does something similar that you can take as a model?

* The code that draws the parts of the domino icons

* The code for the storyboard

* You will find the code in Mondrian:New-Command

To think about

What are the advantages and disadvantages of this kind of feedback? Is it too slow or disruptive [this may depend on the speed of the machine!]. Are there other ways you might imagine improving Mondrian's feedback while a command is being defined?

Other feedback improvements

Mondrian has a simple template-based natural language generator that it uses to generate speech output while you are performing commands in the graphical editor. You will find that code in Mondrian:Narrate.

Caption the storyboard

The task is to add printed natural language text to storyboard the storyboard. Instrument the function that printsout the storyboard panel to call the narrate functions on each expression in the code to generate a caption for each panel. You may have to do a little processing to "clean up" the appearance of the text [punctuation, etc.]

A storyboard stepper

Implement a simple "stepper" for the storyboard stepper that animates frames sequentially instead of shows them side-by-side. Make "tape recorder" style controls that allow you to go forward and backward at various speeds.

To think about

Tradeoffs between time and space are common in interface design decisions. Think about the pros and cons of showing the frames all at once [using up space] or sequentially [using up time].

Task: Extend Mondrian's learning capability

The object of this task is to make a significant extension to Mondrian's learning capability. Mondrian currently learns by recording user actions, and generalizing them according to built-in knowledge about geometric relations, indications from the user about what objects are generalizable, and possibly other domain-specific knowledge.

Similarity-based learning

In contrast to this action-based learning, another major approach in programming by example systems is similarity-based learning. In similarity based learning the system tries to match user actions to previously recorded actions, and , where it can find a match, proposes a generalization. Another choice in similarity-based learning is to try to match states of the interface rather than the actions taken to arrive at the current state.

The task is to add similarity based learning to Mondrian. The idea is that you will add an "agent" that looks for patterns in the history of recorded actions after each action is executed. If it finds a pattern, it can automatically perform the next action in the pattern.

Steps

To do this you will need to:

* Write a matcher that looks for patterns in the command history. This matcher may be heuristic in that it will not always guess correctly.

* Provide some way of invoking the generalized command generated by the matcher. This can be automatic or explicit.

A test case

Here's one test for what the learner should be able to do. If you're defining a command that takes the following arguments:

and you move the two lower leftmost rectangles so that they touch the upper bar,

then the system should be able to infer that the other two should be moved up to touch the bar.

Access to the command history

When Mondrian is defining a command, you can find the object representing the command in (command-being-defined ma). Asking this object for its command-history returns the list of commands.

Experiment with this a little by defining a new command in Mondrian, and querying this after every graphical operation, so you see what it does.

Write a matcher

Now, we will write a matcher that searches for the longest matching pattern in the command history list, and predicts the next instance of the pattern.

Our simple matcher will allow us to find patterns that involve:

* Using up the arguments to commands [first, then second, third argument, etc.]

* Linear displacements [such as evenly spaced objects].

* Iterating through the elements of a group given as the argument to a command.

Start by writing a simple matcher that just detects patterns using Lisp's equal in lists. We only care about patterns in the tail end of a list.

(find-pattern '(a b c d c d)) = (c d)

(find-pattern '(a b c d e c d e)) = (c d e)

Read the chapters in Winston's Lisp book on pattern matching if you want to learn more about how to write matchers.

Now, generalize it to predict the next sequence in the pattern. Start by just predicting from number sequences, but leave room to add more kinds of prediction later.

(find-pattern '(a b c d 1 c d 3)) = (c d 5)

! [One subtlety in writing a matcher for Mondrian's code is that most of the operations generated by Mondrian use the last argument to the function to name the object generated by the function [rectangle-action, line-action, group-action, move-action, etc]. You must make the matcher ignore the last argument for those functions. Expressions that differ only in the last argument should be considered to match. I may supply some code that will do this, or you can determine experimentally where it is ncessary.]

Test it out by invoking your matcher explicitly from the listener, and calling

(eval-history-command

(canvas ma)

(find-pattern

(command-history (command-being-defined ma))

after each graphical operation.

Invocation

Now, hook it up to the interface. There are several ways in which you might do this.

Invocation can either be done automatically [the generalized command is simply executed whever a pattern is found], or you can provide an operation that explicitly asks the agent to guess what the pattern is.. It is easier to make invocation explicit, so that you don't have to worry about stop conditions. Make an icon called Again, that instructs the system to "do the last recognizable pattern over again".

Things to think about

What other kinds of generalizations are plausible? What about stop conditions if you were to make invocation automatic? What about multiple examples? What if you tried to learn by matching the states of the graphical objects instead of the actions? How could you give the user feedback about what the matcher generalized? Could you make the matcher user-extensibe?

To learn more

If you would like to read about some programming by example systems that use similarity-based learning,

* Metamouse, by David Maulsby, SigGraph '89, is a graphical editor that incorporates similarity-based learning.

* Eager, by Allen Cyper, CHI '91, is a programming by example system for Hypercard.