4.889 Intelligent Interface Software Design Workshop
Project 1: Learning in Graphical Editors
You may work alone, or in groups of 2 or 3. If you do team up, try to find someone who complements your skills -- if you are not so experienced in Lisp, team up with someone who is, and vice versa.
Try to be a reflective practicioner while you are working on the problem. Keep a "design notebook" [either electronic or hardcopy] while you are working on the problem and jot down:
* Problems you have while you are working on it, even if you can't deal with these problems immediately [or ever]. Note especially interesting bugs that arise.
* Suggestions for neat features, interesting variants of the problem, etc. even if you don't have time to work on them immediately.
* Interesting design or philosophical issues raised.
These can all serve as fuel for class discussion.
We will start with a relatively simple exercise to make sure you understand the basics of interface programming, and the details of the code provided, and to give you some experience making an extension to an interactive graphical interface. This will be followed by more substantive extensions to the interface. The last exercise implements a significant extension to Mondrian's learning capability, to give you experience with learning techniques.
A suggestion is to add a new graphical primitive, say an operation that draws an oval. An important heuristic in any programming, and especially interface programming is to find some code that "almost kinda" does what you want and use it as a model for the functionality you want to implement. Mondrian's existing Rectangle operation can serve as a model. When you need to do something, try to look at how Mondrian's Rectangle operation accomplishes a similar task.
* Make a domino icon for your new operation
* Make a command object for your new operation
* Define an action routine for your new operation
Define an oval object like Mondrian's rectangle object. The object definition is in the file Suit:Suit-Defs, and the methods in the file Suit:Graphic. Define filled-oval. The graphics function for drawing an oval is called paint-oval and works like paint-rect.
Look at how the rectangle-action function in the file Mondrian: Mondrian-Editor works.
Test out the action routine from the Listener. To do this, you need a canvas object. The variable ma is bound to the application object for Mondrian when it starts up. (canvas ma) will give you the current canvas.
The resource-file and resource-name refer to a resource file where you will keep the domino icon.
You must install the tool on Mondrian's palette. Call the function install-new-tool on your tool object and (palette ma).
You will need to learn about Argument objects and about Mouse-Tracker objects. The definitions are in Suit:Defs. You will need to implement a new kind of Mouse-Tracker object.
It is defined in terms of three functions,
* start-tracking
* move tracker, called each time the mouse moves
* stop-tracking
You can make a tracker that inherits from Mouse-Tracker and redefines any one or all of these without redefining others.
Look in the file Suit:Easel to learn how the system collects arguments and invokes commands, if necessary.
Why wait to fill in the final state? Instead, each tentative final state could be shown after each command. This willanticipate what the domino icon will look like if the user decides to terminate the definition at that moment.
Change Mondrian so that after each command, it fills the right panel of the domino with a miniature copy of the current state immediately after each command.
* Remove the "?" that is currently drawn in the panel.
* Figure out how to insert some code after each command is executed.
* Figure out how to draw a miniature copy of thescreen state.
What code in Mondrian does something similar that you can take as a model?
* The code that draws the parts of the domino icons
* The code for the storyboard
* You will find the code in Mondrian:New-Command
The task is to add similarity based learning to Mondrian. The idea is that you will add an "agent" that looks for patterns in the history of recorded actions after each action is executed. If it finds a pattern, it can automatically perform the next action in the pattern.
* Write a matcher that looks for patterns in the command history. This matcher may be heuristic in that it will not always guess correctly.
* Provide some way of invoking the generalized command generated by the matcher. This can be automatic or explicit.
and you move the two lower leftmost rectangles so that they touch the upper bar,
then the system should be able to infer that the other two should be moved up to touch the bar.
Experiment with this a little by defining a new command in Mondrian, and querying this after every graphical operation, so you see what it does.
Our simple matcher will allow us to find patterns that involve:
* Using up the arguments to commands [first, then second, third argument, etc.]
* Linear displacements [such as evenly spaced objects].
* Iterating through the elements of a group given as the argument to a command.
Start by writing a simple matcher that just detects patterns using Lisp's equal in lists. We only care about patterns in the tail end of a list.
(find-pattern '(a b c d c d)) = (c d)
(find-pattern '(a b c d e c d e)) = (c d e)
Read the chapters in Winston's Lisp book on pattern matching if you want to learn more about how to write matchers.
Now, generalize it to predict the next sequence in the pattern. Start by just predicting from number sequences, but leave room to add more kinds of prediction later.
(find-pattern '(a b c d 1 c d 3)) = (c d 5)
! [One subtlety in writing a matcher for Mondrian's code is that most of the operations generated by Mondrian use the last argument to the function to name the object generated by the function [rectangle-action, line-action, group-action, move-action, etc]. You must make the matcher ignore the last argument for those functions. Expressions that differ only in the last argument should be considered to match. I may supply some code that will do this, or you can determine experimentally where it is ncessary.]
Test it out by invoking your matcher explicitly from the listener, and calling
(eval-history-command
(canvas ma)
(find-pattern
(command-history (command-being-defined ma))
after each graphical operation.
Invocation can either be done automatically [the generalized command is simply executed whever a pattern is found], or you can provide an operation that explicitly asks the agent to guess what the pattern is.. It is easier to make invocation explicit, so that you don't have to worry about stop conditions. Make an icon called Again, that instructs the system to "do the last recognizable pattern over again".
* Metamouse, by David Maulsby, SigGraph '89, is a graphical editor that incorporates similarity-based learning.
* Eager, by Allen Cyper, CHI '91, is a programming by example system for Hypercard.