Henry Lieberman
November 1993
Henry Lieberman
Visible Language Workshop
Media Laboratory
Massachusetts Institute of Technology
20 Ames St.
Cambridge, Mass 02139 USA
Telephone: (1-617) 253-0315;
Fax: (1-617) 258-6264
Electronic Mail lieber@media.mit.edu
* The Mondrian demonstration software, as a Macintosh "application"
* Some Macintosh "resource" files which must be loaded for the application when it starts up.
* Optionally, the Lisp source code for the Mondrian application.
* Written documentation, inlcuding this guide and the article describing Mondrian referenced above.
Mondrian package folder
The folder with the software package will contain
* The Mondrian Application.
* The Henry folder, containing resource files needed by the Mondrian application, and [optionally] the Lisp source code for Mondrian and its support systems.
* [Optionally] A system folder. If you have problems with running your own sytem folder, try this one.
* [Optionally] The Lisp Folder containing the Macintosh Common Lisp system. This is not necessary to run the Mondrian application, but will be if you would like to further modify or extend Mondrian.
Mondrian at startup
The Mondrian window appears.
Rectangle domino
Click for the upper left corner, then drag to the lower right corner, to draw a rectangle.
Drawing a Rectangle
Move domino
Delete domino
Select domino
Group domino
Mondrian also has several other operations that may be loaded dynamically. These will be discussed as they are needed for the example demonstrations below.
Mondrian Menu
You may also see other Mondrian options on this menu.
Layouts Menu
You may also see other images on this menu.
Start by drawing a large rectangle. Choose the Color domino to select a constrasting color for subsequent drawing.
Invoking New Command
Select the rectangle. Then, select the New Command domino. This makes the selected rectangle the argument to the New Command operation.
You are then prompted for the name of the new operation. Type "Arch".
Steps in drawing the Arch
When the three rectangles that compose the Arch are complete, you may Delete the original background rectangle.
We now have the three rectangles, but we really would like the arch as a single object. To get this, Shift-Click on the three rectangles, and choose the Group command.
If you do not, the definition will be removed and its domino will disappear. If you do save the definition, the after panel of the Arch domino will be filled in with the final state of the screen.
Arch domino
Each frame is labeled with the name of the operation chosen. A tiny box in each frame indicates the position of the domino that was selected.
In order to use the speech feature, your Macintosh must have been started with the following in the Extensions folder of the System Folder [these are available from Apple]:
* Speech Manager
* GTEngl
* A voice file, GTEnglFem24 is recommended
Try the Arch example once more. First, make sure that the volume is turned up from the Sound control panel. Just before you invoke the New Command operation and the Arch icon appears, choose Turn on Speech from the Mondrian menu. The machine should now say "Speech is now on".
Below the example of code for the Arch example presented earlier [Actual code may slightly differ]:
_____________________________________________________________
(DEFUN ARCH (INTERACTOR SELECTION)
(RECTANGLE-COMMAND (LEFT-TOP (NTH 0 SELECTION))
(POINT-ON-OBJECT (NTH 0 SELECTION)
12/95 1)
INTERACTOR
'RECTANGLE-5)
(RECTANGLE-COMMAND (LEFT-TOP (FIND-OBJECT-NAMED
INTERACTOR
'RECTANGLE-5))
(POINT-ON-OBJECT (NTH 0 SELECTION)
93/95 8/51)
INTERACTOR
'RECTANGLE-6)
(RECTANGLE-COMMAND (POINT-ON-OBJECT (FIND-OBJECT-NAMED
INTERACTOR
'RECTANGLE-6)
76/91
1/14)
(RIGHT-BOTTOM (NTH 0 SELECTION))
INTERACTOR
'RECTANGLE-7)
(REMOVE-OBJECT INTERACTOR (NTH 0 SELECTION))
(GROUP-COMMAND (LIST (FIND-OBJECT-NAMED INTERACTOR
'RECTANGLE-5)
(FIND-OBJECT-NAMED INTERACTOR
'RECTANGLE-6)
(FIND-OBJECT-NAMED INTERACTOR 'RECTANGLE-7))
INTERACTOR
'GROUP-8))
_____________________________________________________________
Each of the topmost-level calls, such as those to the function Rectangle-Command, represents an action routine invoked by one of Mondrian's iconic commands. The arguments are expressed in terms of the relations inferred by Mondrian, such as Left-Top. Point-On-Object expresses a point relative to an object using the proportions of its X and Y coordinates; (Point-On-Object ... 1/2 1/3) means an point half the way across and a third of the way down on the object.
Suit is independent of the rest of Mondrian, and appears in its own folder.
* The third is Mondrian's learning facilities, mostly in the file New-Command in the Mondrian folder. The file Narrate holds the speech output facility.
EzWin Suit is the successor to EzWin, a user-interface system described in the article [Lieberman 85].
The file Clos-Extensions in the Init directory defines a simpler syntax, Defob, which replaces CLOS's Defclass and automatically defines accesor functions. It also defines an extension that causes objects to print out with their components rather than with a unique-ID index.
Each System object has instance variables to hold names of source code files, pathnames, and other information. There are operations to compile, load, edit, and perform other operations on System objects.
CLOS generally requires that the definition of an object be loaded before it is used, for example, to define a method. The consequence is that object definitions for a single system should be contained in a single file, the Defs- file, which needs to be loaded before source code referencing these objects is loaded or compiled. The methods on the System objects usually take care of this.
Mondrian and Suit each have their own System objects, defined in the files Mondrian-System and Suit-System. You will find Defs files for Mondrian-Defs and Suit-Defs, in their respective folders.
The name "Interactor" was chosen to suggest both "interaction", and also "actor". As in the "actor" systems of Carl Hewitt, an interactor represents a fundamental component out of which everything else is built.
Some objects, like points and colors, are usually "encoded" by Macintosh Lisp into numbers, presumably for "efficiency". This is contrary to the object-oriented paradigm, and here I represent a point by a CLOS object which has its coordinates as instance variables. Rectangle objects hold their Top-Left and Bottom-Right corners., as Point objects.
Objects of class Graphic represent "Platonic" shapes such as points and rectangles. Those of class Geometric can, in addition, have concrete displayable properties such as color or line thickness.
Appear and Disappear take into account the two-and-a-half dimensionality of the display. Objects underneath a Disappeared object will be redisplayed. The algorithm for doing this is simple and straightforward, but perhaps more efficient ones could be found. Appear and Disappear may be changed or redefined in subclasses to implement more sophisticated redisplay algorithms. Overlap testing is done by using bounding boxes, but again, more sophisticated algorithms may be subsituted, by redefining the function Intersect?.
Appear and Disappear are defined in terms of lower-level functions Display and Erase, which do not take into account two-and-a-half dimensionality. Erasing an object will erase any overlapping portions of objects underneath it. Display and Erase with an Interactor as second argument are defined in terms of still lower level functions Display and Erase that take a Macintosh window as a second argument and do display directly to the screen using Macintosh Common Lisp Quickdraw functions.
Because of CLOS's multi-method facility, the output methods may be specialized either on the graphic object argument, or on the Interactor argument. This gives both the object being displayed, and also the containing application a chance to "get a word in" about how display is to occur. This feature is often exploited in applications built with Suit.
The second level is the function Touch, which tells an object "you have been clicked on", with the Interactor as second argument. Thus, Activate gives the Interactor a chance to determine what happens, usually forwarding to Touch, which gives the object itself a chance.
The function Hit? is used for determining what object has been clicked on. The default version dispatches Hit? linearly to each object on the Sensitive list. More efficient algorithms, such as Quad Trees, can be installed simply by replacing the Hit? function.
* Start-Tracking, called once at the start of tracking,
* Move-Tracker, called each time the cursor moves, with old and new positions, and
* Stop-Tracking, called when tracking ends.
Each of these can be redefined to affect the behavior of the tracker, or only the relevant ones, falling back on default behavior for the rest.
Move-Tracker, in turn is defined by
* Erase-Old-Tracker, and
* Draw-New-Tracker
which can be independently redefined.
A Palette is a kind of Radio-Buttons interactor object that collects icons representing commands to be invoked in an interactive application. Individual graphic objects representing commands are known as Tools. Tools declare their argument lists and types of arguments. Each type of argument is a subclass of Argument, and contains a method for obtaining that kind of argument interactively from the user.
A Canvas is also a Radio-Buttons interactor that holds of list of graphical objects to be operated upon by the commands in the Palette. Graphical objects visible in the Canvas may be selected as arguments to the commands in the Palette.
Together, an interface containing a single Palette and a single Canvas is an Easel. Argument-Collector objects manage the interactive loop that selects a tool and its associated arguments, and invokes the command. This is like the "read-eval-print" loop of a Lisp interpreter.
The Trivial-Application object is the simplest possible use. Test-Trivial-Application creates one of these, with a single rectangle object. Clicking on the rectangle or the background simply prints a message in the Listener. Observe how this is done simply by using two Touch methods.
Rectangle-Application is a bit more complicated. It allows you to draw rectangles by rubber-banding their dimensions. You can drag an already-existing rectangle to a new position. Again, it only requires two Touch methods to implement the application. These methods use Mouse-Tracker objects Rubber-Rectangle-Tracker and Drag-Tracker to do their work. The object returned by the tracker is added to the Visible list.
A more substantial example demonstrates an Interactor that can highlight selected objects, defined in the Highlighting file. Radio-Buttons is an Interactor that only permits one Highligted object at a time.
Tracing the various Suit functions or stepping through some of the definitions is a good way to become familiar with the sequence of operations that takes place in the course of a user interaction.
Because the collection of arguments and invocation of commands is separated from the actual action performed by the command, all of the action routines can either be invoked by a user interaction or by a program generated by Mondrian.
The New Command Tool keeps a Command-History produced by the recording and generalization process, and a Snapshot-History which is used to record successive graphical states to produce the definition storyboards of recorded procedured.
The Remember- functions are crucial to this process. For every action performed, there is a Remember- function that yields Lisp code generalizing the particular action. Actions are in general recorded by Remeber-Action, and there are Remember- functions specialized to each kind of action.
Remember- functions for actions in turn call Remember- functions for their arguments. Remember- functions can take advantage of the object hierarchy and be specialized for each type of argument object. Significant-Point determines the relations noticed for a point object. This could be augmented with other relations if desired.
Remember- functions generally take an Environment argument, that records the association between graphical objects and generated code. This is necessary so that when an object is referred to, its intentional description is used, and not the accidental properties of the object, such as its absolute coordinates.
The function Definition-Cartoon draws the frames of the storyboard displayed when you shift-click on a user-defined domino icon.
(Presented as an invited keynote address)
in East-West Conference on Human-Computer Interaction,
St. Petersburg, Russia, August 1992.
Watch What I Do: Programming by Demonstration
edited by Allen Cypher, co-edited by David Kurlander, Henry Lieberman, David Maulsby Brad Myers, and Alan Turransky,
published by MIT Press, April 1993.
Major support for this project at The Visible Language Workshop and the Interface Agents group of the MIT Media Laboratory is provided by research grants from Alenia Corp., JNIDS/ARPA, the National Science Foundation and Apple Computer. Additional funding and equipment grants for the Visible Language Workshop come from Digital Equipment Corp., NYNEX, IBM, the US Dept. of Transportation.