Artificial Intelligence at the Visible Language Workshop
Henry Lieberman
For many graphic designers and artists of the last generation, the idea of computers entering the design process was viewed as threatening. The exploration of visual forms and sense of communication in visual design was viewed as essentially and uniquely human, and the incursion of the computer into the process was viewed by many as demeaning the role of the creative professional. Of all the branches of computer science that proposed relevance to visual design and communication, artificial intelligence was perhaps the most threatening of them all, since it proposed to model the model aesthetic judgment, which, in a pessimistic view, might leave little role for a human designer.
A small handful of visual designers, however, saw it very differently. A
lot had to do with a designer's personal view of the design process itself. The
prevailing culture often viewed design as magic, designers as magicians. But
some viewed the design process as a constant search for the relation between
form and meaning, a continuing exploration of possibilities for expression.
They welcomed tools that could assist a designer in relating form and meaning,
in exploring and evaluating alternatives. They saw the possibility for the
computer to become an intelligent partner in the design process, to extend the
reach and grasp of the human designer. From that point of view, the concerns of
artificial intelligence in representing meaning, inference, and learning are
not foreign to the design process. They are central.
And it was not always the most technologically oriented among designers that had the greatest enthusiasm for the potentials of the new technology. It was the vision of new, dynamic, intelligent, image-making tools that drove a few designers, educated in a culture that was artistic, not technological, to put up with learning the often frustrating details of computers, then as now.
Muriel Cooper was one of those rare pioneers, who had a vision of computer-based tools that could actively participate in the process of dynamic visual expression. She saw no inherent boundary between the visual design process and the software design process, only one imposed on us by the imperfections of our current technology. She was often iconoclastic, challenging the unstated assumptions of both visual designers and computer scientists.
Generating design alternatives
In Muriel's view, the computer is a new design medium, a new kind of brush, a new kind of canvas, a new kind of palette. There is no more reason for a designer to fear computers than for any artist to fear their tools. You learn them and work creatively within their strengths and limitations. The particular strength of the computer as a design tool is its ability to respond dynamically to the user, to remove the burden of tedious, precise or repetitive operations, to become personalized to its users' needs and style of working. Muriel was attracted by Nicholas Negroponte's original vision of the Architecture Machine as an intelligent assistant for architectural design, and envisioned a similar role for an intelligent assistant for visual design tasks.
Muriel made the Visible Language Workshop a meeting ground for people from diverse backgrounds who shared a fascination for the computer as a place where technology and design could converge. Her collaborator in starting the VLW was Ron MacNeil, who came from a photography and printing background, and who also became a strong proponent of the AI approach to design. Designers who had little familiarity with the technology encountered computer scientists who hadn't yet given serious thought to problems of visual design and expression, and both sides learned enormously from one another. The VLW embodied the ideal of interdisciplinary spirit in which the Media Laboratory was founded.
The kids programmed math and physical simulations, but art was also a major activity. I had exhibited some computer art in the Siggraph conference, and some of my students won prizes in Byte magazine's first computer art contest. My first official VLW event was when I was invited by Muriel to participate in a panel on computer art, along with Russell Kirsch, who constructed AI simulations of famous artists' styles. Russell, in his sweet and gentle manner, tore me to shreds for my naivite about artistic traditions and concerns. There was something exciting about the interplay between artistic and computational concerns that I hadn't encountered elsewhere. The panel was part of Muriel's annual summer course, The New Graphics, which introduced people from a variety of backgrounds to computer image making. The activities in this course represented some of Muriel's pioneering effort which established the field of what is now called electronic publishing. The course used the sometimes cranky, sometimes miraculous VLW image editing system, called Sys. The encounter between people of design and computer backgrounds was, for me, exhilarating.
Continuing contact with Muriel and Ron through the incorporation of the VLW of the Media Lab, I started to teach in the introductory VLW course, and participate in thesis committees for VLW students. The course bore the rather nondescript title Computer Graphics Workshop, but it was much more than that. Taken mostly by students immediately after joining the VLW, it was "boot camp" in the VLW philosophy. While much of the course was conducted in the manner of a traditional design course, with design exercises, projects and critiques, Muriel also wanted the students to gain some background and experience in AI issues, and I supplied that perspective, along with Ron and Patrick Purcell, who had done seminal work in studies of design process in architecture. I was part-time between the Media Lab and the AI Lab for a while; then, feeling increasingly that the exciting directions for the future lay in the convergence of artificial intelligence with interactive interface design issues, I joined the VLW full time in 1987.
Layout from sketches
Projects in automatic layout took place in many domains and with several different approaches, explored by many VLW students. Students Timothy Shea, Tom Amari and Alka Badshah were early explorers of rule-based systems for design, with business graphics and packaging being domains of application. Russell Greenlee used sketches of principal elements of layout as a guide, and his system generated sequences of possible layouts for the user to choose from that were consistent with expressed constraints. I had designed an automatic layout system that used best-first search to deal with the common problem of overconstrained space allocation. More recently, Grace Colby applied the techniques of case-based reasoning and constraints to the automatic layout problem. B. C. Krishna looked at the problem of automatically detecting layout constraints from scanned images. Louis Weitzman pursues a linguistic grammar-based approach, with the aid of a parser that can enforce geometric and temporal constraints during automatic layout.
Automatic conversion of layouts
A 3D program representation
I led a project in investigating alternative visual representations, including 3D programming languages, and use of color and translucency. Ron MacNeil's early Tyro system used a representation of visual design elements connected by "spider webs" representing constraints. Projects by students Dorothy Shamonsky, Ming Chen, and Michelle Fineblum also explored issues of visual representation in programming. Storyboards containing visual examples of states of a program appear in my Mondrian system and in Louis Weitzman's VIA. Visualization of rules, constraints and graphic relationships continues to play an important role in several VLW projects.
Muriel provided the insight of a top design professional into the process of design. A task for me and for her VLW collaborators was to "knowledge engineer" some of Muriel's design expertise, and the design expertise she brought to VLW in the form of visitors from the design world and reference materials. The knowledge engineering was trying to come to understand enough about the design process to understand what aspects of could be embodied in intelligent interactive tools. We still have not come very far, relative to the creative powers of an expert designer, but Muriel aided greatly in taking some innovative steps towards intelligent design tools.
Two views of an adaptive presentation
Debra Adams used the observation that type designers often base their designs on a few prototypical letterforms, and she constructed a system that would automatically design a complete font after having been shown designs for the prototype letters. Craig Kanarick built knowledge about the design of charts and spatial data into a case-based presentation tool. Laura Robin, at my suggestion, built a best-first search engine into a hypermedia browser, so that the time and level of detail of a presentation could be automatically adjusted to produce the best possible presentation in a given amount of time and subject to expressed interest in a subset of the topics. Michelle Fineblum also explored adaptive hypermedia presentations.
Open up any book on graphic design, either one intended for teaching beginning design or for communication between seasoned professionals. Unlike books in science and engineering, few principles will be explicitly stated in the text; there will be no equations, no axioms. So how can design students use these books to effectively solve problems? What the books will provide is an abundance of examples: examples of exemplary designs, examples of design variants [or "near misses" in AI terminology]. Good design students are experts in generalizing examples and making analogies from examples to new problems they encounter.
Learning from examples
But why can't we use concrete visual examples in teaching our computers how to do designs, instead of incomprehensible programming languages and rule languages? It is this question that led me to work on the idea of programming by example. In this approach, a designer can use an interactive interface, and a software agent records the actions performed by the user, and can generalize them so that they can be applied to analogous problems in the future.
I [and others in VLW] built several systems that incorporated this approach. Mondrian is an object-oriented graphical editor that can learn new graphical procedures by example. Its base is a MacDraw-like graphical editor, a familiar tool for design professionals. Its learning agent could also be applied to many other sorts of image editing and media editing tools. Mondrian incorporates a learning agent that records and generalizes procedures presented as sequences of interface commands. The user selects objects to represent the examples, and demonstrates a procedure which depends on those objects. The user interface is extended with a new operation that can be applied to different objects in the future.
Mondrian uses a consistent visual language for communicating with the user. Operations are represented with dominoes, before-and-after pictures representing a visual example of the operation, and storyboards, a comic-strip like sequence of frames. It automatically produces natural language descriptions, both written and spoken, and accepts advice from the user via speech recognition. I also built on my previous work with a system called Tinker, which allowed example-based programming with multiple examples, conditionals and recursion.
Learning by examples from maps
Suguru Ishizaki did a thesis on a programming by example system for geographic information systems. Alan Turransky connected his example-based system to produce rules accepted by Louis Weitzman's interactive automatic layout system.
Intelligent interpolation
Sketched examples played an important role in Russell Greenlee's layout-from-sketch system, Steve Librande's intelligent image interpolation system, and Karen Donoghue's gestural animation system. Case-based reasoning, another example oriented technique, was central to Colby's and Kanarick's work mentioned earlier, and Ron MacNeil's Tyro, which constructed new multimedia presentations based on previously presented examples.
Case-based reasoning for presentations