{\rtf1\mac\deff2 {\fonttbl{\f0\fswiss Chicago;}{\f2\froman New York;}{\f3\fswiss Geneva;}{\f4\fmodern Monaco;}{\f13\fnil Zapf Dingbats;}{\f14\fnil Bookman;}{\f15\fnil N Helvetica Narrow;}{\f16\fnil Palatino;}{\f18\fnil Zapf Chancery;}{\f20\froman Times;}
{\f21\fswiss Helvetica;}{\f22\fmodern Courier;}{\f23\ftech Symbol;}{\f31\fnil Optima;}{\f33\fnil Avant Garde;}{\f34\fnil New Century Schlbk;}{\f202\fnil Voyager;}{\f1027\fnil Palex;}{\f1792\fnil Lucida Bright;}{\f2000\fnil Garamond Narrow;}
{\f2007\fnil Delphian;}{\f2017\fnil Lubalin Graph;}{\f2024\fnil Machine;}{\f2029\fnil Nadianne;}{\f2036\fnil Old English Text;}{\f2038\fnil Onyx;}{\f2040\fnil Oxford;}{\f2046\fnil Swing;}{\f2052\fnil Zeal;}{\f6024\fnil Invisible;}
{\f14811\fnil Helvetica Compressed;}{\f14834\fnil H Futura Heavy;}{\f14836\fnil Futura;}{\f14884\fnil Univers 55;}{\f14974\fnil Helvetica Black;}}{\colortbl\red0\green0\blue0;\red0\green0\blue255;\red0\green255\blue255;\red0\green255\blue0;
\red255\green0\blue255;\red255\green0\blue0;\red255\green255\blue0;\red255\green255\blue255;}{\stylesheet{\s243\tqc\tx4320\tqr\tx8640 \f14 \sbasedon0\snext243 footer;}{\s244\tqc\tx4320\tqr\tx8640 \f14 \sbasedon0\snext244 header;}{\f14 \sbasedon222\snext0 
Normal;}}\margl1440\margr2160\widowctrl\ftnbj\fracwidth \sectd \linemod0\linex0\cols1\endnhere {\header \pard\plain \li360\ri-1160\tx7380\tx9799 \f14 {\f16 \par 
}}{\footer \pard\plain \li360\ri-1160\tx7380\tx9799 \f14 {\f16 \par 
}}\pard\plain \qc\li360\ri-1160\tx7380\tx9799 \f14 {\b\f16 Friend-21 Conference\par 
}{\f16 Interaction and Technology\par 
}{\b\f16\fs36 \par 
Visual Programming:\par 
A Vision for the Future\par 
}{\f16 \par 
}{\f16\fs36 Henry Lieberman\par 
}{\f16 \par 
Visible Language Workshop\par 
Media Laboratory\par 
Massachusetts Institute of Technology\par 
Cambridge, Mass. USA\par 
}\pard \qj\li540\ri-1340\tx7560\tx9979 {\f4 __________________________________________________________________\par 
}{\f16 \par 
}{\b\f16 From graphical interfaces to graphical programming\par 
}{\f16 \par 

The advent of high-resolution graphical workstations and direct manipulation interfaces has now led users to expect easy-to-use iconic and pictorial interfaces for a wide variety of applications.  But what about programming itself?  Despite the advances in
 computer graphics and interfaces, and despite the increasingly graphical nature of the programming activity itself, programmers still use interfaces that are limited to "glass teletype" textual type-in and print-out.  \par 
\par 
Visual programming is the idea that
 pictures can be used as a representation for computer programs, and the activity of programming can be performed by manipulating pictures with an interactive graphical editor.   Hopefully, a visual programming system could bring the same kind of productiv
ity leap for programming that real-time display-oriented text editors brought over the previous line-oriented editors, or that desktop-metaphor graphical file systems brought over older command-line shells.   \par 
\par 
But is this expectation realistic?  Is there 
an effective way to map programming concepts like variables, conditionals, and procedures to a visual display, or is programming an inherently verbal activity?  I think visual representations hold great potential for the future, and in what follows, I will
 discuss objections I commonly hear to the idea of visual programming and try to answer them.\par 
\par 
}{\b\f16 Hasn't visual programming been tried before?  \par 
\par 
}\pard \qj\li1160\ri-540\tx5040\tx9179 {\i\f16 We must expect [... from graphical programming]  permanent mental damage from most students exposed to it.\par 
\par 
\tab Edsger Dijkstra, SigCSE 1989\par 
}\pard \qj\li540\ri-1340\tx7560\tx9979 {\f16 \par 

The idea of using pictures as a representation for computer programs is not new.  In fact, the earliest example I have found is the flowchart language GRAIL, dating back to 1966.  The fact that experiments in visual programming go back so far in the histor
y of computing has resulted in some pessimism regarding the future prospects of the idea.  If graphical programming languages have been around so long, why haven't they succesfully displaced textual programming languages by now?  
Even some researchers in the field respond to criticism by hedging about the propsects for widespread visual programming, limiting their claims for it only as a teaching tool or in limited domains.\par 
\par 

I'd like to convince you that the pessimism surrounding graphical programming is misplaced.  Some of these early experiments failed for reasons that are no longer valid, such as lack of sufficient graphics computing power.  Others were more successful in d
emonstrating the feasibility of graphical programming, but did not receive wide publicity and distribution.   Many are so different that conservative people have difficulty changing their ingrained habits enough to accept them.  \par 
\par 
\par 
}{\b\f16 The fundamental problem of visual programming:  \par 
Mapping properties of a program onto properties of pictures\par 
}{\f16 \par 
Programming is the art of constructing static descriptions of dynamic processes to be performed by the computer.  Those static descriptions are traditionally, but need not be, in the form of linear source code in a textual prog
ramming language.   Maybe the reasons are historical.  Computers first had hardcopy text-only printers as output devices, so it was necessary to be able to print the descriptions of programs.  Perhaps if graphic displays had been hooked to computers first,
 we would now think of programs as pictures and it would be hard to imagine how they could be represented only by text!\par 
\par 
Let's look at some experiments that have been done in trying to construct visual programming environments.  I don't have time to consid
er them all, or even all the ones I consider good, but a representative selection is illustrated here.   They all look very different from each other, but some similarities emerge.\par 
\par 

All programming languages, textual or visual, are made of the same kinds of things -- numbers, strings, variables, conditionals, function calls, declarations, and so on.  In designing a textual language, the problem is what to call them and how to string t
hem together.  In visual languages, the problem is how to make pictures that represent them, and how the visual relations between pictures represent the visual relations between program elements.\par 
\par 

In the early days of programing, flowcharts were touted as a conceptual design tool.  It was only natural, then, that the first visual programming systems were based on a graphical flowchart representation.  Early programming languages, like machine langua
ges and Fortran, had tangled control structures which flowcharts helped unravel.  With the rise of "structured" and functional progra
mming, the need for visualizing the linear flow of control lessened, so flowcharts became irrelevant as a visual representation.  However, some people still confuse the notion of visual programming with the idea of "programming with flowcharts".  \par 
\par 

Several more recent visual programming efforts still use a node-and-link visual representation, but these should not be confused with early flowcharts.  Examples include Levitt's Hook-Up, Mainstay's VIP for the Macintosh, Apple's Fabrik, and some of my own
 work. 
 The structure that these diagram is not the flow of control, but the network of function/argument relations.  The function/argument relations are much closer to the way in which programmers in modern programming systems conceptualize the structure of thei
r programs, so it is much better to represent this structure in a direct visual manner.  Some other systems, such as Smith's Pygmalion or Lakin's VennLisp, also use direct visual representation of function/argument relations, but use containment of icons i
nstead of nodes and links.  \par 
\par 

Benefits from a pictorial representation of programs can come with ease of learning, especially for beginning users.  But even experienced users, after they take the effort to "unlearn" the textual representation, might also experience the benefit of being
 able to directly perceive relations in the program that would be difficult if it were shown as text.  Certainly, the node-link or nested element representations make it easier to perceive scoping relations in the program.  Th
ey might also, with suitable operations in the graphical editor, make it easier to browse large programs or control level of detail.  Pictorial representations might also have benefit in international communication for their independence from a particular 
natural language, as do the standard international iconic road signs.  \par 
\par 
Visual representations have already been successful in contexts where either conventional textual representations cannot succeed, or where a simpler domain than full procedural progra
mming is involved.  It could be argued that the success of spreadsheets for many programming tasks is due to the visual feedback of representing both programs and data on a square grid, and immediate display of changes.  Recent additions of charting functi
ons to spreadsheet programs, and extension of spreadsheet languages move them closer to visual programming, although they stop well short of a true programming environment.  Graphical programming environments have also been succesful in teaching programmin
g to young children or beginning users, or in various systems controlled by handicapped users, for whom conventional textual languages would be difficult.  The challenge for the future is to extend these successes into more demanding programming domains.
\par 
\par 
There is now a strong movement to use computer graphics to display simulations of dynamic processes in scientific and engineering fields, under the slogan of }{\i\f16 scientific visualization}{\f16 
.   The benefits from visual representation of complex data, even when the
 mapping of data to graphics is very abstract, as it is in most false-color imagery, are quick visual apprehension of complex relations in the results.  To the extent that computer science is also a science, visual programming  should do for computer scien
tists what scientific visualization already achieves in other fields of science.  \par 
\par 
}{\b\f16 Animated representations of program execution go beyond just graphically representing static programs\par 
}{\f16 \par 
What happens when a visual program is executed?  Now we broaden our c
oncerns from consideration of just the programming language to the whole programming environment.  Some environments merely provide a correspondence between the graphics and a command to execute the program.  But the real power of a visual representation s
hould come from making visible the actual steps of execution of a program.  Animation should be used to make changing graphics reflect the changing state of the computer.  The payoff for this should come when the need arises to debug the program, since deb
ugging is mainly a process of visualizing the operation of a program and matching it to the programmer's expectations.  \par 
\par 

Again, the issue here in designing a dynamic visual representation for execution of programs is mapping the properties of the execution of programs onto visual properties of pictures.  I will show some of my work where I use the idea of graphical substitut
ion of values for the expressions that give rise to them.  This is a natural, intuitive model of the process of evaluation.  Other systems have used moving cursors to show control flow and moving values to show data flow.  \par 
\par 
}{\b\f16 Do graphical program representations take up too much screen space?\par 
}{\f16 \par 

A charge that is frequently levelled against graphical representations is that they are too wasteful of screen space, and that textual programming languages are better because you can "see more of the program on the screen at once".  But how fair is this c
harge?  A closer look shows that an argument based on information density is not decisive.\par 
\par 
First of all, it is necessary to remember that even text itself }{\i\f16 is}{\f16 
 graphics, so that even textual programming languages are graphical representations of a certain kind.  Consider the text of a program in C or Lisp, properly formatted and indented.  There is a lot of visual information in the formatting itself.  Programme
rs who do not know Lisp are often put off by its parentheses, but in fact good Lisp programmers typically ignore parentheses and use the visual cues of vertical indentation to understand the program.   \par 
\par 

If compactness were the only virtue, Lisp or C programs could be made far more compact by deleting excess indentation and all inessential spaces and carriage returns, but the resulting code would be unintelligible to all save the computer.  The most compac
t visual representation possible would be to encode the program using one pixel per bit, which could be learned by people with great difficulty, just as the Morse code was for telegraphy.  That we do not use any of these representations says that
 we are willing, at least in principle, to trade off space for legibility in visual representations, so we should not exclude pictures even if their static density on the screen seems inferior.\par 
\par 

Anyway, it is not just the static information density that matters.  It is very difficult to comprehend all at once an entire page of source code text, no matter what the programming language.   Although text files are oriented toward a linear reading, pro
grammers often need to jump non-linearly from place to place 
in a program when coding and editing.  When programs are larger than one screenful, some means of dynamically moving through them, typically scrolling, multiple files and multiple windows, is employed.   These are themselves (relatively unsophisticated) gr
aphic techniques, which impose arbitrary boundaries between succesive temporal views.  Pictures can have an advantage in dynamic display, because zoom and pan operations, detail elision, motion and animation can provide browsing techniques that are better 
than simple text scrolling and file search in locating and displaying dynamically just the relevant information on the screen.\par 
\par 
}{\b\f16 The development of better graphical editors will enable better graphical programming\par 
}{\f16 \par 

Currently, textual programming language have a real advantage in working with the programming environments that currently exist.  A lot of engineering and optimization work has already been put into text editors that has yet to be performed for graphical e
ditors.  Conventional operating systems al
ready have standard file formats and a host of tools for text files that do not yet exist for graphics.   As an exercise, think of all the tools that your current favorite system supplies for text manipulation.  Is there a graphic equivalent for each of th
em?  Progress is such that the graphic tools will no doubt appear soon, since they are needed for such applications as desktop publishing and CAD if not for programming, but at the moment graphics is at an unfair disadvantage.\par 
\par 
As an example, one requireme
nt that is needed for practical control of graphical representations is zooming for control of level of detail.  Although many systems do provide a graphics zoom operation, many systems are so sensitive to absolute pixel coordinates in bitmaps, fonts, icon
s, and other graphics operations as to make general zooming awkward.  Window systems made the leap away from dependence on absolute screen position, but we have yet to achieve a similar independence from absolute scale. \par 
\par 
}{\b\f16 It is only recently that workstations have reached enough "graphical horsepower" to make graphical programming feasible\par 
}{\f16 \par 
Visual representations are potentially }{\i\f16 better}{\f16 
 than textual representations because  they can provide interactive graphical operations like panning and zooming that would aid navigation through a much larger virtual space than would be feasible for text alone.  Hypertext and hypermedia operations allo
w dynamically switching parts of displays based on user interaction to give selective access to a large information space in
 a way that simply scrolling through text never could.  But these operations are much more computation-intensive than text motion operations, and they critically depend upon the compute power and speed of interactive graphics necessary for updates of large
 portions of the screen area at rates where motion is perceived as being near-instantaneous.\par 
\par 
With hardware graphics accelerators that provide support for very fast block-transfer, line and shape drawing, and other rendering operations, these can now be us
ed as part of the graphics vocabulary for programming.  Before that was true, it was always the case that inserting text into a text editor would be faster, so textual representations always had an unfair advantage.  We have experimented with a three-dimen
sional graphical representation for programming, enabled by a display that performs rendering of three-dimensional objects.  With such hardware, rotating boxes is takes less time than printing out a line of text.\par 
\par 
}{\b\f16 Conservatism is a strong force in preventing the acceptance of visual programming \par 
}{\f16 \par 

Resistance to the idea of visual programming often reminds me of the kind of resistance I encounter to prospects for artificial intelligence.  Seymour Papert talks about the "superhuman human" fallacy:  People expect an intelligent computer to do work surp
assing the best human efforts -- despite the fact that most people themselves couldn't pass this test!  Similarly, people may make demands on a graphical language that no textual language could fulfill either.  People might complain that visual programming
 won't work for programs hundreds of pages in length, despite the fact that only a very few programmers can completely understand such large programs in a conventional language!\par 
\par 

A visual language for programming is like any other language.  It is necessary to invest some effort in learning it before becoming fluent.   Because people aren't accustomed to seeing programs displayed as pictures, it is sometimes difficult for them to c
onceptualize how the pictures relate to the programming task.  This has made people a bit resistant to any proposed visual programming representation if they cannot see immediately how it works.  \par 
\par 

The success of the "desktop metaphor" for manipulating files on a disk has made it part of the computing culture and thus instantly understandable.  People sometimes  expect visual programming to be just as immediately understandable, despite the fact that
 programming is a much more complex task than desktop file manipulation.  The real test of a visual programming system is if a small amount of training is rewarded with a commensurate increase in power.\par 
\par 
\par 
}{\b\f16 A map analogy:  Could a textual language replace a visual one?\par 
}{\f16 \par 

Any proposal to replace textual programming languages with a graphical language faces a formidable challenge.  People already have so much invested in thinking about programs as the source code of a programming language that they find it difficult to conce
ive of how a visual representation could serve.   But the objections posed against a visual representation are often unfair.\par 
\par 
Could a visual representation for programs ever overtake a textual representation?  Let's turn the question around.  
 Suppose we consider a kind of data for which the representation has traditionally been graphical, and ask if we could substitute a textual representation instead.\par 
\par 

The example I wish to consider is representing geographic information in a map.  A map is a graphical representation of geographic relations.  Maps are highly succesful in enabling people to use geographical information.  Some aspects of a map, like the sh
ape of coastlines, are pictorial, but most features of a map are iconic and highly abstract
ed, such as representations of cities, roads, and topographic features.  Maps are a specialized graphical language, and it is necessary to have some experience to be able to read them well.  \par 
\par 

Now suppose we proposed throwing away the graphical representation of maps in favor of a textual representation.  Instead we could represent the geographical relationships by  such textual devices as a table of mileages between cities, or lists of stations
 along train lines or exits along highways.  For certain tasks,
 such as route planning for specific trips, the textual representation might even be more direct.  We could even make an argument that the textual representation holds more information.  If we print the assertions using the smallest type size used to label
 cities on a graphical map, we could perhaps squeeze in a higher density of information onto a given surface area of paper.  \par 
\par 
Yet nobody would agree that we should throw away the graphical representation of maps.  The graphics used on maps is highly effec
tive in making geographical relationships directly perceivable, whereas tracing through mileage tables would be tedious.  Graphics allows direct perception of different levels of detail in the map;  one can focus either on large-scale or small-scale featur
es, whereas a textual representation would display all information at the same size.  \par 
\par 
The textual tables of geometric relations are analogous to the source code of programs as they are currently represented, in textual form.   They display declarative in
fomation in a linear, visually undifferentiated form.  Tracing the control flow of programs is like tracing a route through a mileage table, or making train connections. \par 
\par 

Many of the same conceptual issues facing maps also arise in program representations.  Visually differentiating data at different scales and of different types is important for level of detail control that prevents the user from being overwhelmed with a la
rge quantity of information.   Making proximity and locality relations easily percei
vable is important.  Hopefully, we will develop visual representations that are as effective for programs as maps are for the visual representation of geographical information.  \par 
\par 
}{\b\f16\fs0 Synergy with graphic design will be a powerful force for improving visual programming\par 
}{\f16\fs0 \par 
Recently, I have been working with graphic designers at the Visible Language Worshop of the Media Laboratory at MIT.  I have become convinced that a lot can be learned from graphic design which is applicable to the design of visual interfaces fo
r programming.  \par 
\par 

Graphic design is about the art of arranging visual forms to aid communication, traditionally in print media but now increasingly in electronic media.  Graphic designers invent visual languages, which is not so different in some ways from inventing compute
r languages.  Their expertise is therefore valuable in deciding what kinds of visual representations are appropriate for communicating the ideas inherent in programing systems.  Graphic designers are now routinely called in by software co
mpanies as consultants to critique the appearance of interfaces, but almost always for end-user interfaces and only rarely for interfaces used by programmers.\par 
\par 

Visual languages are also important for extending the computational power of programming to users who, like graphic designers, are visual thinkers.   As a matter of style, some people can better express their thoughts visually than verbally, and these peop
le are often disenfranchised by text-based computer interfaces.   Culturally, computer programmi
ng grew up out of the culture of mathematics and engineering, where people tend to be verbal thinkers.  Now, applications like desktop publishing, multi-media design, animation, and architecture are drawing more visually-oriented people into the programmin
g world.  \par 
\par 
}{\b\f16\fs0 Programming environments can be both textual and visual\par 
}{\f16\fs0 \par 
Finally, we need not make a sharp distinction between textual and graphical programing environments.  I believe the most successful programming environments of the future will present, no
t just a single representation of programs, but many.  The computer can take care of automatically converting one representation to another.  Some of these representations can be textual, others pictorial, others combining both.  \par 
\par 

We have already seen how graphically oriented operations are creeping into programming environments.  Even though the representation of most programs remains textual, the text is manipulated on the screen by graphical desktops, multiple windows, and icons.
  \par 
\par 
Increasingly, progra
ms are beginning to themselves deal with a graphical environment of icons, windows, fonts, bitmaps, and drawings.  The techniques for referencing these objects and operations upon them from a textual programming language -- file names, reference numbers, a
rray data -- are becoming increasingly awkward, so pressure will increase for including direct visual reference and editing for these objects.  The recent flurry of "interactive interface editors" such as the Next Interface Builder is the first step in thi
s direction.  Thus, visual programming will inevitably be adopted little by little, as the osmosis of such tools makes its way into conventional programming systems.\par 
}{\f16 \par 
}{\b\f16 Seemingly radical ideas can achieve sudden acceptance\par 
}{\f16 \par 

It is instructive to follow the history of other seemingly radical ideas in the history of interactive user interfaces.  I predict that the acceptance of visual programming will parallel that of another related interface technique that once seemed hopeless
ly utopian:  hypertext.   The idea
 of hypertext dates back to the 1945 article by Vannevar Bush [Bush 45], and was dramatically popularized by Ted Nelson's book Computer Lib [Nelson 74] in the mid seventies.  The first succesful commercial product in this area, Apple's Hypercard, broke ver
y little new technical ground, but was well engineered and efficiently implemented, and widely distributed at no cost.  I expect the same sort of sudden popularity will occur with a suitably practical visual programming system, without requiring any drasti
c technological breakthrough.     Recent progress in hypertext, and more generally, hypermedia including graphics and sound, is in fact an enabling technology for visual programming.   I would be very surprised if the programming environments of the 21st c
entury did not turn out to have a strong visual component.  \par 
\par 
\par 
}}