Haptics -- the sense of touch--is perhaps the most complicated of the senses in terms of its employment at the human/computer interface. What is especially daunting is the difficulty in devising apparatus that doesn't encumber the user, yet lets the user employ their sense of touch directly and effectively.
The haptic sense is complex as well in that it includes the perception of texture, pressure, temperature, position, and movement. At times, these elements within the haptic sense can be confounded one with the other.
A case in point is the coin-bending illusion, as when you take either side of a small coin between the thumb and forefinger of the hands, press hard, and proceed to move the coin back and forth as if to "bend" it. After a moment or two, you begin to experience the coin at your fingertips as alternately bending now this way, now that way, despite your knowing that the coin is far too rigid and strong for it actually to be bending.
What is happening is the activation of nerves in your finger tips which register deformation in the flesh as you manipulate the coin. Somehow, your brain ascribes at least part of that deformation to a change of shape in the coin.
The effect is possibly related to what occurs in the case of a certain visual phenomenon, an effect called termed "color scission."
The visual effect of color scission was first described in 1925 by Grace and Fritz Heider of Smith College, and revisited later by Metelli (1974, 1970). It works as follows.
Consider a background of two colors, like so:
which you wish to overlay with a broad yellow stripe:
In doing so, you get the result:
But, should you want that yellow stripe to appear as if transparent, you might overlay the background not with a stripe of uniform yellow color but instead with two rectangles of appropriately chosen colors:
...to result in an effect of transparency:
The reason we see the overlain patches--neither of which seems to be particularly "yellowish"--as a single transparent yellow layer is, according to the notion of "color scission," that the brain "splits" the color values of the two overlying patches into two components. One component is attributed to the underlying red or green patch "showing through," and the other component is attributed to there being a transparent yellow film lying over the background colors.
In the case of the coin illusion, we might call it "haptic scission." My guess about what is happening in the case of the bending coin is that the brain ascribes part of the sensed deformation to the fleshy part of the thumb and finger tips but also some part to the coin as well. The resulting experience is that of the thumb and finger tips being depressed this way and that, and also that of the coin itself bending.
As in transparency, the brain differentially ascribes tactile sensation to one's own movements and also assigns movement or deformation to whatever is being probed, in this case the coin.
Another instance of what I term "haptic scission" I suspect occurred in connection with a piece of drafting apparatus with which we used to experiment with way back in early 70's at the former Architecture Machine Group (precursor to the MIT Media Lab). The apparatus in question was a large tilted drafting table. The table had mounted at its left hand edge a long jointed mechanical "arm." At the distal end of the arm was a pen which could be lowered or raised under computer control to produce drawings. There was also a joystick mounted on the end of the arm which you could grasp and whereby you could move the arm around.
However, it was not you who moved the arm, at least not directly. That was done by a set of servo-motors. When you pushed or pulled the joystick in some direction, the computer operating the table would sense the direction in which you were pushing or pulling, as well as the force you were applying. The computer would then actuate the servo-motors to move the arm about.
The table's computer would also take into account whatever "terrain" you might be modeling in its data base. That terrain might contain hilly ups and downs, posess solid barriers beyond it would not let you push, or represent a perfectly flat, uniform plain modelled so as to reflect some (perhaps varying) coefficient of friction about its surface. Accordingly, the force you needed to apply to the stylus to move the arm in any direction could vary from very light to very hard, depending upon what kind of terrain the computer was modeling.We used the drafting table in various projects, one of which was using the joystick to "experience" terrain contours. That is , you would push the arm about using the end of the joystick, and in so doing you would directly feel the contours or whatever geography was modeled in the machine database. As you pushed uphill, the servos controlling the arm's would go slower, and give you an impression of going uphill by increasing the arm's resistance to being moved. Conversely, if you were going downhill, the servos in the arm would go faster, giving you the sensation of the arm being easier to push. The "terrain" being experienced could be concrete, representing either real or simulated geography, or could be abstract, representing for instance, economic trends. The presumption was that we could model a variety of "terrain," and thereby give the user a direct sense of that terrain via direct feedback.
But--as I reflect retroactively on the nature of the apparatus--I am not so sure that the scheme would and could have worked all that well. Why? Precisely because of the servos acting as intermediaries between our fingertips and the (simulated) resistance of the drafting arm: we would be confounding 1) the sense of our fingertips in applying force to the joystick with 2) the resistive force output by the machine as applied to those self-same finger tips.
I wonder how many remember a once-popular game: the Ouida Board? ("Ouida" was often pronounced "WeeGee.")
The game consisted of a flat piece of wood measuring maybe 1-1/2 by 2 feet, on which were stenciled the words "yes" and "no," plus the letters of the alphabet and the series of numbers 0-9. (I may have that not exactly right, having not seen a Ouida Board for some time...). The other piece of apparatus was a small three-legged table called a "planchette." The planchette usually measured about 2-3 inches in size, just big enough for you and one or two others to place their fingertips on. It was often heart-shaped, that is, with a pointed side or end.
The essence of the game was for the players to seek answers to questions, in a fortune-telling sense, by placing their fingertips on the surface of the planchette, concentrate on the question to which they sought the answer, and experience the planchette, under the touch of their fingers, take off, seemingly with a mind of its own and scuttle about the surface of the board, pointing to "yes" or to "no," or possibly spelling out messages by successively pointing to the letters or numbers on the surface of the board.
The fun of the game lay in ascribing the movement of the planchette to "spirits" who knew the answers to their inquiries, or who might be some dead or far off personage one or more of the players were supposedly thinking intently of. Because the players would have their fingertips on the surface of the planchette, no one felt themselves to be "in control" of the movements of the planchettes--though one could exert an extra guiding push if one were so inclined; but, by and large the fun of the game was to simply let the path of the planchette be the resultant of the whims, impulses, and whatever else of the players. Because no one player was in control, the path would be ascribed to no particular person, but to the "spirit" or "spirits" one might be contacting through the board.
A Ouida table...and Quida boat
I once saw on late-night television a large scale version of the Ouida Board principle. The magician was about to show how an ordinary card table could be moving about the floor by it's own power, or conceivably by "spirits." A card table was brought on, and four members of the studio audience were brought up on stage. They were asked each to stand on one side of the table and place both hands on its edge. They were told that certain spirits influences might begin to exert forces on the table, and that they should try to keep the spirits from running away with the table.
After some moments, the table began shifting small distances this way and that, then in larger and larger movements until it was careening about the stage, the four volunteers frantically trying to keep it under control..!!
What was happening, at least to my analysis, was that initially any person about the perimeter of the table would feel minor, extremely small, vibrations through their hands from the table surface arising simply from the contact of the other people's hand contact with the table. In turn, they would try to steady the table, and thus add to the table-surface vibrations their own counter-force. These efforts by the participants to steady the table would gradually increase in force and direction, each person trying to cancel out the unsteadying pushes of the others about the table. Soon, such uncoordinated steadying attempts would escalate to an erratic path of table movement about the stage. A very clever stunt, convincing to participant and viewer alike, that the table was going about of its own accord.
What happens with both the Ouida board and the gyrating table, is that any participant experiences the movement, pressures, and counter-pressures of the planchette or card table as not emanating from themselves or--coherently--from any of the other participant(s). Hence, the planchette or table seems instead to move about on its own.
I might even add in the "gondola effect," which I experienced in Venice at Carnival season. My family and I took a ferry-gondola--called a traghetto--across the Grand Canal. The locals make a point of standing during the trip, and ourselves as well as the boatload of fellow tourists were doing likewise. Someone amidships got a bit panicky, and suddenly sat down. Their motion set the traghetto to rocking, whereupon someone else--heaven knows who--began to shift their standing weight to compensate for the sitter's rocking of the boat. Yet another passenger suddenly shifted their weight to counterbalance what the other was doing, and so on and so forth about the boat. Thus ensued multiple, un-coordinated attempts by the boat's occupants to counter each others' actions and steady the boat; the result was that the boat pitched about willy-nilly. I tried simply to hunker down and pray we didn't all pitch overboard into the canal. After some suspenseful minutes, we did in fact make it to the other side of the canal without us all plunging into the water.
On all three levels--planchettes, card tables, and traghettos--what was happening was an inability of any participating individual correctly to ascribe sensed movement to some coherent source. The sensed force came from everywhere...and nowhere. And, should they decide to exert some force, the sensed feedback from that exertion would be lost amidst the larger complex of forces operating, making the experience all the more mysterious.
On a day-to-day basis, the haptic sense is rarely "pure"--that is, what we experience hapticly we experience rather in a multimodal sort of way.
When we touch a fabric at a clothing store, we see the fabric as well. In sampling texture, as when shopping for kitchen tile, we see as well as feel the texture of the surface, and perhaps hear it as well as we drag our hand across it. That last point bears note: some things we feel by weight or "heft;" others by touch, while keeping our hand steady; and, other surfaces we appreciate by dragging our fingertips to and fro across its surface.
Some time back, my UROP student Mike Agronin and I put together an apparatus designed to be a haptic display. The aim was to present virtual surfaces to the user, much as if they were being offered tile or carpet textures to sample.
The apparatus was to consist of a 1-1/2 diameter plastic disk mounted horizontally on a metal rod. The pad would be grasped between the thumb and middle finger, while the index finger would rest on the central portion of the disk. The user would then move the pad about to sense the virtual surface. A pair of servo motors would control the frictional "resistance" of the virtual surface, not unlike the drafting table apparatus described above. The more the user pressed downwards--there was a strain-gauge to be mounted in the shaft supporting the plastic disk--the more resistance would be applied to the user's movements by the pair of servo motors.
The surface of the pad was perforated, and through the perforations were set the smooth end surfaces of a set of about a dozen or so short metal rods. The other ends of the rods, each about 1/4th inch or so in length, were cemented to the surface of a speaker diaphragm. Some digital sound whose frequency would reflect the "texture" of the surface to be portrayed was played aloud at a rate proportional to the speed at which the user was moving the disk in the x, y plane. The sound signal was directed as well to the diaphragm holding the metal pins, which would vibrate against the surface of the user's index finger to portray in a rough and ready way the texture of the surface.
Additionally, we had a small heat sink available. If you passed a small direct current to the heat sink in one direction the surface of the sink would get warm; pass the current in the opposite direction, and the sink would become cold. The device was intended to act as a tiny refrigerator or a warming pad to be installed in whatever device the experimenter or hobbyist was building. But for us, it would add to the illusion of metal surfaces (cold), perhaps wood or plastic (warmer), and perhaps bring in some of the kind of heat that one might expect to arise from the friction arising from passing one's finger back and forth over a resisting surface.
The illusion of the virtual surfaces was to be visual as well. To do that, we proposed a half-silvered mirror to reflect downward in the same plane as the x, y movement of the plastic disk the face of a CRT bearing the image of the text to be simulated. The user thus would be moving their hand over what they saw to be the surface of some foot-square patch of rug or tile surface, feel its surface on the underside of their index finger, hear their finger tip move over its surface, and feel their finger tip warm up as they moved it above, and--should they press their finger down harder, feel it harder to move about over the rug or tile surface.
Mike, a student in Mechanical Engineering, was very clever and inventive in building the basic apparatus of servos motors, perforated plastic tab with the pin array, and so on. In the interests of time and practicality, though, we had to make some compromises. The horizontal surface became one-dimensional, the user able only to move in a side-to-side direction. We had to abandon adding in the heat sink; it was just too difficult to put it in place in addition to the tactile array of metal pins. What we had at the end of our experiment was a surface the user could sample by moving their finger side-to-side and pressing down with varying force, with simple sound effects and a surface of pins playing on the pad of the index finger.
Now, this apparatus was admittedly very crude--I had only one grad student working on it in the midst of his other academic commitments. But it was fun, a challenge, and it did--with much tweaking and testing-- actually work. I never got the opportunity to set it up formally and test out some subjects to see how "convincing" were the impressions of surface textures it might generate. No one would have been fooled by it, of course, into believing they were appreciating real surfaces, but-- with a dollop of imagination--they might be able to discriminate somewhat reliably amongst a set of contrasting virtual textures.
The multimodal nature of material objects was strongly brought home to me from sampling some desserts and various Boston and Cambridge cheap eats restaurants.
One instance was making gustatory discriminations between different kinds of pie. I could not--by flavor--distinguish between apple, blueberry, rhubarb or whatever...pie. They all tasted --or rather, because they had no real flavor, felt--the same. You needed to have your eyes open to distinguish them. The blueberry was blue in color, the cherry pie was red, and so forth.
Once, I asked in a restaurant what kind of gelatin dessert they had. The waitress replied, "Green, yellow, and red." Not "Lime, lemon, and raspberry (or cherry or strawberry)." She, of course, was right; any one of them was so short of real flavor that you had to see it as well as eat it, and, as you ate, try to instill a modicum of flavor by trying to recall what lime or lemon taste like...
In any event, whether you are eating at Burger King or Four Seasons, the "taste" of food is multimodal, to be sure.
Metelli, F. The perception of transparency. Scientific American, Vol. 230, April, 1974, 91-98.
Metelli, F. An algebraic development of the theory of transparency. Ergonomics, 1970, 13(1), 59-66.