Keywords
human interface, Human Computer Interaction, tangible user interfaces, tactile communication, fingerspelling, deaf-blind, mobility aids, blindness, paging, telecommunications, multimodal sensory perception
BACKGROUND AND STRUCTURE OF THE PAPER
This paper describes the research to date in the ComTouch project, a multimodal sensory communication system to provide an additional channel for emotional communication. The combination method for tactile and audio information is described. The aim is to build a device to send or receive tactile sensations remotely.
This paper begins with a brief review of research on existing tactile languages, and features of such languages that are desired in a tactile remote communication device. Next, a description of the set of available tactile stimuli is also discussed. The combination of tactile stimuli with other sensory modalities is discussed.
Fundamental design issues of what information to encode and how to present the information are discussed. Finally we discuss the possible experiments to assess the performance communication ability of the device. In this paper we describe the process so far in our development of a tactile remote communication device. Scenarios of use for a tactile remote communication are described in order to aid in visualizing the possible usage of the device. A description of the existing state of research is presented. Furthermore, issues of language and implementation of the device are discussed. Finally, experiments are proposed to aid in evaluation of the device.
TACTILE LANGUAGES
The term tactile refers to the sense of touch. Tactile perception is the ability to interpret and give meaning to sensory stimuli in response to tactile stimulation. Tactile language refers to communication methods that employ the sense of touch. Tactile language can be subdivided into two classes of languages, alphabetic and symbolic. The first class, alphabetic language uses the representation of the alphanumeric letters to form words. Examples are chording keyboards, Braille, Moon, and telegraphs.
Conversely, symbolic language represents higher-level concepts that are not mapped directly to words, but rather, ideas and expressive emotions. Examples of symbolic language are facial expressions, hand gestures and body language for expressing interest and emotional state. Research has shown that the rate of transmission of alphanumeric letters is much slower than symbolic language, while accuracy in alphanumeric languages is much higher than in symbolic language (Reed, 1990).
Symbolic and alphabetic language can be combined in a language. For example, Morse code is one example of such a combination of methods. In one study of Morse code, users started out by learning letters individual letters. As the time of usage increased, a symbolic language emerged and it then became hard to distinguish individual letters in transmission. Furthermore, users were able to recognize whole sentences using shorthand and able to perform simultaneous translation of speech in addition to decoding Morse messages (Tan, 1997). Fingerspelling, a tactile language where the pressure and movement of one hand is received on another hand, is another example of a tactile language that has capabilities for both symbolic and alphabetic language.
COUPLING OF TOUCH WITH SENSORY MODALITIES
While the sole sense of touch in communication can be effective (Geldard, 1967), the use of touch in combination with other senses reinforces perception and communication. The following is a brief overview of research on tactile stimulation in combination with other sensory modalities.
Touch and Audition
Audition is typically a broadcast medium. By coupling the audio channel with the private sense of touch, information can draw more attention. Research on vibrotactile devices such as the Tactaid and Tactuator (Tan, 1996), show that speech recognition increases dramatically when audio and touch input are combined in speech reading (Reed 1995, Tan 1997). Tadoma is a method of speech reading using touch, where the receiver places his thumbs on the lips of the speaker, with fingers on the throat. Intonation information was available from the vibrotactile stimuli at levels reliably about 70% (Aeur, et.al, 1999).
Touch and Smell
Experiments on smell and memory prove that there is a connection between memory recall and smell (Aggleton, 1998). Herz has done much work on exploring the coupling of touch and smell in emotion. Ehrlichman (1998) has used smell to recreate moods. Scratch-and-Sniff technology is a fad technology that is no longer popular. Recently, a company called DigiSense has introduced a device for coupling smell with web browsing.
Touch and Vision
Deb Roy has done research on the coupling the audio and visual input to result in faster perception. Combining audio and video senses was done on machine learning and cognition (Roy, 1997), as well as on children and animals have proven the effectiveness of combining audio and visual stimulation on reflexes and perception (?).
DISPLAY OF TACTILE STIMULI
Currently there are two main modes for representing tactile stimuli. One is by means of static contact pressure, and using force feedback to display information. The second means is by use of vibration signals. Furthermore, for each display, there can be variance in the location of the display, then number of stimuli presented on the display, as well as the duration of the information presented.
Static Displays
Much work in virtual reality is concerned with haptic representation of force on the fingertips. Devices such as the Phantom use force-feedback to simulate the reaction force of pressure exerted from a physical object. Assistive devices, such as the Braille Note use arrays of raised pins to represent letters that can be felt underneath the fingertip. In fingerspelling, the pressure and movement of fingers from one hand on another allows communication.
Vibration Displays
The displays for vibrotactile stimuli can be located anywhere on the body. The Tadoma method uses vibration near the mouth for input. Also, distributed multiple stimulations over different areas throughout the body is commonly used. For example, speakers are embedded throughout the body (e.g. wrist, elbows, armpits and lower back) in Gunther’s SkinScape.
The sensory Saltation phenomenon, also called the “cutaneous rabbit” describes the illusion of movement as a result of patterns of stimulation over an area of the skin (Geldard, 1972). The variation of location of vibration can allow gestural information to be displayed. The variation of duration and gaps in the signal may be relevant to the display of tactile stimuli.
FEATURES OF A TOUCH LANGUAGE DEVICE
To aid the design decisions of communication devices, a look at the terminology and features of communication devices are presented below.
Description of Communication Terminology
Direction of communication
Bi-directional means that each participant will have the ability to send and receive signals. In modems and telephones, each device can send or receive information. Unidirectional means that the participation is limited to sending or receiving. For example, broadcasting information, such as radio or television are unidirectional modes of communication.
Management of Flow Control
Asynchronous communication means that users can interrupt each other because they can send and receive at the same time. The users can grab possession of the communication channel at any time, even at the same time. This is opposite to synchronous communication, where one person can transmit without interruption. In such cases, an arbitrator might assign the turn taking.
Mapping of modalities
Symmetric mappings result in direct transmission of the data. An example of symmetric mapping is the inTouch, where rotation input is directly represented in rotation output. Asymmetric mappings are defined by translating the input modality to the output modality. Previous work on symmetric mappings using tangible interfaces were shown to result in users fighting for control .By separating the input and output channels, users will be prevented from interrupting the others transmission.
Data type
Discrete means that the content of the communication is distinguishable into discrete components. Digital communication using bytes, as used in modem communication, allows error detection of lost information in transmission. Analog is the ability to communicate continuous, time-varying signals allows more emotional content in communication.
TACTUAL COMMUNICATION SCENARIOS
Brief descriptions of possible tactile communication scenarios are listed here to highlight the benefits and intended application for ComTouch.
Situations requiring privacy
Where audio communication is impossible, a touch-based device can provide a private channel for communication. For example, one might wish to remain connected even when inside a library. Touch based communication can allow discreet notification of personal messages without broadcasting an interruption to others.
Providing an emotional communication channel
Loved ones, when separated, often want to communicate personal messages (Tollmar, 2000). If the modality of touch was available, being able to express higher-level emotions might be desired. For example, when one partner is in a meeting, the other one might want to express support without interrupting or broadcasting the signal.
Multiplexing of information
In places where remote communication already takes place, touch devices can allow people to further their range of communication by multiplexing existing communication channels For example, politicians would be able to talk and get feedback from their advisors about how the audience is receiving them during a live debate.
Special needs users
Existing technologies limit the ability for the deaf-blind to communicate. In a wireless touch-based communication system, a deaf-blind person could communicate remotely with anyone who has a sense of touch. The ability to broadcast a touch signal might also allow collaboration and communication among remotely located deaf-blind users.
CURRENT DEVICE DESIGN: THE COMTOUCH
Figure 1: Existing protoype describing the input and output vibrotactile mapping of the ComTouch device
Current State of the Prototype
The first prototype is a device that can be used by one hand. The device consists of a pad that allows each finger to squeeze independently. Force sensors under the tip of each finger register the squeeze pressure. The force of squeeze is translated to the intensity of the vibration. A local feedback area allows the user to gauge the force of her signal. The component features of a touch communication language we propose are squeeze force and the duration of vibration under each finger. A local feedback channel exists to allow the user to gauge the force of her signal. When transmitting, one device will register the analog force of pressure from each finger as the object is squeezed and transmit the resulting vibrations under each finger.
Currently, the system allows two users to communicate vibrotactile stimulation to one another by pressing down on the fingertip. The system currently demonstrates the bi-directionality of communication channels, the analog squeeze and vibration qualities of the modalities, the asynchronous communication and the asymmetry of the modalities employed in vibrotactile communication.
Implementation Details
The current device uses a force-sensing resistor (FSR) to measure the pressure. The analog pressure is translated into a voltage drop across the resistor. This voltage drop is the input to a voltage controlled oscillator (VCO) circuit, which translates the pressure into a frequency. When pressed, the voltage output by the VCO is 250 Hz, the frequency to which the output speakers are tuned. The signal is passed through an audio amplifier before the sending the signal through the speakers, which result in the output vibration.
Language of touch
The communication language will have a syntax and grammar such that high-level of concepts are communicated. In addition, it is important that support for alphabetic language exists. Thus, both conceptual content and specific details are communicated. The problem of language design at the moment remains unanswered. Possible solutions are to use the one-handed Braille coding method, or chording keyboard for basic alphanumeric sending. User testing will help determine the possibility of support for higher-level conceptual languages.
Communication Features of A ComTouch Device
Many of the device design features are derived from the example of the modern telephone. The communication channel should be bi-directional and asynchronous, as well as allow for interruption.
Feedback channel for the user
A feedback channel exists for the
user, so that as she is communicating, there is some feedback for what is sent.
Even in telephones there is a small feedback channel so that the user can gauge
how others receive their communication.
FUTURE WORK
The following is a list of issues that remain pending for this research project.
Ergonomics
The current device is meant for single-handed use, but the exact mapping onto the hand has not been decided. First, there is the issue of whether the device should be designed for primarily left- or right-handed use. Second, the implementation of a strap is planned for separating the act holding of the device from listening, as users could inadvertently send signals when simply holding the device. Next, determination of the best position for the input/output sensors is necessary. In particular, some ideas to overlap the input FSRs with the local feedback, or different arrangements of the sensors (e.g. an array similar to Braille, vs. a finger-to-stimuli mapping). Finally, the possibility of ways of reducing crosstalk between actuators has yet to be explored.
Addition of Audio Signal
The current vision is to implement the addition of audio processing into the device. In particular, translating the audio channel into vibrotactile stimuli would allow users who are comfortable with audio expression communicate with users who rely on tactile expression. This way, we would be able to use the comtouch technology with existing wireless communication devices, e.g. a cell phone.
EVALUATION
In order to evaluate the communication ability of the ComTouch device, a sets of hypothesis and tests are proposed below.
1 Simultaneous Transmission of separate modalities
Hypothesis: We can do some simultaneous transmission because of the low bandwidth of attention needed by the device
Test: See if subjects can simultaneously talk and send and receive the ComTouch at the same time.
2 Separate skills vs. attention dividing skill.
Hypothesis: We can do some simultaneous transmission because of the low bandwidth of attention needed by the device
Test: Observe if subjects can simultaneously send and receive ComTouch signals at the same time. It would be interesting to note the turn taking that takes place.
3 Broadcasting Tactile Sign Language is possible
Hypothesis: Tactile broadcasting can be achieved
Test: Have a broadcast session with the ComTouch where there are many receivers and one sender. Detect the rates of accuracy in perception by the receivers.
4 Test of Localization of Input and Output space allows better communication.
Hypothesis: The transfer of knowledge is quicker, as in TUI because the center of attention is not spatially divided over different parts of the body.
Previous research suggests that separating the incoming signal from the transmission signal might be hard if the input and output space is overlapping (e.g. Fogg, 1998), so the separation of input and output space a little bit may or may not be good.
Test: Can be performed by allowing a subject to feed a set of inputs into a computer. The computer then plays it back in random order and it can be determined whether the subject received the output correctly?
5 Touch messaging
Hypothesis: The ability to support replay of messages is possible.
Test: Record a ComTouch message and then replay it to different subjects. Test the accuracy of perception.
6 Continuity and Discreteness of Touch Language corresponds to pattern of linguistic content.
The continuity of inTouch allowed some emotional communication. ComTouch should be able to support both continuous and discrete transmission.
Hypothesis: Transition from discrete language to continuous language can be identified.
Test: Record and make a transcript of usage. Later review the transcript and identify transition from emotional to linguistic content.
7 Test of affective communication
Hypothesis: ComTouch allows for affective communication.
The test would be to observe audio language when use d in conjunction with the ComTouch. Later the experimenter will count occurrences of emotional language, and whether the subjects coincided the emotional component in usage of the device with their audio language.
Figure 2: A conceptual drawing depicting the ComTouch integrated with a cell phone device
CONCLUSION
In conclusion, we have described the research done in developing a vibrotactile remote communication device. By using tactile vibrators coupled with an audio interface, we hope to provide an additional communication channel for all users. In addition, the notion of providing an audio-tactile cross-communication channel between users who are deaf-blind and those who only know speech can broaden understanding and communication between deaf-blind users and others.
One of the ongoing themes in the project is how the information should be presented and used by subjects. The use of vibrators for the low level communication has proved popular in communication devices (e.g. pager), but there are issues of ergonomics, robustness, ease of use and cost if the device is to become widely used. The implementation is far from complete, currently; the audio integration channel is in development. At the moment, we hope that the solution to the ergonomics, language, and performance issues will become clearer as we continue research.
REFERENCES
1. Ackerman, D; A natural history of the senses, New York : Random House, 1990.
2. Aggleton, J.P. and Waskett, L. (1999), The ability of odours to serve as state-dependent cues for real-world memories: Can Viking smells aid the recall of Viking experiences? British Journal of Society, British Journal of Psychology, 90, 1-7.
3. Auer, E.T.,Jr, and Bernstein, L.E. Temporal and spatio-temporal vibrotactile displays for voice fundamental frequency: An initial evaluation of new vibrotactile speech perception aid with normal-hearing and hearing-impaired individuals. Journal of Acoustical of Society of America. 104, 4. October1998, 2477-2489.
4. Boff, K. R., Kaufman, L. and Thomas, J. P. (Eds) Handbook of Perception and Human Performance, 2, Cognitive processes and performance; John Wiley & Sons; New York, 1986
5. Brave, S. and Dahley, A., inTouch: A Medium for Haptic Interpersonal Communication, Extended Abstracts of CHI ‘97, ACM Press, 363-364.
6. Burdea, Grigore C. Force and Touch Feedback for Virtual Reality, John Wiley & Sons; New York, 1996.
7. Burdea, Grigore C. Multimodal Virtual Reality: Input/Output Devices, System Integration and Human Factors, International Journal of Human-Computer Interaction, Special Issue on Human-Virutal Environment Interaction, invited article 8, No. 1, 5-24.
8. Dodge, C. The Bed: A Medium for Intimate Communication. Extended Abstracts of CHI’97, ACM Press, 371-372.
9. Ehrlichman, H. Haplern, Jack N. (1998), Affect and memory: Effects of pleasant and unpleasant odors of retrieval of happy and unhappy memories. Journal of Personality and Social Psychology, 55, 5, 769-779.
10. Fogg, B.J, Cutler, L.D., Arnold, P. and Eisbach, C., HandJive: a device for interpersonal haptic entertainment, CHI ’98, ACM Press, 57-64.
11. Geldard, F., and Sherrick, C., The cutaneous rabbit: A perceptual illusion,“ Science, 178, 1972, 178-179.
12. Geldard, F.A. Body English, Random House, 1967.
13. Green, C.D. Online resources of William James’ The Principles of Psychology (1890) http://psychclassics.yorku.ca/James/Principles/prin20.htm
14. Halloway, M., The Ascent of Scent, Scientific American, online http://www.sciam.com/1999/1199issue/1199profile.html
15. Lederman, S. J. and Klatzky, R. L. Hand Movements: A Window Into Haptic Object Recognition Cognitive Psychology; 19, No. 3, pp. 342-368, 1987.
16. Loomis, J. and Lederman, S.J. Tactual perception.
17. Lumbreras, M. and Jaime Sánchez, J. Interactive 3D sound hyperstories for blind children. Proceedings of CHI ‘99, ACM Press, 318-325.
18. Reed, C.M., Delhorne, L, and Durlach, N. A Study of th Tactual and Visual Reception of Fingerspelling, Journal of Speech and Hearing research, 33, December 1990, 786-797.
19. Reed, C.M., and Delhorne, L.A., Current results of field study of adult users of tactile aids, Seminars of Hearing, 16, 305-315, 1995.
20. Ritm D, A computational model of word learning from multimodal sensory input. International conference of Cognitive Modeling, Groningen, Netherlands, March 2000.
21. Roy, D. "Multimodal Adaptive Interfaces", MIT Media Lab Ph.D. General Exam Paper (also Vision and Modeling Technical Report #438, MIT Media Lab, 1997.
22. Strong, R., and W. Gaver. Feather, Scent, and Shaker: Supporting Simple Intimacy. Proceedings of CSCW '96 (Boston MA, November 1996), ACM Press, 29-30.
23. Sturman, D.J., Zeitzer, David; A Design Method for “Whole-Hand” Human-Computer Interaction. ACM Trans. Inf. Syst. 11, 3 (Jul. 1993), ACM Press, 219-238.
24. Tan, H. Z., Srinivasan, M. A., Eberman, B., and Cheng, B., Human Factors for the Design of Force-reflecting Haptic Interfaces, Proceedings of the ASME Dynamic Systems and Control Division, DSC, 55, 353-359, 1994.
25.Tan, H.Z., Durlach, N., Rabinowitz, W. Reed, C.M, and Santos, J. Reception of Morse Code through Motional, Vibrotactile and Auditory Stimulation Perception & Psychophysics, 59, 7, 1997.
26. Tan, H. Z., Perceptual user interfaces: haptic interfaces; Communications of the ACM 43, 3 (Mar. 2000), 40 – 41.
27. Tan, H., Lim, A., and Traylor, R. A Psychophysical Study of Sensory Saltation with an Open Response Paradigm. ASME Dynamic Systems and Control Division 2000, 2, 1109-1116, 2000.
28. Hong Z. Tan William M. Rabinowitz Nathaniel I. Durlach, Analysis of a synthetic Tadoma system as a multidimensional tactile display, Journal of Acoustical Society of America, 86, No. 3, , 981 – 988, September 1989.
29. Tan, H.Z., Durlach, N.I., Rabinowitz, W.M., Reed, C.M., Information Transmission with a Multi-finger Tactual Display, Proceedings of International Sensory Aid Conference, 1996.
30.Tollmar, K., Junestrand, S., and Torgny, O., Virtually Living Together.DIS’00, ACM Press, 83-91, Brooklyn NY, August 2000.
31.Digiscents, a complete hardware and software platform for integrating sense of smell with multimedia http://www.digiscents.com/prod/index.shtml