Nomadic Radio consists of client and remote server components that communicate over the wireless LAN. The Nomadic Clients, developed in Java, operate on Pentium-based wearable PCs (worn on the waist), such as the VIA Wearable from Flexipc and the Toshiba Libretto 50 mini-notebook PC. The current architecture (shown in Figure 2) relies on server processes, written in C and Perl running on Sun SPARCstations, that utilize the telephony infrastructure in the Media Lab's Speech Interface group [Schmandt94]. The servers extract information from live sources including voice-mail, email, hourly updates of ABC News, weather and traffic reports. The clients, when notified, download the appropriate text/audio files stored on the web server. On the wearable device, IR-based receivers will provide positioning data to a Position Server using the Locust Swarm [Starner97b], a distributed IR location system at the Media Lab. The Position Server will keep track of users throughout the day and provide a means to convey their activity and enable several modes of synchronous and asynchronous communication between them.

A client-server interface to speech recognition and synthesis based on AT&T's Watson API, has been developed. Text files downloaded from the server are converted to synthesized speech. Different voices and volume of feedback can be used to express different information with varying levels of urgency. The system allows speaker-independent recognition using a pre-defined vocabulary. It is currently well integrated with Nomadic Radio, such that most of the functionality can be accessible via speech-only interaction. Recognition accuracy improves if the user explicitly turns recognition on/off via a button press, hence a tactile interface coupled with speech is necessary. Tactile input is provided by a three-button wireless mouse.

An alternative to using a "push-to-talk" button, would be to detect a speaker's voice (using speaker ID and directional audio cues) or head movement towards the microphone. I am considering use of an adaptive speaker-dependent speech recognizer with prosody detection and speaker identification (ID), developed at the Media Lab by Deb Roy and Brian Clarkson [Roy97b]. Prosody would allow us to detect urgency or irritation in the intonation of the user's voice. Speaker ID would allow us to turn on/off recognition only when the user is speaking. A different mode could turn down the volume of all audio sources if it were detected that the user were in a conversation. Currently this system runs on the SGI platform and is being ported to Windows 95/NT (several performance issues need to be addressed before it can be integrated in Nomadic Radio). Another alternative for speech input is the HARK client-server speech recognizer from BBN. Here, all the processing is done on the server, which is well suited for a wearable with CPU and memory constraints.

The acoustic features of the environment i.e. the level of noise or sounds such as conversations or traffic can influence the level and type of information presented. In a related project [Sawhney97c], I found that it is possible to train a system to determine general classes of sounds from acoustic features extracted from the environmental. The results are limited and preliminary, yet offer the potential to provide such audio-based situational awareness in a wearable system.

[Schmandt94] Schmandt, Chris. "Multimedia Nomadic Services on Today's Hardware". IEEE Network, September/October 1994, pp12-21.

[Starner97b] Starner, Thad and Dana Kirsch. "The locust swarm: An environmentally-powered, networkless location and messaging system." Proceedings of the International Symposium on Wearable Computing, IEEE, October 1997.

[Roy97b] Roy, Deb K. and Alex Pentland. "Adaptive Multimodal Interfaces." Proceedings of the Workshop on Perceptual User Interfaces, Banff, Canada, October 1997.

[Sawhney97c] Sawhney, Nitin. "Situational Awareness from Environmental Sounds", Project Report for Pattie Maes, MIT Media Lab, June 1997.

Next: Simultaneous and Spatial Listening

Back: Design of Nomadic Radio

Nomadic Radio Website