An interactive display for face-to-face conversations -- DiMicco, Hollenbach

Second Messenger:
An interactive display for face-to-face conversations

Joan Morris DiMicco, Katherine Hollenbach
MIT Media Laboratory
E15-320, 20 Ames St
Cambridge, MA 02319
617-324-1863
joanie@media.mit.edu, kjhollen@mit.edu

Abstract

The majority of today's communication technology focuses on communicating over distance, yet there are many ways technology can enrich face-to-face conversations. This design exhibit, Second Messenger, demonstrates one such example of augmenting a face-to-face interaction.

By observing the verbal communication between individuals, this application uses a combination of voice recognition technology and data analysis to display a real-time visualization and text-based summary of a conversation. In conjunction with a large shared display, this application provides a method for a group of individuals to review the content of their conversation from different levels of perspective as a means of reflection.

The Interface

Second Messenger's display presents a timeline of who spoke when during a conversation. The timeline is expressed as a vertical stripe with each horizontal bar in the stripe representing approximately 10 seconds of conversation. Within each horizontal bar, the colors represent who was speaking during that time, and in what proportion.

The application is designed to run real-time, as a visual representation of the current social interaction. When running real-time, the timeline slowly builds and increases in length as the group converses. Users can explore the interface and review what was said just minutes previous, as the system is continually collects information about the verbal interaction of the group.

To explore the content of the conversation, users can mouse-over the timeline to scroll up and down through time. By clicking on the timeline, users can view the details of the conversation at that moment. The transcript of the conversation appears to the right of the timeline, again color-coded according to who spoke each utterance. Because of the low-accuracy issues of current speech recognition technology, the application replaces words our speech recognizer transcribes with low-confidence with the string "…" In this way, the display presents phrase snippets and keywords from the conversation, not a complete transcript.

Figures 1-7 show a series of screenshots of the interface during use, with explanations of the visualization. To see a working version of the interface with a static conversation, you can access a Java applet here: http://web.mit.edu/kjhollen/www/second_messenger_applet/. This applet requires a live Internet connection and allows you to browse a previously recorded conversation.

Figure 1: When the group begins conversing, Second Messenger immediately begins building a visualization of the conversation based on who has spoken. In this screenshot, two people have started to talk to each other, represented in red and green.

Figure 2: As the conversation progresses, a third person joins into the conversation and is represented with the color blue.

Figure 3: When the conversation extends the entire screen, the timeline is marked at the top and bottom with relative time stamps.

Figure 4: By clicking on different portions of the timeline, the transcript of the spoken dialogue is shown. The "..." represent portions of the dialogue the speech recognizer detected with low confidence. The remaining words are actual words spoken by the group.

Figure 5: While the words appear muddled to a reader who was not present during the conversation, the text provides many cues to the speakers and allows them to reflect what was just said.

Figure 6: As the conversation progresses, the display continues to update with new content.

Figure 7: By mousing over the timeline, users can scroll up and down through the timeline to view the conversation's entire content .

The Intention of the Interface

Our research focuses on building tools that assist groups in reflecting upon their interaction for the purpose of increasing the diversity of contribution amongst the group members during a meeting. Increasing the diversity of discussion in meetings increases overall meeting quality and outcome (Hackman, 1992) and to see our previous research in this area, please review our earlier publications (DiMicco, 2003).

Our intention in building this interface was to provide a tool for reflecting on group dynamics within the context of a face-to-face setting. Many tools and visualizations have been built for reflecting and gaining insight into large, long-term conversations, of which (Wattenberg, 2003) is one such example. In contrast, this visualization provides a way for a group to understand its small, short-term interactions.

When individuals interact in meetings they are frequently unaware of the style of communication amongst the group members and have difficulty judging how much everyone has been speaking during a meeting. Our hope is that by providing an intuitive way of reviewing this information, groups will be be able to observe dramatic imbalances in the amount each group member has spoken during a meeting.

Figures 8-10 demonstrate how a visual representation of a spoken conversation can reveal different types of group communication styles. These three visual profiles are from three different conversations with three different groupings of individuals. In order, these visualizations show chaotic exchanges, organized turn-taking, and slow transitions from one person to another. Additionally, each of these visual profiles reveals the relative involvement of each person in the conversation and their dominance over entire conversation. By providing these visual profiles, Second Messenger encourages a group to reflect upon group participation and the content of a conversation, allowing for an opportunity for the group to observe an imbalance and correct it.


8: Chaotic exchange with one person contributing less	9: Turn-taking with one person left out	10: Transition of dominance from one person to another
Figures 8-10: Visual profile of three different conversations

Technical overview

Second Messenger is built on a client-server architecture where each conversation participant wears a microphone that sends his/her spoken dialogue to a client machine running IBM's ViaVoice voice-recognition software. ViaVoice transcribes the audio input into a text stream that is then sent to the Second Messenger server for analysis and filtering. Because of the one-client-per-user setup, Second Messenger does not have to perform speaker identification and each client machine utilizes a personally-trained voice-model for each user, eliminating many of the challenges of speech recognition. The limitations of detecting conversational speech still present a challenge for transcript accuracy. Because it is important that Second Messenger detect the amounts and frequencies of speech, but not critical that it have a complete transcript of every word spoken, the application filters ViaVoice's output on the server-side based on the "phrase score" returned for each word. By setting the phrase score threshold quite high, and replacing each low scoring word with "...," we can ensure that the words sent through the semantic filter were actually spoken.

The Second Messenger server software is built using Processing, a Java-based programming language (Fry, 2004-2001). The server uses Processing to incorporate the separate ViaVoice transcripts into its singe visualization.

Conclusion

This project presents a visualization of a face-to-face conversation that conversation participants can contribute to and browse while they are in the midst of conversing. The visualization highlights who has contributed over time and the content of what was said.

The purpose of Second Messenger is to provide a "second opportunity" for members of the group to get their message across to the group. This visualization attempts to do this by presenting a visualization of all of the words spoken during a face-to-face conversation, allowing for the group to observe and reflect upon unique moments in its recent past: moments when one person spoke for a long time, moments when someone who rarely speaks made a contribution to the discussion, and moments when the group made a sudden break from its established social patterns.

References

DiMicco, J. M. and W. Bender (2004). "Second Messenger: Increasing the Visibility of Minority Viewpoints with a Face-to-face Collaboration Tool." In the Proceedings of the ACM Conference on Intelligent User Interfaces (IUI'04), Funchal, Madeira, Portugal.

Fry, B. and C. Reas (2004 - 2001). Processing Programming Language. Massachusetts Institute of Technology and Interaction Design Institute Ivrea, http://processing.org/.

Hackman, J. R. (1992). "Group influences on individuals in organizations." Handbook of Industrial and Organizational Psychology. M. D. Dunnette and L. M. Hough. 3.

Wattenberg, M. and D. Millen (2003). "Conversation thumbnails for large-scale discussions." In the Proceedings of the Conference on Human Factors in Computing Systems (CHI 2003), Ft. Lauderdale, Florida, USA.