HAPPY++

Robert R. Morris | Javier Hernandez | Dan McDuff

About

This project uses your smile to control an audio mix.
As you smile, the song blossoms from a low-fi drum track to a full, hi-fi stereo mix.

Components:

  1. An ordinary webcam
  2. Computer vision algorithms to track multiple smiles
  3. Audio software to map smile intensities and durations to musical parameters

The technology was deployed as an installation for the MIT Media Lab Spring Sponsor Event. Technology from the MIT Mood Meter was used to track multiple faces at once. This allowed people to control the audio mix as a group. As the group became happier, the music responded in tandem, growing louder and more vibrant.

Outcomes

Perhaps the most interesting outcome from this project was the sensation that the technology was somehow eavesdropping on conversations. Even when a group stopped attending to the installation, the technology continued to track everyone's facial expressions. If the group erupted in laughter and began smiling, the technology immediately followed suit and the music became more resplendent. It was almost as though the technology was in tune with the affective contours of the conversation and had the ability to react just like everyone else, albeit in a language of its own.

An interesting follow-up to this work would be to create a suite of affective sounding-boards in the home - for instance, chairs, TVs, and other household objects that are programmed to react sympathetically to the emotional tone of the people in the room.