About
MirrorFugue is a set of interfaces for the piano to visualize the gesture of a performance. Based on the idea that the visibility of gesture contributes to learning and synchronization, MirrorFugue displays the hand and body movements of piano playing using metaphors from the physical world to connect musicians from disparate spaces and times.
We designed two modes to visualize the hand gesture of a performance, which we term "Reflection" and "Organ". Inspired by the reflective surface on a lacquered grand piano that mirrors the keyboard and player's hands, Reflection mode shows the mirrored keyboard and hands of a performance. Organ mode displays the unaltered, top-town image of the collaborator's keyboard like the aligned and offset keyboards of an organ.
MirrorFugue can be used in remote lessons to enable teachers and students to see each other's playing. It could also be used to playback recordings of the self and others. A student can use MirrorFugue for practice support by recording a portion of a piece playing a duet with the recording. By displaying the body and the hands at full scale on the surface of the instrument, MirrorFugue evokes the presence of a pianist with a sense of intimate emotional expression.
Images
Closeup of Organ Mode
Organ Mode in Action
Reflection Mode Diagram
Hands-only Prototype - 1
Hands-only Prototype - 2
Hands-only Prototype - 3
Credits
- Advisors
- Hiroshi Ishii, MIT Media Lab
- Tod Machover, MIT Media Lab
- Ken Perlin, NYU Media Research Lab
- Donal Fox, MIT Music Dept.
- Thanks to
- Lauren Gust, Piano case design
- Paula Aguilera, Video Content
- Jonathan Williams, Video Content
- Avid, Keyboard donations
- Monica Gallegos, Photo Credit
- Brygg Ullmer, Photo Credit
- Julia Ma, Photo Credit
- Playing Piano in Photos
- Ken Perlin
- Donal Fox
- Marvin Minsky
- Akiko Sugaya
- Alisa Ishii
- Lena Ishii
Publications
[pdf] Xiao Xiao and Hiroshi Ishii. 2010. MirrorFugue: communicating hand gesture in remote piano collaboration. In Proceedings of the fifth international conference on Tangible, embedded, and embodied interaction (TEI '11). ACM, New York, NY, USA, 13-20.
[pdf] Xiao Xiao and Hiroshi Ishii. 2011. Duet for solo piano: MirrorFugue for single user playing with recorded performances. In Proceedings of the 30th international conference extended abstracts on Human factors in computing systems(CHI EA '11). ACM, New York, NY, USA [pdf] Xiao Xiao. MirrorFugue: Communicating Presence in Musical Collaboration Across Space and Time. MAS Thesis. Massachusetts Institute of Technology, USA. 2011.