Gestural Media Framework
Performance artists working with digital media often use human gestures to control and shape that media as part of a theatrical, choreographic, or musical performance. In my own work, I have found that strong, semantically-meaningful mappings between gesture and sound or visuals are necessary to create compelling performance interactions. However, the existing systems for developing mappings between incoming data streams and output media have extremely low-level concepts of gesture; the actual programming process focuses on low-level sensor data, such as the voltage values of a particular sensor. Thus, it is challenging to later reconfigure or expand gestural mappings, or to experiment with flexible mappings in a rehearsal setting. Also, much of the expressive content of movement is lost when we only focus on sensor readings rather than meaningful gestures or qualities of gesture.
As part of my master's thesis work at the MIT Media Lab, I created The Gestural Media Framework, a new development environment for gestural control of media in rehearsal and performance, allowing users to create clear and intuitive mappings in a simple and flexible manner by using high-level descriptions of gestures and of gestural qualities. Numerous mapping systems for digital media have been developed, but few of these systems incorporate gesture representations, or even much conception of gesture. I designed a system with a vocabulary of abstracted gesture and gesture quality objects, allowing for higher-level control of mappings. As part of the evaluation of the current version of this system, I choreographed a set of four pieces for public performance that uses this system to map dancers’ movements to control sound and visual elements, including video projection.
This dance/theater performance work, Four Asynchronicities on the Theme of Contact, was performed in Februrary 2010. In this performance piece in four movements, the performers' gestures and qualities of movements are used to control and shape media elements in real time, exploring the shifting and complex relationships between individuals, between the body and technology, and between the past, present, and future.
The First Movement was a duet in which the performer's movement affected the theatrial lighting.
In the Second Movement, a solo performer shaped a complex soundscape through her dance.
The Third Movement was a duet where the performers created a musical score by the variations of their interactions.
In the Fourth Movement, the quickly shifting relationships between all five performers were echoed by fluid projected visualizations.
An early version of the Gestural Media Framework mapping system.
Related publication: Jessop, E. "A Gestural Media Framework: Tools for Expressive Gesture Recognition and Mapping in Rehearsal and Performance." M.S. Thesis. MIT Media Laboratory, 2010.