with Simon Shelley
New: this project is also covered on d-touch.org.
audio d-touch is a collection of 3 tangible interface applications for music composition and performance: the Augmented Stave, the Tangible Drum Machine and the Physical Sequencer.
They can be though as toy-like computer music instruments. Each instrument includes a number of wooden blocks, and the interaction takes place on a flat surface, like a table top. The blocks represent different sounds, by spatially arranging the blocks on the surface users can create sequences of piano notes, drum sounds or generic audio clips.
Here are some illustrative videos (clik on them to play). Please see also the Physical Sequencer Live.
Augmented Stave
divx video (15 megs)
Tangible Drum Machine
divx video (17 megs)
Physical Sequencer
divx video (36 megs)
A standard personal computer observes the blocks through a low cost web-cam, and thanks to the d-touch computer vision system it can localize them precisely. The information about the position and orientation of each block is used to control a digital audio synthesis process. The system was designed to be extremely low cost (it only requires a consumer-grade PC with a sound card and a web-cam, inexpensive wooden bricks tagged with the markers and a printed piece of paper covering the interactive surface), robust and easy to set up, which makes it ideal for domestic and school environments.
A paper describing audio d-touch was presented at the DAFx03 Conference, London, September 2003. Informal observations revealed that the mapping strategy employed make the interfaces very intuitive and simple to use, ongoing research aims at generalizing these observations.
The design of audio d-touch was inspired by a passion for electronic music (electronica) and minimalism. Sharing the same musical interests Simon Shelley got immediately excited about the project: the collaboration resulted in the incredibly fast production of a number of prototypes, thanks also to the great encouragement that we received from the rest of the Media Engineering Group in York. From the technical point of view audio d-touch is written entirely in C++ in Linux, using a multi-threaded architecture to achieve low latency and robustness.
If you are interested in using audio d-touch for performance or research feel free to contact me.