In the early stages of my PhD, I was working on tools for creating, testing and iterating on augmented reality music videos. I called this system Cybernetic Media Production. Before discarding the idea, I developed a custom augmented reality framework. Here it is in action:

This is how it works:

  • I mounted an HTC Vive “Puck” Tracker on a DSLR camera
  • I wrote a utility that uses Valve’s OpenVR C++ library to access the HTC Vive tracking data
  • I created a 3D scene using the open source tools in OpenFrameworks.
  • Using the tracking data from the camera to I position a virtual camera in the 3D Space
  • Using the tracking data from the HTC Vive controller, I put virtual objects into the scene, and layer them over the live feed from the DSLR

During development, I discovered that it is the coincidence of many small details that makes the AR illusion effective. The latency of the video feed has to be perfectly matched with the latency of the tracking data. The field of view of the virtual camera needs to be carefully calibrated with the “real” camera. The virtual objects need to be smooth and I found that it was important to optimize the rendering pipeline. Efficiently rendering pretty, smooth lines in 3D is surprisingly difficult.

The idea behind Cybernetic Media Production was to procedurally create many different versions of a music video, and then A/B test the audio and video content the same way that big companies like Amazon and Microsoft test different versions of their websites. Microsoft was able to use A/B testing to increase advertising revenue by hundreds of millions of dollars. At the time, I was thinking “shouldn’t musicians be able to do the same thing?”

I believe that Cybernetic Media Production has commercial value. I also believe that some version of this practice is likely inevitable. The technology isn’t quite there to make it happen yet.

Initially, I intended to spend my PhD developing the technology to make Music Video A/B testing a reality. For every creative or ethical question I answered, another obstacle emerged. What kinds of artistic parameters should be A/B tested? Will A/B tested content really offer new creative opportunities? The major web companies do a good job of measuring and manipulating the public for profit, but what are the creative advantages to thinking about artwork the way that the Big Four think about web pages?

With guidance from my advisor and dissertation committee, questions like these led me to reconsider. Eventually I did move my dissertation work away from what you might call the Amazon of music production toward the development of infrastructure to support the Wikipedia of music production. Ask me about this journey if your are curious – It’s an interesting story, and it speaks to unique characteristics within the MIT Media Lab’s unconventional academic environment.

However, I was able to re-purpose the my AR framework for something Joyful. In the video below, the projections are entirely created by members of the audience who are manipulating the 3D Vive controllers in real time.