Computational Plenoptic Imaging

SIGGRAPH 2012 Course
Monday, 6 August 2012, 9:00-10:30 am
Los Angeles Convention Center, Room 408B

Abstract and Motivation

Spawned by the introduction of the Lytro light field camera to the consumer market and recent accomplishments in the speed at which light can be captured [Raskar et al. 2011] , a new generation of computational cameras is emerging. By exploiting the co-design of camera optics and computational processing, these cameras capture unprecedented details of the plenoptic function - a ray-based model for light that includes the color spectrum as well as spatial, temporal, and directional variation. Although digital light sensors have greatly evolved in the last years, the visual information captured by conventional cameras has remained almost unchanged since the invention of the daguerreotype. All standard CCD and CMOS sensors integrate over the dimensions of the plenoptic function as they convert photons into electrons; in the process all visual information is irreversibly lost, except for a two-dimensional, spatially-varying subset - the common photograph. In this course, we review the plenoptic function and discuss approaches that aim at optically encoding high-dimensional visual information that is then recovered computationally in post-processing.

This course is intended to review the state of the art in joint optical light modulation and computational reconstruction of visual information transcending that captured by traditional photography. In addition to the plenoptic dimensions, we also consider high dynamic range image acquisition as common sensors have a limited dynamic range. In contrast to prior courses on general computational photography [Raskar and Tumblin 2006,2007] and a recent survey on the topic [Wetzstein et al. 2011], this course gives a broad, well-structured, and intuitive overview of all aspects of plenoptic image acquisition and focuses on two recent developments: light field acquisition and ultra-fast cameras. We unveil the secrets behind capturing light at a trillion frames per second and the Lytro camera. Our course serves as a resource for interested parties by providing a categorization of recent research and help in the identification of unexplored areas in the field.

We will discuss all aspects of plenoptic image acquisition in detail. Specifically, we begin by giving an overview of the plenoptic dimensions and show how much of this visual information is irreversibly lost in conventional image acquisition. We proceed by discussing the state of the art in joint optical modulation and computation reconstruction for the acquisition of high dynamic range imagery as well as spectral information. Two parts, focusing on light field acquisition and ultra-fast optics respectively, will unveil the secrets behind imaging techniques that have recently been featured in the news. We outline other aspects of light that are of interest for various applications and wrap the course up with a short summary, while leaving enough time for questions and a short discussion.


This introductory-level course has no prerequisites.

Course Notes and Slides:

SIGGRAPH 2012 Computational Plenoptic Imaging Course - 5 Imaging at a Trillion Frames per Second [please see Ramesh Raskar's TED talk]

SIGGRAPH 2012 Computational Plenoptic Imaging Course - 6 Inside the Lytro Light Field Camera [slides not publicly available]


G. Wetzstein, I. Ihrke, D. Lanman, W. Heidrich, K. Akeley, R. Raskar. Computational Plenoptic Imaging. ACM SIGGRAPH 2012 Course, 2012.


      author = {G. Wetzstein and I. Ihrke and D. Lanman and W. Heidrich and K. Akeley and R. Raskar},
      title = {{Computational Plenoptic Imaging}},
      howpublished = {ACM SIGGRAPH Course Notes},
      year = {2012},


Gordon Wetzstein . MIT Media Lab

Bio: Gordon Wetzstein is a Postdoctoral Researcher at the MIT Media Lab. His research interests include light field and high dynamic range displays, projector-camera systems, computational optics, computational photography, computer vision, computer graphics, and augmented reality. Gordon received a Diplom in Media System Science with Honors from the Bauhaus-University Weimar in 2006 and a Ph.D. in Computer Science at the University of British Columbia in 2011. His doctoral dissertation focuses on computational light modulation for image acquisition and display and won the Alain Fournier Ph.D. Dissertation Annual Award. He organized the IEEE CVPR 2012 Workshop on Computational Cameras and Displays and won a best paper award for "Hand-Held Schlieren Photography with Light Field Probes" at ICCP 2011, introducing light field probes as computational displays for computer vision and fluid mechanics applications.

Ivo Ihrke . Saarland University

Bio: Ivo Ihrke heads the research group "Generalized Image Acquisition and Analysis" within the Cluster of Excellence "Multimodal Computing and Interaction" at Saarland University. He is an associate Senior Researcher at the MPI Informatik, also associated with the Max-Planck Center for Visual Computing and Communications. Prior to joining Saarland University he was a postdoctoral research fellow at the University of British Columbia, Vancouver, Canada, supported by the Alexander von Humboldt-Foundation. He received a MS degree in Scientific Computing from the Royal Institute of Technology (KTH), Stockholm, Sweden (2002) and a PhD (summa cum laude) in Computer Science from Saarland University (2007). His main research interest are the modeling of forward and inverse light transport processes and computational algorithms for solving these large scale problems in the context of computational imaging, measurement, and display.

Douglas Lanman . Nvidia Research, formerly MIT Media Lab

Bio: Short Biography Douglas Lanman is a Postdoctoral Associate at the MIT Media Lab. His research is focused on computational imaging and display systems, including light field capture, automultiscopic (glasses-free) 3D displays, and active illumination for 3D reconstruction. He received a B.S. in Applied Physics with Honors from Caltech in 2002 and M.S. and Ph.D. degrees in Electrical Engineering from Brown University in 2006 and 2010, respectively. Prior to joining MIT and Brown, he was an Assistant Research Staff Member at MIT Lincoln Laboratory from 2002 to 2005. Douglas has worked as an intern at Intel, Los Alamos National Laboratory, INRIA Rhône-Alpes, Mitsubishi Electric Research Laboratories (MERL), and the MIT Media Lab. He presented the "Build Your Own 3D Scanner" course at SIGGRAPH 2009 and SIGGRAPH Asia 2009 and the "Build Your Own 3D Display" course at SIGGRAPH 2010, SIGGRAPH 2011, and SIGGRAPH Asia 2010.

Wolfgang Heidrich . University of British Columbia

Bio: Professor Wolfgang Heidrich holds the Dolby Research Chair in Computer Science at the University of British Columbia. He received a PhD in Computer Science from the University of Erlangen in 1999, and then worked as a Research Associate in the Computer Graphics Group of the Max-Planck-Institute for Computer Science in Saarbrucken, Germany, before joining UBC in 2000. Heidrich's research interests lie at the intersection of computer graphics, computer vision, imaging, and optics. In particular, he has worked on High Dynamic Range imaging and display, image-based modeling, measuring, and rendering, geometry acquisition, GPU-based rendering, and global illumination. Heidrich has written over 100 refereed publications on these subjects and has served on numerous program committees. He was the program co-chair for Graphics Hardware 2002, Graphics Interface 2004, and the Eurographics Symposium on Rendering, 2006.

Kurt Akeley . Lytro, Inc.

Bio: A pioneer in the field of computer graphics and a founding member of Silicon Graphics (later known as SGI), Kurt has helped develop innovative products like SGIs RealityEngine and the OpenGL graphics system. Hes contributed as well to 3D display technology, NVIDIA GPUs, and the Microsoft Research lab. He is an indefatigable traveler and prefers to be on the move when closer to home as well, often scheduling "walking meetings" while exploring the tranquil neighborhoods near Lytros Mountain View office. Kurt earned his Ph.D. in computer science from Stanford University and his B.E.E. from the University of Delaware. In 2005, he was awarded membership in the National Academy of Engineering. SIGGRAPH honored him with the Computer Graphics Achievement Award in 1995, and hes also been inducted as a Fellow of the Association for Computing Machinery.

Ramesh Raskar . MIT Media Lab

Bio: Ramesh Raskar joined the Media Lab from Mitsubishi Electric Research Laboratories in 2008 as head of the Labs Camera Culture research group. His research interests span the fields of computational light transport, computational photography, inverse problems in imaging and human-computer interaction. Recent projects and inventions include transient imaging to look around a corner, a next generation CAT-Scan machine, imperceptible markers for motion capture (Prakash), long distance barcodes (Bokode), touch+hover 3D interaction displays (BiDi screen), low-cost eye care devices (Netra,Catra), new theoretical models to augment light fields (ALF) to represent wave phenomena and algebraic rank constraints for 3D displays (HR3D). He is a recipient of TR100 award from Technology Review, 2004, Global Indus Technovator Award, top 20 Indian technology innovators worldwide, 2003, Alfred P. Sloan Research Fellowship award, 2009 and Darpa Young Faculty award, 2010. Other awards include Marr Prize honorable mention 2009, LAUNCH Health Innovation Award, presented by NASA, USAID, US State Dept and NIKE, 2010, Vodafone Wireless Innovation Award (first place), 2011. He holds over 40 US patents and has received four Mitsubishi Electric Invention Awards. He is currently co-authoring a book on Computational Photography.


Course Material