Dappled Photography: Mask Enhanced Cameras for
Heterodyned Light Fields and
Coded Aperture Refocusing

Ashok Veeraraghavan, Ramesh Raskar, Amit Agrawal, Ankit Mohan and Jack Tumblin

ACM SIGGRAPH 2007

Low res pdf,   Matlab Code and Images,    High res pdf

Slides,     SIGGRAPH Talk Slides



Updates:
Frequently Asked Questions
Movie of Siggraph presentation (Quicktime, 24MB, 25 minutes)
Errata: Equations 11 and 12 have error in published version which has been corrected in online pdfs. Detailed derivation of equations 10,11 and 12 (not in paper)
Analysis of Aliasing and the Band-limited Lightfield Assumption
History of Integral and Lightfield Imaging (External Link)
Datasets and How to create a LF camera in 35 seconds and a $5 mask.
Coded Aperture Refocussing




Can we capture additional information about a scene by inserting a patterned mask inside a conventional camera?

We use a patterned attenuating mask to encode the light field entering the camera. Depending on where we put the mask, we can effect desired frequency domain modulation of the light field. If we put the mask near the lens aperture, we can achieve full resolution digital refocussing. If we put the mask near the sensor, we can recover a  4D light field without any additional lenslet array. See
Slides  for more details.

What happens when you insert a mask in a camera?
Encoded Blur or Light Field Camera
 


Abstract

We describe reversible modulation of 4D light field by inserting a patterned planarmask in the optical path of a lens based camera. We can reconstruct the 4D light field from a 2D camera image without any additional lenses as required by previous light field cameras. The patterned mask attenuates light rays inside the camera instead of bending them, and the attenuation recoverably encodes the ray on the 2D sensor. Our mask-equipped camera focuses just as a traditional camera might to capture conventional 2D photos at full sensor resolution, but the raw pixel values also hold a modulated 4D light field. The light field can be recovered by rearranging the tiles of the 2D Fourier transform of sensor values into 4D planes, and computing the inverse Fourier transform.

We also show how a broad-band mask placed at the lens enables us to compute refocusing at full sensor resolution for images of layered Lambertian scenes. This partial encoding of 4D ray-space data enables editing of image contents by depth to remove or suppress unwanted occluders, yet does not require computational recovery of the complete 4D light field.





Heterodyne Light Field Camera


Light field camera built with View Camera and a $70 Canon Flatbed Scanner
Light Field Camera can cature full resolution image and can achieve extreme deblurring and programmable depth of field
Light Field Camera, i.e. with mask close to the sensor, can cature 4D-Ray representation (at 1/9th reduced resolution) as well as a 2D full resolution image. Thanks to a complete 4D representation of light field, it can achieve extreme deblurring in complex situations and support programmable depth of field.





Full resolution Digital Refocussing using the Encoded Blur Camera (Click on photos to download full resolution versions)
 
Camera with Mask Coded Aperture Defocussed photo taken with coded aperture camera and deblurring result
Input Photo (Full Resolution)                                          Digitally Refocussed Photo (Full Resolution)
Encoded Blur Camera, i.e. with mask in the aperture, can high spatial images frequencies in the defocus blur. Notice the glint in the eye. In the misfocused photo, on the left, the bright spot appears blurred with the bokeh of the chosen aperture (shown in the inset). In the deblurred result, on the right, the details on the eye are correctly recovered. [Hires Photos]




Differences in photos taken with our Encoded Blur Camera as compared to a traditional camera

Note that when the lens is out of focus, the blur is disc shaped in a traditional camera as shown in the middle figure. With our camera, due to the mask in the aperture, the shape of the defocus blur, or Bokeh, changes according to the mask pattern. Effectively, high spatial frequencies are preserved using the mask, which are otherwise lost. Post-processing in software can recover the sharp image of the scene.


In-focus photo

 Out of focus photo taken with a traditional camera

Out of focus photo taken with Encoded Blur Camera





Graphical Illustration of Computing 4D Light Field from 2D Photo






Consider a scene consisting of several cones at different depths. If we take an image with a traditional camera, we get the photo shown above. The magnitude
of the 2D FFT of the image shows high energy at low frequencies. This is a well known fact that most energy in natural images is concentrated
in low frequencies.


Now consider taking a 2D photo of the same scene with our Heterodyne Light Field Camera. In this new camera, we place a cosine mask near the
sensor. We get an image shown as below.







Although this image looks similar to the previous photo captured using a traditional camera, notice the effect of mask on the input photo. The mask
casts a soft shadow on the sensor. It dapples the light reaching the sensor. In theory, mask modulates the incoming 4D light field to make spectral
replicas.


Now consider the 2D FFT of this new image. Notice that the spectral replicas are clearly seen!!!. Compare this FFT image with the previous FFT image to
visualize the differences.
These replicas are due to the cosine mask placed near the sensor. The cosine mask modulates the incoming 4D light field.
In this example, the mask has 4 harmonics, or 4*2+1 = 9 impulses in its frequency response. Thus, we get 9*9=81 replicas in both x and y direction.


Now lets see how we can obtain the 4D light field from the modulated 2D photo captured from our Heterodyne Light Field Camera. The entire algorithm
is shown graphically below.










Steps:

1. Compute the 2D FFT of the captured 2D photo.
2. We know that we will get 9*9 replicas due to the physical mask placed near the sensor. Rearrange these 81 tiles into 4D.
3. Compute the inverse 4D FFT to get the light field.


Click on the above image to see a video (ViewCones1.avi) of different 'views' obtained from the light field. These views are essentially images that we
would obtain if we look through a narrow aperture on the lens at different positions.
Video Showing Refocusing from Back to Front

Gif Animation showing Refocusing
from Back to Front (Loop time is about 10 seconds)


Gif Animation showing Views 
obtained from light field


Frequently Asked Questions




Related Project: Coded Exposure Photography: Flutter Shutter Camera for Motion Deblurring
In flutter shutter camera, we use a broadband code (a code with flat frequency response) as a binary sequence to open and close the shutter in a rapid sequence. In this paper, for full resolution digital refocusing, we use a coded aperture where we use a broadband code for opacity of the mask.

In the other part of the paper, however, we use a narrow-band code (a code with energy in only a few narrow band of frequencies) for the mask placed near the sensor to obtain the 4D light field from a single 2D photo.


Related Papers:

Coded Exposure Photography: Motion Deblurring using Fluttered Shutter, SIGGRAPH 2006
Resolving Objects at Higher Resolution from a Single Motion-Blurred Image, CVPR 2007


Related Presentations:

Less is More: Coded Computational Photography
, from Keynote at ACCV 2007
Computational Photography, Siggraph 2007 course material



Back to Other Projects in Computational Photography
Contacts: Ramesh Raskar, Amit Agrawal