Near Depth and 2D Motion Invariant Photography

Near-Invariant Blur for Depth and 2D Motion via Time-Varying Light Field Analysis

Yosuke Bando^1,2 Henry Holtzman² Ramesh Raskar²
¹TOSHIBA Corporation ²MIT Media Lab

Recently, several camera designs have been proposed for either making defocus blur invariant to scene depth or making motion blur invariant to object motion. The benefit of such invariant capture is that no depth or motion estimation is required to remove the resultant spatially uniform blur. So far, the techniques have been studied separately for defocus and motion blur, and object motion has been assumed to be 1D (e.g., horizontal). This paper explores a more general capture method that makes both defocus blur and motion blur nearly invariant to scene depth and in-plane 2D object motion. We formulate the problem as capturing a time-varying light field through a time-varying light field modulator at the lens aperture, and perform 5D (4D light field + 1D time) analysis of all the existing computational cameras for defocus/motion-only deblurring and their hybrids. This leads to a surprising conclusion that focus sweep, previously known as a depth-invariant capture method that moves the plane of focus through a range of scene depth during exposure, is near-optimal both in terms of depth and 2D motion-invariance and in terms of high frequency preservation for certain combinations of depth and motion ranges. Using our prototype camera, we demonstrate joint defocus and motion deblurring for moving scenes with depth variation.

Keywords: extended depth of field, defocus and motion deblurring, depth and motion-invariant photography

Paper (13MB, PDF):
ACM Transactions on Graphics, Vol. 32, No. 2, Article 13, 2013.

Standard camera image Depth and 2D motion-invariant capture Deblurred image with a single PSF
- Standard camera image suffers from both defocus and motion blur if the scene has moving objects at various depths (left).
- Blur can be made nearly identical irrespective of scene depth or motion by sweeping focus during exposure (center).
- Sharp image can be recovered by spatially-uniform non-blind deconvolution with a single, a priori known PSF (right).
Video (2.1MB, WMV)
30 sec presentation at SIGGRAPH 2013 Technical Papers Fast Forward session.
Talk slides at SIGGRAPH 2013 (35MB PPTX, 2MB PDF)
Poster (3.5MB, PDF):
IEEE International Conference on Computational Photography, Posters/Demos, 2013.
How simple it is:

Traditionally, if an image is degraded by defocus and motion blur caused by differently moving objects at various depths,
one first needs to identify the PSF for each local window in the image,
because the PSF varies across the image depending on object depth and motion.
This is extremely difficult as it amounts to estimating different depth and motion of multiple objects from a single image.
One then needs to deconvolve the image with the identified spatially-varying PSF.

We propose to capture an image in a depth and 2D motion-invariant way,
so that the PSF stays nearly constant across the image irrespective of object depth or motion.
Then the blur can be removed by spatially-uniform deconvolution with that constant PSF, which can be known beforehand.
There is no need for identifying the PSF (no need for estimating object depth or motion).

Principle:

For a static scene point, during exposure, the instantaneous PSF changes its size through time while the plane of focus sweeps a range of depth. Such a sweep can be implemented by moving the sensor (or equivalently the lens) along the optical axis during exposure (called focus sweep). The resultant focus sweep PSF is the integral (~average) of the instantaneous PSF, which has a high peak in the center and a fast decaying tail. The focus sweep PSF is nearly invariant to the depth of the point as long as the point is located well within the swept depth range, so that the instantaneous PSF becomes a delta function at some moment during exposure. This fact was known previously.

For a moving scene point, the instantaneous PSF changes not only its size but also its center position. In the figure, the point is moving downwards and so is the instantaneous PSF center, as indicated by the red arrow. Nevertheless, the integrated focus sweep PSF is nearly identical to the static point case. The PSF does get elongated vertically, but the effect is negligible as long as the speed of the moving point is well below the rate at which the instantaneous PSF size changes. This fact is demonstrated both theoretically and experimentally in the paper.

Prototype camera:

We implemented a focus sweep camera by moving the lens during exposure.
The Arduino board is configured to send focus commands to the lens when the shutter release button is pressed.
How to move the lens (0.7MB, PDF)
To those who are interested in controlling Canon EF lenses externally.
See also this lens disassembly instruction related to this project.
How to achieve perfect invariance:
In a follow-up work we derive a way to modify focus sweep such that it achieves perfect invariance at least in theory, rather than near-invariance.

home


Standard camera image	Depth and 2D motion-invariant capture	Deblurred image with a single PSF