Extracting Depth and Matte using a Color-Filtered Aperture

Yosuke Bando		Bing-Yu Chen		Tomoyuki Nishita
TOSHIBA Corporation The University of Tokyo		National Taiwan University		The University of Tokyo

This paper presents a method for automatically extracting a scene depth map and the alpha matte of a foreground object by capturing a scene through RGB color filters placed in the camera lens aperture. By dividing the aperture into three regions through which only light in one of the RGB color bands can pass, we can acquire three shifted views of a scene in the RGB planes of an image in a single exposure. In other words, a captured image has depth-dependent color misalignment. We develop a color alignment measure to estimate disparities between the RGB planes for depth reconstruction. We also exploit color misalignment cues in our matting algorithm in order to disambiguate between the foreground and background regions even where their colors are similar. Based on the extracted depth and matte, the color misalignment in the captured image can be canceled, and various image editing operations can be applied to the reconstructed image, including novel view synthesis, post-exposure refocusing, and composition over different backgrounds.

Keywords: computational photography, computational camera, color filters, color correlation, depth estimation, alpha matting

Paper (3.5MB, PDF):
ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia 2008), Vol. 27, No. 5, Article 134 (9 pages), 2008.
CG Paper Grand Prix presented by The Society for Art and Science, Japan.


Lens with RGB filters	Captured image	Extracted depth map	Extracted alpha matte
From a single photograph captured through a color-filtered aperture, its depth map and alpha matte can be extracted automatically.

Principle:

	For a scene point at the focused depth, light rays in the Red band and those in the Green band (the Blue band is omitted for 2D illustration) converge to the same point on the image sensor.
	For a scene point off the focused depth, light rays in the two bands reach different positions on the image sensor, resulting in a color shift (resulting from geometric optics, not from chromatic aberration).
	As a result, a scene point farther than the focused depth is observed with A rightward shift in the Red plane, An upward shift in the Green plane, and A leftward shift in the Blue plane according to the arrangement of the color filters. We exploit these depth-dependent color misalignment cues for depth and matte extraction.

Video (21.7MB, WMV):
Demonstration of our color-filtered aperture photography with various applications including:
- Reconstruction of color-aligned images
- Interactive novel view synthesis
- Interactive post-exposure refocusing
- Composition over different backgrounds
- Video matting
Fast-forward presentation (7.2MB, WMV):
50-sec presentation in the fast-forward session.
Talk slides (13.2MB PPT, 4.7MB PDF)
Additional results:
Results for photos taken at the conference venue.
How to disassemble the lens (1.4MB, PDF):
To those who are interested in exploring/reproducing coded/color-filtered aperture techniques.
Depth estimation code (3.3MB, ZIP):
C source code of the color-filtered aperture-based depth estimation.

home