Ubiquitous Video Communication with the perception of eye contact

Download DIVX Video presented at Ubicomp 2003

Abstract

This project consists of the design of a video conferencing system for future ubiquitous computing environments that can guarantee two remote conversants the ability to establish eye contact. Unlike prior work, eye contact can be achieved even as people move about their respective environment engaging in everyday tasks. We describe the ideal design for future environments with ubiquitous computing components and then present the implementation and testing of one key component of the proposed system. Based on prior studies specifying the conditions required for the perception of eye contact, we experimentally show that a system that simultaneously uses a new display technology, visual head tracking, digital compositing, and miniature optical sensors embedded directly in surfaces offers the promise of unconstrained video conferencing with natural eye contact

Description

The idea is to improve eye contact by embedding a miniature camera for example in a wall, and then use the everywhere display to project the image of the remote conversant so that, the pinhole camera is always aligned with the middle part of the eyes of the remote conversant. Then, the conversants are always staring at the image of the other conversant and at the same time staring at the camera. If any of the conversants start moving, the alignment between the center part of the eyes and the pinhole camera is lost. In order to solve this problem, we implemented head tracking using the CAMSHIFT algorithm to correct for head movements.

The system consists of the following components:

1. The Everywhere display (IBM)
2. head tracking
3. digital compositing (Chroma keying)
4. miniature optical sensors

The Everywhere display is a new display technology developed by IBM which is able to display images and real-time video in any surface. Please take a look at the following video for a demostration of the Everywhere display.

Head tracking

We implemented the Continuously Adaptative Mean Shift Algorithm (CAMSHIFT) originaly developed by Intel. We track the area of the face, cut it, and paste it always in the same position in the projected image comming from the Everywhere display. The result is that even if the conversants move, the tracking always corrects for translation movements of the face keeping the alignment between the pinhole camera and the center part of the eyes at all times. For a demonstration of the head tracking results, please take a look at the next two videos

Digital compositing (chroma keying)

Consists of chroma keying in order to eliminate the background of the images. A black screen is placed behind the conversants so that a computer program can eliminate the background by simple color diferencing.

Miniature optical sensors

Miniature cameras are embedded in the sufaces that are going to be used to project the videoconferece images. The camera is a miniature Sony color camera.

Once the system has been set up, we can for example, use the wall as a mirror. The miniature camera is embedded in the area of the wall