Face Analysis
Affective Computing, MIT Media Lab
Fully Automatic Upper Facial Action Recognition
Ashish Kapoor, Yuan Qi and Rosalind W. Picard
Appears in IEEE International Workshop on Analysis and
Modeling of Faces and Gestures , Oct 2003
This paper provides a new fully automatic framework to
analyze facial action units, the fundamental building blocks of facial
expression enumerated in Paul Ekman's Facial Action Coding System (FACS). The
action units examined in this paper include upper facial muscle movements such
as inner eyebrow raise, eye widening, and so forth, which combine to form facial
expressions. Although prior methods have obtained high recognition rates for
recognizing facial action units, these methods either use manually pre-processed
image sequences or require human specification of facial features; thus, they
have exploited substantial human intervention. This paper presents a fully
automatic method, requiring no such human specification. The system first
robustly detects the pupils using an infrared sensitive camera equipped with
infrared LEDs. For each frame, the pupil positions are used to localize and
normalize eye and eyebrow regions, which are analyzed using PCA to recover
parameters that relate to the shape of the facial features. These parameters are
used as input to classifiers based on Support Vector Machines to recognize upper
facial action units and all their possible combinations. On a completely natural
dataset with lots of head movements, pose changes and occlusions, the new
framework achieved a recognition accuracy of 69.3% for each individual AU and an
accuracy of 62.5% for all possible AU combinations. This framework achieves a
higher recognition accuracy on the Cohn-Kanade AU-coded facial expression
database, which has been previously used to evaluate other facial action
recognition system.}
Postscript .
PDF
Demonstration Movie: Download
(AVI 2.9 MB)
Real-Time, Fully Automatic Upper Facial Feature Tracking
Ashish Kapoor and Rosalind W. Picard
Appears in: Proceedings of The 5th International Conference
on Automatic Face and Gesture Recognition May, 2002
ABSTRACT
Robust, real-time, fully automatic tracking of facial features is required for
many computer vision and graphics applications. In this paper, we describe a
fully automatic system that tracks eyes and eyebrows in real time. The pupils
are tracked using the red eye effect by an infrared sensitive camera equipped
with infrared LEDs. Templates are used to parameterize the facial features. For
each new frame, the pupil coordinates are used to extract cropped images of eyes
and eyebrows. The template parameters are recovered by PCA analysis on these
extracted images using a PCA basis, which was constructed during the training
phase with some example images. The system runs at 30 fps and requires no manual
initialization or calibration. The system is shown to work well on sequences
with considerable head motions and occlusions.
PDF
Demonstration Movie: Download
(MPEG 5.2 MB)
A Real-Time Head Nod and Shake Detector
Ashish Kapoor and Rosalind W. Picard
Appears in: Proceedings from the Workshop on Perspective
User Interfaces, November 2001
ABSTRACT
Head nods and head shakes are non-verbal gestures used often to communicate
intent, emotion and to perform conversational functions. We describe a
vision-based system that detects head nods and head shakes in real time and can
act as a useful and basic interface to a machine. We use an infrared sensitive
camera equipped with infrared LEDs to track pupils. The directions of head
movements, determined using the position of pupils, are used as observations by
a discrete Hidden Markov Model (HMM) based pattern analyzer to detect when a
head nod/shake occurs. The system is trained and tested on natural data from ten
users gathered in the presence of varied lighting and varied facial expressions.
The system as described achieves a real time recognition accuracy of 78.46% on
the test dataset.
PDF
Demonstration Movies: