%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%
%%  File         :      Gesture Book Chapter
%%  Start Editing:      9/15/95
%%
%%  Author       :      Sandy Pentland <sandy@media-lab.media.mit.edu>
%%  Note         :    
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%

%\documentstyle[cup7b,david,epsf,psfig]{cuphci}
%\documentstyle[11pt,psfig]{article}




\author[A. Pentland]{Alex P. Pentland\\Perceptual Computing Section, The Media 
Laboratory\\
Massachusetts Institute of Technology, 20 Ames St., Cambridge, MA 02139\\
sandy@media.mit.edu, http://www-white.media.mit.edu/vismod/demos/ive/}
\chapter{Smart Rooms: Machine Understanding of Human Behavior}

\begin{abstract}
I describe progress in building computer systems that understand
people, and can work with them in the manner of an attentive
human-like assistant.\footnote{This research was supported by ONR,
ARL, and ARPA under the MURI and FERET contracts, and by BT (British
Telecom) and Texas Instruments.  Portions of this text have appeared in
Scientific American \cite{G-Pentland96}.} To accomplish this, I have built a
series of real-time experimental testbeds, called {\sl Smart Rooms}.
These testbeds are instrumented with cameras and microphones, and
perform audio-visual interpretation of human users.  Real-time
capabilities include 3-D tracking of head, hands, and feet, and
recognition of hand/body gestures.  The system can also support face
recognition and interpretation of face expression.

\end{abstract}


\section{Introduction}

My goal is to make it possible for computers to function like
attentive, human-like assistants.  I believe that the most important
step toward achieving this goal is to give computers an ability that I
call {\sl perceptual intelligence}.  They have to be able to
characterize their current situation by answering questions such as
who, what, when, where, and why, just as writers are taught to do.  

In the language of cognitive science, perceptual intelligence is the
ability to solve the frame problem: it is being able to classify the
current situation, so that you know what variables are important, and
thus can act appropriately.  Once a computer has the perceptual
intelligence to know who, what, when, where, and why, then simple
statisical learning methods have been shown to be sufficient for the
computer to determine what aspects of the situation are significant,
and to choose a helpful course of action \cite{Pen-maes-book}.

In order to interact with people, the most important answers to
``who, what, when, where, and why'' are those concerned with human
behavior.  Consequently, I have focused my work on the perception of
human behavior.  I have concentrated on giving computers the ability
to recognize people, to understand our expressions and gestures, and
hear the tone and emphasis of the human voice \cite{G-Pentland96}.  If I can
give computers the ability to perceive and categorize human
interactions in a human-like manner, I expect that the computer will
be able to work with people in a natural and common-sense manner.

%In conducting this project I have discovered that the swiftest and
%most reliable progress can be made by creatively assembling 2-D image
%processing modules \cite{Pen-azriel,Pen-ballardbrown}, rather than by
%developing exotic new mathematical formulations, detailed
%understanding of photometry, or using sophisticated 3-D
%representations. Perhaps the success of 2-D methods is not so
%surprising; after all, what we know of brain function usually looks
%more like parallel image processing than abstract or logical
%calculations. [That appearance, of course, could be deceiving.]

%
%The following sections of this paper I will describe how 2-D picture
%processing modules can be assembled to obtain building blocks of
%perceptual intelligence, and how they can be used to help
%build virtual creatures that interact with people in a compelling and
%lifelike manner.  Although the building blocks described here are
%limited in comparison to human abilities, and only cover a scattering
%of human perceptual abilities, I believe that they are the first steps
%down a path to a human-style perceptual intelligence.


\subsection{Experimental Testbeds}

To carry out this plan of research I have created a series of
experimental testbeds that are instrumented with cameras and
microphones, thus allowing the computer to see, hear, and interpret
users' actions.  These testbeds are ``smart rooms'' that help users
navigate the World Wide Web, to teleconference in virtual worlds, to
find multimedia information, and to interact with artifical life
agents \cite{G-Pentland96}.  The key idea is to develop perceptual
intelligence: because the environment knows something about what is
going on, it can act and react appropriately.

The idea of a smart room is a little like having a butler; i.e., a
passive observer who usually stands quietly in the corner but who is
constantly looking for opportunities to help.  Imagine a house that
always knows where your kids are, and tells you when they might be
getting in trouble.  Or a car that knows when you are sleepy and
should stop for coffee (the interior of a car is a particularly
important type of smart room).

%Smart clothes, on the other hand, are more like a personal assistant.
%That is, they are like a person who travels with you, seeing and
%hearing everything that you do, and who tries to anticipate your needs
%and generally smooth your way.  For instance, by attaching a camera to
%your eyeglasses, then our face recognition software can help you
%remember the name of the person you are looking at, by whispering
%their name in your ear.  Or, by building a microphone into your
%jacket, then our word-spotting and database search software can
%provide remind you of facts relevant to the current conversation
%\cite{Pen-rememberance}.

%Our current smart clothes prototypes project a rather CyberPunk look
%(see Figure \ref{Pen-fig:borgs}).  However all the components are
%shrinking fast, so that the coming of continuous computing is not far
%off.  Already two of the Media Lab Cyborgs, Thad Starner at the
%extreme right and Steve Mann at the extreme left, wear their devices
%all day every day.

The first prototype of a smart room was developed in 1991 by Pattie
Maes, Trevor Darrell, Bruce Blumberg and myself as part of the ALIVE
(Artifical Life Interactive Video Environment) project; now there are
smart rooms in Japan, France, England, and several places in the U.S.
They are linked together by ISDN telephone lines to allow shared
virtual environment and cooperative work experiments.

%Our smart clothes project was started in 1992 by graduate students
%Thad Starner and Steve Mann, and is supervised by Rosalind Picard and
%myself.

\subsection{The Modeling and Estimation Framework}


The general theoretical approach I have taken in these ``smart
environment'' testbeds is to perform a maximum {\sl a posteriori}
(MAP) interpretation on very low-level, 2-D representations of regions
of the image data.  The appearance of a target class $\Omega$, e.g.,
the probability distribution function $P({\bf x}|\Omega)$ of its
image-level features ${\bf x}$, can be characterized by use of a
low-dimensional parametric appearance model.  Once such a probability
distribution function (PDF) has been learned, it is straightforward to
use it in a MAP estimator in order to detect and recognize target
classes.  Behavior recognition is accomplished in a similar manner;
these parametric appearance models are tracked over time, and their
time evolution $P({\bf x}(t)|\Omega)$ characterized probabilistically
to obtain a {\sl spatiotemporal behavior model}.  Incoming
spatiotemporal data can then be compared to the spatiotemporal PDF of
each of the various behavior models using elastic matching methods
such as dynamic time warping \cite{G-Darrell93} or Hidden Markov
Modeling \cite{G-starner95}.

The use of parametric appearance models to characterize the PDF of an
object's appearance in the image is related to the idea of view-based
representation, as advocated by Ullman \cite{Pen-ullman91} and Poggio
\cite{Pen-poggio90}.  As originally developed, the idea of view-based
recognition was to accurately describe the spatial structure of the
target object by interpolating between various views.  However, in
order to describe natural objects such as faces or hands, I have found
it necessary to extend the notion of ``view'' to include
characterizing the range of geometric and feature variation, as well
as the likelihood associated with such variation.

This approach is typified by my face recognition research
\cite{G-TurkPentland91,Pen-moghaddam95}, which uses linear combinations of
eigenvectors to describe a {\sl space} of target appearances, and then
characterize the PDF of the targets appearance within that space.
This method has been shown to be very powerful for detection and
recognition of human faces, hands, and facial expressions
\cite{Pen-moghaddam95}.  Other researchers have used extensions of this
basic method to recognize industrial objects and household items
\cite{G-murase_nayar_95}.  Another variation on this approach is to use linear
combinations of examples rather than eigenvectors; this type of
appearance modeling has demonstrated a power similar to that of the
eigenvector-based methods \cite{G-Darrell93,Pen-jones}, although it
necessarily has a lower efficiency.

\subsection{The Computer Architecture}

Sensing and interpretation of human movement is accomplished by a
modular computer architecture, which can be configured in a variety of
ways, as more or less interpretation capability is required.  The
basic element is an SGI Indy computer; these computers are connected
by video, audio, ISDN, and ethernet networks.  This allows each
computer to independently access whatever portion of the audiovisual
input it needs, to share control information among the other
computers, and to connect with other, distant installations over ISDN phone
lines.  Normally each computer is dedicated to perform only one type
of interpretation.  For instance, if video tracking, audio input, and
gesture interpretation capabilities were desired, then three computers
would be used, one for each task.

\section{Building Blocks of Perceptual Intelligence}

The following sections will briefly describe each of a set of programs
that can find, track, and interpret human behavior \cite{G-Pentland96}.  The
last section will illustrate how these programs have been assembled
into a real-time ``virtual dog'' that interacts with people in an
interesting and lifelike manner.  In order of description, the modules
are:
\begin{itemize}

\item Pfinder, a real-time program that tracks the user, and
recognizes a basic set of hand gestures and body postures,

\item the face processor, an interactive-time program that recognizes
human faces,

\item the expression processor, an interactive-time program 
that uses motion-energy templates to classify people's facial
expressions,

\item vision-directed audio, a real-time system that uses the head
position information provided by Pfinder to steer a phase-array
microphone,

\item behavior recognition, real-time systems that examine users'
movements in order to determine what they are doing.

\end{itemize}

For additional detail the reader should examine the numerous papers,
technical reports, interactive demos, and computer code available at
our web site at http://www-white.media.mit.edu/vismod or
by anonymous FTP from whitechapel.media.mit.edu.


\subsection{Pfinder}

Pfinder (``person finder'') is a real-time system for tracking and
interpretation of people. It runs at 10Hz on a standard SGI Indy
computer, and has performed reliably on thousands of people in many
different physical locations \cite{Pen-ali96,G-wren95,G-Pentland96}.  The system uses a
multi-class statistical model of color and shape to segment a person
from a background scene, and then to find and track people's head and
hands in a wide range of viewing conditions.  Pfinder produces a
real-time representation of a user useful for applications such as
wireless interfaces, video databases, and low-bandwidth coding,
without cumbersome wires or attached sensors.

Pfinder employs a maximum {\sl a posteriori} (MAP) approach by using
simple 2-D models to perform detection and tracking of the
human body. It incorporates {\sl a priori} knowledge about people
primarily to bootstrap itself and to recover from errors.

In order to present a concise description of Pfinder, I will only
describe its representations and operation in the ``steady state''
case, where it is tracking a person moving around an office
environment, neglecting the problems of initially learning a
person's description. 

\begin{figure}[t]
\centerline{\psfig{figure=Pentland_figs/raw.002.eps,width=35mm}
            \psfig{figure=Pentland_figs/seg.002.eps,width=35mm}
            \psfig{figure=Pentland_figs/fig2.eps,width=35mm}}
\caption[]{Analysis of a user in the ALIVE environment.  The
             frame on the left is the video input (n.b. color image
             shown here in black and white for printing purposes), the
             center frame shows the support map $s(x,y)$ which
	     segments the user into
             blobs, and the frame on the right showing a person model
             reconstructed from blob statistics alone (with 
             contour shape ignored).}
\label{Pen-fig:occlude1}
\end{figure}

\subsubsection{Modeling The Person}

Pfinder models the human as a connected set of 2-D {\sl blobs}, an old
concept, but one to which in recent years little serious attention has
been paid in favor of stronger local features like points, lines, and
contours.  The blob representation that we use was originally
developed by me in 1976 \cite{Pen-pentland76} for application to
multi-spectral satellite (MSS) imagery.  The PDF of each blob
is characterized by the {\sl joint} distribution of spatial $(x,y)$
and color features.  Color is expressed in the $YUV$ space, which is a
simple approximation to human color sensitivity.

We define ${\bf m}_{ k}$ to be the mean $(x,y,Y,U,V)$ of blob $k$, and
${\bf K}_{ k}$ to be the covariance of that blob's distribution.
%Because of
%their different semantics, the spatial and color distributions are
%assumed to be independent. That is, ${\bf K}_{ k}$ is block-diagonal,
%with uncoupled spatial and spectral components.
The mean of a blob expresses the concept ``color {\sl a} at location
{\sl b},'' while the covariance expresses how the color and brightness
of the blob changes across its surface.  For instance, the blob
statistics can express that one side is brighter than the other
(perhaps due illumination from the side), or that the color changes
from top to bottom (perhaps due to light reflected from the floor).

Each blob also has associated with it a {\sl support map} that
indicates exactly which image pixels are members of a particular blob.
Since the individual support maps indicate which image pixels are
members of that particular blob, the aggregate support map $s(x,y)$
over all the blobs represents the segmentation of the image into
spatial/color classes.

%Each blob can also have a detailed representation of its shape and
%appearance, modeled as differences from the underlying blob
%statistics.  The ability to efficiently compute compact
%representations of people's appearance is useful for low-bandwidth
%telecommunications, such as our demonstration of a shared virtual
%envionments at SIGGRAPH '95 \cite{Pen-Siggraph95}.

The statistics of each blob are recursively updated to combine
information contained in the most recent measurements with knowledge
contained in the current class statistics and the priors.  Because
the detailed dynamics of each blob are unknown, we use approximate
models derived from experience with a wide range of users.  For
instance, blobs that are near the center of mass have substantial
inertia, whereas blobs toward the extremities can move much faster.

\subsubsection{Modeling The Scene}

It is assumed that the majority of the time Pfinder will be processing a
scene that consists of a relatively static situation such as an
office, and a single moving person.  Consequently, it is appropriate
to use different types of model for the scene and for the person.

The surrounding scene is modeled as a texture surface; each
point on the texture surface is associated with a mean color value and
a distribution about that mean.  The color distribution of each pixel
is modeled with the Gaussian described by a full covariance matrix.
Thus, for instance, a fluttering white curtain in front of a black
wall will have a color covariance that is very elongated in the
luminance direction, but narrow in the chrominance directions.

In each frame visible pixels have their statistics recursively
updated using a simple adaptive filter.  This allows us to compensate
for changes in lighting and even for object movement.  For instance,
if a person moves a book it causes the texture map to change in both
the locations where the book was, and where it now is.  By tracking
the person we can know that these areas, although changed, are still
part of the texture model and thus update their statistics to the new
value.  The updating process is done recursively, and even large
changes in illumination can be substantially compensated within two
or three seconds.

\subsubsection{The Analysis Loop}

Given a person model and a scene model, Pfinder then acquires a new
image, interprets it, and updates the scene and person models.  To
accomplish this there are several steps:
\begin{enumerate}

\item First, the appearance of the users is predicted in the new image
using the current state of our model.  This is accomplished using a
set of Kalman filters with simple Newtonian dynamics that operate on
each blob's spatial statistics.  

\item  Next, for each image pixel the likelihood that it is a
member of each of the blob models and the scene model is measured.
Self-shadowing and cast shadows are a particular difficulty in
measuring this likelihood; this is addressed by normalizing the hue and
saturation by the overall brightness.

\item Resolve these pixel-by-pixel likelihoods into a support map,
indicating for each pixel whether it is part of one of the blobs or of the
background scene.  Spatial priors and connectivity constraints are
used to accomplish this resolution.

\item Update the statistical models for each blob and for
the background scene; also update the dynamic models of the blobs.

\end{enumerate}

For some applications, Pfinder's 2-D information must be processed
to recover 3-D geometry.  For a single calibrated camera this can be
accomplished by backprojecting the 2-D image information to produce
3-D position estimates using the assumption that the user is standing
on a planar floor. When two or more cameras are available, the hand,
head, etc., blobs can be matched to obtain 3-D estimates via triangulation.

Figure \ref{Pen-fig:occlude1} illustrates Pfinder's functioning.  At the
left is the original video frame (shown in black and white rather than
the original color).  In the middle is the resulting segmentation into
head, hands, feet, shirt, and pants.  At the right are
one-standard-deviation ellipses illustrating the statistical blob
descriptions formed for the head, hands, feet, shirt, and pants.  Note
that despite having the hands either in front of the face or the body
a correct description is still obtained.  For additional detail see
references \cite{Pen-ali96,G-wren95,G-Pentland96}.


\begin{figure}[t]
\begin{center}\hfil
(a)\psfig{figure=Pentland_figs/face_processor.ps,width=4.2in}\hfil
\begin{tabular}{cccc}
\psfig{figure=Pentland_figs/white.ps,height=2.4cm,clip=t} &
\psfig{figure=Pentland_figs/white_ms_eigsearch.ps,height=2.4cm,clip=t} &
\psfig{figure=Pentland_figs/white_scalewarp.ps,height=2.4cm,clip=t} &
\psfig{figure=Pentland_figs/white_eigsearch.ps,height=2.4cm,clip=t} \\
(b) & (c) & (d) & (e)
\end{tabular}
\end{center}
\caption[]{(a) The face processing system,
(b) original image, (c) position and scale estimate, (d) normalized
head image, (e) position of facial features.}
\label{Pen-fig:face_processor}
\end{figure}

\begin{figure}[tb]
\begin{center}
\begin{tabular}{c}
(a) \psfig{figure=Pentland_figs/feret_8 eigfaces.ps,height=2.4cm,clip=t}\\
(b) \psfig{figure=Pentland_figs/feret_photobook2.ps,width=10cm,clip=t} 
\end{tabular}
\end{center}
\caption[]{(a) The first eight eigenfaces, (b) searching 
for similar faces in a database, using the Photobook image
database tool \cite{Pen-pentland96}.}
\label{Pen-fig:feret}
\end{figure}

\subsection{Face Recognition}

Once the rough location of the person's head is known, one can attempt
to recognize their face.  As with Pfinder, a maximum {\sl a
posteriori} (MAP) approach is applied to this problem.  This has been
accomplished by developing a method for determining the probability
distribution function for face images within a low-dimensional
eigenspace.  Knowledge of this distribution then allows the face and
face features to be precisely located, and compared along meaningful
dimensions.  The following gives a brief description of this system;
for additional detail see \cite{Pen-moghaddam95}.

\subsubsection{Face and Feature Detection}

The standard detection paradigm in image processing is that of
normalized correlation or template matching.  However this approach is
only optimal in the simplistic case of a {\em deterministic} signal
embedded in white Gaussian noise. When we begin to consider a target
{\em class} detection problem --- {\it e.g.,} finding a generic human
face or a human hand in a scene --- we must incorporate the underlying
probability distribution of the object of interest.  Subspace or
eigenspace methods, such as the KLT and PCA, are particularly
well-suited to such a task since they provide a compact and {\em
parametric} description of the object's appearance and also
automatically identify the essential components of the underlying
statistical variability.

In particular, the eigenspace formulation leads to a powerful
alternative to standard detection techniques such as template matching
or normalized correlation.  The reconstruction error (or residual) of
the KLT expansion is an effective indicator of a match.  The residual
error is easily computed using the projection coefficients and the
original signal energy. This detection strategy is equivalent to
matching with a linear combination of {\em eigentemplates} and allows
for a greater range of distortions in the input signal (including
lighting, and moderate rotation. and scale). Some of the low-order
eigentemplates for a human face are shown in Figure
\ref{Pen-fig:feret}(a).  In a statistical signal detection
framework, the use of eigentemplates has been shown to be orders of
magnitude better than standard matched filtering \cite{Pen-moghaddam95}.

Using this approach the target detection problem can be reformulated
from the point of view of a MAP estimation problem.  In particular,
given the visual field, estimate the position (and scale) of the
subimage which is most representative of a specific target class
$\Omega$.  Computationally, this is achieved by sliding an $m$-by-$n$
observation window throughout the image and at each location computing
the {\em likelihood} that the given observation $\bf{x}$ is an
instance of the target class $\Omega$ --- {\it i.e,}
$P(\bf{x}|\Omega)$. After this probability map is computed, the
location corresponding to the highest likelihood can be selected as
the MAP estimate of the target location.
%Note that the likelihood map is in
%general evaluated over the parameter space affecting the object's
%apperance and in addition to spatial position can include scale and
%rotation.

\subsubsection{The Face Processor}

This MAP-based face finder has been employed as the basic building block of
an automatic face recognition system.  The function of the face finder
is to very precisely locate the face, since Pfinder produces only
low-resolution estimates of head location.  The block diagram of the
face finder system is shown in Figure~\ref{Pen-fig:face_processor} which
consists of a two-stage object detection and alignment stage, a
contrast normalization stage, a feature extraction stage, followed by
recognition (and, optionally, facial coding.)
Figure~\ref{Pen-fig:face_processor}(b)-(e) illustrates the operation of
the detection and alignment stage on a natural image containing a
human face.

The first step in this process is illustrated in
Figure~\ref{Pen-fig:face_processor}(c) where the MAP estimate of the
position and scale of the face are indicated by the cross-hairs and
bounding box. Once these regions have been identified, the estimated
scale and position are used to normalize for translation and scale,
yielding a standard ``head-in-the-box'' format image
(Figure~\ref{Pen-fig:face_processor}(d)).  A second feature detection
stage operates at this fixed scale to estimate the position of 4
facial features: the left and right eyes, the tip of the nose and the
center of the mouth (Figure~\ref{Pen-fig:face_processor}(e)).  Once the
facial features have been detected, the face image is warped to align
the geometry and shape of the face with that of a canonical model.
Then the facial region is extracted (by applying a fixed mask) and
subsequently normalized for contrast.  This geometrically
aligned and normalized image  
%(shown in Figure~\ref{Pen-fig:white_normed}(a)) 
is then projected onto the set of eigenfaces shown in Figure
\ref{Pen-fig:feret}(a). 

The projection coefficients obtained by comparison of the normalized
face and the eigenfaces form a feature vector which accurately describes
the appearance of the face.  This feature vector can therefore be used
for facial recognition, as well as for facial image coding.
Figure~\ref{Pen-fig:feret}(b) shows a typical result when using the
eigenface feature vector for face recognition. The image in the upper
left is the one to be recognized and the remainder are the most
similar faces in the database (ranked by facial similarity, left to
right, top to bottom).  The top three matches in this case
are images of the same person taken a month apart and at different
scales.  The recognition accuracy of this system (defined as the
percent correct rank-one matches) is 99\% \cite{Pen-moghaddam95}.

\begin{figure}[t]
\begin{center}
\begin{tabular}{cccc} 
\psfig{figure=Pentland_figs/ie-surprise.ps,width=2.0cm}\hspace{-8mm} &
\psfig{figure=Pentland_figs/ie.sp.me.bw.ps,width=2.0cm}\hspace{-8mm}  &
\psfig{figure=Pentland_figs/ie-smile.ps,width=2.0cm}\hspace{-8mm}  &
\psfig{figure=Pentland_figs/ie.sm.me.bw.ps,width=2.0cm}\hspace{-8mm} \\
{(a) Surprise} & 
{(b) Motion Energy} &
{(c) Smile} &
{(d) Motion Energy} 
\end{tabular}
\\
\begin{tabular}{cccc} 
\psfig{figure=Pentland_figs/temp.sm.bw.ps,width=2.0cm} &
\psfig{figure=Pentland_figs/temp.sp.bw.ps,width=2.0cm} & 
\psfig{figure=Pentland_figs/temp.di.bw.ps,width=2.0cm} &
\psfig{figure=Pentland_figs/temp.an.bw.ps,width=2.0cm} \\
{(e) Happiness}& {(f) Surprise}& {(g) Disgust} & {(h) Anger}
\end{tabular}
\end{center}
\caption[]{Determining expressions from
video sequences. (a) and (c) show expressions of smile and surprise,
and (b) and (d) show the corresponding spatio-temporal motion energy
pattern.  (e) - (h) show spatio-temporal motion-energy 
templates for various expressions.}
\label{Pen-fig:s-t-templates}
\end{figure}


\begin{table}
\begin{tabular}{||l|c|c|c|c|} 
\hline\hline
Expressions & Smile & Surprise & Anger & Disgust \\
\hline
Template &
\psfig{figure=Pentland_figs/gh.sm.12.ps,width=1.6cm,clip=t} &
\psfig{figure=Pentland_figs/kr.sp.10.ps,width=1.6cm,clip=t} &
\psfig{figure=Pentland_figs/cd.an.06.ps,width=1.6cm,clip=t} &
\psfig{figure=Pentland_figs/sn.di.12.ps,width=1.6cm,clip=t} \\
\hline\hline
Smile      & {\bf 12} & 0        & 0        & 0        \\
\hline
Surprise   & 0        & {\bf 10} & 0        & 0        \\
\hline
Anger      & 0        & 0        & {\bf 9}  & 0        \\
\hline
Disgust    & 0        & 0        & 1        & {\bf 10} \\
\hline\hline
Success    & 100\%     & 100\%   & 90\%    & 100\%  \\
\hline\hline 
\end{tabular} %%%%
\caption[]{Results of facial
expression recognition using spatio-temporal motion energy templates.
This result is on based on 12 image sequences of smile, 10 image
sequences of surprise, anger, disgust, and raise eyebrow. Success rate
for each expression is shown in the bottom row. The overall
recognition rate is 98.0\%.}
\label{Pen-table:classification-temp}
\end{table}

\subsection{Expression Recognition}

Once the face and its features have been accurately located, one can
begin to analyze its motion to determine the facial expression.  This
can be done by using spatio-temporal motion-energy templates of the
whole face \cite{Pen-irfan94,Pen-irfan95}.  For each facial expression, the
corresponding template expresses the {\sl peak amount} and {\sl
direction} of motion that one would expect to see at each point on the face.
Figures \ref{Pen-fig:s-t-templates}(a) - (d) show Irfan Essa making two
expressions (surprise and smile) and the corresponding motion-energy
template descriptions.

These simple, biologically-plausible motion energy ``templates'' can
be used for expression recognition by comparing the motion-energy
observed for a particular face to the ``average'' template for each
expression.  Figures \ref{Pen-fig:s-t-templates}(e) - (h) show the
motion-energy templates for several expressions.  To classify an
expression one compares the observed facial motion energy with each of
these motion energy templates, and then picks the expression with the
most similar pattern of motion.

This method of expression recognition has been applied to a database
of 52 image sequences of 8 subjects making various expressions.  In
each image sequence the motion energy was measured, compared to each
of the templates, and the expression classified, generating the
confusion matrix shown in Table \ref{Pen-table:classification-temp}.  This
table shows just one incorrect classification, giving an overall
recognition rate of 98.0\%.  For additional details see
\cite{Pen-irfan94,Pen-irfan95}.



\begin{figure}
\centerline{
\psfig{figure=Pentland_figs/micarray.ps,height=5cm,clip=t}}
%\centerline{
%(b) \psfig{figure=Pentland_figs/audio.ps,height=1.5in,clip=t}}
\caption[]{Geometry of the vision-directed phased-array microphone
system}
% (b) Block diagram of the 3-D audio output system.
\label{Pen-fig:audio}
\end{figure}
  
\subsection{Vision-Driven Audio}

Audio interpretation of people is as important as visual
interpretation.  Although much work as been done on speech
understanding, virtually all of this work assumes a closely-placed
microphone for input and a fixed listening position.  Speech
recognition applications, for instance, typically require near-field (
$< 1.5m$) microphone placement for acceptable performance.  Beyond
this distance, the signal-to-noise ratio of the incoming speech effects
the performance significantly; most commercial speech-recognition
packages typically break down over a 4 to 6 DB range.

The constraint of near-field microphone placement makes audio
difficult to integrate into normal human life, so it is necessary to
find a solution that allows the user to move around with
no noticeable degradation in performance.  The solution to this problem
is to use the head-tracking ability of Pfinder to steer the audio
input so that it focuses on the user's head.

There are several potential ways to steer audio input.  One is to use
a highly directional microphone that can be panned using a motorized
control unit, to track the user's location.  However, this requires a
significant amount of mounting and control hardware, is limited by the
speed and accuracy of the drive motors, and can only track one user at
a time.

It is therefore preferable to have a directional response that can be
steered electronically.  This can be done with the well-known
technique of beam forming with an array of microphone elements, as
illustrated in Figure \ref{Pen-fig:audio}(a). Though several microphones
need to be used for this method, they need not be very directional and
they can be permanently mounted in the environment.  In addition, the
signals from the microphones in the array can be combined in as many
ways as the available computational power is capable of, allowing for
the tracking of multiple moving sound sources from a single microphone
array.  For additional detail, see references \cite{Pen-ali96,Pen-basu96}.

%\subsection{Audio Output}
%
%The audio output system can position sounds at arbitrary azimuths and
%elevations around a listener's head.  The system uses stereo
%loudspeakers arranged conventionally (at $\pm 30$ degrees with respect
%to the listener).  The system works by reconstructing the acoustic
%pressures at the listener's ears that would occur with a free-field
%sound source at the desired location.  This is accomplished by
%combining a {\em binaural spatializer} with a {\em transaural audio
%system}, as shown in Figure \ref{Pen-fig:audio}(b).
%
%The spatializer convolves the source sound with the direction
%dependent filters that simulate the transmission of sound from
%free-field to the two ears.  The resulting binaural output of the
%spatializer is suitable for listening over headphones. In order to
%present the audio over loudspeakers, the output of the spatializer is
%fed to a transaural audio system which delivers binaural signals to
%the ears using stereo speakers.  The transaural system filters the
%binaural signals so that the crosstalk leakage from each speaker to
%the opposite ear is canceled.

\subsection{Recognizing Human Behaviors}


\begin{figure}
\centerline{\hfil
(a) \psfig{figure=Pentland_figs/ASL.ps,height=3.5cm}\hfil
(b) \psfig{figure=Pentland_figs/sirca1.eps,height=3.5cm}\hfil}
\caption[]{(a) Real-time reading of American Sign Language (with Thad
Starner doing the signing), and (b) real-time 
classification of driver's actions in a driving simulator.}
\label{Pen-fig:action}
\end{figure}

Work on recognizing body position, face, expression, and sound are
only the first steps toward perceptual intelligence.  To
make computers really useful, these basic perceptual functions need to
be integrated with higher-level models of human behavior, so that we can
begin to understand what the person is {\sl doing.} 
%This is the most
%difficult challenge facing the Looking at People project: attempting
%to understand what a person is thinking, doing, and intending.  Not
%just what word they uttered, or what motion they performed, but what
%they {\sl meant}.

%The technical difficulty, of course, is that we generally do not
%understand humans well enough to model their behavior accurately.
%This difficulty has forced most attempts at human-machine interface to
%adopt the simplest possible model of the human, such as a random
%number generator or a temporal filter.  Such simple models, however,
%make it impossible to build a system that takes real advantage of the
%human's abilities.

A general approach to interpreting human behavior is to directly
extend the MAP techniques used for recognition of face, expression,
and body pose.  That is, to track the 2-D parametric appearance models
over time, and then probabilistically characterize the time evolution
of their parameters.  This approach produces the PDF of the behavior
as both a function of time and 2-D appearance.  Behaviors can then be
recognized by comparing them to these learned models using MAP
methods.  To obtain high accuracy behavior recognition, one must use
an elastic spatio-temporal matching technique such as dynamic time
warping \cite{G-Darrell93} or Hidden Markov Modeling \cite{G-starner95}.

To recognize particular gestures or behaviors, for instance, the
person is modeled as a Markov device, with internal mental states that
have their own particular distribution of appearance and inter-state
transition probabilities.  Because the internal states of a human are
not directly observable, they must be determined through an indirect
estimation process, using the person's movement and vocalizations as
measurements.  One efficient and robust method of accomplishing this
is to use the Viterbi recognition methods developed for use with
Hidden Markov Models (HMM).

%A simple example of this type of human
%model would be if each of the internal states were a standard
%quadratic controller, each using different dynamics and measurements,
%together with a network probabilistic transitions between them.
%To integrate this human model into a large software system it is
%necessary to know which controller is currently ``in charge,'' so that
%the remainder of the system can configure itself to achieve optimum
%overall performance.  
%
%In this manner we can produce a human-machine system that is able to
%dynamically reconfigure itself based on the internal state of the
%human operator.  This allows much higher performance than is possible
%with a fixed model of the human (assuming similar model complexity).

This general approach is similar to that taken by the speech
recognition community.  The difference is that here internal state is
not thought of as being just words or sentences; the internal states
can also be actions or intentions.  Moreover, the input is not just
audio filter banks but also facial appearance, body movement, and
vocal characteristics such as pitch as inputs to infer the user's
internal state.  Two good examples of that employ this approach to
behavior recognition are reading American Sign Language (ASL)
\cite{G-starner95}, and interpreting automobile driver's behavior
\cite{Pen-ivhs}.

The ASL reader is a real-time system that performs high-quality
classification of a forty-word subset of ASL using only the hand
measurements provided by Pfinder (e.g., hand position, orientation,
and width/height ratio).  Thad Starner is shown using this system in
Figure \ref{Pen-fig:action}(a).  The accurate classification performance
of this system is particularly impressive because in ASL the hand
movements are rapid and continuous, and exhibit large coarticulation
effect.

The second system interprets people's actions while driving a car
\cite{Pen-ivhs}.  In this system the driver's hand and leg motions were
observed while driving in the Nissan Cambridge Basic Research Lab's
driving simulator (see Figure \ref{Pen-fig:action}(b)).  These
observations are used to classify the driver's action as quickly as
possible.  The goal is to develop safer cars by having the automobile
``observe'' the driver, continuously estimate the driver's internal
state (what action they are taking), and respond appropriately.

I found that the system was surprisingly accurate at identifying which
driving maneuver the driver was beginning to execute (e.g., passing,
turning, stopping, or changing lane).  A recognition accuracy of
$95.24 \% \pm 3.1 \%$ was obtained at 0.5 seconds after of the
beginning each maneuver...long before the major, functional parts of
the maneuver were executed.  This experiment shows that driver's
execute {\sl preparatory} movements that are reliable indicators of
which maneuver they are beginning to execute.

\section{Putting It All Together}

Given these building blocks of perceptual intelligence, it is possible
to try to ``put it all together'' and actually build autonomous
creatures (``agents'') that interact with people in a natural,
life-like manner without use of goggles, position sensors, or wires.
The ALIVE experiment, which stands for ``Artificial Life Interactive
Video Environment,'' has accomplished this goal \cite{G-maes96}.  Rather
than try to model human-like agents immediately, however, the
creatures in ALIVE are (mostly) animals.  The hope was that in this
manner we could focus on the basics of achieving natural, life-like
behavior, without being distracted by the difficulties of modeling
natural language or complex application domains.

ALIVE uses a single video camera to obtain a color image of a person, which
is then placed (with correct depth and occlusion) within a 3-D
graphical world.  The resulting image is projected onto a large screen
that faces the user and acts as a type of ``magic mirror'' (Figure
\ref{Pen-alive}): the user sees herself surrounded by objects and
agents. No goggles, gloves, or wires are needed for interaction with
the virtual world.  The creatures in this virtual world are
autonomous, behaving entities that use the camera and microphones to
sense the actions of the participant, interpret them, and react to
them in real time.   The perceptual intelligence of these agents is truly
active and purposive \cite{Pen-bajcsy,Pen-aloimonos}

%\subsection{Modeling Agents}

The creatures in ALIVE are modeled as autonomous semi-intelligent
agents using principles drawn from the ethology literature
\cite{Pen-lorenz}.  They have a set of internal needs and motivations,
a set of sensors to perceive their environment, a repertoire of
activities that they can perform, and a physically-based motor system
that allows them to move in and act on the virtual environment. A
behavior system decides in real-time which activity to engage in to
meet the internal needs of the agent and to take advantage of
opportunities presented by the current state of the environment. The
agent's kinematic state is updated according to the motor activities
associated with the chosen behaviors, and the agent rendered on the
graphics display.  The user's location and hand and body gestures
affect the behavior of the agents, and the user receives visual as
well as auditory feedback about the agents' internal state and
reactions.

The ALIVE system is built using Blumberg's and Maes's behavior
modeling tools for developing semi-intelligent autonomous agents
\cite{Pen-bruce-SAB,Pen-maes-book}; its goal is to produce agents that select
an optimal activity on every time step given their internal needs and
motivations, past history, and the perceived environment with its
attendant opportunities, challenges, and changes.  The activities are
chosen such that the agents neither dither among multiple activities,
nor persist too long in a single activity. The behavior system is
capable of interrupting a given activity if a more pressing need or an
unforeseen opportunity arises.

When using the behavior toolkit to build an agent, the designer specifies:

\begin{itemize}

\item{{\em The motivations or internal needs of the agent.}
Internal needs are modeled as variables which may vary over time. For
example, the ``dog'' agent in ALIVE has the internal need to receive
attention from the user. Whenever the user pats the dog, this variable
will temporarily decrease in value and as a result the dog will be
less motivated to seek human attention.}

\item{ {\em The activities and actions of the agent.}  An agent's set
of activities is organized as a loose hierarchy with the top of the
hierarchy representing more general activities and the leaves
representing more specific activities. For example, the dog agent has
several top-level activities such as Playing, Feeding, Drinking,
etc. A top-level activity such as Feeding has several children:
Chewing, Preparing-to-Eat, and Searching-for-Food.  Searching-for-Food
in turn has three children: Wander, Avoid-Obstacles, Move-to-Food, and
so on.  The lowest-level activities control the motor system, for
example, making the dog move a little to the left or right, or making
it bark in a certain way.}

\item{{\em The virtual sensors of the agent.}  The sensors of the
agent are uniform between the real world and the virtual world.  For
instance, a distance sensing action aimed at the user will use the
camera to estimate distance, whereas the same action aimed at another
virtual creature will fire a ray in the direction of other creature
and measure the distance to intersection with the other creature's
bounding box.}

\end{itemize}

Given the above information, the behavior system selects the
activities most relevant to the agent at a particular moment in time
given the state of the agent, the situation it finds itself in, and its
recent behavior history.  One of the design goals of the behavior
system, and the modeling system in general, was that it be compatible
with a physically-based simulated motor system such as those proposed
by Raibert \cite{Pen-raibert} or Zeltzer \cite{Pen-zeltzer}.  Consequently,
the ALIVE agents use a combination of physically-based and kinematic
modeling to implement the motor system; Featherstone's spatial algebra
\cite{Pen-featherstone} and an adaptive-step-size RKF numerical integrator
are used to model the movement of the agent in response to forces
applied by its driving motor.  Details of the behavior and modeling
systems are reported upon in \cite{Pen-bruce-SAB} and \cite{Pen-bruce-AAAI-SS}.

The ALIVE system represents the user as a ``special'' sort of 3-D
agent.  The position and state of the user-agent are based on the
information computed by the vision system (e.g., position, gesture)
and the auditory system (e.g., speech recognition, pitch recognition
\cite{Pen-basu96}).  This allows the artificial agents to sense the user
using the same virtual sensors that they use to detect virtual objects
and other virtual agents. The person agent is rendered in the final
image using the live video image of the actual user mapped to the
correct place in the 3-D virtual world.


\begin{figure}[tb]
\centerline{
(a) \psfig{figure=Pentland_figs/composite.sit.eps,width=5cm}
\hfill
(b) \psfig{figure=Pentland_figs/composite.right.eps,width=5cm}}

\centerline{
(c) \psfig{figure=Pentland_figs/composite.shake.eps,width=5cm}
\hfill
(d) \psfig{figure=Pentland_figs/composite.beg.eps,width=5cm}}

\caption[]{(a) Image of user is composited in 3-D with computer graphics;
here the dog responds to pointing gesture by sitting.
(b) Another example of a recognized gesture; the dog walks 
in direction indicated by user.
(c) The dog shakes hands with user; note that this behavior depends on
knowing the stance of the user.
(d) The dog standing on hind legs to mimic user's gesture.}
\label{Pen-alive} 
\end{figure}

In the most recent version of ALIVE the user interacts
with a virtual dog, as shown in Figure \ref{Pen-alive}. The dog has
a repertoire of behaviors that include playing, feeding, drinking,
receiving attention, and sleeping.  The dog also uses auditory input,
consisting of simple verbal commands that are recognized using a
commercial speech recognition system, and produces auditory output,
consisting of a wide variety of prerecorded samples which are played
at appropriate times. The dog has both interactive behaviors and
autonomous action; while its primary goal is to play with the user,
internal motivations (e.g., thirst) will occasionally override.

This system has been installed in several public exhibitions, which
has allowed us to gather experience with thousands of users in the
system. The overwhelming majority report that they enjoy interacting
with the system and consider the actions and reactions of objects and
agents believable.  I consider people's reactions to these systems to
be a sort of informal Turing test; for instance, people attribute all
sorts of intentions, goals, and reasoning ability to the dog, that is,
they treat it as if it were an intelligent creature.


\section{Conclusion}

I have described my progress toward building computer systems that
understand people, and can work with them in the manner of an
attentive human-like assistant.  To accomplish this I have built a
series of experimental testbeds, called {\sl Smart Rooms}, that are
able to answer some of the who, what, where, when, and why questions
with respect to human behavior.

These Smart Rooms have used cameras and microphones to accomplish
real-time 3-D tracking of head, hands, and feet, as well as
recognition of hand and body gestures.  The system can also support
face recognition and interpretation of facial expression.  By
developing the perceptual intelligence needed to interpret human
behaviors, and coupling them to a simple set of motor reactions, we
have been able to build autonomous creatures that interact with people
in an interesting and even lifelike manner.  My hope is that this is
the first step toward the development of truly useful computer
assistants.



