%\newcommand{\Vector}[1]{{\bf{#1}}}
%\newcommand{\Matrix}[1]{\Vector{#1}}


\author{C.J. Taylor, A. Lanitis, T.F. Cootes, G. Edwards \& T. Ahmad  \\
\\University of Manchester}
\chapter{Model-Based Interpretation of Human Faces and Hand Gestures}


\begin{Abstract}
Face images and hand gestures are an important means of communication between humans. 
Automatic face and gesture recognition systems could be employed similarly 
for contact-less human-machine interaction systems. However, developing such 
systems is difficult, because faces and hands are both complex and highly variable 
structures. We describe how flexible models can be used to represent varying 
appearance of faces and hands and how these models can be used for tracking 
 and interpretation. Experimental results are presented for face pose recovery, 
face identification, expression recognition, gender recognition and gesture 
interpretation.
\end{abstract}


\section{Introduction}

This paper addresses the problem of locating and interpreting faces and hand 
gestures in images. By interpreting face images we mean recovering the 3D pose, 
identifying the individual and recognizing the expression and gender; for the hand images 
we mean recognizing the configuration of the fingers. In both cases different 
instances of the same class are not identical; for example, face images belonging 
to the same individual will vary because of changes in expression, lighting 
conditions, 3D pose and so on. Similarly hand images displaying the same gesture 
will vary in form. 

        We have approached these problems by  modeling the ways in which the 
appearance of faces and hands can vary, using parametrised deformable models 
which take into account all the main sources of variability.  A robust image 
search method ~\cite{Tay-Cootes_cvgip95,Tay-Cootes_bmvc94} is used to fit the models to new face/hand 
images recovering 
compact parametric descriptions. Given this compact coding, standard statistical 
pattern recognition techniques can be used to perform the task of interpretation.

        For our face interpretation experiments we have used a database containing 
690 face images from 30 individuals. We have described the contents of the 
database elsewhere ~\cite{Tay-Lanitis94}.  (The database is publicly available on the Word Wide 
Web at: http://peipa.essex.ac.uk/ftp/ipa/pix/faces/manchester).

        In the remainder of the paper we review some of the most relevant 
literature on face coding and interpretation, describe our approach in more 
detail, and present results for coding and reconstruction, pose recovery, person 
identification, gender recognition, and expression recognition. We also describe 
how our approach can be applied to the problem of gesture recognition and present 
preliminary results for this problem.


\section{Background}

In this section we briefly review previous work on face image interpretation 
and coding. Comprehensive reviews for these topics can be found elsewhere ~\cite{G-afgr95,Tay-
Chellapa95,Tay-Samal92,Tay-Valentin94}.


\subsection{Locating Facial Features}

Methods based on deformable templates have proved useful for locating facial 
features. Kass et al~\cite{G-kass87:snakes} describe the use of active contour models - snakes 
- for tracking lips in image sequences.  They initialize a snake on the lips 
 in a face image and show that it is able to  track lip movements accurately. 
Because snakes do not incorporate prior knowledge about expected shapes, this 
approach is easily confused both by other structures present in the image and 
occlusion. Yuille et al~\cite{Tay-Yuille92a} describe the use of deformable templates, based 
on simple geometrical shapes, for locating eyes and mouths. These templates 
are similar to snakes, in that they can deform and move. Yuille's models incorporate 
shape constraints but it is difficult to ensure that the form of a given model 
is sufficiently general or that an appropriate degree of variability has been 
allowed.

        Human faces are characterized by constrained geometrical relationships 
between the positions of facial features. Some systems exploit these constraints 
to locate groups of features~\cite{G-Brunelli93,Tay-Craw_eccv92}. For example, once one feature has been 
located, the positions of other features can be predicted and their search 
areas reduced significantly.

\subsection{Coding and Reconstruction}

Kirby and Sirovich~\cite{Tay-Kirby90} propose the decomposition of face images into a weighted 
sum of basis images (or eigenfaces) using a Karhumen-Loeve expansion. They 
code a face image using 50 expansion coefficients and subsequently reconstruct 
an approximation using these parameters.

        Model-based coding and reconstruction has also received a considerable 
attention in the literature. Models based on the physical and anatomical structure 
of faces~\cite{Tay-Terzopoulos93} and 3D models~\cite{Cip-Li:93} have been used to account for the 
variability in appearance due to changes in pose and expression.

\subsection{Interpretation}

Face image interpretation techniques can be divided into two main categories: 
those employing geometrical features and those using grey-level information.

        Techniques based on geometrical features use a number of dimensional 
measurements~\cite{G-Brunelli93}, or the locations of a number of key points~\cite{Tay-
Craw_bmvc92,Cip-Gee:94d,Tay-Yacoob94} in 
single images or in image sequences, for face image interpretation.

        Techniques based on grey-level information show more promise. Turk 
and Pentland~\cite{Mas-turk-pentland91} describe face identification using an eigenface expansion. 
The eigenface weights are used for classification. Many researchers have built 
on this approach~\cite{Tay-Cottrell91,Tay-Craw_bmvc92}, which is perhaps the most successful to date. 
Other 
approaches based on grey-level information, include template matching~\cite{G-Brunelli93} and 
filtering with multi-scale Gabor filters~\cite{Tay-Lades93}.

\section{Overview of our Approach}

Our approach can be divided into two main phases: modeling, in which flexible 
models of facial appearance are generated, and interpretation, in which the 
models are used for coding and interpreting face images.

\subsection{Modeling}

We model the shapes of facial features and their spatial relationships using 
a single 2D flexible shape model (a Point Distribution Model)~\cite{Tay-Cootes_cvgip95}. The model 
is generated by a statistical analysis of the positions of landmark points 
over a set of training face shapes. The model describes the mean shape and 
a set of linearly independent �modes� of shape variation. Any  face shape can 
be represented using a weighted sum of the mean and modes of variation controlled 
by a small number of model parameters.

        We have augmented our shape model with flexible grey-level models (of 
similar form to the shape model) using two complementary approaches. In the 
first we generate a flexible grey-level model of �shape-free� appearance by 
deforming each face in the training set to have the same shape as the mean 
face. In the second approach we use a large number of local profile models, 
one at each landmark point of the shape model. The first approach is more complete 
but the second is more robust to partial occlusion. 

        Shape and grey-level models are used together to describe the overall 
appearance of each face; collectively we refer to the model parameters as appearance 
parameters. It is important to note that the coding we achieve is reversible 
- a given face image can be reconstructed from its appearance parameters.

\subsection{Interpretation}

When a new image is presented to our system, facial features are located automatically 
using Active Shape Model (ASM)  search~\cite{Tay-Cootes_cvgip95,Tay-Cootes_bmvc94} based on the 
flexible shape model 
obtained during training. This results in automatically located model points which
are transformed into shape model parameters. Grey-level information at each 
model point is collected and transformed to local grey-level model parameters. 
Then the face is deformed to the mean face shape and the grey-level appearance 
is transformed into the parameters of the shape-free grey-level model. The 
resulting set of appearance parameters can be used for image reconstruction, 
person identification (including gender recognition), expression recognition 
and pose recovery.

\section{Modeling Faces}

We have described elsewhere~\cite{Tay-Lanitis94,Tay-Lanitis95} the generation of flexible models of facial 
appearance. In this section we provide a brief introduction.

\subsection{Flexible Models}

A flexible model~\cite{Tay-Cootes_cvgip95} can be generated from a set of training examples, each 
represented by $N$ variables. The average example 
(${\bf \hat{X}}$) is calculated and the deviation 
of each example from the mean is computed. A principal component analysis of 
the covariance matrix of deviations reveals the main modes of variation. Any 
training example ${\bf X}_{i}$ can be approximated using:

\[
	{\bf X}_i = {\hat{\bf X}} + {\bf Pb}
\]

Where ${\bf P}$ is a matrix of unit eigenvectors of the covariance of deviations and 
${\bf b}$ is a vector of eigenvector weights 
(these are referred to as model parameters). 
By modifying ${\bf b}$, new instances of the model 
can be generated. Since the columns 
of ${\bf P}$  are orthogonal ${\bf P}^T {\bf P}={\bf I}$, 
the equation shown previously can be solved with respect to ${\bf b}$.


\[
	{\bf b} = {\bf P}^T({\bf X}_i-{\hat{\bf X}})
\]


Therefore, training examples can be transformed to model parameters. 
Usually, the number of eigenvectors needed to describe the variability within 
a training set is much smaller than the original number of parameters, so that 
training examples can be approximated with a small number of model parameters 
$b_1 \ldots b_t: t<N$.
The same method can be used to train both shape and grey-level models. For 
shape models  the variables are point co-ordinates and for grey-level models 
the variables are based on grey-level intensities.

\subsection{Modeling Shape}

We have built a flexible shape model (or Point Distribution Model)~\cite{Tay-Cootes_cvgip95} 
representing a line drawing of the face using 152 points (see figure~\ref{Tay-fig:points_overlayed}) on each 
of 160 training examples. The model can approximate the shape of any face in the training set using just 16 
modes of variation/parameters; the effect of varying some of the model parameters is shown in 
figure~\ref{Tay-fig:shape_modes_variation}.



\begin{figure}
\centerline{\epsfxsize=1in
\epsffile{Taylor_figs/fig1}}
\caption{Shape model points}
\label{Tay-fig:points_overlayed}
\end{figure}


\begin{figure}
\centerline{\epsfxsize=2.0in
\epsffile{Taylor_figs/fig2}}
\caption{The main modes of shape variation}
\label{Tay-fig:shape_modes_variation}
\end{figure}




\subsection{Modeling Shape-Free Appearance}


We wish to model grey-level appearance independently of shape. To do this we 
have applied a warping algorithm~\cite{Tay-Bookstein89} to deform all training images to the mean 
shape,  in such a way that changes in grey-level intensities are kept to a 
minimum. Training images were deformed to the mean shape and grey-level intensities 
within the face area were extracted. Each training example was represented 
by a vector containing the normalized grey-level at each pixel in the patch. 
A flexible grey-level model was generated for our database; only 79 modes of 
variation/parameters were needed to explain 95\% of the variation. Each parameter 
is responsible for a specific mode of variation, as shown in Figure ~\ref{Tay-fig:grey_level_modes}.


\begin{figure}
\centerline{\epsfxsize=2.0in
\epsffile{Taylor_figs/fig3}}
\caption{The main modes of grey-level variation}
\label{Tay-fig:grey_level_modes}
\end{figure}



\subsection{Modeling Local Appearance}


To model local grey-level variations we overlaid shape model points on the 
corresponding training images and extracted grey-level profiles perpendicular 
to the boundary, as shown in Figure ~\ref{Tay-fig:profile_exraction}. A flexible grey-level model  was built 
for the profile at each model point; most of these models needed four modes of 
variation/parameters to explain 95\% of the variation.


\begin{figure}
\centerline{\epsfxsize=3.0in
\epsffile{Taylor_figs/fig4}}
\caption{Extraction of a grey profile at a model point}
\label{Tay-fig:profile_exraction}
\end{figure}



\section{Locating Facial Features}


The shape model and local grey-level models described above can be used to 
locate all the modeled features simultaneously ~\cite{Tay-Cootes_cvgip95,Tay-Cootes_bmvc94}. The 
mean shape model 
is placed in the image and is allowed to interact dynamically until it fits 
to the image evidence. Each iteration involves two main steps: calculating 
a new suggested position for each model point based on matching the local grey-level 
models, followed by moving and deforming the model in order to place each point 
as close as possible to its new preferred position. During this process the 
shape model is only allowed to deform in ways which are consistent with the 
training set. The robustness, speed and accuracy of this basic algorithm can 
be improved using a multi-resolution approach ~\cite{Tay-Cootes_bmvc94}. The fitting procedure is 
illustrated 
in Fgure ~\ref{Tay-fig:fitting_shape_model} and described in detail in ~\cite{Tay-Cootes_cvgip95,Tay-
Cootes_bmvc94}.  We have shown ~\cite{Tay-Cootes_bmvc94} that model points in new face images can 
be located with an average accuracy of about 3 pixels,  using this method.


\begin{figure}
\centerline{\epsfxsize=3.0in
\epsffile{Taylor_figs/fig5}}
\caption{Fitting the shape model}
\label{Tay-fig:fitting_shape_model}
\end{figure}



\section{Recovering Appearance Parameters}

New face images can be coded in terms of model parameters. This procedure is 
summarized in Figure ~\ref{Tay-fig:calc_appear_params}. The shape model is fitted automatically to the 
new 
face and the shape model parameters corresponding to the extracted shape are 
computed. At each located shape model point, grey-level information from a 
profile perpendicular to the boundary is extracted and the model parameters 
of the local grey-level model are calculated. The warping algorithm is applied 
to produce a shape-free face and the shape-free grey-level model parameters 
are calculated. Together the three sets of model parameters constitute a full 
set of appearance parameters; in the following sections we describe how we 
use these parameters to reconstruct and interpret face images.


\begin{figure}
\centerline{\epsfxsize=3.5in
\epsffile{Taylor_figs/fig6}}
\caption{Calculating the appearance parameters}
\label{Tay-fig:calc_appear_params}
\end{figure}


\section{Reconstructing Face Image Sequences}

The flexible shape model can be used for tracking face movements in a video 
sequence. An example of tracking is shown in Figure~\ref{Tay-fig:face_tracking}. At each frame the 
appearance 
parameters can be calculated and used for reconstructing the face. To reconstruct 
a face the shape-free grey-level parameters are used to generate the shape-free 
grey-level appearance, and the shape model parameters to define the shape of 
the face. The warping algorithm~\cite{Tay-Bookstein89}  is applied to deform from the average 
face shape to the shape corresponding to the shape parameters given.


\begin{figure}
\centerline{\epsfxsize=3.0in
\epsffile{Taylor_figs/fig7}}
\caption{Face tracking using a flexible shape model}
\label{Tay-fig:face_tracking}
\end{figure}



        Figure ~\ref{Tay-fig:reconstructions} shows examples of reconstructions from face image sequences. 
The images in these sequences are {\it new} images  of individuals who were in the 
training set. The total number of parameters needed for coding and reconstructing 
faces using this method, is 99 (16 shape parameters, 4 2D orientation parameters 
and 79 shape-free-grey-level parameters). It should be noted that in the reconstructed 
images, the pose, expression and individual appearance are faithfully reproduced 
thus a face image interpretation system based on appearance parameters should 
be feasible.

\begin{figure}
\centerline{\epsfxsize=3.0in
\epsffile{Taylor_figs/fig8}}
\caption{Example of tracking and reconstruction of face image sequences}
\label{Tay-fig:reconstructions}
\end{figure}


\section{Recovering Pose}

Once the shape model has been fitted to a new face, the recovered shape model 
parameters can be used to determine the pose. As  shown in Figure ~\ref{Tay-fig:shape_modes_variation}, 
the first 
(b1) and third (b3) shape model parameters are responsible for controlling 
the apparent changes in shape due to nodding and turning. Pose recovery can 
be based on the values of these parameters, calculated for a new face outline. 
To calibrate the system, we captured a series of face images in which an individual 
was asked to rotate his head in both the vertical and horizontal direction, 
from -20 degrees to +20 degrees by looking at a number of grid points on a 
wall; each grid point corresponded to a known combination of horizontal and 
vertical rotation angles. We fitted the shape model to these images automatically 
and recorded the values of the first and third shape parameters. When a new 
face is presented the shape model is fitted, the resulting numerical values 
for b1 and b3 are recorded and the pose angles are calculated based on the 
calibration.

        We tested the accuracy of pose recovery on 30 new test images obtained 
similarly to the calibration set. The results obtained are summarized in Figure 
~\ref{Tay-fig:pose_recovery_res} and show robust recovery of pose angles. Figure~\ref{Tay-
fig:pose_recovery_ex} shows examples of test 
images and the pose angles computed.


\begin{figure}
\centerline{\epsfxsize=3.0in
\epsffile{Taylor_figs/fig9}}
\caption{Pose recovery results}
\label{Tay-fig:pose_recovery_res}
\end{figure}

\begin{figure}
\centerline{\epsfxsize=3.0in
\epsffile{Taylor_figs/fig10}}
\caption{Pose recovery example}
\label{Tay-fig:pose_recovery_ex}
\end{figure}



\section{Identifying Individuals}

We have developed a face identification system which uses appearance parameters. 
A classifier was trained by computing the appearance parameters for all training 
images (10 images for each of the 30 individuals) in our database and establishing 
the distribution of appearance parameters for each individual. Discriminant %%%Discriminating?
analysis techniques  were applied in order to enhance the effect of the inter-class 
(between person) variation parameters using the Mahalanobis distance measure.

        We performed person identification experiments on the test set of our 
face database which contains images with a wide range of expressions, poses 
and lighting conditions. When all three types of appearance parameters were used,
92\% of the test images were classified correctly. (More detailed results of 
the classification experiments are given in~\cite{Tay-Lanitis94}).

        For access control applications it is important that person identification 
systems have the ability to reject faces which do not look similar to any of 
the database entries. We performed a preliminary experiment in which images 
were rejected if the minimum Mahalanobis distance was not within acceptable 
limits. The results of this experiment are shown in Figure~\ref{Tay-fig:changing_rejection}. When the 
threshold 
was set to 0.8 the correct classification rate was 100\%. In this case, about 
55\% of the test images are rejected since they are not similar enough to any 
of the individuals in the training set, which is not surprising bearing in 
mind the appearance variations between training and test images in our database.
 

\begin{figure}
\centerline{\epsfxsize=3.0in
\epsffile{Taylor_figs/fig11}}
\caption{The effect of changing the rejection threshold}
\label{Tay-fig:changing_rejection}
\end{figure}




\section{Gender Recognition}

Gender recognition can also be implemented in our framework, since some appearance 
parameters reflect inter-gender differences. We trained the system using the 
appearance parameters of images from 16 male and 4 female subjects in our 
database, using a similar approach to the person identification experiments. 
A correct classification rate of 94\% was obtained when we tested the system 
on 100 images of unseen individuals.
 
\section{Expression Recognition}

Expression recognition, particularly from static images, is a difficult  problem. 
Even human observers often fail to agree in expression classification experiments. 
We have addressed the problem  by establishing the distribution of appearance 
parameters over a selected training set for each expression category, so that 
the appearance parameters calculated for a new face image could be used for 
determining the expression.



\begin{figure}
\centerline{\epsfxsize=3.0in
\epsffile{Taylor_figs/fig12}}
\caption{Faces displaying the seven expressions we used in the expression recognition experiment}
\label{Tay-fig:7_expressions}
\end{figure}


        We asked five observers to classify the expression of each of our training 
and test face images using the seven psychologically recognized categories ~\cite{Mas-ekman-friesen75} 
shown in Figure ~\ref{Tay-fig:7_expressions}. For our experiments we used the images for which at 
least four of the observes agreed (139/300 training and 118/300 test images). 
A peak classification rate of 74\% was obtained. We also asked two more observers 
to classify the expression of the faces in each of the 118 test images on which 
we tested our classification system. They achieved 80\% and 83\% correct classification. 
The classification rate obtained from the automatic system compares well with 
these results.

\section{Gesture Recognition}


The framework we have described can also be used for gesture
recognition ~\cite{Tay-Ahmad95,Tay-Heap95}. We trained a flexible shape model using hand shapes 
extracted from a sequence 
of moving hands, using 89 landmarks on each training shape; just nine shape parameters 
were needed to explain most of the variability in the training set. In the 
sequence the subject was asked to animate the five gestures shown in Figure~\ref{Tay-
fig:example_gestures}. 
 These gestures were intended to be useful in mouse-less interaction with a 
windows-based operating system and represent actions such as point, point-and-click, 
grab, and swap.

\begin{figure}
\centerline{\epsfxsize=3.0in
\epsffile{Taylor_figs/fig13}}
\caption{The gestures used in our experiments}
\label{Tay-fig:example_gestures}
\end{figure}

 
\begin{figure}
\centerline{\epsfxsize=3.0in
\epsffile{Taylor_figs/fig14}}
\caption{The main modes of shape variation for the  hand  model}
\label{Tay-fig:gesture_modes}
\end{figure}



        The main modes of shape variation of the model are shown in Figure ~\ref{Tay-fig:gesture_modes}. 
Since shape information is sufficient for classifying gestures, we just used 
a shape model for classification, though local grey-level models were 
used for model fitting. Tracking was performed using model-based image search 
as described earlier for face images. The resulting shape parameters were used 
to determine the gesture in each frame using a Mahanabolis distance classifier. 
In practice, the constraints imposed by the multi-gesture shape model were 
relatively weak because, in order to span the full range of training examples, 
the linear sub-space of the model also included regions which corresponded 
to �illegal� examples of the hand shape. This lack of specificity (to legal 
hand configurations) led to lack of robustness in model-based tracking. We 
dealt with the problem by using five models for tracking, each trained using 
a large number of examples of one gesture and fewer examples of the other gestures. 
Each model was thus reasonably specific to the particular class of gesture 
but was able to model transitions to the other gestures. During tracking, each 
frame was classified using the multi-gesture model and the resulting gesture 
class was used to select the gesture-specific model to use for tracking the 
hand shape in the following frame. Figure~\ref{Tay-fig:hand_tracking} shows results of hand tracking 
using this method. 


\begin{figure}
\centerline{\epsfxsize=3.0in
\epsffile{Taylor_figs/fig15}}
\caption{Example of hand tracking}
\label{Tay-fig:hand_tracking}
\end{figure}



        The system was tested on a previously unseen  hand  image sequence 
containing 811 frames; 480 of those frames were rejected because they contained 
transitions between gestures. The rest of the frames were classified correctly.


\section{Conclusions}


We have presented an automatic face processing system which can be used for 
locating facial features, coding and reconstruction, recovering pose,  recognizing 
expressions and identifying the individual in an image. We believe that our 
system is unusual in that it can cope successfully with almost all the aspects 
of face image processing, within a unified framework.

        The face processing procedures described are fully automatic; errors 
in the classification experiments  may be caused either by failure in locating 
landmarks accurately, or by failure of the classification algorithm. We do 
not distinguish between the two cases, since we believe that locating facial 
characteristics automatically is an important aspect of an integrated face 
processing system.

        We have also described preliminary results for interpreting gestures, 
using  a similar approach. Our results for most of the applications are promising, 
and demonstrate the potential of our approach in real life applications. 


\section*{Acknowledgments}

Most of the work described was supported by a University of Manchester Research 
Studendship and an ORS award to Andreas Lanitis. Further research in the area 
is supported by EPSRC and British Telecom. Discussions with Professor  J.F.W. Deakin 
and Dr. J.F. Whittaker  from the Dept. of Psychiatry, Manchester Royal Infirmary, 
UK, have been particularly useful for the expression recognition experiments. 
We are grateful to all volunteers who provided face and hand images for our 
experiments.



