%\documentstyle[cup7b,david,epsf]{cuphci}

%\begin{document}

%\bibliographystyle{plain}

\author[Cipolla, Gee and Hollinghurst]{R. Cipolla, A.H. Gee and N.J. Hollinghurst
\\University of Cambridge} 
\chapter{Pointing and Face Gestures in Interactive Robotics}
%\setcounter{page}{3}
%\setcounter{equation}{7}

\begin{abstract}
Computer vision provides many opportunities for novel man--machine
interfaces.  Pointing and face gestures can be used as a simple,
passive means of interfacing with computers and robots. In this paper
we describe two novel algorithms to track the position and orientation
of the user's hand or face in video images. This information is used
to determine where the hand or face is pointing. This can be used in
interactive robotics to allow a user with manipulation disabilities or
working in hazardous environments to guide a robot manipulator to pick
up a simple object of interest.
\end{abstract}


\section{Introduction}
\label{Cip-sec:intro}
There has been a lot of interest lately in the use of hand and face
gestures as a means for interfacing with computers and robots: they
are intuitive for the operator, and provide a rich source of
information to the machine. This type of interface is particularly
appropriate in applications such as virtual reality, multimedia and
teleoperation. It is also important as a means of helping individuals
with manipulation disabilities.

Most current commercial implementations rely on sensors that are
physically attached to the hand, such as the `DataGlove'. More
recently, systems have been proposed using {\em computer vision} to
observe the hand. Some require special gloves with attachments or
markings to facilitate the localization and tracking of hand parts,
but others operate without intrusive hardware. This is attractive
because it is convenient for the user and potentially cheaper to
implement.


In this chapter we describe two novel algorithms to track the position
and orientation of the user's hand or face in video images. This
information is used to determine where the hand or face is
pointing. This can be used in {\em interactive robotics} to allow a
user with manipulation disabilities or working in hazardous
environments to guide a robot manipulator to pick up a simple object of
interest.
 
We describe preliminary implementations of these algorithms. We also
describe an interactive robotic inspection system being designed at
Cambridge to provide a means for a physically disabled person to
perform the job of visual inspection of hybrid micro-circuits.

\section{Pointing Interfaces}
\label{Cip-sec:point}

Here we present an experimental system that uses {\em pointing\/}
gestures to guide a robot to pick and place objects on a table 
top~\cite{Cip-Cipolla94,Cip-Cipolla96}.
A pair of cameras view the table and pointing hand in stereo.
Active contours are used to track the hand in real time.
By simple geometry we can calculate where the hand is pointing
in the robot's workspace, without camera calibration, to an accuracy
of about 10mm.

\subsection{Geometrical Framework}

A single view of a pointing hand is ambiguous: its distance from the
camera cannot be determined, and the `slant' of its orientation cannot
be measured with any accuracy. This means that the
�piercing point�, where the line defined by the hand intersects the
work surface, is constrained to a line, which is the projection of
the hand's line in the image (see figure \ref{Cip-geom_fig}). A second
view is needed to fix its position in two dimensions \cite{Cip-quan}.

\begin{figure}[bhpt]
\centerline{\epsfysize=1.8in \epsffile{Cipolla_figs/geom.eps}}
\caption{Relation between lines in the world, image and ground planes}
\label{Cip-geom_fig}
\end{figure}

Consider a pair of pinhole cameras viewing a planar surface (such as the
robot's work surface). The viewing transformations can be modelled by
plane projectivities, and there exists a projective transformation
that maps one image to the other. This transformation can be computed
by observing a minimum of four points on the plane. We exploit this to
transform the constraint lines into a common `canonical' view of the
plane \cite{G-Mundy:92}, and hence find their intersection (figure \ref{Cip-cheat_fig2}).

\begin{figure}[bhpt]
\centerline{\epsffile{Cipolla_figs/example_a.eps} \hspace{4mm}
\epsffile{Cipolla_figs/example_b.eps}}
\centerline{(a) \hspace{4cm}(b)}
\caption{Uncalibrated stereo pair of pointing hand and ground plane.}
\label{Cip-cheat_fig}
\end{figure}

\begin{figure}[bhpt]
\centerline{\epsfxsize=5cm \epsffile{Cipolla_figs/example_c.eps}}
\caption{Combining constraints from two views of a pointing hand}
\label{Cip-cheat_fig2}
\end{figure}

The piercing point can then be projected back into the two images; if the four
reference points are known, its world coordinates can be calculated,
otherwise visual feedback can be used to guide a robot manipulator to
this point \cite{Cip-Hollinghurst94}.

\subsection{Tracking Mechanism}

We use a type of active contour 
tracker~\cite{G-kass87:snakes,Cip-Cipolla:92a} to track the 
image of a hand in the familiar `pointing' gesture.
The tracker is based on a template, representing the shape of the
occluding contours of an index finger and thumb 
(figure \ref{Cip-template_fig}).

\begin{figure}[hbtp]
\centerline{\epsfxsize=5cm \epsffile{Cipolla_figs/oldshowoff.ps} ~~ 
\epsfxsize=5cm \epsffile{Cipolla_figs/newshowoff.ps}}
\centerline{(a) \hspace{3in} (b)}

\vspace{-2pt}
\caption{The finger-tracking active contour (a) in its canonical frame,
(b) after an affine transformation in the image plane (to track a
rigid motion of the hand in 3D).}
\label{Cip-template_fig}
\end{figure}

The tracker's motion is restricted to 2D affine
transformations in the image plane, which ensures that it keeps its
shape while tracking the hand in a variety of poses
\cite{Cip-Blake:93}
(this approach is best suited to tracking small planar objects, but
also works well with fingers, which are cylindrical).

\subsection{Implementation}

The system is implemented on a Sun SPARCstation 10 with a Data Cell
S2200 frame grabber. Images are provided by two monochrome CCD
cameras, which view the operator's hand and the working area from
a distance of about 1.6 metres. The angle between the cameras is about
$30^{\sf o}$.
A Scorbot ER-7 robot arm is also controlled by the Sun
(figure \ref{Cip-equip_fig}).

\begin{figure}[hbtp]
\centerline{\epsffile{Cipolla_figs/camset3c.ps}}
\caption{Arrangement of the stereo cameras, table and robot}
\label{Cip-equip_fig}
\end{figure}

The four reference points were defined by observing the robot as it
moved to four known points 50mm above the table-top. Hand-trackers
were initialised by the operator.

Figure \ref{Cip-robot_cheat_fig} shows the system in operation. The white
rectangle represents the images of the reference points in the two
views, and the overlaid square shows the position of the indicated
point on the plane. Movements of the operator's hand cause corresponding
movements of this point in real time.

\begin{figure}[hbtp]
\centerline{\epsfxsize=5cm \epsffile{Cipolla_figs/retry2c.ps} ~~
\epsfxsize=5cm \epsffile{Cipolla_figs/retry1c.ps}}
\centerline{(a) \hspace{3in} (b) \vspace{-2pt}}

\caption{Gestural control of robot position for grasping,
seen in stereo.}
\label{Cip-robot_cheat_fig}
\end{figure}

The robot is instructed to move repeatedly to where the hand is
pointing. One can thus position the gripper with sufficient accuracy
(about 10mm) to pick up small objects in the workspace.

\section{Face Tracking and HCI}

The ability to detect and track a person's face is potentially very
powerful for human-computer interaction. For example, a person's gaze
can be used to indicate something, in much the same manner as pointing
(Section~\ref{Cip-sec:point}). Gaze aside, the head orientation can be
used in virtual holography applications~\cite{G-Azarbayejani:93}, or to
guide a remote, synthesised ``clone'' face for low bandwidth video
conferencing~\cite{Cip-Koch:93,Cip-Li:93}. In addition, a head tracker could
provide a very useful computer interface for physically handicapped
people, some of whom can only communicate using head gestures. We have
developed techniques for inferring the gaze direction of arbitrary
faces in static images~\cite{Cip-Gee:94d}, and also tracking familiar
faces as they move at high speed in image
sequences~\cite{Cip-Gee:96b}.

\subsection{Detecting Facial Features}

The face tracking algorithm has been specially developed to cope with
the poorly defined features and unpredictable motion associated with
faces. In each frame, the tracker tries to detect six features on the
subject's face --- see Figure~\ref{Cip-fig:features}. Dark-pixel finders
(which simply return the location of the darkest pixel within their
local search windows) are used to locate the eyes, nostrils and the
shadow below the lower lip. A coarse correlation-based detector is
used to detect the centre of the eyebrows using a dark-light-dark
correlation template. While this set of features has proved adequate
for our own trials, it is possible that other features, coupled with
suitable feature detectors, could be used to great advantage on other
faces.

\begin{figure}
\centerline{\epsfxsize=2.0in
\epsffile{Cipolla_figs/feat_framed.eps}}
\caption{Features and search windows for face tracking.}
\label{Cip-fig:features}
\end{figure}

\begin{figure}
\centerline{\epsfxsize=3.5in \epsffile{Cipolla_figs/nose_framed.eps}}
\begin{center}
\begin{tabular}{ccc}
\makebox[1in]{Cycle 250} & \makebox[1in]{Cycle 900} &
\makebox[1in]{Cycle 1600} 
\end{tabular}
\begin{tabular}{c}
\epsfxsize=4in
\epsffile{Cipolla_figs/track_err.eps}
\end{tabular}
\end{center}
\caption{Tracking error of a typical feature detector.}
\label{Cip-fig:err}
\end{figure}

\subsection{Robust Pose Estimation}

In theory, the pose of the face can be calculated up to a two-fold
ambiguity using just three of these features, a geometric model of the
face and the ``alignment'' algorithm~\cite{Cip-Huttenlocher:90}. In
practice, the feature detectors are unreliable, and we have to draw on
more than three measurements. Typical non-stationary measurement noise
is illustrated in Figure~\ref{Cip-fig:err}, which shows the tracking error
of one of the nostril detectors. The detector performs well when the
nostril is visible, finding the nostril position to within, typically,
three pixels. When the head is turned so that the nostril is less
prominent, between cycles 500 and 1100, the feature detector starts to
show a bias as it finds instead the shadow around the outline of the
nose. There are also many isolated outliers (gross errors) in the x
and y traces. This sort of behaviour is typical of simple visual
feature detectors, and demonstrates that it is unwise to assume
zero-mean Gaussian noise on the detector outputs. A standard
least-squares fit to the six measurements is therefore inappropriate.

Instead we use RANSAC~\cite{Cip-Fischler:81}, a robust regression scheme,
to accurately estimate the face's pose. All combinations of three
measurements out of the six available are considered in each frame. For
each set of three measurements, the two solutions proposed by the
alignment algorithm are assessed on the basis of consensus: if the
pose is also consistent with some of the other measurements, this
constitutes an extended consensus which lends support to that pose
hypothesis. 

\begin{figure}
\centerline{\epsfxsize=4in \epsffile{Cipolla_figs/ransac_ex.eps}}
\caption{Using consensus to select the best pose.}
\label{Cip-fig:cons}
\end{figure}

The notion of consensus is illustrated in Figure~\ref{Cip-fig:cons}, which
shows a frame from a typical image sequence. A set of facial features
is detected in the image using imperfect feature detectors whose
outputs are displayed as crosses. All measurements are good except at
the left nostril (measurement 5). If measurements 1, 2 and 3 are used
to estimate the pose, the model back-projects into the image as shown
on the left (back-projected model points are displayed as
squares). Five of the measurements are consistent with this pose. If
measurements 1, 3 and 5 are used to estimate the pose, then the
consensus set is only three, as shown on the right. Thus consensus can
be used to select the best pose from the $2 \times ^{n}C_{3}$
candidate solutions, reliably rejecting outlying measurements.


\subsection{Head Gesture Interfaces}

A prime application of the head tracking algorithm lies at the
front-end of a head gesture computer interface. Such interfaces are of
particular value to physically handicapped people, who might have
considerable difficulty using conventional input devices. For example,
the JESTER system~\cite{Cip-Perricos:96}, currently under commercial
development, uses head gestures to replace the mouse in standard
PC-based Windows applications. In its standard configuration, JESTER
senses head pose using a commercially available magnetic transducer,
such as the Polhemus 3Space Isotrak. Unfortunately, magnetic pose
transducers cost many times more than the standard PC equipment they
are connected to. Far more desirable would be a vision-based
transducer, especially now that many computer systems are being
supplied with video cameras and framegrabbers as standard (primarily
for video mail purposes).

\begin{figure}
\centerline{\epsfxsize=4in \epsffile{Cipolla_figs/track_sequence.eps}}
\caption{Some frames from a successful head tracking sequence.}
\label{Cip-fig:frames}
\end{figure}

We have integrated the visual face tracker into the JESTER system, and
found that its output is good enough to enable successful recognition
of a variety of head gestures. Some typical frames from a head gesture
sequence, along with the tracker's orientation estimates (illustrated
drawing pins in the top left hand corner of each frame) can be found
in Figure~\ref{Cip-fig:frames}. Note the tracker's tolerance of mild
occlusion of the face. It remains to be seen whether the tracker
is sufficiently robust to cope with the unpredictable head movements
of severely handicapped people.

\section{Conclusions}

Pointing can be used successfully to specify positions on a
two-dimen\-sional workspace for a robot manipulator. The system
described in Section 2 is simple and intuitive for an operator to use,
and requires practically no training. There is no need for camera
calibration because no 3D calculations are involved.  By tracking at
least four points on the plane it could be made invariant to camera
movement.

Although subjective pointing direction depends upon eye as well as
hand position, it is not necessary to model this phenomenon. Instead,
by providing the operator with feedback about the {\em objective}
pointing direction (e.g.~having a robot follow the pointing hand in
real time), the hand can be aligned with any desired object un the
workspace. Points can then be indicated with sufficient accuracy to
guide simple pick-and-place operations that might otherwise have been
specified using a teach pendant.

A person's gaze can be used to indicate something, in much the same
manner as pointing. The algorithm presented in Section 3 is capable of
estimating the pose of the user's face in realtime.  We hope to
integrate these into the Interactive Robotics Inspection System to
expand the opportunities of individuals with manipulation
disabilities.


\section*{Acknowledgements}

We gratefully acknowledge the financial support of EPSRC, and the
donation of a robot manipulator by the Olivetti Research Laboratory,
Cambridge. This research has benefited from discussions with
Mr.~Masaaki Fukumoto and Dr.~Yasuhito Suenaga of the NTT Human
Interface Laboratories.

%\bibliography{cipolla} 

%\end{document}

