\author[K. Mase]{K. MASE \\ATR Media Integration \& Communications
Research Laboratories}


\chapter{Human Reader: A vision-based man-machine interface}
%\setcounter{page}{3}
%\setcounter{equation}{7}

\begin{Abstract}
This chapter introduces the Human Reader project and some research results of
human-machine interfaces based on image sequence analysis. Real-time responsive
and multimodal gesture interaction, which is not an easily achieved
capability, is investigated. Primary emphasis is placed on real-time
responsive capability for head and hand gestural interaction as applied to
the project's Headreader and Handreader. Their performances 
are demonstrated
in experimental interactive applications, the CG Secretary Agent and the
FingerPointer. Next, we focus on facial expression as a rich
source of nonverbal message media. A preliminary experiment in 
facial expression research using an optical-flow algorithm is
introduced  to show what kind of information can be extracted from facial
gestures. Real-time responsiveness is left to subsequent research work, some
of which is introduced in other chapters of this book.  Lastly,  new
directions in vision-based interface research are discussed based on these
experiences.
\end{abstract}


\Section{Introduction}
Human body movement plays a very important role in our daily
communications. Such communications do not merely include human-to-
human interactions but also human-to-computer (and even to inanimate objects)
interactions. We can easily infer people's intentions from their gestures.
I believe that a computer possessing �eyes�  in addition to a mouse and
keyboard, would be able to interact with humans in a smooth, enhanced and
well-organized way by using visual input information. If a machine can
sense and identify an approaching user, for example, it can load
his/her personal profile and prepare the necessary configuration before
he/she starts to use it. If it senses the direction in which the user is
looking, it could infer the user's attention and intention.  The signals
from a human are essentially multimodal because he/she may use every
expressive physical
function to compose a message in any kind of interaction. Therefore, the
computer sensors and interface should be able to deal with the multimodality
of human communications.  Computers having a �face� and a �body� will
also be a great help in facilitating human-computer interaction in various
situations by providing the sensation of intimacy with the computer.

There have been many reported concepts of novel interfaces
combining speech with nonverbal media such as facial expressions and
gestures. Only a few attempts have been made, however, to actually build
and test such systems. The author and his colleagues proposed a new 
interface concept in which the system uses both visual and audio media
for human-computer interaction to provide multimodality to the interface
\cite{Mas-mase92ae,  Mas-sue-mase-fuku-hohiji93ae}. We named this interface system the
Human Reader (first described in \cite{Mas-mase-sue-aki87a}). It extracts
and interprets human actions from vision and speech sensors, and, can accordingly,
read the various information emitted from users, such as commands,
intentions, emotions, etc., and transfer it to computer application programs.
We have developed several modules, called {\it ``part-readers''}, in the
first step and then integrated them to realize examples of systems with
multimodal interfaces.

In this chapter, we will overview the results of the Human Reader project.
First, we discuss the multimodal message interpretation process and
describe the configuration of the Human Reader. Then we describe the methods
to realize real-time two-part readers,  Headreader and Handreader, and
their real-time implementations as well as their applications in the context
of a CG Secretary and a Finger-pointer. Next, we introduce a non-real-time
part-reader, which we call the Facereader. Finally, new directions in interface research
are discussed.

\Section{Multimodal Interface and Human Reader}
\subsection{Multimodal Interface}
Multimodality does not merely mean using multimedia. Rather, it means
that one or a set of elemental actions of a medium (e.g., part) can be
interpreted differently by the context and/or the situation. For instance, a
hand raising gesture  with the index finger up could mean, (i)
counting, (ii) pointing, (iii) shaping, or (iv) emphasizing. The same
action can mean
different things. As humans are used to communicating in a multimodal
manner, a multimodal interface is expected to provide natural
interaction with the benefits of redundancy, unburdening and
summation\cite{Mas-bolt87}.

\begin{table}
\caption{Human Gesture Media and Messages}
\label{Mas-table:gid}
%\tablewidth=11cm
\begintable
Gesture Media| Messages/ Functions|Example actions\crthick
Locator|Indicate location in space|Pointing,\nr
 | |Head orientation  \cr
Selector|Select from a list| Hand spelling\cr
Valuator|Indicate quantity| Dial manipulation\cr
Imager|Indicate general |Body language, \nr
 |images and concepts| facial expression
\endtable
\end{table}

If we view human actions as message media
it is helpful to categorize the kind of message we are extracting from
the actions according to the ``graphical input devices'' used.
Table~\ref{Mas-table:gid} is a categorized list of human gestural media
for conveying messages \cite{Mas-mase92ae}. These media are {\it selector,
locator, valuator} and {\it imager}. {\it Selector} media include
gestures that select between two or more states. {\it Locator} media
contain pointing actions mainly used to indicate special location. 
{\it Valuator} media contain the gestures that indicate quantity, 
e.g., size of things. The last media, {\it Imager}, do not appear 
in the graphical input device list and include gestures that visualize
shapes, actions or sensitivities such as �round shape� or �difficult
status''. Most body language and facial expressions are categorized as
belonging to an Imager medium, and thus it has strong expressive power;
however, recognition of such sensitive states is quite difficult.

Psychological studies have categorized gestures differently.  Ekman
proposed five media, i.e.,  Emblem, Body manipulator, Illustrator,
Emotional expression,  and Regulator\cite{Mas-ekman77,Mas-vargas87e}.  McNeill
categorized hand gestures with voice in more detail, i.e.,  Iconic,
Metaphoric, Beat,  and Deictic\cite{Mas-cassell94}. These categories cover
most human gestures. However, the graphical input device metaphor makes
it easy to understand the functionality of each category because we are
already very familiar with those physical devices.  Psychological categorization
will become more important as the interfaces evolve and distinctions within
the �imager�  media become necessary in the future.





\subsection{Graceful Interaction and Human Reader}
Humans emit various messages from the different body parts. Figure
\ref{Mas-fig:human-reader} shows a few examples of messages, such as
attention and nodding messages from head. We have designed an interface
system that exploits human actions as media conveying the message of
operation. The process starts with sensing of visual/auditory signals,
continues with extraction of element actions, and finishes with integration
and interpretation. Our approach is building {\it  �part-readers�}, which 
correspond to each body part, and then integrating the part-readers to make 
an interface system.

For example, {\it Headreader} extracts head action, {\it Facereader}
recognizes facial expressions, {\it Lipreader} recognizes spoken words from
lip actions, and {\it Handreader} reads hand-pointing actions.

\begin{figure}
\centering
\centerline{\epsfxsize=9cm \epsffile{Mase_figs/humanmessage.eps}}
\caption{Human Reader$:$ Messages from the human body}
\label{Mas-fig:human-reader}
\end{figure}

When exploiting computer vision technology for a human-machine interface, 
its design should have the following features.
\begin{enumerate}
\item Real-time interaction
\item Evolutional potential
\item Robust processing
\end{enumerate}
The currently proposed algorithms and/or systems have achieved these
requirements in the following ways: 
\begin{enumerate}
\item Real-time by Simplicity: complexity is minimized for real-time
processing because latency is the most want-to-avoid factor in interaction.
Sensory devices should provide a high sampling rate, ideally over 100
pps. However, the more vision algorithms are integrated, the more time is
consumed. 
\item Evolutional by Portability: portable programming is
necessary because the progress of hardware technology is
vary rapid. Portable systems/programs make quick improvements possible.
\item Robust by Constraint: many usage conditions are often
constrained to achieve quick interaction and robustness, e.g. user
group and number, background, illumination and lighting, action,
point of interest, resolution and accuracy, and scenario.
\end{enumerate}

The following sections present examples of implementation. Real-time 
interaction has been realized for Headreader and Handreader. Facereader, 
however, has been realized as off-line processing in the Human 
Reader projects to pursue benefits of its functionality rather 
than the real-time feature. Recent progress in facial expression research
for real-time interaction is covered in another chapter by Alex Pentland.







\Section{Headreader}
Head action is the basis of upper body gestures along with hand
gestures. For example, although facial expression and eye movements
convey much information, their action is always
relative to head movement. Headreader 
\cite{Mas-mase-sue-aki87a} extracts head action by using
the contrast between the head and the face.
% 
% Several works based purely on image analysis have reported head motion 
% detection\cite{Mas-akita84}, head motion recognition for sign language
% understanding, and human motion
% understanding\cite{Mas-orourke-badler80, Mas-leung-yang87}. Akita\cite{Mas-akita84} 
% attempted to analyze human head orientation in two-dimensional (2-D) space. 
% An assumption of head rigidity is often employed, where three-dimensional (3-D)
% head movement is recovered by using several facial feature points,
% e.g. lip extreme, eye lids, etc.



\subsection{Detection of Head Movement}
It is said that a human being has the ability to roughly perceive a person's face direction, even if the person 
is too
distant to know who it is. This indicates that humans can perceive
head orientation without identifying the eyes and mouth. Rather, we can
assume that  the contrast between hair and face and the
appearance of hair style are sufficient for perceiving head orientation.

\begin{figure}
\centerline{\epsfxsize=7cm \epsffile{Mase_figs/headread_rotation_xy.eps}}
(a) rotation around y-axis\\[3mm]
\centerline{\epsfxsize=7cm \epsffile{Mase_figs/headread_rotation_z.eps}}
(b) rotation around z-axis
\caption{Estimating rotational parameters}
\label{Mas-fig:headreader-algorithm}
\end{figure}

\subsubsection{Head Projection Model and Motion Parameters}
Headreader uses a simple elliptical head model that has a face and hair. 
Such a head model projected onto a view plane forms areas of the face and
of the whole head. If we consider the centroids of projected areas (the 1st order
momentum), the relative position changes as the head rotates. This is
illustrated in Figure~\ref{Mas-fig:headreader-algorithm}. The changes are
used to calculate the head motion parameters, mostly 3-D rotational
parameters. Translational parameters are clearly obtained from the
position of the head centroid and the change in head area size.
The rotational and translational parameters, $(R_x, R_y)$ and $(T_x,
T_y, T_z)$,  can be defined as follows: 
\begin{eqnarray} % equations 
 R_x(t) & = & g_{rx}\left((\,\Delta y_h(t) - \Delta
y_f(t)\,)/\sqrt{S_h(t)}\right), \label{Mas-eq:rx}\\
 R_y(t) & = & g_{ry}\left((\,\Delta x_h(t) - \Delta
x_f(t)\,)/\sqrt{S_h(t)}\right), \label{Mas-eq:ry}\\ 
 T_x(t) & = & g_{tx}\left(\Delta x_h(t)/\sqrt{S_h(t)}\right), \label{Mas-eq:tx}\\
 T_y(t) & = & g_{ty}\left(\Delta y_h(t)/\sqrt{S_h(t)}\right), \label{Mas-eq:ty}\\
 T_z(t) & = & g_{tz}\left(\sqrt{\Delta S_h(t) / S_h(0)}\right), \label{Mas-eq:tz}
\end{eqnarray}
where $\left(x_h(t),y_h(t)\right)$ and $\left(x_f(t),y_f(t)\right)$
are the area centroid positions of head and face, respectively, at
time $t$ on the view plane coordinate system.  $\Delta$ shows the
difference in the amount at time $t=0$.  (For example, $\Delta x_h(t)
= x_h(t) - x_h(0) \sqrt{S_h(t) \over S_h(o)} $.)  Similarly, $S_h(t)$
is the size of the head area.
%
$\{g\}$ are mapping functions, which can be represented by a linear
function in the most simplified case, except that $g_{rx}$ is a
combination of two linear functions intersecting at $g(0)=0$.
$g_{tx}, g_{ty}, g_{tz}$ are also linear functions.
Generally, the $\{g\}$s of equations \ref{Mas-eq:rx} and \ref{Mas-eq:ry} are
represented by inverse trigonometric functions.

When the middle position of neck level is provided, we can compute the 
rotational parameter around the z-axis by the following equation:
\begin{eqnarray} % eqs. 7,8
 R_z(t)=tan^{-1}\frac{x_h(t)-x_n(t)}{y_h(t)-y_n(t)} -
	tan^{-1}\frac{x_h(0)-x_n(0)}{y_h(0)-y_n(0)},
\label{Mas-eq:rz}
\end{eqnarray}
where $(x_n(t), y_n(t))$ is the coordinate of the neck mid-point.
This is illustrated in Figure~\ref{Mas-fig:headreader-algorithm}(b). 
In this case, we substitute $\Delta x_h(t)$, $\Delta x_f(t)$, $\Delta
y_h(t)$, $\Delta y_f(t)$ in equations \ref{Mas-eq:rx} and \ref{Mas-eq:ry} for
computing $R_x(t)$ and $R_y(t)$ with the ones in the rotated
coordinate system.
% 
% with the following equations:
%\begin{eqnarray} % eqs. 9,10
%  R_x(t) & = & g_{rx}\left((\,\Delta y_h'(t) - 
% \Delta y_f'(t)\,)/\sqrt{S_h(t)}\right), \label{Mas-eq:rx-prime} \\
%  R_y(t) & = & g_{ry}\left((\,\Delta x_h'(t) - 
% \Delta x_f'(t)\,)/\sqrt{S_h(t)}\right) \label{Mas-eq:ry-prime}. 
%\end{eqnarray}
% These use a new coordinate system rotated by $ R_z(t)$ around a point 
% on the view plane.
% The positions of centroid, $(x_h',y_h')$
% and $(x_f',y_f')$,  in the rotated coordinate system can be used to
% compute $\Delta x_h'(t)$, $\Delta x_f'(t)$, $\Delta y_h'(t)$, and $\Delta
% y_f'(t)$. 
For example, 
$\Delta x_h'(t) = x_h'(t) - x_h(0) \sqrt{S_h(t) \over S_h(0)} $.


\subsection{Vision algorithm and Implementation}
This section discusses the vision algorithm that obtains parameters for
the transformations described in the previous section. The algorithm
is based on binary thresholding and region extraction. The following
constraints are assumed:
\begin{enumerate}
\item camera parameters are constant,
\item background scene is stable and illumination does not change,
\item head image above shoulder is always available,
\item contrast in  brightnesses between hair and skin is high,
\item hair style is similar to that of the head model used.
\end{enumerate}
In addition to the general conditions (i) to (iii) for image sequence
processing, conditions (iv) and (v) are limiting constraints of the user group that were necessary for the 
proposed
method to achieve real-time interaction. As described
below, however, this method is less affected by the 
use of eyeglasses or long hair covering the eyes.
\begin{figure}
\centerline{\epsfxsize=9cm \epsffile{Mase_figs/image_proc_diagram.eps}}
\caption{Image processing diagram of headreader}
\label{Mas-fig:headreader-diagram}
\end{figure}

\subsubsection{Head Area Extraction}
First, subtracting the background image $B(x,y)$ from the input image 
$F_t(x,y)$ produces the silhouette image $s_t(x,y)$:
\begin{eqnarray}
 s_t(x,y) = | f_t(x,y) - B(x,y) |,
\end{eqnarray}
where $t$ is time. 
Second, scanning and thresholding of the silhouette image
extracts head area $H(t)$:
\begin{eqnarray} % eq.12
H(t) \equiv \{(x,y)|s_t(x,y) > T_h\},
\label{Mas-eq:ht}
\end{eqnarray}
where $T_h$ is a previously given threshold.
From  acquisition conditions (i) and (ii), $T_h = 0$ is sufficient.
However, the acquisition condition may vary because of camera variation,
noise, unstable illumination, etc. Accordingly, we use a positive value for $T_h$.
The silhouetting and head extraction proceed simultaneously line-by-line from the top of the image, and 
searching for edge pixels proceeds from
both sides of the
image to find the longest run. 
% 
% Edge pixels are located  by checking
% pairs of pixels separated by $M$ pixels. The edge is defined as the 
% outer-most pixel that  satisfies the head area
% condition. That is, $s_t(x,y)>T_h$ and $s_t(x+M,y)>T_h$.
% By searching for such edge pixels from both sides of the image, we get left
% and right edge pixels of the head area, or a run segment defined by these
%pixels. 
Figure~\ref{Mas-fig:headip} shows an example result of area extraction. 
 
\begin{figure}
\centerline{\epsfxsize=9cm \epsffile{Mase_figs/thresh_head.eps}}
\caption{Result of head and facial area extractin}
\label{Mas-fig:headip}
\end{figure}


%The value of $M$ is usually set to between  2 and 5. 
The head area defined by
the above algorithm, $\hat{H}(t)$, is a set of run segments. It is,
therefore, different from the $H(t)$ defined by equation~\ref{Mas-eq:ht}.
The neck position is defined where the length of run segment is
minimal or zero within the region  far from the head top. 
% 
% The parameters of the head, i.e. area size and centroid coordinates, are:
%\begin{eqnarray}
% S_h(t) = \sum_{(x,y) \in \hat H(t)} 1,\\
% x_h(t) = \frac{1}{S_h(t)}\sum_{(x,y)\in \hat H(t)} x,\\
% y_h(t) = \frac{1}{S_h(t)}\sum_{(x,y)\in \hat H(t)} y.
%\end{eqnarray}


When acquiring a frontal image  as $f_0(x,y)$ at time $t=0$,
the process computes the threshold $(T_f)$ for facial area extraction
and sets the reference parameters $S_h(0),x_h(0)$ and $y_h(0)$.
Automatic threshold selection\cite{Mas-ohtsu79}
is employed to chose $T_f$ from the grey-level histogram of 
$\hat H(0)$.


\subsubsection{Facial Area Extraction}
Facial area extraction is also based on thresholding. The facial area
$F(t)$ is defined by the following.
\begin{eqnarray}
F(t) \equiv \{(x,y) |f_t(x,y) > T_f,\, (x,y)\in \hat H(t)\}.
\end{eqnarray}
This is  true if  the hair is brighter than the face.
Otherwise, the inequality is reversed.
% 
% The parameters of face, area size and centroid coordinates are:
%\begin{eqnarray}
% S_f(t) = \sum_{(x,y) \in F(t)} 1, \\
% x_f(t) = \frac{1}{S_f(t)}\sum_{(x,y)\in F(t)} x, \\
% y_f(t) = \frac{1}{S_f(t)}\sum_{(x,y)\in F(t)} y.
%\end{eqnarray}
% 
% The facial area darkens as the face tilts down under ordinary
% lighting conditions. If we want to know the face-down area only, rather than 
% the precise movement, it is possible to re-extract the facial area
% with a dynamic threshold $T_f$. It may decrease the value if the 
% extracted facial area is anomalously small.


\subsubsection{Implementation}
The feature extraction algorithm and the movement calculation algorithm
were implemented on a workstation (SUN4/260) in C language. Grey
images are acquired with an image input system. The image
size is $128 \times 120$ pixels with 8 bits per pixel. The system
transfers input images to the main memory of the workstation, where the
entire image processing operation is performed. Image processing takes
$0.14\sim 0.16$ seconds per frame. Image
acquisition itself takes about 0.12 seconds of the total processing
time. Thus, the actual image processing is considered to proceed at video
rate.

It would seem at a glance that the accuracy of the processing result is
independent of the processing speed. However, we can exploit the
coherence between successive frames of an image sequence. For instance,
we can reduce the processing time for head area search if we focus on
the area around the area extracted in the previous frame; in this case, the
coherence between the processing frames increases. Thus, we can obtain a good
processing loop-back. This idea is incorporated in the program.

\subsection{Experiment of Motion Detection and Discussion}
% A computer graphics head model was used to simulate head motion and
% to evaluate the image processing and motion detection
% algorithms.  The head model is based on
% measurements of actual humans and has hair generated by the method
% introduced by Watanabe et al.\cite{Mas-hohiji-sue89a}. Thus, the head
% shape is very realistic.
% Figure \ref{Mas-fig:cg-head} shows  examples of the CG head model. 
% The variations in centroid position with the rotation of the model
% were computed and are plotted in Figure \ref{Mas-fig:plot-of-cg}.  The
% graphs show the estimated rotation angle against the actual rotation
% of the model.  Figures \ref{Mas-fig:plot-of-cg}(a) and (b) show the
% rotations around the x-axis and the y-axis, respectively.
% 
% The curve of the rotation around the x-axis is approximately straight, as
% seen in Figure \ref{Mas-fig:plot-of-cg}(a). On the other hand, a straight
% line does not accurately model the curve of the y-axis rotation, as seen in
% Figure~\ref{Mas-fig:plot-of-cg}(b). Since the difference is rather small,
% it is possible to use a linear approximation for motion estimation in
% combination with  thresholding to identify 
% simple head actions such as up, down, left and right.

%\begin{figure}
% \epsfile{file=Mase_figs/calibration.eps,width=8cm}
% \caption{Plots of rotation angle and centroid position movement for CG face}
%\label{Mas-fig:plot-of-cg}
%\end{figure}

\begin{figure}
\centerline{\epsfxsize=9cm \epsffile{Mase_figs/realimage.eps}}
(a) Samples of image sequence and centroids
\centerline{\epsfxsize=5cm \epsffile{Mase_figs/headread_result_graph.eps}}
(b) Estimated rotational parameters
\caption{Estimated rotational parameters from real human images}
\label{Mas-fig:experiment}
\end{figure}
% 
% The slope of $R_x$, Rx (uncalibrated) in
% Figure~\ref{Mas-fig:plot-of-cg}(b), varies with rotation.  This is because
% of the vertically asymmetrical shape of the hair. In other words,
% rotation and centroid position are not independent of one another
% within the coordinate system, as we assumed in equations
% \ref{Mas-eq:rx-prime} and \ref{Mas-eq:ry-prime}.  Thus, we need to add a
% compensating term that reduces the variation.  The new estimation
% function for $R_x(t)$ that replaces equations \ref{Mas-eq:rx} and
% \ref{Mas-eq:rx-prime} is
% \begin{eqnarray} %eq.20
%  R_x(t) = g_{rx}\left((\Delta y_h'(t) -
% \Delta y_f'(t)\,)/\sqrt{S_h(t)}\right) + c_{ry}\, R_y(t), \label{Mas-eq:rxm}
% \end{eqnarray}
% where $c_{ry}$ is the compensating function which depends on hair
% style. The figure also shows the graph of $R_x(t)$ calibrated by a
% constant  $c_{ry}$ value of  0.4. In order to increase  estimation accuracy,
% a compensating table should be constructed by using the un-calibrated
% $R_x(t)$. Finally, the compensated rotation angles should be calculated in the
% order  $R_z, R_y,$ and $R_x$.


Figure \ref{Mas-fig:experiment} is the result of motion extraction from
real images, (a) illustrates the input images and
extracted centroid positions, and (b) plots variations of estimated
rotation angles. The extraction results are quite good. 
Since this experiment did not
measure the original rotation  angles, the estimation accuracy is unknown.
However, the graph shows that head movement was well tracked.
Equation \ref{Mas-eq:rxm} is used in  motion calculation to ensure hair style 
compensation, which is obtained through a simulation with a CG head
model motions.
\begin{eqnarray} %eq.20
 R_x(t) = g_{rx}\left((\Delta y_h'(t) -
\Delta y_f'(t)\,)/\sqrt{S_h(t)}\right) + c_{ry}\, R_y(t), \label{Mas-eq:rxm}
\end{eqnarray}
where $c_{ry}$ is the compensating function which depends on hair
style. 
% 
% The figure also shows the graph of $R_x(t)$ calibrated by a
% constant  $c_{ry}$ value of  0.4. In order to increase  estimation accuracy,
% a compensating table should be constructed by using the un-calibrated
% $R_x(t)$. Finally, the compensated rotation angles should be calculated in the
% order  $R_z, R_y,$ and $R_x$.

\subsubsection{Real-time Analysis and Synthesis}
The Headreader was connected to a graphics
workstation (GWS) to complete the experiment on the analysis and
synthesis of head motion.
% Figure \ref{Mas-fig:real-time-system} gives an appearance of the
% experimental system. 
The system\cite{Mas-mase-hohiji-sue90a} takes face images with the TV
camera of a low-cost picture phone and sends extracted motion
information to the GWS, which duplicates the original motion with a CG
head model.  The 3-D head model consists of about 600 trigonal patches
and is displayed on the GWS in real-time.
The system is a prototype of a future intelligent picture coding
system, e.g.,  a model-based image coding scheme.
% 
% The data is transmitted over the network in a 3-D transformation matrix.
% When native transformation parameters are transmitted in terms of
% character strings, the baud rate is about 2,500 bps. Data of facial
% expressions should be added in a practical picture coding system.

\subsection{Interpretation of Head Action and Applications}
An application of the Headreader to a human-machine interface is
investigated in this section. 

\subsubsection{Command Interpretation}
Head actions as nonverbal message media convey the location message by gazing
and the selection message by nodding and shaking for �yes/no.�

First, the head rotation angles are quantified to identify the gaze
direction. For example, if we need a  $2$ by $2$ selection window on
screen, trinary quantification of angle $\theta$ is performed with 
thresholding parameters $\{th\}$s:
$$\theta > th^+,\, th^+ > \theta > th^-,\, th^- > \theta.$$

Next, changes in rotational angles around $x$- and $y$-axes are
interpreted for the �yes/no� messages.  For example, a DP matching
scheme can be used. In this chapter, however, we introduce a method that uses
local temporal statistics of rotational parameters to achieve 
real-time responses. The statistics used are the means ${m}$ and
variances ${var}$ of
angles and their temporal derivatives through the past several frames.
The following equations defines the mean values, $m_{rx}$ and
$m_{\Delta rx}$, and the variances, $var_{rx}$  and $var_{\Delta rx}$. 
\begin{eqnarray}
m_{rx}(t) & = & \frac{1}{t_c}\sum_{T=t-t_c}^t R_x(T),\\
var_{rx}(t) & = & \frac{1}{t_c}\sum_{T=t-t_c}^t (m_x(t) - R_x(T))^2,\\
m_{\Delta rx}(t) & = & \frac{1}{t_c}\sum_{T=t-t_c}^t (R_x(T)-R_x(T-1)),\\
var_{\Delta rx}(t) & = & 
   \frac{1}{t_c}\sum_{T=t-t_c}^t (m_{\Delta rx}(t) - R_x(T)+ R_x(T-1))^2,
\end{eqnarray}
where $t_c$ is the number of frames, which is set to 6 in the 
following experimental system to cover an approximately 1.0 second
interval. 

The following logical equations finally give the answer of  �Yes� and
``No''. It checks the values of means and variances obtained by the
equations above with pre-defined thresholding parameters, $th$s.
\begin{eqnarray*}
if(var_{rx} > th_1 \wedge var_{\Delta rx} > th_2 \wedge m_{rx}
< th_{down} \wedge m_{rx} > th_{up}) & \Rightarrow & \mbox{Yes}\\
if(var_{ry} > th_3 \wedge var_{\Delta ry} > th_4 \wedge m_{ry}
< th_{right} \wedge m_{ry} > th_{left}) & \Rightarrow & \mbox{No}
\end{eqnarray*}

\subsubsection{Headreader Interface of CG Secretary}
The nodding and shaking head actions are used to express
�yes/no�' messages even by children and speech-impaired persons. Head shaking is the most basic action of 
interaction.
A computer graphics secretary agent (CG Secretary) interacts with the user
in front of a screen equipped with a TV camera.  
The tasks of the CG Secretary are:
\begin{enumerate}
\item to start a session when it detects a user within the view range,
\item  to check a video-mailbox and notify the user of its contents,
\item to ask the user his/her work menu choice and provide the program/workplace,
\item to watch for interruptions by the user,
\item to close the session when the user leaves the view range.
\end{enumerate}

\begin{figure}
\centerline{\epsfxsize=4.5cm \epsffile{Mase_figs/cgsecretary.eps}}
\caption{Interface of electronic CG Secretary $:$ Recognition of  �Yes/No� and
menu selection from head actions  (notification of a video-mail message)}
\label{Mas-fig:cg-secretary}
\end{figure}

\begin{figure}
\centering
\centerline{\epsfxsize=10cm \epsffile{Mase_figs/headreader_seq.eps}}
\caption{Snapshot of head actions with CG Secretary}
\label{Mas-fig:cg-secretary-seq}
\end{figure}

In order to make the interaction possible based on this scenario, the
Headreader was used as an eye of the CG Secretary in the following
way:
\begin{enumerate}
\item user's presence: checking head area size,
\item user's choice of menu: checking head direction
as representing gaze,
\item user's command: interpreting ``yes/no'' actions.
\end{enumerate}

Figure~\ref{Mas-fig:cg-secretary} shows the setup of the CG Secretary
application system, and Figure~\ref{Mas-fig:cg-secretary-seq} shows the
selected scenes of user interaction with the CG Secretary:  head shaking
to stop the video program on screen and a menu selection. The
video-mail and video programs are stored in a remote-controlled
digital VTR. The previously recorded voice-messages of the CG Secretary
are stored as sound files in the voice generator workstation. These
are played by the control commands issued by the message integration
and application server. Figure~\ref{Mas-fig:cg-secretary-system} is an
integrated illustration of the CG Secretary and FingerPointer
prototypes. Note that the CG Secretary system did not employ the Voice Listener
and the Handreader.


\begin{figure}
\centerline{\epsfxsize=10cm \epsffile{Mase_figs/cgsecretarysys.eps}}
\caption{System configuration of CG Secretary and
FingerPointer prototypes}
\label{Mas-fig:cg-secretary-system}
\end{figure}


\Section{Handreader}
Human hands are truly a multimodal message conveying device. They are
used for such purposes as indication, counting, shaping, manipulation, gesturing, etc. 
Hand-gesture-based command interfaces have been proposed in many virtual
reality systems and in computer aided design tools. In these systems,
however, the user must wear special physical devices such as gloves and 
magnetic devices. 

Handreader\cite{Mas-fuku-mase-sue94} involves computer image processing of 
hand and finger images. This system allows the user to input commands based on
recognition of finger pointing and several hand gestures in a meeting
space or living room. The user's pointing actions are captured by a
pair of stereoscopic TV cameras: one mounted on the ceiling and
the other on the side wall. 

As a prototype of a gesture interface, Handreader is integrated with
a Voice Listener and called
``Finger-Pointer'' (see Figure~\ref{Mas-fig:cg-secretary-system}).  Gestures used in this system are 
classified as 
``locator'', ``selector'', and to some extent ``valuator'', according to the
categorization given in  Table~\ref{Mas-table:gid}.
Voice commands complement the gestural commands, and thus a multimodal
interface is realized. This section first introduces the vision
algorithms used for extracting gesture messages and then describes a novel 
method to determine pointing direction called ``Virtual Projection
Origin (VPO)''. Next, the method of integrating voice and gesture media 
and examples of integrated commands are presented.  

\begin{figure}
\centerline{\epsfysize=4cm \epsffile{Mase_figs/figure3.eps}}
(a)  determining finger tip location
\centerline{\epsfysize=4.8cm \epsffile{Mase_figs/figure4.eps}}
(b)  detection of  thumb up/down
\centerline{\epsfysize=5.2cm \epsffile{Mase_figs/figure5.eps}}
(c) detection of number of fingers
\caption{Recognition of hand gestures}
\label{Mas-fig:handgesture}
\end{figure}

\subsection{Hand Extraction with Stereo Vision}
As shown in Figure~\ref{Mas-fig:cg-secretary-system}, Handreader uses
two cameras to obtain the three-dimensional position of the hand and fingers. The
camera on the wall is used to capture hand gestures as well as the
position of the finger tip. The other camera on the ceiling captures the 
horizontal finger tip position. The Infra-red(IR) LED array light sources
are installed by the cameras in order to make the hand area extraction
algorithm simple and robust against change in illumination. The light 
is unnoticiable to the users, but a video camera with an IR filter can
detect hand position very easily.

Figure~\ref{Mas-fig:handgesture} (a) illustrates the method of determining
finger tip location. First, it binarizes the two images captured by the cameras 
with a fixed threshold and extracts hand regions. Then, each image is 
scanned to find the tip that is closest to the screen. The candidate tip pixel is verified to determine 3-D 
position, based on the
length and thickness of the index fingers.
Figure~\ref{Mas-fig:handgesture} (b) illustrates the method of
determining thumb up/down action. The fan shaped scan area is constructed
based on the finger tip and the center of  wrist and scanned to search 
for the thumb tip pixel. This action is used to send a switcher
message. Similarly, hand indication of numbers can be recognized by
counting the number of outstretched fingers, as illustrated in
Figure~\ref{Mas-fig:handgesture} (c) .

\begin{table}
\caption{Command examples of combined hand gestures and voice}
\label{Mas-table:commands}
\tablewidth=11cm
\parasize=4cm
\begintable
Hand gesture   | 	Voice example 	      \| Interface command \crthick
\para{Locator (indication)}
	       | ``This'', ``From'', ``To''   \| Point \cr
Selector (thumb up/down)	
	       |	none		      \| Menu, Button\cr
Valuator (finger number) 
	       |	``Forward this many'' \| Dial 
\endtable
\end{table}

%\subsubsection{Virtual Projection Origin}
How do we draw a virtual line between the finger and target 
in the distance? Often, we use a base reference point on or near the body and
draw a line through the finger tip and the base reference point. The location of
the base reference point, or 
the Virtual Projection Origin (VPO), depends on the person, pointing style,
etc. However, since we can assume it is stable in a distinct session,
a pre-session calibration to define VPO is sufficient. Four
corners of a target screen are used for such a calibration in which the
user is instructed to point to each indicated corner for the system to find the VPO. An experimental 
analysis with 20 subjects showed that VPO is mostly
distributed along the  line of the eye on the pointing hand's side and
the shoulder.  

\subsection{Finger-Pointer : multimedia and multimodal interface}
Handreader provides multimodal information, i.e., valuator,
locator, switcher. Voice information 
will give a clue for distinguishing the hand gestural commands. 
The Finger-Pointer provides integration of voice and pointing gestures 
by using a speech recognition unit. Table~\ref{Mas-table:commands}
lists examples of combinations of gesture and voice.   

The Finger-Pointer realizes inter-channel synchronization between
voice and image to avoid mismatch between the messages provided from
the Handreader and the Voice-Listener. It introduces �time-tags�
that use a common master clock. Each part-reader (here, one
of the readers processes voice) applies a time-tag when it recognizes a
message. The message integration server will use the time-tags when
it integrates the output of the part-readers.

\begin{figure}
\centerline{\epsfysize=5cm \epsffile{Mase_figs/fpointer.eps}}
\caption{Finger-Pointer: interface for presentation system using hand gestures and voice}
\label{Mas-fig:finger-pointer}
\end{figure}


\subsection{Application of Finger-Pointer}
Three experimental application systems use the Finger-Pointer
interface. 

First, an electronic presentation system was constructed.
The user can control page forwarding/backwarding, select commands in 
the listed menu, mark with a bullet on a slide, draw an
underline and so on. In page controlling, a skip-page number can be
specified by the finger numbering
gesture. Figure~\ref{Mas-fig:finger-pointer} is a snapshot of the system being used. 

\begin{figure}
\centerline{\epsfxsize=8cm \epsffile{Mase_figs/fpointer3.eps}}
\caption{Finger-Pointer: interface to VCR remote and Space Sketchpad}
\label{Mas-fig:fpointer_video}
\end{figure}


Next, a video browsing system was constructed. The system allows the user
to use hand gestures to remote control a VCR; for example: Play, Stop
and special search operations similar to those offered by a
``Shuttle-Ring'' of real VCRs. 
Figure~\ref{Mas-fig:fpointer_video} (a) is a snapshot of controlling the
VCR.

Third, a user may want to write a character or draw simple graphics in
space. A Space Sketchpad is the third application that can be realized
with the Finger-Pointer. 
Figure~\ref{Mas-fig:fpointer_video} (b) is a snapshot of the system being used. 


\Section{Facereader}
Facial expression is a major medium
for non-verbal communication.  It complements verbal
communication and can  convey sufficient information by itself.
 Thus, the ability of
understanding expressions would make communication smoother and 
more accurate.  Advanced research into man-machine interfaces must tackle
this issue. For instance, future computers and household appliances
should automatically respond to annoyed users by offering
helpful information.  This section presents a method that uses optical
flow to estimate facial muscle movements \cite{Mas-mase91c} and then
recognizes these movements as various facial
expressions.

The human face has several distinctive features such as eyes, mouth
and nose.  We need to extract their movements and shape-deformations
in order to recognize facial expressions.  Many techniques have been developed for
extracting facial features and finding fiducial points using image
processing and computer vision for automatic facial recognition
\cite{Mas-sakai-nagao-kanade72}. 
We use optical flow to extract the muscle movement. 
% 
% However, extraction accuracy is still insufficient 
% because of the deformation and low contrast of feature boundaries.
% These movements are the result of various combinations of contractions
% and/or relaxations of the facial muscles.  
It seems more straightforward to extract muscle movement from skin
displacement than to estimate each muscle action from feature point
movement.  Optical flow information can be extracted easily, even at
low contrast edges. Furthermore, the facial skin at the cheek and
forehead has the texture of a fine-grained organ, which is a key
factor for optical flow estimation.
% 
% This section first briefly addresses the research on facial expressions
% as reported in various fields. A conventional optical flow algorithm
% is then applied to image sequences of facial expressions. 

We investigated the recognition of facial expressions in two ways: the
top-down and bottom-up approaches. The top-down method is applied to
estimate facial muscle movement, and we found that the obtained
optical flow information was sufficient to extract information of
facial muscle movements as a description of facial expressions.
Furthermore, we constructed a bottom-up expression recognition system
to categorize emotional expressions.  
% 
% The results of a preliminary
% experiment show an accuracy rate of nearly 80\% in recognizing four
% types of expressions: happiness, anger, disgust and surprise. The
% accuracy rate increased to 86\% when arguably vague expressions were
% excluded from the evaluation data.
\subsection{Facial Expression and Optical Flow}
% Facial expression has been a research subject in the field of
% psychology since Darwin's work on emotional expression in the 19th
% century\cite{Mas-ekman73}.  Ekman and Friesen\cite{Mas-ekman-friesen78}
% recently developed a system named FACS (Facial Action Coding System) to
% systematically analyze facial expressions in the field of
% psychoanalysis. Their system made it possible for computer graphics
% researchers to treat facial expressions as an engineering subject. 
% FACS was soon used in generating facial
% animation\cite{Mas-platt-badler81,Mas-waters87,Mas-choi-hara-takebe90e}.  
% Model-based picture 
% coding architecture, which is based on analysis and synthesis of the human
% figure, face and facial expressions\cite{Mas-hara-aizawa-saito89a}, is  now of
% interest to the computer graphics and computer vision communities
%  because it provides many unsolved and practical problems.  
% 
% A few attempts to describe facial expressions from feature
% deformations have been reported\cite{Mas-pilowsky86}.
% ``Action units'' (AUs: elements of expression in FACS) are sometimes
% used to describe such deformations.  However, an automatic AU extraction
% method has not yet been established because of the difficulties of
% precise facial feature shape extraction by image processing. Our
% optical flow approach can extract muscle movements, which can be
% interpreted as appropriate action units in combination with feature
% positioning.  Moreover, as Bruce noted\cite{Mas-bruce}, ``there has been
% rather little exploration of the question of whether ``action units''
% are the units whereby we categorize expressions.'' 
% AUs and other descriptions\cite{Mas-mase90e} are the basis of categorizing 
% expressions by the use of knowledge about facial anatomy and expressions.
% This can be called a top-down approach. 
% We leave the final step of categorization of expression from these 
% descriptions of muscle movements to future collaboration  with
% psychologists.
% However, it is still a very interesting issue to evaluate how well the
% computer can  
% categorize facial expressions.
% Thus, we investigated 
% a different way of describing facial expressions. 
% The technique is also derived from optical flow and leads us to
% the direct 
% categorization of facial expressions. Because we use a classical pattern
% recognition approach in this technique, it can be called a bottom-up
% approach. Consequently, we present two ways of analyzing facial expressions
% from optical flow.
% 
% Choi~et~al.\cite{Mas-choi-hara-takebe90e} used the geometric parameters of
% facial features, such as lip height and angle of eyebrow, and assigned
% changes to appropriate AUs. However, it is very difficult to extract such
% facial features by image processing techniques, so they were extracted 
% interactively.  Kass~et~al.\cite{Mas-kass-witkin-terzopoulos87a} used
% snakes to determine the deformation of facial features such as lips.
% Terzopoulos and Waters used that information to duplicate facial
% expressions in computer animation\cite{Mas-terzopoulos-waters90}. However, 
% it is  difficult to apply snakes to the facial parts having less
% explicit boundaries, such as the lower lip and nose wings.

Considering that expressions are formed by the contraction of facial
muscles, it is reasonable to assume that deformation of the skin
provides valuable information on muscle action.  That is, muscle
contractions are triggered by the nerve impulses generated by
emotions, and the skin is deformed by the contraction of those muscles
under the skin. Because most muscles associated with expressions are
attached to some facial feature, e.g.,  lips and eyes, the features are
deformed. Simultaneously, the deformation can be seen on the cheeks and
forehead as dimples or wrinkles. Even the conventional gradient based
optical flow algorithm introduced by Horn and Schunck can extract skin
motion from the subtle texture of these regions. The advantage of the
optical flow algorithm is that there is no need to extract and trace
particular points in the image sequence.  Mase and
Pentland  \cite{Mas-mase-pentland90a} exploited this concept
to extract lip movements and for application to computer lip reading.

%\subsubsection{Facial Muscles and Optical Flow}
%There are many optical flow estimation algorithms.  They can be
%divided into three general groups: gradient-based, correlation-based,
%and filtering-based. The conventional gradient-based algorithm
%proposed by Horn and Schunck\cite{Mas-horn-schunck81} is simple and works
%well for facial skin deformation, except for the aperture problem and
%motion discontinuity. The correlation-based method is also simple, but
%its computational cost is large and it prefers highly textured regions
%for flow extraction.  Although the filtering based method has the
%desirable property of computational stability for flow
%estimation\cite{Mas-heeger88}, many frames are necessary to compute a flow
%field. This is unacceptable when analyzing facial expressions because
%their duration is too short to use the assumption of smooth
%translation.  Even though facial skin is a non-rigid deformable
%object, we can consider its motion field to be locally smooth.  Thus
%Horn and Schunck's gradient-based optical flow
%algorithm\cite{Mas-horn-schunck81} was adopted.  The original ``two-frame-based'' algorithm was used 
in our experiments despite the many
%improvements that have been proposed to overcome its drawbacks. This is because the errors in flow 
estimation do not seem to significantly affect the
%final recognition result.



\subsection{Extraction of Muscle Movement}
\subsubsection{Optical Flow Muscle Model}
% The muscles of expression interact in a very complex way under the
% facial skin. The primary muscles involved in expressions are  Orbicularis
% Oris around the lips, Venter Frontalis in the forehead, and Buccinator,
% Zygomaticus and Anguli Oris connected to the mouth.  Figure
% \ref{Mas-fig:muscle}\cite{Mas-mori50e} illustrates 
% the locations of these muscles.  Since several muscles
% contribute locally to one expression, it is of little use to measure all
% muscle movements.  Mase and Pentland\cite{Mas-mase-pentland90a} analyzed
% optical flow patterns and extracted two principal motions around the lips
% for speech articulation.
% \begin{figure}
% \centering
% \epsfile{file=Mase_figs/muscles.eps,width=8cm}
% \caption{Schematic of facial muscles}
% \label{Mas-fig:muscle}
% \end{figure}



First, in order to see how well optical flow information can capture muscle
movement,  the amount of muscle contraction was computed from optical
flow data.  We used windows that define the major
directions of muscle contraction (skin deformation). 
This allows the effect of each muscle to be determined.
The average length of directional components in the major
window axis is calculated from the optical flow vectors in each window.
\begin{equation}
m_{i,t} = \frac{1}{S_i} \int_{window_i} {\bf u_t( x)} \cdot {\bf n}_i \,\,  d{\bf x},
\end{equation}
where ${\bf u_t(x)} \equiv (u_t(x,y), v_t(x,y))$ is the optical flow field
computed from two successive frames at times $t$ and $t+1$.  $S_i$ and ${\bf n}_i$ are the area
size and the normal vector of the $i$-th window $(i=0,...,11)$,
respectively.  
Figure \ref{Mas-fig:muscle-motion} (lower-right) uses arrows to indicate
the direction of muscle contraction, 
which is equivalent to $-{\bf n}_i$. Windows are manually located
using feature points as references. For example, window 0 is
placed on
the line connecting the lip's left extremity and the left eye's  left extremity.
$m_i$ is the estimated muscle 
movement in the $i-$th window and is a function of time (frames). 
Although the movement is a velocity, it is also regarded as an
averaged displacement in a unit period.  


\begin{figure}
\centerline{\epsfxsize=10cm \epsffile{Mase_figs/happy.eps}}
\caption{Windows for measurement and estimated muscle motions}
\label{Mas-fig:muscle-motion}
\end{figure}

Figure \ref{Mas-fig:muscle-motion} shows a recorded image
and estimated muscle velocities during one full cycle of a ``happy''
expression of a woman. The white and black arrows are computed optical flow
and estimated velocities, respectively. The optical flow field is
computed for each pixel and is shown for every three pixels. Image size
is 256 by 256 pixels.  The estimated motions are functions of time. 
From this preliminary experiment, it can be stated that the information of
optical flow data  sufficiently  reflects the muscle movement invoked by
expressions. 

%\begin{figure}
%\centering
%\vspace{6cm}
%\epsfile{file=Mase_figs/happy_plot1.eps,width=8cm}
%\caption{Plots of estimated motion (happiness, a woman)}
%\label{Mas-fig:plot-1}
%\end{figure}

% 
% In the experiment, the muscle windows were manually located .  Since
% they were placed with reference to the major feature points, automatic
% window positioning would be possible by adopting one of the techniques
% created for face recognition systems.
% In this case, highly accurate feature extraction is not
% necessary because the perturbation of window positioning caused by
% the extraction process is absorbed by averaging the  optical flow over
% the window.

\subsubsection{FACS-based Description of Expression from Muscle Model}
% 
% Describing expressions with codes is a step in expression
% recognition. Muscle-model descriptions based on optical flow
% data are already known, as shown in the previous section. On the other
% hand, FACS-based description has been widely used by computer vision
% and graphics researchers as well as the psychology community. Accordingly, it
% is of some value in correlating our muscle-model-based description to
% FACS's Action Units, although this is not an easy process. 

In this
section, we introduce an association table between our muscle-model
and a subset of AU (Action Unit), and show an example of describing expression with
AUs.

The major AUs for six fundamental expressions (happiness, anger,
surprise, disgust, sadness and fear) are first extracted from the
literature\cite{Mas-choi-hara-takebe90e}. They are AUs 1, 2, 4, 5, 6, 7,
9, 10, 12, 15, 17, 20 , 25 and 26.  An association was then formed
between our muscle model and the extracted AUs by referring to the FACS
manual and to literature in anatomy. Because AU descriptions are mostly
static and only partly dynamic, our dynamic muscle model cannot be
directly associated with a particular AU.  The relationship between action
and its effect on shape must be considered to develop intuitively.
Moreover, our muscle model does not cover the whole facial area, thus
some AUs are not included.

The muscle action derived from our muscle model can be associated with
several action units as listed in Table \ref{Mas-table:facs}.  
Since 
lip-part (AU25) and jaw-drop (AU26) are not distinguishable  from
$m_8$ (lower part of mouth) of the muscle model,
we score AU26 for positive $m_8$. $m_9$ seems to be related mainly to
the Orbicularis oris and partly to other muscles, and is often
observed when the  mouth opens. Therefore, AU25 (lip part) was
associated with  negative
$m_9$. It may be better to assign AU25 only when negative $m_9$ and
positive $m_8$ are observed, but we only consider $one$ to $N$ associations of
muscle action and AUs in this discussion for simplicity.
%AUs 2, 5, 7 and 20 are not covered because windows are not placed 
%around the eyes, lip sides and outer brow in the current implementation.

\begin{table}
\caption{Relationship between Muscle Windows and Action Units}
(AU$n$$^-$ occurs if $m_i < 0$, and AU$n$$^+$ occurs if $m_i > 0$.)
\vspace{3mm}
\label{Mas-table:facs}
\begintable
Window\|Muscles|Actions and Results	| Closest AUs\crthick
$m_0$(left)\|Zygomaticus mj|pull up lip extreme  	|AU6$^-$(cheek raiser)\crnorule
$m_4$(right)\|Orbicularis oculi|raise cheek 		|	\cr
$m_1$(left)\|Zygomaticus mj |pull lip |AU12$^-$(lip corner puller)\crnorule
$m_5$(right)\|Buccinator|	corners backward	|	\cr
$m_2$(left)\|Levator labii|	raise upper lip	   	| AU10$^-$(upper lip raiser)\crnorule
$m_6$(right)\|		|				|	\cr
$m_3$(left)\|Depressor  |depress lip corners	|AU15$^-$\crnorule
$m_7$(right)\|Angli oris|				|(lip corner depressor)	\cr
$m_8$	\|Mentalis  	|raise chin			|AU17$^-$(chin raiser)\crnorule
	\|Jaw action	|drop jaw and 	 	|AU26$^+$(jaw drops),\crnorule
	\|Dipressor labii	|part lips 		|[AU25$^+$(lips part)]\cr
$m_9$	\|Orbicularis oris|relax upper lip or		|AU25$^-$(lips part)\crnorule
	\|(relaxed)	|part lips			| 	\cr
$m_{10}$\|Depressor sup|make wrinkle at |AU9$^-$(nose wrinkler)\crnorule
	\|Levator labii|root of nose			|	\cr
$m_{11}$\|Ventor frontalis|raise brow or		|AU1$^-$(inner brow raiser)\crnorule
	\|Depressor sup|lower brow		|AU4$^+$(brow lowerer)
\endtable
\end{table}

Thus, for example, estimated muscle
motions can be scored, for example, as AUs
6(0.6)+12(0.35)+17(0.28)+25(0.8)+26(0.20)
if we consider  only
strong actions\cite{Mas-mase91c}.  The figures in the parenthesis, e.g. 0.6, indicate
the AU's strength, which is equated to  the maximum absolute
velocity between the frames containing neutral and maximum positions
of expression. The scored AU combination is fairly close to the
combinations of AUs 6+12+26 or AUs 6+12+17, which often 
appear with the happiness expression as shown  in the FACS's AU
photographs.
Because the example score was computed from an image sequence, it is
natural to include several ``happy'' expressions. Unfortunately, 
AU25(0.8) cannot be clearly interpreted in such expressions.


%\subsubsection{Recognition of Expression with  Optical-flow Muscle Model}
%There are a few ways of recognizing expressions from the descriptions
%introduced previously. For instance, it is possible to develop feature
%vectors from those descriptions and then to feed them into orthodox
%discriminant functions of a categorization system.  The possible
%feature vectors are as follows. (I) If the expressions are
%simple (one expression), we can develop a vector consisting of muscle
%velocities at the time when one arbitrary muscle reaches its
%maximum velocity. (II) Since the data consists of velocity, it becomes
%zero at the maximum extent of the expression. The integral of the
%velocity is roughly regarded as the displacement. The integrals
%of the velocity until it reaches zero are used as
%feature pattern.  (III) The waveform pattern of each velocity may be
%used directly as the feature pattern. (IV) We relate each muscle movement
%to a corresponding AU, which becomes an element of the feature
%vector\cite{Mas-choi-hara-takebe89e}.

%However, we did not actually construct a top-down recognition system
%by using these descriptions. These descriptions are more suitable to
%build a higher-level pattern recognition system that positively
%exploits knowledge of anatomy and psychology than to building an orthodox pattern recognition
%system. This would require collaboration with experts in these fields
%because even FACS does not explicitly specify relations
%between AUs and emotions, which is the goal output of our recognition system.  Moreover, the 
estimated muscle movement is
%constrained by the window position, size and direction so that these
%movements may fail to catch useful information for recognition. Again,
%we need the help of experts to optimize the window specifications.

%The next section presents a statistical (bottom-up) approach to
%expression recognition in which evenly divided rectangular regions
%cover the entire face as muscle windows and the most active regions
%are selected from the sample data to develop the feature vector. The
%feature vectors of test data are categorized based on the
%k-nearest-neighbor rule.  Because this approach is free from any a
%priori knowledge of generating expressions except for skin
%deformation, it is possible to construct a recognition system without
%reference to expression experts.  With this approach, the larger the
%set of well-defined sample data, the more robust the feature vector
%becomes, which enhances overall recognition performance.

\subsection{Recognition of Expression by k-Nearest-Neighbor Rule}
The previous
section presented a top-down method that  describes facial expressions
in terms of facial muscle movements, which could later be related to
categorized expressions such as �happy� and �anger.�
This section presents, an expression recognition system based on the
classical bottom-up pattern 
recognition approach.

The expression recognition system presented here directly categorizes 
facial expressions into emotional categories by deriving a 
feature vector from sample image sequences.

\subsubsection{Feature Vector Selection}
A non-parametric pattern recognition method, the k-nearest-neighbor
rule\cite{G-Duda73}, is used to classify the image sequences of
various facial expressions. As is well known, the nearest-neighbor rule
is a sub-optimal procedure for classification.

First, we define a $K$-dimensional feature vector whose elements are
derived from optical flow data of sample image sequences. 
% 
% As shown in
% the previous section, the time sequence of flow data (velocity) is
% known at each point. To overcome the undesirable behavior of flow
% calculation at edges such as occluding boundaries, the local average
% is taken. The treatment of time sequenced data is also difficult, so
The time information is discarded by treating the flow data at each
frame as independent. Consequently, the first and second degrees
of moment (means and covariances) of flow data are calculated in each
spatially and temporally local region. Suppose the image sequence is
$N$(column) by $M$(row) pixels with a period of $T$(frames) for the
$k$-th expression sample. The image is evenly divided into $n$ by $m$
regions by a ratio of $r= N/n = M/m$. The resulting basic feature data
is the set of
\begin{eqnarray}
\mu_{u,i,j} &=& \frac{1}{T} \frac{1}{r^2} 
	\sum_{0 < t \leq T} \sum_{(x,y)\in R(i,j)} u_t(x,y), \label{Mas-eq:muu}\\
\mu_{v,i,j} &=& \frac{1}{T} \frac{1}{r^2} 
	\sum_{0 < t \leq T} \sum_{(x,y)\in R(i,j)} v_t(x,y), \\
\sigma_{uu,i,j} &=& \frac{1}{T} \frac{1}{r^2} 
	\sum_{0 < t \leq T} \sum_{ R(i,j)} (u_t(x,y) - \mu_{u,i,j})^2, \\
\sigma_{uv,i,j} &=& \frac{1}{T} \frac{1}{r^2} 
	\sum_{0 < t \leq T} \sum_{ R(i,j)} 
	(u_t(x,y) - \mu_{u,i,j}) (v_t(x,y) - \mu_{v,i,j}), \\
\sigma_{vv,i,j} &=& \frac{1}{T} \frac{1}{r^2} 
	\sum_{0 < t \leq T} \sum_{ R(i,j)} (v_t(x,y) - \mu_{v,i,j})^2,
\end{eqnarray}
where $\mu_{u,i,j}$ and $\mu_{v,i,j}$ are the means of optical flow of
horizontal and vertical components, respectively, at the $(i,j)$-th
divided region $R(i,j)$, and $\sigma_{uu,i,j}$, $\sigma_{uv,i,j}$ and
$\sigma_{vv,i,j}$ are the covariances of optical flow. This results in
a $K(=5mn)$-dimensional vector whose dimensionality should be reduced.
As a convention, we denote the vector by
\begin{eqnarray}
{\bf F} = \{f_1, f_2, ...,f_k,...,f_K\}.
\end{eqnarray}

To reduce the dimensionality of the vector, we introduce the following criterion
function of separation goodness for the $k$-th element of ${\bf F}$.
\begin{eqnarray}
J(k) = \frac{var_B(k)}{var_W(k)},
\end{eqnarray}
where $var_B(k)$ is the between-class variance of $f_k$ and $var_W(k)$
is the within-class variance of $f_k$ as defined by the following
equations.  For the c-class problem,
\begin{eqnarray}
var_W(k) & = & \sum_{i=1}^{c} 
	(\frac{1}{n_i}\sum_{f_k\in \theta_i}(f_k - \bar{f_{k,i}})^2),\\
var_B(k) & = & \sum_{i=1}^{c} (\bar{f_{k,i}} - \bar{f_k})^2,
\label{Mas-eq:varbk}
\end{eqnarray}
where $\theta_i$ is the set of class $i$, $\bar{f_{k,i}}$ is the
mean of $f_k$ within the class $i$ and $\bar{f_k}$ is the total mean
of $f_k$\cite[sec. 4.11]{G-Duda73}.  We choose $\hat{K}$ elements for
the objective feature vector
\begin{eqnarray}
 \hat{\bf F} = \{f_1,...,f_{\hat{K}}\},
\end{eqnarray}
which consists of
elements that provide the $\hat{K}$ largest values of the  criterion function
$J(k)$.

This operation yields the feature vector that should most perfectly
classify the sample pattern from the basic data.  We can confirm its
completeness by applying the nearest-neighbor rule to the sample data.
This feature vector was constructed for each test data and was used in
recognition tests. Experimental results from actual data are shown in
the next section.


\subsubsection{Experimental Results}
Image sequences  of various facial expressions of the
author were collected;
% \footnote{He made expressions based on particular actions
%described in the literature\cite{Mas-ekman-friesen75}}
 20 as sample data and 30 as test data. The instructions described in
\cite{Mas-ekman-friesen75} were followed to make
expressions of various emotions. The sample data consisted of the four
classes ($c = 4$) of expression, i.e. happiness, anger, disgust, and
surprise. Each class has five image sequence samples.  Parameters for
the feature vector were developed from the sample data set as
described in the previous section. The recognition test was then
executed using the test data set. The recognition was based on the
k-nearest-neighbor rule for the feature vectors of the test data
derived from optical flow fields.

The image sequences  were acquired off-line with a digital video tape
recorder and then stored as image files for further processing.
The head of the subject was fixed to prevent  global flotation,
which would have significantly degraded  optical flow calculation.
Ordinary and constant illumination was applied to the facial  area.
Because the video signal is interlaced,  the original image was
sub-sampled at half the signal rate.
 Finally,  image sequences ($M = 256$ by $N = 240$ pixel size
and $T = 30$ frames length) were extracted for each expression instance.
Target expressions in  the video sequences were manually spotted.

\subsubsection{Feature Extraction from Sample Data}
The optical flow of each image sequence was computed with
Horn-Schunck's algorithm and the maximum number of iterations was set to 20.
The optical flow field was divided into $m = 16$ by $n = 15$ rectangular regions
and feature parameters were computed using Equations \ref{Mas-eq:muu}-\ref{Mas-eq:varbk}.
The dimension of the feature vector was set to ($\hat{K} = 15$), which was
selected so as not to 
exceed the number of samples. 

\begin{figure}
\centerline{\epsfxsize=5cm \epsffile{Mase_figs/feature_vec.eps}}
\caption{Locations of selected feature vector}
\label{Mas-fig:feature-vector}
\end{figure}
Figure \ref{Mas-fig:feature-vector}
illustrates the positions of elements chosen as the feature vector.

Since we use the three nearest samples for voting, the data is labeled
�rejected� only if three different classes are simultaneously voted to it.
Nineteen image sequences out of 20 were classified correctly in the
experiment (i.e. 95\% recognition accuracy for 20 sample data sets)
and one sample was rejected.
The minimum distance to the feature vectors of sample data $\{\hat{\bf
F}_n\}$ from test data $\hat{\bf F'}_n$
is calculated by the following formula;
\begin{eqnarray}
D({\hat {\bf F'}}) = \min_{\hat {\bf F}_n} \|\hat{\bf F}_n - \hat{\bf F'}_n\|^2,
\end{eqnarray}
where $\hat{\bf F}_n$ is the normalized feature vector of  $\hat{\bf F}$
such that each element of sample data $f_{\hat{k}}$ has the normal
distribution, 
$p(f_{\hat{k}}) \sim N(0,1)$.
%Table \ref{Mas-table:sample-nn} lists the nearest neighborhood and its distance.
When  the k-nearest-neighbor rule is applied to the test data,  the
same means and variances are used for the normalization of test data.
 
\subsubsection{Recognition of Test Data}
Ten people categorized the 30 test image sequences
% in two ways; first
%into the original four  classes and then 
into five classes, which were the original four plus the class of  
�unknown� expressions.
The test data is categorized if more than six people choose the same class,
otherwise it is categorized as unknown.

Accordingly, the data is not evenly distributed over the classes.
Table \ref{Mas-table:co-matrix} 
shows final  recognition results 
illustrated by the correlation matrix between the computer-recognized result
and human assessments. 
%Table \ref{Mas-table:co-matrix}(a) 
%lists results of the four-class categorization, while 
Table \ref{Mas-table:co-matrix} %(b) 
lists corresponding results of the five-class categorization.


\begin{table}
\centering
\caption{Correlation matrix of recognition results}
\label{Mas-table:co-matrix}
% (a) Four classes (number of test data)
% \tablewidth=11cm
% \begintable
% human $\setminus$ computer|	      happiness|  anger |  surprise|   disgust|  total\crthick
% happiness|      7 |             |          |          |    7 \cr
% anger|	          |       4	|   1      |   1      |  6\cr
% surprise|         |        2    |     9    |          |    11\cr
% disgust|     1    |       1     |          |    4     |        6\crthick
% total|          8|         7 |        10 |       5    |    30
% \endtable
% \vspace{5mm}
Five classes (including unknown; number of test data)
\tablewidth=10cm
\begintable
human $\setminus$ computer|	      happiness|  anger |  surprise | disgust| total \crthick
happiness|      7     |         |         |          |     7   \cr
anger|	              |   4	|           |  1    |    5 \cr
surprise |            |    2    |     5    |        |      7 \cr
disgust|	     |         |           |    3   |     3 \crthick
subtotals|       7     |    6    |     5    |     4  |      22 \cr
unknown |       1     |    1    |     5     |    1  |      8 \crthick
total  |        8     |    7     |    10    |    5  |      30
\endtable
\end{table}



Table \ref{Mas-table:co-matrix} shows that facial expression
recognition performance is fairly good for the four categories tested.
The overall recognition  accuracy
% in the case of (a) was $24/30$, or 80\%,  while in the case of  (b) it 
was $19/22$, or 86\%, by taking only known categories into consideration.

\Section{Other partReaders}
Other partReaders were investigated in the Human Reader project, such
as the People Image Reader and the Person Image Reader. 

The People Image Reader is a locator system, which is derived to detect the
position and motion of one or more human subjects. Capabilities include
detection of human presence, counting subjects, and understanding the
direction of crowd movement. One application of People Image Reader is
pedestrian counting\cite{Mas-sato-mase-tomono-ishii93}.

The Person Image Reader is similar in concept to face
recognizers\cite{Mas-turk-pentland91}. The face functions both as a
selector that matches faces with names to identify
individuals\cite{Mas-aka-sasaki-fuka-masui-sue92} and as an imager that sends
messages about types of facial features and expressions (round,
wrinkled, pleasant)\cite{Mas-birdwhistell}. This correlation of face,
identity, and name is indispensable to the achievement of a secure yet
convenient interface. Also, a person's facial expression contributes as
much to establishing a basis for conversation as do clothing and
posture. However, research on the development of such an imager has
yet to appear.


\Section{MIC Interface Agent}
We do want to have a �natural interface� that is comparable to
human-to-human interactions. However, what in fact is a �natural interace�?
We rely on the human capabilities of intelligence, adaptivity and
autonomy to achieve such a natural interface to specific tasks in
the real world. The most important point is that the interface should
know about the task it is interfacing to the human. Without knowing the
task, it loses the context, and thus cannot properly interpret messages. The Part-readers presented in the 
previous sections are not 
the interface yet, but simply visual sensors. When they are integrated
with an application module, they can act as a real interface, such as the CG 
Secretary, the FingerPointer, etc.

\begin{figure}
\centerline{\epsfxsize=7cm \epsffile{Mase_figs/MICinterface.eps}}
\caption{MIC (multimodal, interactive/intimate and creative) interface}
\label{Mas-fig:micinterface}
\end{figure}

Consequently, {\it ``Interface Agent''} is a solution to providing a
natural interface because it implies the agent may include a part
of the task, or at least it should act autonomously based on the
situation. The naive but worthy expectations of the Interface
Agent can be considered as the following:

\begin{enumerate}
\item visual perceptibility,
\item translating machine interface,
\item social, intelligent and adaptive capability,
\item intimacy.
\end{enumerate}

%\begin{figure}
% \centering
% \epsfile{file=Mase_figs/geek_walk.eps,width=5cm}\\
% {\tiny \copyright 1996 Zabranska and Bruderlin, ATR}
%\caption{A computer generated agent character that looks like it is
%drawing attention to the ground}
%\label{Mas-fig:geek}
%\end{figure}


Thus, in Interface Agent research, I think much attention should be
given to 
% we will need to consider 
not only
multimodality and interactivity but also
creativity of human. The MIC  (multimodeal, interactive and creative) interface
is the interface that 
we pursue in researching and developing the Interface Agent
(Figure~\ref{Mas-fig:micinterface}). 
% 
% The life-likeness and/or believablily of the agent\cite{Mas-nakatsu-mase96a} is
% an aspect of the MIC interface (see Figure~\ref{Mas-fig:geek} as an
% example of a believable guide agent). 


\Section{Conclusion}
If we can provide a warm and communicative environment, the other can
read even subtle 
messages.  The objective of the Human Reader project and the subsequent
research of the MIC Interface is to provide such an environment in a
human-computer interaction scheme. This chapter presents results of
the Human Reader project, focusing on the multimodal and interactive
aspects. Computer vision or image processing capability is not yet
sufficient to provide such a naturally communicative
atmosphere\cite{G-Pentland96}. However, such intelligent
vision sensors will inevitably be improved and used in human-computer
interfaces in the future. Computer vision technology will be used as
the ``eyes'' of the Interface Agent, which will be, I believe, the
interface we are looking for.



\Section*{Acknowledgment}
The Human Reader project (1985 -- 1992) was carried out at NTT Human Interface
Laboratories in collaboration with Yasuhito Suenaga, Yasuhiko
Watanabe, Masaaki Fukumoto, Atsushi Satoh and many other NTT
researchers. The author would like to thank Alex Pentland for his
support of lip reading research, which became a major thrust of this
research work. The author would also like to thank 
Ryouhei Nakatsu and other ATR MI\&C members for their support and
discussions on the MIC Interface research at ATR.


