
%\documentstyle[cup7b,david,psfig]{cuphci}

% for latex209 version <24 May 1989>
\makeatletter
\def\@bibitem#1{\item\if@filesw \immediate\write\@auxout
       {\string\bibcite{#1}{\the\c@enumiv}}\fi\ignorespaces}
\makeatother
%

%\begin{document}

\newcommand{\bold}[1]{\mbox{\boldmath{#1}}}
\newcommand{\dfrac}[2]{\displaystyle\frac{#1}{#2}}

\author[Masahiko Yachida and Yoshio Iwau]%
{Masahiko Yachida and Yoshio Iwai\\
  Department of Systems Engineering, Osaka University and \\
  Laboratories of Image Information Science and Technology}

\chapter[Towards Measurement of Human Information ...]{%
Towards Measurement of Human Information by Network of Active
  Cameras for Human-Machine Interaction}


\begin{abstract}
This paper describes the work on human-computer interaction in our 
laboratories.   
Recognition of human expressions is necessary for human-computer
interactive applications. 
A vision system is suitable for recognition of human expression  since
this involves passive sensing and the human gestures of hand, 
body and face that can be recognized without any discomfort for the user.
The computer should not restrict the movements of the human to the
front of the computer.
Therefore, we study methods of looking at people using a network
of active cameras.

% $B$3$NO@J8$G$O!"2f!9$N8&5f<<$G9T$J$o$l$F$$$k%R%e!<%^%s%$%s%?!
<%U%'%$%9 (B
% $B$N8&5f$K$D$$$F=R$Y$k!#?M4V$H5!3#$H$N%$%s%?!<%U%'%$%9$O<+
A3$J>uBV$G9T (B
% $B$J$o$l$?$[$&$,$h$$!#$=$3$G!"2f!9$O%;%s%5!<$H$7$F%+%a%i$rMQ$$$F?
7$7$$ (B
% $BJ}K!O@$r8&5f$7$F$$$k!#?M4V$H$N%$%s%?%i%/%7%g%s$r3NN)$9$k$?$
a$K$O?M$N (B
% $BI=>p$d<j$NF0$-!"BN$NF0$-
$J$I$r7W;;5!$,M}2r$9$kI,MW$,$"$k!#$b$A$m$s!" (B
% $B?M4V$O$8$C$H$O$7$F$$$J$$$N$G!"7W;;5!$NJ}$+$i@Q6KE*$K4Q;!$r9T$
J$&J}K! (B
% $B$r8!F$$7$F$$$k!# (B
\end{abstract}

\section{Introduction}
\label{Yac-sec:yi-intro}

%\begin{multicolpar}{2}
Sensing of human expressions is very important for human-computer
interactive applications such as virtual reality, gesture
recognition, and communication.
A vision system is suitable for human-computer interaction since this
involves passive sensing and the human gestures of the hand, 
body, and face that can be recognized without any discomfort for the
user. 
We, therefore, use cameras for the sensors in our research
to estimate human motion and gestures.

% $B?M4V$NI=8=$rG'<1$9$k$3$H$O!"%P!<%A%c%k%j%"%j%F%#$d%8%'%9%
A%c!<G'<1!"2qOC (B
% $B$J$I$N2qOCE*$J%"%W%j%1!<%7%g%s$K$H$C$F=EMW$G$"$k!# (B  
% $B2hA|%7%9%F%`$O!"<uF0E*$J%;%s%7%s%0$G$"$j!"%f!<%6!<$K2?$i (B
% $BIT2w46$rM?$($:$K!"?M4V$NI=>p!"%8%'%9%A%c!<!"F0:n$J$I$r (B
% $B<+F0E*$KG'<1$9$kJ}K!$9$k$3$H$,$G$-$k$N$G!" (B
% $B%^%s%^%7%s%$%s%?!<%U%'!<%9$K$OE,$7$F$$$k!# (B
% $B$=$3$G!"2f!9$N8&5f$G$O%;%s%5$H$7$F%+%a%i$rMQ$$$k!# (B
% $B$=$7$F!"2f!9$O?M4V$NI=8=$N$J$+$G$b!" (B
% $B4i$NI=>p!"<j$V$j!"?H?6$j$KCmL\$7!"$=$l$i$N7W;;5!$G$N (B
% $BG'<1J}K!$K$D$$$F@bL@$9$k!# (B

Facial expression is a natural human expression and 
is necessary to communicate such emotions as {\em happiness, surprise,}
 and {\em sadness} to others.
A large number of studies have been made on machine recognition of
human facial expression.
Many of them are based on multi-resolution monochrome images and
template pattern matching techniques%
~\cite{Yac-Yang93,Yac-Tsukamoto93,G-Brunelli93}.
This kind of approach needs some average operation on the face model
or blurring of the input image to cope with the different appearance
of faces in the images. 
This causes face detection to often fail by detecting false
``faces'' when the input images have complex backgrounds.
We describe a method for face detection using fuzzy theory and 
a method for recognition of facial expressions by using a
potential net.

% $B4i$NI=>pG'<1$K4X$7$F$O!"=>Mh$?$/$5$s$N8&5f$,9T$o$l$F$-$?!# (B
% $B$=$l$i$NB?$/$OB?=E2rA|EY$NGr9u2hA|$G%F%s%W%l!<%HK!$r (B
% $B85$H$7$?$b$N$G$"$C$?!#$3$N<o$N%"%W%m!<%A$O (B
% $B2hA|Cf$K$"$k$$$m$$$m$J4i$NCj=P$N$?$a$K (B
% $B4i7A>u$KJ?6Q2==hM}$r;\$7$?$j!" (B
% $BF~NO2hA|$r$\$+$7$?$j$9$kI,MW$,$"$k!# (B
% $B$=$7$F!"$3$N=hM}$O!"J#;($JGX7J2<$G$O4i$K;w$?$b$N$NB8:_$G (B
% $B$7$P$7$P<:GT$5$;$k860x$H$J$k!# (B
% $B$^$?!"F~NO2hA|$O@5LL$G$^$C$9$0$K8~$$$?4i$,2>Dj$5$l$F$$$?!# (B
% $B$=$3$G2f!9$O%U%!%8!]M}O@$K$h$kH)!"H1!"4i$N0BDj$7$?8!=P$H!" (B
% $B%"%/%F%#%V%+%a%i$K$h$k4i$N8~$-$NJd@5$H (B
% $B%]%F%s%7%c%k%M%C%H$K$h$kA4BN$NJQ2=NL$r9MN8$7$?I=>pG'<1
K!$r (B
% $B@bL@$9$k!# (B

Hand gesture is also one of the typical methods of nonverbal
communication for human beings and we naturally use various gestures
to express our own intentions in everyday life.
Several model-based methods for the recognition of hand gestures from 
images have been proposed.
Rehg et al.\cite{G-rehg95} used a kinematic model and fitted it to
input images of fingers.
Kuch et al.\cite{G-Kuch95} used a calibrated 3-D model and fitted
the silhouette of this model to an input image.
It is a good strategy to have a accurate model and a kinematic
constraint of the known articulated or known complex object. 
We also describe how to acquire a shape model of an articulated
object and how to fit the model to a real image.

% $B<j$N%8%'%9%A%c!<$b$^$?!"?M4V$K$H$C$F$OE57?E*$J%N%s%P!<%P%
k%3%_%e%K%1!< (B
% $B%7%g%s$N0l$D$G$"$j!"<+J,$N0U?^$rI=8=$9$k$?$a$K$$$m$$$m$J%8%'%9
%A%c!<$r (B
% $BMxMQ$7$F$$$k!# (B
% $B4v$D$+$N8&5f$G$O<j$N%8%'%9%A%c!]$NM}2r$K%b%G%k$r;H$C$?$b$
N$,$"$k!# (B
%Rehg $B$i$O%-
%M%^%F%#%C%/%b%G%k$rMQ$$$FF~NO2hA|$H0lCW$5$;$?!# (B
%Kuch $B$i$O#3Kg$N2hA|$+$i<j7A>u$N#3<!85%b%G%k$r3MF@$7!" (B
% $B$=$N%7%k%(%C%H$rF~NO2hA|$H%U%#%C%F%#%s%0$7$?!# (B
% $B4{CN$N4X@aJ*BN$d4{CN$NJ#;(J*BN$N7A>u$d1?F094B+$r;}$D$3$H$O 
(B
% $BNI$$@oN,$G$"$k!# (B
% $B2f!9$b4X@aJ*BN$+$i$N7A>u%b%G%k$N3MF@$H$=$N%b%G%k$r (B
% $B<B2hA|$K0lCW$5$;$kJ}K!$K$D$$$F@bL@$9$k!# (B

Needless to say, considering some interactive systems requiring
symbolic meanings of gestures, accurate joint angles are not
necessarily required. 
Processing time is important rather than
measurement accuracy in such an interactive system.
From this point of view, Darrell et al.\cite{G-Darrell93} presented a
method to recognize gestures directly from gray-scaled images instead
of geometrical information of feature points. 
In the method, a gesture is represented by a set of images,
and correlation scores between input images and models
are calculated by special hardware.
In this paper, we also present a new method for recognizing gesture 
of arms and legs in real-time with multiple template models.
In our method, the models for specific human body parts are used, and
the disparity between an input image and the models  
is analyzed by a template matching method.

% $B$b$A$m$s!"$"$k<o$N%8%'%9%A%c!<$N5-
9fE*0UL#$@$1I,MW$J%$%s%?%i%/%F%#%V%7 (B
% $B%9%F%`$G$O!"@53N$J4X@a3QEY$H$$$&$b$N$OI,MW$J$$!# (B
% $B$=$N$h$&$J%7%9%F%`$G$O!"7WB,@:EY$h$j$b=hM};~4V$NJ}$, (B
% $B=EMW$G$"$k!# (B
% $B$3$N4QE@$+$i!" (BDarrell  $B$i$OGr9u2hA|$+$i (B
% $BD>@\%8%'%9%A%c!<$rG'<1$9$kJ}K!$rDs0F$7$F$$$k!# (B
% $B$3$NJ}K!$G$O%8%'%9%A%c!<$r2hA|Ns$GI=8=$7!" (B
% $BAj4X$r7W;;$9$k$3$H$GG'<1$7$F$$$k!# (B
% $B2f!9$O%F%s%W%l!<%H$rMQ$$$??7$7$$J}K!$r>R2p$9$k!# (B
% $B$3$NJ}K!$G$O%b%G%k$OBN$N3FItJ,$rI=8=$7 (B
% $B%b%G%k$H2hA|$H$N%^%C%A%s%0$GG'<1$r$9$k!# (B

% $B$7$+$7!"8D!9$N8&5f$O$=$l$>$lJL8D$K9T$o$l$?$b$N$G!" (B
% $BE}9g2=$5$l$?8&5f$O$^$@9T$o$l$F$$$J$$!# (B
% $B$3$3$G$O!"2f!9$,8&5f$r9T$C$F$$$k!">e$K=R$Y$?$h$&$J8D!9$NMWAG5;
=Q$N@bL@ (B
% $B$H!"$=$NMWAG5;=Q$rE}9g$7$?%7%9%F%`$K$D$$$F@bL@$9$k!# (B
%\end{multicolpar}

\section{Facial Expression and Gesture Recognition}
\label{Yac-sec:yi-face-rec}

%\begin{multicolpar}{2}
The face is an important object for extracting a lot of information
about a person, so many studies have been made on facial image recognition.
Especially, many researchers have payed considerable attention to
facial expression.
We describe a method for recognition of facial expression and also
describe the method for detection of faces in this section. 
Detecting human faces in images is a key problem in human-computer
interaction and is an essential step for face recognition.

% $B<+A32hA|$+$i?M4V$N4i$r8!=P$9$k$3$H$O!" (B
% $B%R%e!<%^%s%$%s%?!<%U%'!<%9$N8&5fJ,Ln$K$*$1$k (B
% $B=EMW$J2]Bj$N0l$D$G$"$j!"8D?M<1JL$HI=>pG'<1$N85$H$J$k!# (B
% $B$=$3$G!"I=>pG'<1$N@bL@$NA0$K4i$N8!=P$H;Q@*Jd@5$K$D$$$F@bL
@$9$k!# (B

% $B$=$3$G!"<!$N$h$&$J@oN,$G4i$NI=>pG'<1$r9T$&!# (B( $B$3$3$O%+%
C%H$9$k (B)
%\end{multicolpar}
%\begin{enumerate}
%\item  $B<+A32hA|$+$i$N4i8!=P (B
%\item  $B4i$NDI@W$H%/%m!<%:%"%C%W (B
%\item  $BI=>pG'<1 (B
%\end{enumerate}

\subsection{Face Detection}
%  $B8b$5$s$N8&5f (B

%\begin{multicolpar}{2}
It is difficult to detect human faces in an image with a complex
background since the faces are not always distinct from others.
We, therefore, propose a method for face detection by using 
color information and fuzzy pattern matching.
%The process of the face detection is shown in
%Fig. \ref{Yac-fig:facedetectionflow}. 

% $B4i$O8D?M:9$,$"$j!"2hA|Cf$N4iFCD'$,$$$D$bA/L@$G$"$k$H$O (B
% $B8B$i$J$$$N$G!"J#;($JGX7J$+$i4i$rCj=P$9$k$N$OFq$7$$!# (B
% $B$=$3$G!"?'>pJs$H%U%!%8%#%Q%?!<%s%^%C%A%s%0$N<jK!$r (B
% $BMQ$$$?J}K!$rDs0F$9$k!# (B
% $B4i8!=P=hM}$NN.$l$r?^ (B\ref{Yac-fig:facedetectionflow} $B$K<($9!# (B
%\end{multicolpar}

%\begin{figure}[htbp]
%  \begin{center}
%    \leavevmode
%    \epsfile{file=Yachida_figs/fig1.ps,width=6cm}
%    \caption{The overview of the face detection system}
%    \label{Yac-fig:facedetectionflow}
%  \end{center}
%\end{figure}

\subsubsection{Uniform Chromatic System}

%\begin{multicolpar}{2}
%Since the terms ``skin color'' and ``hair color'' are subjective
%concept obtained by human's observation through eyes, the color space
%used in the processing of information about skin color and hair color
%should agree with human senses in order to obtain more reliable
%result. 
In our research, we use the perceptually uniform color space
proposed by Farnsworth's to represent colors in an image.
Such a non-linear color space as Farnsworth is called {\em
  perceptually uniform chromatic system} or UCS. 
The main advantage of UCS is that the distribution of ``skin'' color
becomes concentric and can be expressed by a Gaussian distribution
function, so we can reliably detect ``skin'' color. 

% $BH)?'$H$$$&8@MU$O!"?M4V$NL\$h$jF@$i$l$?<g4QE*$J35G0$G$"$k!# (B
% $B=>$C$FH)?'$N>pJs$r=hM}$9$k$N$KMQ$$$k?'6u4V$H$7$F!" (B
% $B?M4V$N;k3PFC@-$H0lCW$9$k?'6u4V$NJ}$,K>$^$7$$$H$$$($k!# (B
% $B$3$3$G$O!" (BFarnsworth  $B$,Ds0F$7$??M4V$N;k3PFC@-
$H0lCW$9$k?'6u4V$rMQ$$$F!" (B
% $BF~NO2hA|$N?'$rI=8=$9$k!# (B
% $B$=$N$h$&$JHs@~7A$J?'6u4V$rMQ$$$k$3$H$G!" (B
%CIE  $B6u4V$G$OEyJ}E*$G$J$$J,I[$r$9$kH)?'$H$$$&<g4QE*$J?'$, (B
% $BEyJ}E*$JJ,I[!"$9$J$o$AF1?41_>e$NJ,I[$K$J$k$N$G!" (B
% $BMF0W$KCj=P$,2DG=$K$J$k!# (B

We developed two models called the {\em skin color distribution model}
and the {\em hair color distribution model}, where the values are between
0.0 and 1.0 since it is difficult to represent skin or hair color exactly.
We can use these distribution models to estimate the
degree of how well a color looks like skin or hair color. 

% $B$=$3$G!"2f!9$O0J2<$N$h$&$K!"H)?'J,I[%b%G%k$HH1?'J,I[%b%G%k$r:n@.
$7$?!# (B
% $B$^$?!"$3$NJ,I[%b%G%k$O$I$l$/$i$$H)?'$H$K$F$$ (B
% $B$k$+!"H1?'$H$K$F$$$k$+$NI>2A$H$7$F$b;H$&$3$H$,$G$-$k!# (B
%\end{multicolpar}

\subsubsection{Selection of Face Candidates}

We have currently prepared five basic head shape models, each of them
containing $m\times n$ square cells of same size. There are two kinds
information in each cell in a basic model:

\begin{enumerate}
\item the ratio of skin color area to total area,
\item the ratio of hair color area to total area.
\end{enumerate}

The basic models are scalable. Therefore we can use them to detect
faces of any arbitrary size. 

The faces in images are extracted by comparing the head shape models with
the extracted skin color like regions and the hair color like regions
using a fuzzy pattern matching method based on a {\it binary fuzzy
  relation}. 

To detect regions containing faces, we 
use the face models to scan the entire image in order to estimate the
matching degree between the face model and each rectangular region in
the image. We use a matrix to represent the matching degrees of all
the rectangular regions in the image. If a local maximum of matching
degrees is greater than a certain threshold, we will consider that the
rectangle may contain a human face and treat it as a face
candidate. 
%An example of estimated matching degrees and extracted face
%candidates are shown in figure 2. 

\subsubsection{Verification by Facial Parts}

In most cases, the facial features are not clear in the image, 
thus eyes and eyebrows (also nose and mouth) cannot be detected
separately from an image with high reliability. 
In our research, we first try to
extract two regions that contain one eye and one eyebrow, and one
region that contains the nose and the mouth.

Since the position and size of facial features such as eyes and mouth,
do not differ very much for different persons, we can predict their
position and size in the image when the position and the size of the
face are known. %(See figure 5). 

%Since there are many longer horizontal edges near the eyes, eyebrows,
%and mouth, we use a mask of the size $7 \times 3$ pixels. 
%In order to use color information efficiently, the gradient is not
%only for brightness(Y), but also for color. In other words, we use
%among R, G, B to do edge extraction:

%\begin{equation}
%Strength = \triangle I_1 + \triangle I_2 \times 2 + \triangle I_3
%\times 2 + \triangle I_4 \times 3 + \triangle I_5 \times 2 + \triangle
%I_6 \times 2 + \triangle I_7
%\end{equation}

%where 

%\[
%\triangle I_i =\sqrt{(R_ci-R_ai)^2 + (G_ci-G_ai)^2 + (B_ci-B_ai)^2}.
%\]

%We define an eye-eyebrow detector that calculates the sum of the
%value of the horizontal element of the gradient of RGB in
%regions. The nose-mouth part is detected in a similar way. 
%Figure 6
%shows the output of the eye-eyebrow detector and the nose-mouth
%detector when applied to the face candidate extracted from an input image.


%\subsection{Face Verification Using Relational Facial Model}

%The geometrical relations between the two eyes and the mouth on a
%human face are approximately the same for different persons. 
We build a simple model to represent the geometric relations among the
two eye-eyebrow parts and the nose-mouth part of a human face.
% as shown in figure 7. 
We call it the {\it relational facial model} or simply {\it
RFM}. It is a scalable and rotation independent model. 
%
The face verification is carried out by fitting the {\it RFM}\/ to the
extracted eye-eyebrow parts and nose-mouth part. It determines whether
the face candidate is really a human face or not based on the the
estimation of difference between the model and the facial
features. 
%The {\it RFM} is fitted to the extracted facial features as
%follows:
%% (See figure 7):

%\begin{enumerate}
%\item Finding the middle point ($M_i$) between two eye-eyebrow
%parts.
%\item Drawing a straight line $L_1$ between $M_i$ and the
%nose-mouth part.
%\item Drawing a line $L_2$ through $M_i$ and perpendicular to line
%$L_1$.
%\item Drawing a line $L_3$ through $e_l$ and perpendicular to line
%$L_2$ and finding the intersection $e'_l$ of $L_3$ and $L_2$.
% The same for the right eye.
%%\item Drawing a line $L_4$ through $e_r$ and vertical to line
%%$L_2$ and finding out the crossing point $e'_r$ of $L_4$ and $L_2$,
%%and do the same for the right eye.
%\item Fitting the RFM with the extracted features by setting
%$E_l$ at $e'_l$ and $E_r$ to $e'_r$.
%\end{enumerate}

The difference $D$ between the {\it RFM}\/ and the extracted facial
features is estimated as the sum of the distances between the various
{\it RFM} facial features and their corresponding extracted facial
features divided by the distance between the two eye-eyebrow parts of
the {\it RFM}. % as follows:
%
%If $D$ is less than 0.4, we consider that the face candidate is a
%reality human face.
We consider that the face candidate is a true human face if $D$ is
less than a certain threshold value.

\subsection{Recognition of Facial Expression}
\label{Yac-sec:yi-rec-face}
% $BLZB<$5$s$N8&5f (B

After detection of a face, recognition of facial expressions can be
done. Conventional methods for recognition of facial expressions 
examine geometric relations established by facial features, such as
eyes, eyebrows and a mouth, and variation of the locations 
of the features. However, estimation of their precise positions
and shape attributes in real images is very difficult since abrupt
rigid motion of the head including facial expressions  degrades facial
images. 

We have suggested a new approach to treat facial images
as whole patterns of potential fields induced on the retina of living
things~\cite{Yac-Matsuno94,Yac-Matsuno95}.
The potential field is sampled by a potential net and analyzed in the
principal pattern space by the Karhunen-Lo\`{e}ve expansion method. 
%In this paper, we present an extended improvement of our approach with
%which we can roughly estimate the strength of the facial expression of
%happiness.
%To achieve more detailed recognition results,
%we introduce an energy pattern model of facial images, 
%and the model is obtained from accumulated energy on nodes of a
%potential net. 
%The virtue of our approach is to analyze an expression as a whole
%pattern. It means that we can understand facial expressions by using
%only one facial image. 

\subsubsection{Potential Net and Netted Energy Model}

%Fig.\ref{Yac-fig:carp_field} shows the potential
%field on the retina of a carp when the patterns which cause the
%M\"uller-Lyer Visual Illusion is shown. 
%Psychologists also have studied the induction of
%the potential field on human retinas and think that the field is
%one of the factors which cause the visual illusion. 
%We have applied the idea to recognize facial
%expressions and have shown that the potential field is effective to 
%extract global features of more complex patterns.

%\begin{figure}[htbp]
%  \begin{center}
%    \leavevmode
%    \vspace{3cm}
%    %  $B%]%F%s%7%c%k%U%#!<%k%I$N3( (B
%    \caption{The potential field on the retina of a carp.}
%    \label{Yac-fig:carp_field}
%  \end{center}
%\end{figure}

To model the potential field, we used a physical model, called
a potential net. The potential net is a two dimensional mesh of which
each node is connected to four neighbors by the springs.
We put this net on an edge image of human face, 
then the net is deformed by the image force from the potential field
so as to reflect local  
deformation over the whole net through springs. 
The final configuration of the nodes represents the overall pattern of 
the potential field and can be used for various applications such as
structural sampling of image~\cite{Yac-Terzopoulos91b}. 
%In this paper, 
We adopt an improved structure of the potential net
model % as shown in Fig.\ref{Yac-fig:structure}, 
of which each node is connected to eight neighbors by springs to
sample the potential field in detail.   

%\begin{figure}[htbp]
%  \begin{center}
%    \leavevmode
%    \epsfile{file=Yachida_figs/structure.ps,width=4.0cm}
%    \caption{An improved potential net model}
%    \label{Yac-fig:structure}
%  \end{center}
%\end{figure}

%\subsection{Energy Model}

Our new approach of modeling potential field is called the netted energy
model, which is considered as actualization of accumulated energy at
the nodes of the potential net.
The energy is easily calculated by the following equation:
\begin{equation}
    E_{k,l}=
        \sum_{n_{i,j}\in{N_{k,l}}}
        \sum_{a=1}^{8}(S\frac{1}{2}k{l_{i,j}^{a}}^{2})/{N_{k,l}}
    \label{Yac-eq:energy}
\end{equation}
where $N_{k,l}$ is the number of nodes in a small block of $(k,l)$,  
$l_{i,j}^{a}$ is the expanded or contracted length of springs which are 
connected to a node, and $S$ is the sign of the deformed direction
of the spring; plus for contraction and minus for expansion.

Figure \ref{Yac-fig:tuka_sample} shows a deformed net and gray-level
conversion of it's netted energy.  In the figure, bright regions express
highly accumulated energy induced by edge components of facial organs
or furrows of facial expressions. The model is a very coarse
representation for a facial image, but we can notice that individuality
disappears in the model, and useful information for determining facial
expressions is well sampled from the input image.

\begin{figure}[htbp]
  \begin{center}
    \leavevmode
    \begin{tabular}{cc}
        \psfig{figure=Yachida_figs/face/fig3a.ps,height=3cm} &
        \psfig{figure=Yachida_figs/face/fig3b.ps,height=3cm}
    \end{tabular}
    \caption{A deformed potential net and an energy model}
    \label{Yac-fig:tuka_sample}
  \end{center}
\end{figure}

\subsubsection{Representative Model for Facial Expressions} 

%In our previous work, we analyzed four different expressions:
% \em{happiness}, \em{anger}, \em{surprise}, and \em{sadness}.
%It was needed to prepare representative models, and in this paper we
%also employ the same scheme for the netted energy model.

Our purpose of the work is to estimate the strength of facial expressions.
However, it is very difficult even for human beings to recognize
subtle changes in {\em anger}, {\em surprise}, and {\em sadness}. 
So here we restrict the object of recognition to one expression: 
{\em happiness} and attempt to roughly recognize its strength.

Facial expression is classified into four degrees of strength:
{\em expressionless, minute, moderate}, and {\em maximum}
happiness. 
From the training facial images, we first obtained the representative
models according to the four degrees of expression strength with
the following equation:
\begin{equation}
      {E}_{d}=
        \frac{1}{h_d}\sum_{{E_{k,l}}{\in}{\phi_d}}E_{k,l}
\end{equation}
where ${\phi_{d}}$ is the set of energy models for the
facial images in the $d$-th strength grade training set which contains
${\phi_{d}}$ images. Figure \ref{Yac-fig:average} shows the models
represented in the netted energy model.
%Simply speaking, these models are the average
%of a few people's energy model $E_{d}$ for four levels of
%happiness. 
\begin{figure}[htbp]
  \begin{center}
    \leavevmode
    \begin{tabular}{cc}
        \psfig{figure=Yachida_figs/face/fig4a.ps,height=3cm} &
        \psfig{figure=Yachida_figs/face/fig4b.ps,height=3cm} \\
        \psfig{figure=Yachida_figs/face/fig4c.ps,height=3cm} &
        \psfig{figure=Yachida_figs/face/fig4d.ps,height=3cm} 
    \end{tabular}
    \caption{The models represented in the netted energy model.}
    \label{Yac-fig:average}
  \end{center}
\end{figure}

\subsubsection{Principal Component Analysis on Netted Energy Models}

We treat the net deformation as a overall pattern of facial feature.
Namely, we extract the global features of the face in the potential
net by the energy model and apply the principal component analysis to
represent the expression strength in low dimensional vector space. The
following summarize the procedure:

\begin{enumerate}
\item compute the mean energy $\mu$ of the four representative energy
models $E_d$,
\item compute the covariance
 matrix $\bold{C}$ of the training image set:
\begin{equation}
\bold{C}=\frac{1}{4}\sum_{d=1}^{4}(E_{d}-\mu){(E_{d}-\mu)}^{T}
        =\bold{A}\bold{A}^{T}.
\end{equation}

\item compute the eigenvalue ${\lambda}_{f}$ and the eigenvector  
$\bold{U}_{f}$ of the covariance  matrix $\bold{C}$
%\begin{eqnarray}
%\bold{C}\bold{U}_{f}
%        & = & {\lambda}_{f}\bold{U}_{f} \\
%\bold{U}_{f}^{T}\bold{U}_{f} & = & 1  \\
%\end{eqnarray}
%where eigenvector $\bold{U}_{f}$ is obtained from the following K-L 
%expansion procedure:
%\begin{eqnarray}
%\bold{A}^{T}\bold{A}\bold{W}_{f} & = & \Phi_{f}\bold{W}_{f} \\
%\bold{A}\bold{A}^{T}\bold{A}\bold{W}_{f}
%& = & \Phi_{f}\bold{A}\bold{W}_{f} \\
%\bold{U}_{f} & = & \bold{A}\bold{W}_{f}. 
%\end{eqnarray}
\item Select $G$ significant eigenvectors,
$\bold{U}_{1},\bold{U}_{2},\ldots,\bold{U}_{G}$
\item a $G$-dimensional vector ${\bold
    R}_{df}={(r_{d1},r_{d2},\ldots,r_{dG})}^{T}$, 
  which is regarded as the feature vector for the netted energy model
  of facial 
  expressions in the training set, is computed by  
  \begin{equation}
    {r}_{df}=\bold{U}^{T}_{f}({E}_{d}-{\mu})
    \hspace{10mm} f=1,\ldots,G.
    \label{Yac-eq:disparity_ab}
  \end{equation}
\end{enumerate}

Following the above PCA procedure, 
a low dimensional space spanned by the G
feature components is determined. We call this space {\em Emotion
  Space} because 
the projection of a energy model into this space reflects the global
features of the facial expression in the training set. 
%Two principal
%components, feature vectors are shown in Fig.\ref{Yac-fig:top_eigen}. 
%And also the Emotion Space exhibited by top three principal components
%is shown in Fig.\ref{Yac-fig:strength_space}. 
%From the figure we can notice that minute
%and moderate expressions are in the neighborhood in (a), however, two
%categories are distinctly separated by third component as in (b).  

%\subsubsection{Experimental Results}

%  $B<B837k2L:\$;$k (B

\subsection{Facial Gesture}

%\begin{multicolpar}{2}
%  $BI=>pG'<1$d4iG'<1$K$7$F$b!"B?$/$N=>Mh8&5f$O (B
%  $B@5LL2hA|$r2>Dj$7$F$$$?!#$7$+$7!" (B
%  $B?M4V$,F0:n$r$7!"%8%'%9%A%c!<$r9T$J$C$F$$$k;~$O (B
%  $B$=$N$h$&$JM}A[E*>u67$O5/$3$j$=$&$K$J$$!#$=$3$G!" (B
Most existing face recognition systems like facial expression
recognition, personal identification system, require images
containing faces in the frontal pose and at fixed size.
Therefore, the camera must keep taking the frontal
view of the face by changing the view point, view direction or zooming 
factor when the user's head moves.

In our system, we use active cameras which can follow the motion of the 
user's head. 
For this purpose, we need information about the pose of the
face relative to the camera. 
The relative pose can be decomposed into the three rotation elements:
the $X$ axis and $Y$ axis of the head coordinate system and  
the $Z$ axis of the camera coordinate system.
%
% $B4i2hA|$rCj=P8e!"E,@Z$J;Q@* (B( $B@5LL8~$-
!K$K4i2hA|$rF@$k$h$&$K (B
% $B%+%a%i$N;kE@!"J}8~$rJQ$($?$j!"%:!<%`Hf$rJQ$($?$j$9$kI,MW$,$"$k!# (
B
% $B2f!9$N%7%9%F%`$G$OG=F0E*%+%a%i$rMQ$$$F!" (B
% $BMxMQ<T$NF,$NF0$-$KDI=>$G$-$k$h$&$K$9$k!# (B
% $B$=$N$?$a$K$O!"G=F0E*%+%a%i$rMQ$$%+%a%i$KBP$9$k4i$N;Q@*$r (
B
% $B?dDj$9$kI,MW$,$"$k!# (B
% $BAjBPE*$J;Q@*$OJ*BN7O$N (B$X$,$Y$ $B$H%+%a%i7O$N (B$Z$ $B
2sE>$N#3$D$KJ,2r$G$-$k!# (B 
%
%Here 
We describe two approaches for estimating the rotational
component of the $Z$ axis and the $Y$ axis of the
camera coordinate system. 
%We use the information about the skin part and the hair part of the
%face to estimate the rotational components because they can be
%extracted from input images more reliably than the individual facial organs.

% $B%+%a%i$KBP$9$k4i$N;Q@*$O!"F,It:BI87O$N (B$x$,$y$ $B<42s$j$N2sE
>MWAG$H!" (B
% $B%+%a%i:BI87O$N (B$z$ $B<42s$j$N2sE>MWAG$K$h$jI=8=$5$l$k (B( 
$B?^ (B\ref{Yac-fig:face-rot}
% $B;2>H (B) $B!# (B
% $B$3$3$G$O!"F,It:BI87O (B
% $B$N (B$y$ $B<42s$j$N2sE>MWAG!JF,$N?eJ?J}8~$N2sE>!K$H%+%a%i:BI
87O$N (B$z$ $B<42s$j$N (B
% $B2sE>MWAG!J4i$r:81&$K79$1$k2sE>!K$r?dDj$9$k%"%W%m!<%A$K$D$$$
F=R$Y$k!# (B
% $B4i$N3F4o41$h$j$b>e$G=R$Y$?H)?'NN0h$HH1?'NN0h$N>p (B
% $BJs$,0BDj$7$F8!=P$G$-
$k$N$G!"2f!9$O?'>pJs$r4i$N;Q@*?dDj$KMQ$$$k!# (B
%\end{multicolpar}

\begin{figure}[htbp]
  \begin{center}
    \leavevmode
    \psfig{figure=Yachida_figs/head_cam.ps,width=6.0cm}
    \caption{The rotational components of a face}
    \label{Yac-fig:face-rot}
  \end{center}
\end{figure}

\subsubsection{Tracking of Face}

%\subsubsection{ $BF,It:BI87O$N (B$y$ $B<42s$j$N2sE>MWAG$N?dDj (B}

%\begin{multicolpar}{2}
When a face in an image appears in frontal pose,
the skin part will be at the center of the face and
the hair part will be around the face
since the face is approximately symmetric.
Thus the $X$ coordinate of the center of the skin area and that of
the center of the hair area will be same. 
If the head rotates from frontal to the left horizontally, 
the skin part will appear on the right and the hair part on the left.
If the $X$ coordinate of the center of the skin area will move to
right,
the center of the hair area will move to the left relative to the
center of the whole face area. 
Therefore, we can determine whether a face is in frontal pose by
checking the difference between the $x$-coordinates of the center of
hair area and the one of the skin area. 

% $B4i$,%+%a%i$KBP$7$F@5LL$r8~$$$F$$$k$H$-
$r9M$($k$H!"F@$i$l$?4i2hA|Cf$G$O!" (B
% $BH)?'NN0h$,F,It$NCf?4It$K$"$j!"H1?'NN0h$,F,It$N<~JUItJ,$K$"$k$G$"$m$&
!# (B
% $B$7$+$b!"$=$l$i$NNN0h$O:81&BP>N$K8+$($k$G$"$m$&!#4i$,:8$r8~$$$F$$
$k$H$- (B
% $B$O!"2hA|Cf$G$OH)?'NN0h$OF,It$N1&$K8=$l!"H1?'NN0h$O:8$K8=$l$k$@$
m$&!#5U (B
% $B$K!"4i$,1&$r8~$$$F$$$l$P!"H)?'NN0h$O:8$K!"H1?'NN0h$O1&$K8=$l$k$@$
m$&!# (B
% $B$3$N$3$H$+$i!"$=$l$>$l$NNN0h$N=E?44V$N4X78$J$I$rD4$Y$l$P!" (B$y
$ $B<42s$j$N (B
% $B2sE>MWAG$r?dDj$G$-$k$3$H$,M=A[$5$l$k (B
%\end{multicolpar}

\begin{figure}[htbp]
  \begin{center}
    \leavevmode
    \psfig{figure=Yachida_figs/fuku-center.ps,width=8.0cm}
    \caption{The relationship among head regions}
    \label{Yac-fig:head-rel}
  \end{center}
\end{figure}


%\subsubsection{ $B%+%a%i:BI87O$N (B$z$ $B<42s$j$N2sE>MWAG$N?dDj 
(B}

%\begin{multicolpar}{2}
Here we introduce a simple method for estimating the rotational
component of the head around $Z$ axis of the camera coordinate system. 
We can estimate the head rotation by inferring the direction of the
face pattern in the image since the rotation of the head around the
$Z$ axis causes the rotation of the face pattern in the image.
We, therefore, can infer the direction of the face pattern in the
image by estimating the axis of least inertia of the head region
extracted from the image since the shape of the head is symmetrical
and can be considered as an elliptical sphere.

% $B$3$3$G$O!"%+%a%i:BI87O$N (B$Z$ $B<42s$j$N4i$N2sE>@.J,$r?dDj$9$k
4JC1$JJ}K!$r (B
% $B>R2p$9$k!# (B
% $BF,It2sE>$O4i$N4o41$N2sE>$r0z$-5/$3$9$N$G!"4i$N2sE>NL$r (B
% $B?dDj$9$k$3$H$,$G$-$k!# (B
% $BF,It$N7A>u$O!"F,$NCf?4<4$K4X$7$F$[$\:81&BP>N$G$"$j!"$^$?$[$\BJ1_B
N$H$7 (B
% $B$F8+$J$9$3$H$,$G$-
$k!#$3$N$3$H$+$i!"4i$,%+%a%i:BI87O$N (B$z$ $B<42s$j$K2sE> (B
% $B$r9T$C$?>l9g!J$D$^$jF,$r79$1$k1?F0$r$7$?>l9g!K!"2hA|Fb$NF,It$rBJ1_$H8
+ (B
% $B$J$7$?$H$-
$ND9<4$NJ}8~$rD4$Y$l$P!"$3$N (B$z$ $B<42s$j$N2sE>MWAG$r?dDj$9$k$3 
(B
% $B$H$,$G$-$k!# (B
%\end{multicolpar}

%\begin{figure}[htbp]
%  \begin{center}
%    \leavevmode
    
%    \caption{Results of estimated axis}
%    \label{Yac-fig:head-axis}
%  \end{center}
%\end{figure}

%

\section{Hand Gestures}
\label{Yac-sec:yi-hand-ges}

%\begin{multicolpar}{2}
We also use hand gestures to express our own intentions.
Hand gestures are also important for human-computer interaction 
to operate objects.

% $B<j$N7A>u$d%8%'%9%A%c!<$rF~NO<jCJ$H$9$kJ}K!$b$^$?!" (B
% $B%3%s%T%e!<%?$H$N%$%s%?!<%U%'!<%9$,<+A3$J7A$G<B8=$9$k$3$H$
K (B
% $B$D$J$,$j!"Hs>o$K=EMW$J0UL#$r;}$C$F$$$k!# (B
%\end{multicolpar}

\subsection{3-D Hand (With Fingers) Motion and Position Estimation}

Correct hand pose and position are sometimes needed in such
applications as measurement systems, and robot learning systems.
We describe a method for measurement of hand pose in this section.

\subsubsection{Reconstruction of Hand Shape From a Monocular Image}

\begin{figure}[htbp]
  \begin{center}
    \leavevmode
    \begin{tabular}{cc}
        \psfig{figure=Yachida_figs/CameraModelII.ps,width=4.0cm} &
        \psfig{figure=Yachida_figs/ObjectModel.ps,width=4.0cm} 
    \end{tabular}
    \caption{The camera coordinate system (a) and the hand model
      coordinate system(b)}
    \label{Yac-fig:camera}
  \end{center}
\end{figure}

%\begin{multicolpar}{2}
We consider that the object-centered coordinate system $O_O-X_OY_OZ_O$
whose origin $O_O$ is fixed at the bottom of the palm as illustrated
in Figure  \ref{Yac-fig:camera}.
We define the position of $\bold{P}_C(t)$ at time $t$ on the
camera-centered coordinate system and
the position of $\bold{P}_O(t)$ ,which corresponds to $\bold{P}_C$,
in the object-centered coordinate system (shown in Fig. \ref{Yac-fig:camera}). 

The relation between $\bold{P}_C(t)$ and $\bold{P}_O(t)$ is given by

%Fig. \ref{Yac-fig:ObjectModel}  $B$N$h$&$KJ*BN:BI87O (B $O_O-
X_OY_OZ_O$  $B$r9M$($k!# (B
% $BJ*BN:BI87O$N86E@$O<j$NJ?$NIU$1:,$K$"$k$H$9$k!# (B
%$\bold{P}_C(t)$  $B$r;~9o (B $t$ 
 $B$K$*$1$k%+%a%i:BI87O$N0LCV$H$7!" (B
%$\bold{P}_O(t)$  $B$r$=$l$KBP1~$9$kJ*BN7O$N0LCV$H$9$k!# (B
% $B$3$NFs$D$N4X78$O<!$N<0$GM?$($i$l$k!# (B
%\end{multicolpar}

\begin{equation}
  \label{Yac-eqn:camerafrommodel}
  \bold{P}_C(t) = R(t)\cdot\bold{P}_O + \bold{T}(t)
\end{equation}
%\begin{multicolpar}{2}
  where $R(t)$ and $\bold{T}(t)$ are the $3\times3$ rotational matrix 
  and translational components at time $t$ between these coordinate
  systems, respectively.

%   $B$3$3$G!" (B$R(t)$  $B$H (B $\bold{T}(t)$ 
 $B$O$=$l$>$lJ*BN:BI87O$H%+%a%i:BI87O (B
%   $B$H$N4V$N (B $3\times3$  $B$N2sE>9TNs$HJB?J@.J,$G$"$k!# (B
%\end{multicolpar}
%\begin{figure}[htbp]
%\begin{center}
%\psfig{figure=Yachida_figs/ObjectModel.ps,width=6.5cm}
%\end{center}
%\caption{The hand model coordinate system}
%\label{Yac-fig:ObjectModel}
%\end{figure}

%\begin{multicolpar}{2}
  Next, we represent a human hand by the following parameters shown in 
  Figure \ref{Yac-fig:PrivateModel}. 
%
%   $BK\O@J8$G$O?M4V$N<j$r0J2<$K<($9%Q%i%a!<%?$GI=8=$9$k!# (B
%\end{multicolpar}
\begin{list}{}{
\setlength{\leftmargin}{7em}
\setlength{\listparindent}{0em}
\setlength{\itemindent}{\listparindent}
\setlength{\labelsep}{1em}
\setlength{\labelwidth}{\leftmargin}
\setlength{\itemsep}{0em}
\setlength{\parsep}{0em}
}
\item[$\bold{P}_{Oi}(t)$] the location of the feature point $i$ in
  the object-centered coordinate frame at time $t$,
\item[$l_{ji}$] the length of the $j$-th link of the finger $i$,
\item[$W_{ji}$] the width of the $j$-th link of the finger $i$,
\item[$\bold{r}_{Fi}, O_{Fi}$] the location of the tip $\bold{r}_F$ 
  and the bottom $O_F$ of the finger $i$ in the object-centered 
  coordinate frame.
\end{list}
\begin{figure}[htbp]
  \begin{center}
    \begin{tabular}{cc}
        \psfig{figure=Yachida_figs/PrivateModel.ps,width=4cm} &
        \psfig{figure=Yachida_figs/InitialAnalysis.ps,width=4cm} 
    \end{tabular}
    \caption{Illustration of the parameters of the hand shape model
      (a) and the reconstruction of hand shape (b)}
    \label{Yac-fig:PrivateModel}
    \label{Yac-fig:InitialAnalysis}
  \end{center}
\end{figure}

%  $B;X$N%Q%i%a!<%?$N?dDj (B
%\begin{multicolpar}{2}
At the first frame, the user's hand must be placed at the initial
position.  We measure the finger parameters as follows:

%  Fig. \ref{Yac-fig:InitialImage-trans} $B$N$h$&$K=i4|;Q@*$O (B
%   $B%+%a%i$N2hA|LL$KJ?9T$G$"$k$H$9$k!# (B
%   $B0J2<$K3F=hM}$K$D$$$F@bL@$9$k!# (B
%\end{multicolpar}

%\begin{list}{}{
%\setlength{\leftmargin}{1em}
%\setlength{\listparindent}{1em}
%\setlength{\itemindent}{\listparindent}
%\setlength{\labelsep}{1em}
%\setlength{\labelwidth}{\leftmargin}
%\setlength{\itemsep}{0em}
%\setlength{\parsep}{0em}
%}
%\item[1. Determination the X axis of the object coordinate system.] \hfill \\

\begin{description}
\item[Determination the X axis of the object coordinate system:]
first, we calculate the centroid and the moment axis of the binarized
hand region.  The moment axis of the binarized hand
region is regarded as the X axis of the object coordinate system.
At this point, the direction of the X axis is not determined.
%\item[1.  $BJ*BN7O (B X $B<4$N7hDj (B] \hfill \\
% $B<j$NFsCM2=NN0h$N=E?4$H47@-<g<4$r5a$a$k!# (B
% $B47@-<g<4$rJ*BN7O$N (B X $B<4$H$9$k!# (B
% $B$3$N;~E@$G$OJ}8~$O7hDj$5$l$F$$$J$$!# (B
%\item[2. Generation of the finger tip and finger bottom pairs.]
%\hfill \\ 
\item[Generation of the finger tip and finger bottom triplets:]
we find a triple of one convex feature point and two concave feature
points along the contour of the hand. % (. Fig. \ref{Yac-fig:InitialAnalysis}). 
The middle point between two concave feature points is the
bottom of the finger and the convex feature is the tip of the finger.
Then, we determine the length and the direction of this finger from
the line connecting this middle point and the finger tip.
%\item[2.  $B;X@h (B- $BIU$1:,AH$N@8@. (B] \hfill \\
% $BFLFCD'E@ (B( $B;X@h8uJd (B) $B$H!" (B
%X $B<4$NJQF0NL$NId9f$,H?E>$9$k1zFCD'E@$r (B
% $BNX3T$K1h$C$FN>B&$K#2$D8+$D$1$k!# (B(cf. Figure \ref{Yac-
fig:InitialAnalysis})
% $B1zFCD'E@$NCfE@$rIU$1:,8uJd$H$7!";X$N<4$ND9$5$HJ}8~$r7hDj$9$k!# 
(B
%\item[3. Extraction of the finger.] \hfill \\
%\item[4. Finding the thumb.] \hfill \\
\item[Finding the thumb:]
we regard the tip-bottom pair which is nearest from the
centroid of the hand region as the thumb. 
%\item[4.  $B?F;X$N7hDj (B] \hfill \\
% $B$5$i$K!";X@h (B- $BIU$1:,AH$N$J$+$G!"J?9T;X72$K6a$/!" (B
% $B<4$NCfE@$,=E?4$K6a$$$b$N$r?F;X$H$9$k!# (B
%\item[5. Determination of each finger] \hfill \\
\item[Determination of each finger:]
then, the tip-bottom pairs are considered as the fore finger,
middle finger, ring finger, little finger in order of the
distance from the thumb. 
%\item[5.  $B3F;X$N7hDj (B] \hfill \\
% $BCj=P$7$?;X8uJd$KBP$7!"?F;X$HF10lJ}8~$G?F;X$K6a$$=g$K!" (B
% $B?M;X$7;X!"Cf;X!"Lt;X!">.;X$H$9$k!# (B
%\end{list}
\end{description}

%\begin{figure}[htbp]
%\begin{center}
%\psfig{figure=Yachida_figs/InitialAnalysis.ps,width=6.5cm}
%\end{center}
%\caption{Illustration of the reconstruction of hand shape}
%\label{Yac-fig:InitialAnalysis}
%\end{figure}

%  $B?dDj$7$?7k2L (B

\subsubsection{Motion and Position Estimation From Monocular Sequence
  of Images}

%\begin{multicolpar}{2}
  In general, we cannot estimate translational components of motion
  because of an inherent depth/speed scaling ambiguity in monocular
  vision.
  To overcome the scaling ambiguity, our system makes a shape model of
  the user's hand from the input image put at a known initial position.
  This is a reasonable approach for the human interface, so as not to
  give the users any discomfort. 

%   $B0lHL$KC14c$G$O (B depth/speed scaling 
 $B$N[#Kf$5$K$h$j!"1?F0$NJB?J@.J,$r (B
%   $B5a$a$k$3$H$O$G$-$J$$!# (B 
%   $B$3$NLdBj$KBP=h$9$k$?$a$K!"%7%9%F%`$N5/F0;~$K!" (B
%  
 $B=i4|0LCV$K$"$k%f!<%6!<$N<j$N2hA|$+$i%b%G%k$r@8@.$9$k$3$H$G2r7h$
7$?!# (B
%  
 $B$3$l$O%f!<%6!<$r@)8B$7$J$$$H$$$&$3$H$G!"%R%e!<%^%s%$%s%?!<%U
%'%$%9$H (B
%   $B$7$F$O%j!<%:%J%V%k$J%"%W%m!<%A$G$"$k!# (B

  We assume that the initial position of the user's hand (${\bold
    T}(0)$ and $R(0)$) is given in advance.
  Then, $\bold{P}_{Oi}, l_{ji}, W_{ji}, \bold{r}_{Fi}, O_{Fi}$, and
  $\phi_{ji}(0)$ can be measured %by the Initial Analyzer of this system
  when the user places his hand at a certain distance from the camera. 

%   $B=jDj$N1|9T$-$K%f!<%6!<$,<j$r$*$/$3$H$G!" (B$\bold{P}_{Oi}, l_{ji},
%  W_{ji}, \bold{r}_{Fi},  O_{Fi}, \phi_{ji}(0)$  $B$N7WB,$r9T$&!# (B
%   $B$J$*!"<j$NJ?$N=i4|0LCV!"=i4|;Q@* (B $\bold{T}(0), R(0)$ 
 $B$OM?$($i$l$F$$ (B
%   $B$k$b$N$H$9$k!# (B

%  After obtaining these initial values, the system estimates the
%  following parameters for each succeeding frame.

%  
 $B$3$l$i$N%Q%i%a!<%?$,F@$i$l$?8e!"%7%9%F%`$O0J2<$N%Q%i%a!<%?$r0J
8e$N%U (B
%   $B%l!<%`$G?dDj$9$k!# (B 
%\end{multicolpar}

%\begin{list}{}{
%\setlength{\leftmargin}{7em}
%\setlength{\listparindent}{0em}
%\setlength{\itemindent}{\listparindent}
%\setlength{\labelsep}{1em}
%\setlength{\labelwidth}{\leftmargin}
%\setlength{\itemsep}{0em}
%\setlength{\parsep}{0em}
%}
%\item[$\bold{T}(t), R(t)$] the position of the hand at time $t$.
%\item[$\dfrac{d\theta_j(t)}{dt}, \dfrac{d\bold{T}(t)}{dt}$]
%the motion of the hand at time $t$.
%\item[$\phi_{ji}(t)$] the angle of the joint $j$ of the finger $i$ at
%  time $t$.
%\end{list}

%\section{Position estimation of fingers}

%\begin{multicolpar}{2}
  It is not easy to estimate the finger position analytically 
  because the degree of freedom of the fingers is too high.
  Therefore, to estimate the finger position, we use
  the energy minimization method by using model matching.
  To use the energy minimization method we must define an
  evaluation function. 
  The evaluation function $E_{total}$ consists of {\em the feature
    position constraint} $E_F$, {\em the contour constraint} $E_C$,
  and {\em the finger position constraint} $F_B$.
%
%   $B;X$N1?F0$O<+M3EY$,9b$/2r@OE*$K5a$a$F$b@:EY$,4|BT$G$-
$J$$!#$=$3$GK\%7 (B
%  
 $B%9%F%`$G$O%(%M%k%.!<:G>.2=$NJ}K!$K$h$j7A>u%b%G%k$H$N%^%C
%A%s%0$r9T$& (B
%   $B$3$H$G;Q@*$r?dDj$9$k!#%(%M%k%.!<I>2A4X?t (B $E_{total}$ 
 $B$G$OFCD'E@$N0L (B
%  
 $BCV94B+!"NX3T@~94B+!";X$NAjBP0LCV94B+$N#3$D$N94B+>r7o$r@_$1$k!
# (B
%\end{multicolpar}
%
%\begin{multicolpar}{2}
  We estimate the angle $\phi_0, \phi_1, \phi_2$ and $\phi_3$ by
  minimizing $E_{total}$.
  
%   $B$=$7$F!" (B  Fig. ()  $B$K<($94X@a3QEY (B $\phi_0, \phi_1, \phi_2, 
\phi_3$  $B$r (B 
%  $E_{total}$  $B$r:G>.2=$9$k$3$H$G5a$a$k!# (B 
%\end{multicolpar}

%\subsection{Evaluation functions}
\paragraph{Evaluation functions}

%
%  $BFCD'E@0LCV94B+ (B
%
%\begin{multicolpar}{2}
  Let $\bold{r}_i$ be the position of each feature point $i$ on the
  image coordinate system and $\bold{m}_i$ be the position, which
  corresponds to $\bold{r}_i$, on the model coordinate system.
  {\em The feature position constraint} is defined as follows:
%
%   $B3FFCD'E@ (B $\bold{r}_i$ 
 $B$K4X$7$F!"$=$l$KBP1~$9$k%b%G%k7O0LCV$r (B
%  $\bold{m}_i$  $B$H$9$k!#FCD'E@$N0LCV94B+$O0J2<$N<0$GI=$5$l$k!# (B
%\end{multicolpar}
\begin{equation}
  \label{Yac-eqn:featureconstraint}
E_F = \frac{1}{2n}\sum_{i}^{n}(\bold{r}_i-P\bold{m}_i)^T
K_F(\bold{r}_i-P\bold{m}_i)
\end{equation}
%\begin{multicolpar}{2}
  where $K_F$ is the weight matrix, and $P$ is the projection function
  from the model coordinate system to the image coordinate system.

%   $B$3$3$G!" (B$K_F$  $B$O=E$_9TNs!" (B $P$ 
 $B$O%b%G%k7O$+$i2hA|7O$X$N<LA|$G$"$k!# (B

  The above equation (\ref{Yac-eqn:featureconstraint}) is minimal when the
  projected position of the model point and the position of the
  feature point are equal.

%   $B$3$N<0$O%b%G%k7O$N0LCV$N2hA|LL>e$G$NEj1FE@$H (B
%   $B<B:]$NFCD'E@$,0lCW$7$?;~$K:G>.$H$J$k!# (B
%\end{multicolpar}
%
%  $BNX3T@~94B+ (B
%
%\begin{multicolpar}{2}
  {\em The contour constraint} indicates if the contour of the
  model and the edge extracted from the real image are matched.
  Let $\bold{n}$ be the normal vector on the location $c$ on the 
  contour $C$ generated from the shape model.
  {\em The contour constraint} is expressed by the following equation:
%
%   $BNX3T@~94B+$H$O<B:]$N2hA|$+$iF@$i$l$?NX3T@~$H (B
%   $B%b%G%k$+$i@8@.$5$l$?NX3T@~$,BP1~$7$F$$$k$+$r<($9!# (B
%   $B7A>u%b%G%k$+$iF@$i$l$k2hA|>e$K$*$1$kNX3T@~$r (B $c$ $B!" (B
%   $BNX3T@~>e$N0LCV (B $dc$  $B$NK!@~%Y%/%H%k$r (B 
$\bold{n}(dc)$  $B$H$7!" (B
%   $BNX3T@~94B+$O0J2<$N<0$GI=$5$l$k!# (B
%\end{multicolpar}
\begin{equation}
E_C = - \frac{1}{L}\int_C|\frac{\partial}{\partial\bold{n}}
G\star I|^2 dc
\end{equation}
%\begin{multicolpar}{2}
  where $L (= \int_C dc)$ is the total length of the contour.
  This equation is minimal when edges actually exist on the generated
  contour. 

%   $BNX3T@~$ND9$5$r (B $L = \int_c dc$  $B$H$9$k!# (B
%  
 $B$3$N<0$O%b%G%k$+$i@8@.$5$l$?NX3T@~>e$K<B:]$K%(%C%8$,$"$C$?>l
9g$K!" (B
%   $B:G>.$K$J$k!# (B
%\end{multicolpar}
%
%  $B;X$NAjBP0LCV94B+ (B
%
%\begin{multicolpar}{2}
  {\em The finger constraint} binds the fingers not to move freely.
  {\em The finger constraint} is given by the following equation:
%
%   $B;X$NAjBP0LCV94B+$O$=$l$>$l$N;X$NIU$1:,$,<+M3$K1?F0$G$-
$J$$$h$&$K (B
%   $B94B+$9$k!# (B
%   $B;X$NAjBP0LCV94B+$O0J2<$N<0$GI=$5$l$k!# (B
%\end{multicolpar}
\begin{equation}
E_B = \frac{1}{2}\sum_{i\not= j}
\left\{(\bold{r}_i-\bold{r}_j)-(\bold{r}_i^0-\bold{r}_j^0)
\right\}^TK_B\left\{(\bold{r}_i-\bold{r}_j)-({\bold
    r}_i^0-\bold{r}_j^0) 
\right\}
\end{equation}
%\begin{multicolpar}{2}
  where $\bold{r}_i$ is the bottom position of the finger $i$, and
  $\bold{r}_i^0$ is the bottom position, which was already estimated
  by the Initial Analyzer, at the first frame.

%   $B;X$NIU$1:,0LCV$r (B $\bold{r}_i$  $B$H$9$k=i4|0LCV (B $\bold{r}_i^0$ 
 $B$G=i4| (B
%   $B2hA|2r@O$+$iF@$i$l$k!# (B
%\end{multicolpar}

%\subsection{Search algorithm}
\paragraph{Search algorithm}

%\begin{multicolpar}{2}
  We use a hill-climbing method to look for the minimal solution of
  each finger. 
  The system ends the search if the iteration count is greater than a
  given value or the amount of the change of the energy cost function is
  less than a given threshold value.
  
%  
 $BC5:w$K$O;3EP$jK!$rMQ$$$k!#8=:_$N%Q%i%a!<%?$N6aK5$N%(%M%k%.!<
$rI>2A$7!" (B
%  
 $B:G5^9_2<J}8~$r7hDj$9$k!#;XDj$5$l$??t$@$1C5:w$9$k$+!"%(%M%k%.!<4X?t
$N (B
%   $BJQF0$,ogCM$h$j>.$5$/$J$C$?>l9g!"C5:w$r=*N;$9$k!# (B
%\end{multicolpar}

\subsubsection{Experimental Result}

%  $B<B837k2L:\$;$h$& (B
We show an experimental result on a real image sequence.
We used 80 frames, some of which are shown in Figure \ref{Yac-fig:yi-sequence}.
The left-top image is the 1st frame and the right-bottom image is the
80th frame.
The estimated shape model and position are superimposed
on the input image.

\begin{figure}[htbp]
  \begin{center}
    \leavevmode
    \psfig{figure=Yachida_figs/Result1.ps,width=8cm}
    \caption{The estimated shape model and position are superimposed on
      the input image.} 
    \label{Yac-fig:yi-sequence}
  \end{center}
\end{figure}

\subsection{Manual Alphabets Recognition Using Colored Glove}
%\subsection{ $B%+%i!<%0%m!<%V$K$h$k;XJ8;z$NG'<1 (B}
\label{Yac-sec:yi-coloredglove}
%  $B8-$A$c$s$N8&5f (B

In the previous section, we described the method for correct
measurement of hand pose. 
That method is not good for real time applications.
In this section, we describe a method for real time recognition of
hand gestures with saving of image processing time.

% $BA0@a$N<B83$G$O!"%b%G%k$rMQ$$$F@53N$J7WB,$r9T$J$C$?!# (B
% $B$7$+$7!"%b%G%k$N:GE,2r$rC5$9;~4V$,$+$+$j (B
% $B%j%"%k%?%$%`$N$h$&$J%"%W%j%1!<%7%g%s$G$O8=<BE*$G$O$J$$!#
 (B
% $B$3$N@a$G$O2hA|=hM}$N;~4V$r@aLs$7!"9bB.$KG'<1$9$kJ}K!$K (B
% $B$D$$$F@bL@$9$k!# (B

\subsubsection{Expression of manual alphabets}
%\subsubsection{ $B%+%i!<%0%m!<%V$K$h$k;XJ8;z>pJs$NI=8= (B}
\label{Yac-sec:yi-ExpressionGesture}

We show the colored gloves used in our work in
Figure \ref{Yac-fig:ColoredGlove}. 
Each finger is divided into two regions by a PIP joint.
Thus four fingers are divided into eight regions.
The thumb is divided into two regions by an IP joint.
We call each of the four finger regions and the thumb region ``finger
regions''.  
The hand without fingers is divided into two regions: a wrist part and
the palm.
In total, the hand is divided into twelve regions.
We paint all twelve regions different colors, so that we can easily
find each region. 

\begin{figure}[htbp]
  \begin{center}
    \leavevmode
    \psfig{figure=Yachida_figs/ColorGloveRegion.ps,width=5.0cm}
    \caption{The colored glove used in our experiments}
    \label{Yac-fig:ColoredGlove}
  \end{center}
\end{figure}

The image features of a hand gesture acquired from a colored glove
are shown in Figure  \ref{Yac-fig:features}.
We calculate the following features:
\begin{enumerate}
\item the areas of the 10 finger regions,
\item the 10 vectors between the finger regions and the wrist part. We
  call these vectors ``wrist vectors'',
\item the 5 vectors between the base of the finger and the tip of the
  same finger. We call these vectors ``finger vectors'',
\item the 10 vectors between the tips of the fingers. We call these
  vectors ``tip vectors'',
\end{enumerate}
We use the distance and angle of a vector as features, so we get two
features per vector.
In total, 60 features are measured per hand gesture.
When the area of a region is approximately zero, the features related
to this region are treated as invalid.

\begin{figure}[htbp]
  \begin{center}
    \leavevmode
    \psfig{figure=Yachida_figs/colorglovefeature.ps,width=5.0cm}
    \caption{Features acquired from real images}
    \label{Yac-fig:features}
  \end{center}
\end{figure}

\subsubsection{Recognition of Manual Alphabets by Decision Tree}
%\subsubsection{ $B7hDjLZ$K$h$k;XJ8;z$NG'<13X=, (B}
\label{Yac-sec:yi-learning}

Hand gestures are recognized by a decision tree made from the image
features described in section \ref{Yac-sec:yi-ExpressionGesture}.
We use the decision tree, as opposed to other pattern recognition
methods (such as the nearest neighbor method), as it is efficient
using only a few selected important features. 

We automatically make the decision tree using an improved version of
the ID3 method.
ID3 is a method for constructing a decision tree from positive
examples which include pairs of features and values.
We use the image features, described in the previous section, as ID3's 
attributes.
ID3 classifies the example set $C$ into clusters $C_i$ by the threshold value
$v_i$ of the feature $i$ which maximizes the gain of information. 
The ID3 method takes no account of the distance between clusters in
classification, so the robustness grows worse as noise in the data
increases. 

We redefine the evaluation function $F$ for feature selection as the
following equation:
\begin{eqnarray}
  F & = & E \times D \\
  E & = & - \sum^{patterns}_{k=1}P_{S,C_k}log_2P_{S,C_k} -
  \sum_{v_j \in Range(A_i)} \frac{|S_{ij}|}{|S|}I(S_{ij}) \nonumber
\end{eqnarray}
where $E$ is the gain of information, $D$ is the normalized
distance between the two clusters, $P_{S,C_k}$ is the frequency of the
pattern $C_k$ in the example set $S$, and $S_{ij}$ is the subset
consisting of the examples in which the feature $A_i$ has value $v_j$.

In addition to the feature selection, we must determine the threshold
value for dividing clusters for constructing the decision tree.
When one cluster is divided by a certain attribute value,
the number of the divided clusters becomes too large because ID3's
attributes described in the previous section are numerical.
Thus we need to gather attribute values into segments, and to regard 
each segment as one attribute value (like symbolization).
Furthermore, we select only one threshold value to restrict
division of the example set into only two clusters.

With the above idea, the feature and the threshold value are selected
by the following procedure:
\begin{enumerate}
\item for each image feature $F_i$ ($0 \leq i \leq 59$), determine the 
  threshold value $T_i$ by the following procedure: \label{Yac-proc:1}
  \begin{enumerate}
  \item for each hand gesture $j$ ($0 \leq j \leq 25$), make the
    minimum cluster $L_j$ of all examples of the gesture $j$ and
    calculate the 
    maximum value $L_{j,max}$ and the minimum value $L_{j,min}$ of the
    image feature $F_i$ for all examples in the cluster $L_j$.
    When either $L_{j,max}$ or $L_{j,min}$ is not available, the image
    feature $F_i$ is excluded from the selection,
  \item sort the cluster $L_j$ by $L_{j,min}$ in increasing order,
  \item compare $max(L_{k(0),max},...,L_{k(j),max})$ and
    $L_{k(j+1),min}$. If $max(L_{k(0),max},...,L_{k(j),max}$ is
    greater than $L_{k(j+1),min}$, the 
    clusters $k(j)$ and $k(j+1)$ are combined. repeat this procedure from $k(j) 
    = 0$ to 24. (shown in Fig. \ref{Yac-fig:combining}),
  \item when the number of the clusters $N$ is
    \begin{enumerate}
    \item [$N = 1$] the image feature $F_i$ is excluded from the
      selection.
    \item [$N = 2$] the threshold value $T_i$ is given by
      $(L_{0,max}+L_{1,min})/2$ 
    \item [$N > 2$] select $k$ which maximizes the distance between
      two clusters $D = (L_{k+1,min}-L_{k,max})$. the threshold value
      $T_i$ is given by $(L_{k,max}+L_{k+1,min})/2$ 
    \end{enumerate}
  \end{enumerate}
\item calculate the gain of information $E$ and the distance between
  two clusters $D$ and the evaluation value $F$ for each image feature
  $F_i$. Select the $k$-th feature $F_k$ which maximizes the
  evaluation value  
  $F$ and select the threshold value $T_k$ of the $k$-th feature
\item divide the example set into clusters by the selected feature
  $F_k$ and the threshold value $T_k$. If all leaf nodes of the
  decision tree contain examples of the same gesture, the procedure
  ends. Otherwise continue to \ref{Yac-proc:1}.
\end{enumerate}
\begin{figure}[htbp]
  \begin{center}
    \leavevmode
    \begin{tabular}{cc}
        \psfig{figure=Yachida_figs/cluster-integration.ps,width=4.0cm} &
        \psfig{figure=Yachida_figs/threshold-determination.ps,width=4.0cm} 
    \end{tabular}
    \caption{Integration of the clusters (a) and determination of the
      threshold value (b)}
    \label{Yac-fig:combining}
  \end{center}
\end{figure}

\subsubsection{Learning Manual Alphabets by CAD Model}
%\subsubsection{CAD  $B%b%G%k$rMQ$$$?3X=, (B}

Learning from positive examples needs many examples.
It is difficult to make all the necessary examples from real scenes
because it is tedious for humans to make many gestures.
Thus we automatically make the example set using a CAD model.

We added Gaussian noise to the value of image features to take
into consideration the noise of real images, by the
following equation:
\begin{eqnarray}
    P^{\prime}_{ij} & = & P_{ij} + 1.5{\sigma_j}n (area) \\ 
    & = & P_{ij} + {\sigma_j}n (distance, angle) \nonumber
\end{eqnarray}
where $P_{ij}$ is the image feature value, 
$P^{\prime}_{ij}$ is the image feature value
added the Gaussian noise N(0,$\sigma_j$), and $\sigma_j$ is the
standard deviation of the real image feature $j$ of the example sets. 

\section{Body Motion and Gesture Recognition}
\label{Yac-sec:yi-body-motion}
%  $BEOn5$5$s$N8&5f (B

%\begin{multicolpar}{2}
Human body motions are also estimated by the same method for hand
gesture recognition.
The degree of freedom of arm motion is less than that of hand motion.
Hence, we can apply more simple and faster methods for body gesture
recognition than the above methods.
We describe the real-time body gesture recognition by using template
matching. 

% $BBN$NF0$-$NI|85$b<j$NF0$-$NI|85K!$HF1$8<jK!$,MxMQ$G$-$k!# (B
% $B$7$+$7!"<j$N$h$&$KJ#;($J%*%/%k!<%8%g%s$,H/@8$7$J$$$N$G!" (B
% $B$b$&>/$7C1=c$G9bB.$JJ}K!$G4v$D$b$N%8%'%9%A%c!<$r (B
% $BG'<1$9$k$3$H$,$G$-$k!# (B
% $B$3$N@a$G$O%F%s%W%l!<%H$r;H$C$?%8%'%9%A%c!<G'<1K!$r (B
% $B@bL@$9$k!# (B
%\end{multicolpar}

\subsection{Pose Estimation With Template Matching}

\begin{figure}[htbp]
  \begin{center}
    \leavevmode
    \begin{tabular}{ccccc}
      \psfig{figure=Yachida_figs/l_arm110.0000.ps,width=1.0cm} &
      \psfig{figure=Yachida_figs/l_arm110.0001.ps,width=1.0cm} &
      \psfig{figure=Yachida_figs/l_arm110.0002.ps,width=1.0cm} &
      \psfig{figure=Yachida_figs/l_arm110.0003.ps,width=1.0cm} &
      \psfig{figure=Yachida_figs/l_arm110.0004.ps,width=1.0cm} 
    \end{tabular}
    \caption{An example of left arm template models for pose
      estimation with pose numbers.} 
    \label{Yac-fig:left-arms}
  \end{center}
\end{figure}
Our method for gesture recognition is fundamentally
based on template matching technique.
The advantage of the technique is that 
the algorithm is very simple, so that a matching result
can be calculated by small scale hardware system 
like {\em ReMOT-M}~\cite{Yac-Lee94}. 
In addition, since we compare similarities
between a binarized input image and the template models,
the influences caused by different persons or clothing can be ignored.
Figure \ref{Yac-fig:left-arms} shows an example of template models employed 
in conventional template matching to estimate the poses of the left arm. 

In template matching, 
we calculate the disparity between a partial region $R(x,y)$ in an input image
$I$ and a model image $T_{i}$ as the result of the following equation:
\begin{equation}
        d(R(x,y), T_{i}) = \sum_{m,n}|I(x+m,y+n) - T_{i}(m,n)|.
        \label{Yac-eq:disparity}
\end{equation}
In the above equation, $d(R(x,y), T_{i})$ represents the disparity in 
$R(x,y)$ which is a partial region of $I$ whose size is the
same as $T_{i}$,
and $I(x,y)$ and $T_{i}(m,n)$ show intensities of
an input image and a model image, respectively. 
Here we assume that the intensity of $I(x,y)$ and $T_{i}(m,n)$ is 0 or 1,
that is $I(x,y)$ is obtained by binarizing a gray-scaled input image 
by a certain threshold value.
By using $d$, we can obtain the disparity $D$ between an input image
$I$ and a model image $T_{i}$ by searching the smallest $d$ in a
search area $S$ as the following equation:
\begin{equation} 
        D(T_{i}) = \min_{(x,y) \in S}d(R(x,y), T_{i})
        \label{Yac-eq:disparity2}
\end{equation}
where, $S$ represents the search area in the input image $I$.

When the disparity $D$ is calculated for all model images
$T_{0}, \cdot \cdot \cdot, T_{n-1}$,
we can get the best pose number $N$ as
\begin{equation}
        N = Pose(T),
        \label{Yac-eq:pose}
\end{equation}
where T is the $i$-th template model and $i$ minimizes $D(T_{i})$.
The function $Pose()$ gives the pose number shown in
Fig.\ref{Yac-fig:left-arms}, such as $Pose(L_{13}) = 3$. 
From this procedure, we obtain the best pose number.
However, the conventional template matching has some problems
in human gesture recognition.
In the following section, we describe them in detail.


\subsubsection{Problems in Template Matching}


The disparity calculated in the previous section, however, is very 
unstable since binarized input images have a lot of noise such as small
dots and dumpy shapes caused by inadequate threshold values for
various clothes in the binarizing process.
This suggests that to estimate the pose of articulated objects
correctly, the disparity should be calculated only in specific regions
of interest, such as arms or legs in our case. 

The other problem is how to calculate a disparity with odd shaped
templates which are not adequately represented by rectangular
patterns. 
Especially, to avoid interference from invalid regions such as
the background, we should consider the non-rectangular decomposition
of an object.  
For this problem, we usually employ a sequential masking operation.

Figure \ref{Yac-fig:masktemp} shows an example of a masked template.
In the figure, (a) represents a masked template for an arm of the
template model (b). Using mask templates and
template models, the correct disparity in specific region
can be calculated. 

Although we can eliminate the influences of mismatch in template
matching processing by using mask templates, the computation cost
increases.
%By masking and matching process, the computation amount becomes more
%than twice as much as in conventional template matching method.

\begin{figure}[htbp]
  \begin{center}
    \leavevmode
    \psfig{figure=Yachida_figs/masktemp.ps,width=6.0cm}
    \caption{An example of mask template and template model. (a) shows
      an example of mask template, (b) shows an example of a template
      model masked by (a).} 
    \label{Yac-fig:masktemp}
  \end{center}
\end{figure}

\subsection{Maskable Templates}

In order to solve the problems in template matching for pose
estimation, we propose a
method to integrate the target template model and mask template into one
--- a maskable template model. 
Then, by selecting proper template intensities for the region: object
region, 
non-object region and mask region, we can derive that the matching
process can be realized without obvious masking operation.

A maskable template model is generally defined as shown in
Fig.\ref{Yac-fig:masktemp};
a rectangle-shaped template composing of 
an arbitrarily-shaped matching region $\Omega_p$ and 
a masking  area $\Omega_m$.
A matching region $\Omega_p$ is composed of object regions and
non-object regions, and it is the specific region which we should
obtain the disparity.

Assume that the value for region $\Omega_m$ is $V_m$, and $\Omega_p$ 
have  binary values $V_{p0}$ for non-object regions and $V_{p1}$ for
object regions.
With this assumption, equation (\ref{Yac-eq:disparity}) can be rewritten as 
the sum of the disparity of the mask 
region $\Omega_m$ and matching region  $\Omega_p$,
\begin{equation}
  \label{Yac-eq:t_matching2}
  d(R(x,y), T_{M_{i}}) = d_{\Omega_m}(T_{M_{i}}) + d_{\Omega_p}(T_{M_{i}}).
\end{equation}
%\begin{eqnarray}%\label{Yac-eq:rmrp}
%\lefteqn{d(R(x,y), T_{M_{i}})}  \nonumber \\
%& = & \sum_{T_{M_{i}}(m,n) \in \Omega_m}{|I(x+m,y+n)-T_{M_{i}}(m,n)|} 
\nonumber \\
%        &   &    + \sum_{T_{M_{i}}(m,n) \in \Omega_p}{|I(x+m,y+n)
%        -T_{M_{i}}(m,n)|}\label{Yac-eq:t_matching1} \nonumber  \\
%        & = & d_{\Omega_m}(T_{M_{i}}) + d_{\Omega_p}(T_{M_{i}})\label{Yac-
eq:t_matching2}.
%\end{eqnarray}
$d(R(x,y),T_{M_{i}})$ is the disparity we can get by using a maskable
template model $T_{M_{i}}$ and equation (\ref{Yac-eq:disparity}), and 
$d_{\Omega_p}$ is the
disparity in a matching region $\Omega_p$, and
$d_{\Omega_m}$ is the disparity in a mask region $\Omega_m$.

%Input image pixels $I(x,y)$ in a mask region $\Omega_m$ have values
%$V_{p0}$ and $V_{p1}$, so $d_{\Omega_m}(T_{M_{i}})$  can be further
%divided into two parts. 
%If we consider one part is corresponding to those points which have
%the value of $V_{p0}$, and those having the value of $V_{p1}$.
%\begin{eqnarray}
%\lefteqn{d_{\Omega_m}(T_{M_{i}})}  \nonumber \\
%%        & = & \sum_{T_{M_{i}}(m,n) \in \Omega_m}
%%               {|I(x+m,y+n)-T_{M_{i}}(m,n)|} \nonumber \\
%        & = & \sum_{T_{M_{i}}(m,n) \in \Omega_m \wedge I(x+m,y+n) = 
%                V_{p0}}{| V_{p0} - V_m |} + \nonumber \\
%        &   & \sum_{T_{M_{i}}(m,n) \in \Omega_m \wedge I(x+m,y+n) = V_{p1}}{| 
V_{p1} - V_m |}
%\label{Yac-eq:t_matching3}
%\end{eqnarray}
%Here, if we let
%\begin{equation}
%        V_m = \frac{V_{p0} + V_{p1}}{2}\label{Yac-eq:m_value}
%\end{equation}
%and substitute it in equation (\ref{Yac-eq:t_matching3}),we can
%get following result
%\begin{eqnarray}
%\lefteqn{d_{\Omega_m}(T_{M_{i}}) } \nonumber \\
%%& = &    \sum_{T_{M_{i}}(m,n) \in \Omega_m \wedge I(x+m,y+n) = V_{p0}}{| 
V_{p0} - \frac{V_{p0} + V_{p1}}{2} |} \nonumber \\
%%        &   & + \sum_{T_{M_{i}}(m,n) \in \Omega_m \wedge I(x+m,y+n) = 
V_{p1}}{| V_{p1} - \frac{V_{p0} + V_{p1}}{2}|} \nonumber \\
%        & = & \sum_{T_{M_{i}}(m,n) \in \Omega_m \wedge I(x+m,y+n) = 
%                                V_{p0}}{\frac{| V_{p0} - V_{p1}|}{2}}  \nonumber \\
%        &   & + \sum_{T_{M_{i}}(m,n) \in \Omega_m \wedge I(x+m,y+n) = 
V_{p1}}{\frac{| V_{p1} - 
%                                        V_{p0} |}{2}} \nonumber \\
%        & = &   V S_{\Omega_{m(i)}}     
%        \label{Yac-eq:t_matching4}
%\end{eqnarray}
%where
%\begin{equation}
%        V = \frac{|V_{p0} + V_{p1}|}{2}.\label{Yac-eq:m_value2}
%\end{equation}
%$S_{\Omega_{m(i)}}$ is the number of pixels of mask region in the model
%$T_{M_{i}}$.  
As  $V_{p0}$ and $V_{p1}$ are constants, and $S_{\Omega_m(i)}$ can be
calculated beforehand, %the equation (\ref{Yac-eq:t_matching4}) can be
$d_{\Omega_m}(T_{M_{i}}) = K_{i}$ can be expressed as a constant $K_{i}$. 
%\begin{equation}\label{Yac-eq:const}
%        d_{\Omega_m}(T_{M_{i}}) = K_{i}.
%\end{equation}
With this result and equation (\ref{Yac-eq:t_matching2}),
the disparity in the matching region $\Omega_p$ is obtained as
\begin{equation}
        d_{\Omega_p}(T_{M_{i}}) = d(R(x,y),T_{M_{i}}) - K_{i}. \label{Yac-
eq:m_result}
\end{equation}
Finally, from equation (\ref{Yac-eq:m_result}) and equations
(\ref{Yac-eq:disparity2}) -- (\ref{Yac-eq:pose}), we can select the best
model using the disparity in only matching regions.

%From equation (\ref{Yac-eq:m_result}), we must perform the extra subtraction 
%compared with the conventional
%template matching.
%However, in equations (\ref{Yac-eq:disparity2}) -- (\ref{Yac-eq:pose}), 
%if we define the constant $K_{i}$ equal for the all models,
%we can delete the constant $K_{i}$ from equation (\ref{Yac-eq:m_result})
%In other words, we define the values of $V$ and
%$S_{\Omega_m(i)}$ equal for all models.
%So equation (\ref{Yac-eq:m_result}) can be rewritten as 
%\begin{equation}
%        d'_{\Omega_p}(T_{M_{i}}) = d(R(x,y), T_{M_{i}})\label{Yac-eq:m_result1}
%\end{equation}
%where $d'_{\Omega_p}(T_{M_{i}})$ represents the simplified matching result.

%Thus, using maskable template models, 
%the matching result of specific matching region can be obtained 
%within same time as the conventional template matching process.

\subsection{Pose Estimation of Arms Using Maskable Template Models}


When we use a conventional template matching method, it is impossible to
prepare all 
the template models representing every state of a certain gesture.
As shown in Figure \ref{Yac-fig:left-arms}, we prepared a few models
for the motion of raising and lowering arms.
This means that models are very sparsely matched with input image and
if the arm is
placed between two models,  matching will be often unsuccessful; thus we can not
get correct recognition results. 
In order to obtain more detailed and robust pose information 
without adding any template models,
we adopt an interpolation method using maskable template models.

%Fig.\ref{Yac-fig:leftarm}(a) represents an example of maskable template models
%for the motion raising and lowering arms.
%In each model in Fig.\ref{Yac-fig:leftarm}(a), the gray region shows
%mask region. Here, notice that the mask region encloses the white object
%region and all the mask regions are overlapped with the object regions of
%neighboring models as seen in Fig.\ref{Yac-fig:interregion}.
%For example, the mask region in L22 covers the region in which it is possible
%to place an arm 
%of the object region in L21 and L23. 
%Consequently, if the pose is changed in order of L21, L22, and
%L23, we can obtain pliable variation of disparity by adopting a simple
%interpolation method described in the followings.

%Fig. \ref{Yac-fig:interpolation} shows a typical transition of
%disparity $D$ according to pose change. 
Assuming that present pose is
$I_{c}$, then we can obtain  disparity $D(T_{M_{i}})$ with the
maskable template model $T_{M_{i}}$. 
Undoubtedly, the disparity $D(T_{M_{i}})$ has the minimum
value in comparison with the rest disparities estimated from the other
models. 
If the pose changes to pose $T_{M_{i-1}}$, $D(T_{M_{i-1}})$
starts to decrease while $D(T_{M_{i+1}})$ and $D(T_{M_{i}})$ increase.
%(pose number is $I_{b}$ in Fig.\ref{Yac-fig:interpolation}). 
If the pose keeps
changing in same direction, the value of $D(T_{M_{i}})$ becomes equal to
that of $D(T_{M_{i-1}})$ and then we can get  pose number $I_a$.
Ultimately $D(T_{M_{i-1}})$ becomes the smallest value among all other
disparities. Of course, assuming that sufficient number of models are
prepared, namely that pose difference between models is small, we can
get a partially linear loci of disparities as shown in the figure.
Therefore, the pose similar to $T_{M_{i}}$ can be decided by referencing
the some disparities of neighboring model templates.

Consequently, we can interpolate pose between models by comparing
disparity of neighboring models.
%and getting a modification number $N'$ as
%\begin{equation}
%        N' = \frac{P - N}{2*max(P, N)},
%\end{equation}
%where
%\begin{eqnarray}
%        P & = & D(T_{M_{i-1}}) - D(T_{M_{i}}) \\
%        N & = & D(T_{M_{i+1}}) - D(T_{M_{i}}).
%\end{eqnarray}
%\begin{equation}
%        Max(P, N) = \left\{
%        \begin{array}{ll}
%        P & P \geq N\\
%        N & P < N .
%        \end{array}
%        \right.%.
%\end{equation}

%In the equations, $D(T_{M_{i}})$ is the smallest disparity at the
%present time. 
%We just add the number $N'$ to the pose number $N$ in equation
%(\ref{Yac-eq:pose}), 
%and finally get the pose number as the estimated pose.

\subsection{Application: Gesture game system}
In this section, we explain a real-time interactive system 
which can recognize user's gesture
and control a television game by using the recognition results.
Fig.\ref{Yac-fig:game-system} shows the system's overview.
It consists of the following units.
\begin{description}

\begin{figure}[htbp]
  \begin{center}
    \leavevmode
    \psfig{figure=Yachida_figs/system.ps,width=8.5cm}
    \caption{Real-Time Gesture Game System}
    \label{Yac-fig:game-system}
  \end{center}
\end{figure}

\item[Silhouette Generating Unit:]
This unit generates the silhouette image by 
thresholding the intensity difference
between the input image and the background image 
by using a Pipe-Line image processing system, MaxVideo 200(Datacube Inc.).
To avoid flickering due to the user's shadow,
we use a black curtain as a background in our experiment.

\item[Gesture Recognition Unit:]
This unit recognizes the user's gesture with the method described
in the previous section. Template matching is performed with a specific hardware
{\em ReMOT-M} system~\cite{Yac-Lee94}, and pose numbers of the arms
and legs for every image frame are obtained in real-time. 
A sequential pose transitions are interpreted as a gesture and control
codes are encoded and then transmitted to the {\em Game Manipulation Unit}.

\item[Game Manipulation Unit:]
This unit emulates the button operation of a television game
by a personal computer. It receives the control code and
translates it to the practical button operation in sequence.
\end{description}

%Input images with user's gesture is captured by CCD camera 
%and sent to the {\em Silhouette Generation Unit}.
%It generates silhouette images and also extracts 2-D position data of
%an object. 
%The images and positions are transmitted to {\em Gesture Recognition
%  Unit} via video cable and Ethernet, respectively.
%The position information is used to roughly determine initial search
%area and to recognize the position of user. 
%The {\em Gesture Recognition Unit} produces pose numbers for every
%input image and estimates the gesture.
%The specific pose changes are recognized as the gesture, 
%and the recognition results and the user's position information are 
%encoded into operation codes for a game and transmitted 
%to the {\em Game Manipulation Unit} via Ethernet.
%The {\em Game Manipulation Unit} receives the gesture code 
%and the position information code, and 
%sends the actual button operation to the game set 
%which is connected to a personal computer through an interface board.

\section{Network of Active Cameras}
\label{Yac-sec:yi-synthesized}
%  $BLyFb$N8&5f (B

%\begin{multicolpar}{2}
Conventional methods for human gesture recognition are 
individually developed in separate fields.
These methods assume that human faces are
always taken closely from the front of human.
and often fail a gesture recognition in real dynamic scenes. 

% $B=>Mh8&5f$O8D!9$NMWAG5;=Q$r8&5f$7$F$$$?!# (B
% $B$7$+$b!"$=$l$i$O@5LL$K8~$$$?%/%m!<%:%"%C%W2hA|$r (B
% $B2>Dj$7$F$$$?$N$G!"B>$N;Q@*$N>l9g$K$OIT8~$-$G$"$C$?!# (B

For the above reason, human-machine interfaces which processes all
gestures at the same time are not constructed yet.
We, therefore, have made a active camera system to observe a user from 
different directions by using multiple active cameras.
%
% $B$b$7!"A4$F$N>pJs$r0lEY$K3MF@$G$-$l$P!" (B
% $B$h$jM-8z$J%$%s%?%U%'!<%9$,9=C[$G$-$k$O$:$G$"$k$,!" (B
% $B>e$N$h$&$JM}M3$GIT2DG=$G$"$C$?!# (B
% $B$=$3$G!"2f!9$OJ#?t$NG=F0E*%+%a%i$rMQ$$$F (B
% $B?MJ*$rB?J}8~$+$i4QB,$7$F!"?M4V$NF0:n$rB*$($k (B
% $B%7%9%F%`$r;n:n$7$F$$$k!# (B
%
% $B$=$N$h$&$JB?4c$GG=F0E*%+%a%i%7%9%F%`$rMQ$$$k$3$H$G (B
% $B%*%/%k!<%8%g%s$N1F6A$r<u$1$J$$J}8~$+$i$N4Q;!$d!" (B
% $B6/D44QB,$K$h$k8zN(E*$J4Q;!!"Cm;kNN0h$N$_$N=hM}$K$h$k9bB.2=!" (
B
% $B$^$?!"A4J}0L$N%$%a!<%8%;%s%5$K$O$J$$9b2rA|EY$rF@$k$3$H$,$G$-
$k!# (B
%
% 
 $B2f!9$O!"FC$KJ#?t$N%+%a%i$K$h$k?MJ*$N6(D44QB,$K$D$$$F8&5f$r9T$J$
C$F$$ (B
% 
 $B$k!#$3$3$G$O!"?MJ*$N8!=P$HDI@W!"6(D44QB,$K$h$kLr3d$NJQ2=$K$D$$
$F@bL@ (B
%  $B$9$k!# (B
%\end{multicolpar}
%
%\subsection{Configuration of active cameras}
%
%\begin{multicolpar}{2}
% $B%"%/%F%#%V%+%a%i%7%9%F%`$N%O!<%I%&%'%"9=@.$r (B
% $B?^ (B\ref{Yac-fig:active-system} $B$K<($9!# (B
% $B%"%/%F%#%V%+%a%i$N%=%U%H%&%'%"9=@.$r (B
% $B?^ (B\ref{Yac-fig:} $B$K<($9!# (B
%
Fig. \ref{Yac-fig:active-system} shows the hardware configurations.
%Fig. \ref{Yac-fig:software-system} shows the software configurations.
%\end{multicolpar}
\begin{figure}[htbp]
  \begin{center}
    \leavevmode
    \psfig{figure=Yachida_figs/ActiveCamHW.ps,width=6.5cm}
    \caption{Hardware configuration of Active Camera System}
    \label{Yac-fig:active-system}
  \end{center}
\end{figure}

%\begin{figure}[htbp]
%  \begin{center}
%    \leavevmode
%    \caption{Software configuration of Active Camera System}
%    \label{Yac-fig:software-system}
%  \end{center}
%\end{figure}

\subsection{Autonomous Tracking of Human}

%\begin{multicolpar}{2}
We use color information for detecting the human in a scene.
The tracking objects are quickly detected by referencing
object-non-object color tables.
All active cameras greedily trace the objects in the field of view
by changing the focus, zoom, direction, and iris.
If a camera finds that some object can't be traced, 
that camera requests help from another camera via a communication
network. 
Once this negotiation is established, the roles of cameras are
instantly changed.

% $B?MJ*$N8!=P$K$O?'>pJs$rMQ$$$k!# (B
% $BDI@W$9$kJ*BN$OJ*BN (B/ $BHsJ*BN?'%F!<%V%k$r;2>H$7$J$,$i (B
% $B9bB.$K8!=P$5$l$k!# (B
% $B$9$Y$F$NG=F0E*%+%a%i$O%U%)!<%+%9!"%:!<%`!"J}8~!"%"%$%j%9$r
 (B
% $BJQ$($J$,$iM_D%$C$FDI@W$9$k!# (B
% $B$b$7!"%+%a%i$,J*BN$rDI@W$G$-$J$$$H$-$E$$$?$i!" (B
% $B%M%C%H%o!<%/$rDL$8$F1g=u$rMW5a$9$k!# (B
% $B$3$N8r>D$,@.8y$9$k$H%+%a%i$NLr3d$O=V;~$KJQ$o$k!# (B
%\end{multicolpar}

\subsection{Dynamic Role Assignment Among Active Cameras}

The differing roles of active cameras are {\em locking target,
  searching target,} and {\em idle}.
The number of cameras tracing the same target is fixed at a certain
value given by a higher module such as face detection unit, 
hand motion estimation unit, or body gesture recognition unit, 
since it is meaningless to trace only one object by all cameras.
So each camera trades the right of tracking via the lossless
network and changes it's role when these events occur:
a target is lost, a target is found, or when a call for help is
requested. 
  

\section*{Acknowledgment}

We have summarized the recent research works (shown below) done by 
Man-machine interaction group of Laboratories of Image Science \&
Technology and Computer vision \& image media group of Department of
Systems Engineering, Osaka University.  
We thank Haiyuan Wu, Takeshi Fukumoto, Satoshi Kimura, Ken Watanabe,
Takahiro Watanabe, and Tsutomu Yabuuchi for their original
contributions~\cite{Yac-Watanabe95,Yac-Kimura95,Yac-Wu95,Yac-Chen95,Yac-
Chen95b,Yac-k-wata95-E}
and Nels Benson for checking the English.



%\bibliographystyle{plain}
%\bibliography{eccv-ws}

%\end{document}


