\author[K. Kakusho, N. Babaguchi \& T. Kitahashi]{K. KAKUSHO,
N. BABAGUCHI \& T. KITAHASHI\\Osaka University}
\chapter[A Task-Oriented and Multimodal Approach]
{A Task-Oriented and Multimodal Approach 
toward Recognizing Movements of Humans
--- A Case Study for Social Dancing ---}
%\setcounter{page}{3}
%\setcounter{equation}{7}

\begin{Abstract}
In this chapter, we focus on recognizing movements of multiple humans
from a real image sequence.  Most of the previous work related to this
problem assumes to recover detailed motion description that could be
widely applicable to various cases of human movement recognition, but
only deals with fairly simple motions of a single object such as a
hand or a body.  We take a task-oriented and multimodal approach to
cope with the difficulty of the problem especially for recognizing
complicated movements of multiple humans from a real image sequence.
As our task-oriented approach, we first clarify what kind of movements
are assumed to be recognized, and then focus on the bare minimum of
the information to recognize the assumed movements.  As our multimodal
approach, we employ acoustic data together with an image sequence to
increase the information available in the process of recognizing
movements.  We take social dancing as an example of somewhat
complicated movements of multiple humans, and show that the problem of
its recognition becomes easy enough to solve with simple and
familiar computer vision techniques based on this approach.
\end{abstract}

\section{Introduction}
\label{Kak-Sec: introduction}

From the conventional viewpoint, recognition of a human movement from
its image sequence is realized through two processes.  One, is recovering the 
three-dimensional(3D) posture of a part, or the
whole, of a human body from each frame of an image sequence, and the
other is recognizing motion elements that constitute the whole
movement from the sequence of the recovered 3D postures.  We call
these two processes {\it motion analysis} and {\it motion recognition}
respectively.

Previous work on human movement recognition mainly focuses on the motion
analysis~\cite{Kak-Akita, Kak-Leung, Kak-ORourke, G-rehg94a, Kak-Seki}.  
The work assumes to
recover detailed description of human postures using an articulated
posture model with many degrees of freedom without due consideration
of the information that is actually required to recognize the
movements.  The motion analysis using an articulated posture model
in general requires considering huge numbers of possible
correspondence between the model and an image.  To decrease the
difficulty of the process, most of the previous work assumes a single
body in fairly simple motion, such as walking frontoparallel to the
camera~\cite{Kak-Niyogi, Kak-Rohr}.  Other work assumes an engineered
situation in which a marker is attached to each junction of the
human's body in motion~\cite{Kak-Campbell, Kak-Rashid}.

In this chapter, we take a task-oriented and multimodal approach
aiming at recognizing more complex movements of multiple humans from a
real image sequence for more practical and realistic formulations.
The task-oriented approach was originally proposed as a new paradigm
against Marr's framework for general-purpose vision systems, and its
effectivity was shown in the problems of object recognition with
different tasks~\cite{Kak-Ikeuchi}.  The basic idea of this approach is
that the task to be accomplished specifies the object description to
be recovered from an input image, and a vision system is to be built
so as to acquire that description.  Since human movement recognition
can be viewed exactly as the same problem as model-based object
recognition in the sense that their major difficulty exists in finding
a single correct correspondence between a model and an image
efficiently, this approach is expected to bring a realistic solution
of the human movement recognition problem.

In our task-oriented approach to human movement recognition, we first
clarified what kind of human movements are assumed in the task of motion
recognition, and focused on the bare minimum of information that should
be acquired from an image sequence by motion analysis in order to
accomplish the assumed task.  What is to be recovered from each image
frame in motion analysis is determined by what movement is aimed to be
recognized and what description is required to recognize the movement.
The same detail is not required in recovery of motion description for
recognizing all kinds of human movements.  Although some movements may
not be recognized without their motion description in full detail,
others could be correctly recognized only from their much less
detailed description.  It follows that motion analysis and motion
recognition so closely depend on each other that we cannot discuss
these two processes separately even if we only aim to realize either
one of them.  The interdependency among series of processes is often
pointed out in discussions on other problems of computer vision.  We
discuss both motion analysis and motion recognition from the
task-oriented viewpoint in order to clarify the motion description that is
recovered by motion analysis.

In this chapter, we assume the task of recognizing movements of social
dancing as a typical example of somewhat complicated movements by
multiple humans.  It is not aimed at the task of analyzing
each dancer's precise posture but is aimed at recognizing what dance is
performed by a pair of dancers.  What is primarily required for that
task is the description of the motions of the pair rather than the motions of
each dancer.  In addition, we consider using music as different data
from images in modality in our multimodal approach for the
problem of recognizing social dancing.  Since social dancing is
performed in rhythm with music, that probably provides useful
information about the motions.

In the remainder of this chapter, we will show that our above
expectations for the task-oriented and multimodal approach to motion
analysis and recognition are fulfilled through discussion on a method
of analyzing and recognizing motions of social dancing, given a real
image sequence and an acoustic signal of music.  In Section \ref{Kak-Sec:
Social Dancing}, we will discuss motion elements features which are
useful in discriminating one from another.  In Section \ref{Kak-Sec:
Assumptions}, we describe the assumptions for the problem
in this chapter.  In Section \ref{Kak-Sec: Segmentation}, we will propose
a method to estimate the turning of each motion element of social
dancing by extracting the beat of music from its acoustic signals.  In
Section \ref{Kak-Sec: Recognition by MD and RA}, we will present motion
recognition from rough motion description of the position and the
orientation of a pair of dancers on the floor.  In Section \ref{Kak-Sec:
Recognition by Pivoting Feet}, we discuss motion recognition with
slightly more detailed motion description using the positions of the
male dancer's pivoting feet.  The experimental results of all the
processes described in Sections \ref{Kak-Sec: Segmentation}, \ref{Kak-Sec:
Recognition by MD and RA}, \ref{Kak-Sec: Recognition by Pivoting 
Feet} are presented in \ref{Kak-Sec: Results}.  In section \ref{Kak-Sec: 
Conclusions},
we will discuss the future steps of this work.

\section{Movements of Social Dancing}
\label{Kak-Sec: social dancing}

\subsection{Classification of Motion Elements}
\label{Kak-SubSec: figure classification}

The movement of social dancing consists of motion elements called {\it
figures}.  In this chapter, we define recognition of social dancing as
acquiring a sequence of these figures' names from a real image
sequence.  There are two primal features that are useful to diminish
the figures.  One of them is the direction from the initial position
to the final position of a pair of dancers in a figure based on the
initial orientation of the male dancer in the figure(see Figure \ref{Kak-Fig:
MD and RA}(a)).  We call this feature the {\it movement direction} of
a figure.  The other feature is the angle between the orientations of
the pair at the beginning and the end of each figure (see
Figure \ref{Kak-Fig: MD and RA}(b)).  We call this angle the {\it rotation
angle}of a figure.

\begin{figure}
\begin{tabular}{cc}
\epsfxsize=4.6cm
\epsffile{Kakusho_figs/motion_direction.eps}&
\epsfxsize=3.5cm
\epsffile{Kakusho_figs/rotation_angle.eps}\\
{\small (a)}&{\small (b)}
\end{tabular}
\caption{(a) Movement direction and (b) rotation angle of a figure}
\label{Kak-Fig: MD and RA}
\end{figure}

These two features of the basic figures of waltz and blues are
summarized in Table \ref{Kak-Tab: figure list}, where counterclockwise
rotations and clockwise ones are denoted by positive
values and negative ones respectively .

\begin{table}
\begin{center}
\caption{Basic figures of waltz and blues}
\label{Kak-Tab: figure list}
{\footnotesize
\input{Kakusho_figs/figure_list}
}
\end{center}
\end{table}

\subsection{Combination of Figures}
\label{Kak-SubSec: figure combination}

In the movement of social dancing, all figures can neither precede nor
follow each figure to constitute a proper movement of social dancing.
The basic relation that constrains proper combinations of figures 
comes from the fact that all the steps for performing figures are
always taken by different feet alternatively, as in a gait.  Thus, when
a figure ends with a step by the right foot for example, only the
figures that start with the left foot are allowed as the next step.
Since the feet used for the first step and the last step are specified
by the figures as shown in Table \ref{Kak-Tab: figure list}, we can
restrict the possible figures using this relation as constraints
together with the two features described above.

\subsection{Positions of Pivoting Feet}
\label{Kak-SubSec: pivoting feet}

In order to discriminate the figures that have the same movement
direction and rotation angle and begin with steps taken by the
same foot, we can employ positions of a dancer's pivoting feet in a
figure.  Using black spots, Figure \ref{Kak-Fig: Precise figure models} shows the 
positions of a male dancer's pivoting feet needed to perform the basic 
figures of a waltz.  The origin of the coordinate is at the center of the
initial position of a pair of dancers in each figure, and the
horizontal axis and the vertical axis are set forward and to the right in
the initial orientation of the male dancer in each figure.  
%%%% (awkward sentence.  I wasn’t able to understand enough to 
make adjustments) For
example, CC and WK, (who cannot be discriminated by their rotation
angles and movement directions), together with the feet, used for their
first and the last steps, can be discriminated from each other by this
description.

\begin{figure}
\centerline{\epsfxsize=8cm \epsffile{Kakusho_figs/precise_model.eps}}
\caption{Positions of a male dancer's pivoting feet for waltz}
\label{Kak-Fig: precise figure models}
\end{figure}

\section{Assumptions of the Problem}
\label{Kak-Sec: assumptions}

An acoustic signal of music and a color image sequence are assumed as
the auditory and visual input data.  The camera is fixed at a
configuration with no roll about the optical axis, and only the same
single pair of dancers is in motion in the scene.  We also assume that
the camera parameters, the background image, the colors of the
dancers' costumes, the classification of the dance (waltz, tango,
etc.), and the frequency range of acoustic hardware noises can all be
given.  Most of these assumptions could be satisfied even when we
recognize social dance from video images taken in a
ballroom.

\section{Segmentation of a Movement into Motion Elements}
\label{Kak-Sec: segmentation}

\subsection{Beat Tracking}
\label{Kak-SubSec: beat tracking}

In order to recognize figures in a movement performed by a pair of
dancers, we have to segment the movement into its motion elements to
be recognized as figures.  Since each figure of social dancing is
performed so that it begins and ends on the beats of music, we can get
information about the moments of the beginning and the end of each
figure from the beat of music.  Extracting the beat of music from its
acoustic signal is called {\it beat tracking}, and has been discussed
as one of the key techniques for automatic transcription from acoustic
signals into a music score in the community of music information
processing~\cite{Kak-Desain, Kak-Katayose1, Kak-Katayose2}.  

Since the beat lengh for social dancing in general varies from 0.25 to
0.8 seconds, the beat of music is acquired by finding local peaks 
power with an acoustic signal of the same interval in this
range(Fig.\ref{Kak-Fig: beat tracking}).

\begin{figure}
\centerline{\epsfxsize=6cm \epsffile{Kakusho_figs/beat_tracking.eps2}}
\caption{Tracking next beat}
\label{Kak-Fig: beat tracking}
\end{figure}

\subsection{Segmentation into Figures}
\label{Kak-SubSec: figure tracking}

Movements of social dancing in most cases stop or slow down with the
moments of the figures turning.  We employ the difference in
gray-level between adjacent frames together with the beat of music as
clues to extract those moments for segmentation without motion
recognition.

All the figures listed in Table \ref{Kak-Tab: figure list} are performed
during $3 \sim 6$ beats.  The frames corresponding to the turnings of
figures are detected by finding local minima of the difference between
adjacent image frames with the intervals of $2 \sim 7$ beats
considering errors within one beat (see Figure \ref{Kak-Fig: figure
tracking}).

\begin{figure}[h]
\centerline{\epsfxsize=6cm \epsffile{Kakusho_figs/figure_tracking.eps2}}
\caption{Tracking figures}
\label{Kak-Fig: figure tracking}
\end{figure}

\section{Motion Recognition by Movement Directions and Rotation
Angles}
\label{Kak-Sec: recognition by MD and RA}

\subsection{Recovering the Pair's Position on the Floor}
\label{Kak-SubSec: position recovery}

As mentioned in Section \ref{Kak-SubSec: figure classification}, movement
directions and rotation angles are useful features to discriminate
figures.  They are obtained by accumulating the change in the position
and the orientation of the pair throughout each motion element whose
beginning and end are given by the segmentation in the previous
section.  In order to recover the position and the orientation of a
pair of dancers, the region corresponding to the pair is extracted
from each frame of an input image sequence at first.  We call this
region the {\it human region} of a frame.  Given the background image,
the human region is readily extracted from each image frame as the colors of 
the
pixels are different from those of the pixels at the same %%%% awkward 
sentence below
positions in the background image more than a threshold.  Since the
camera parameters are assumed to be given, a rough position of the
pair on the floor is obtained by projecting the bottom of the human
region of each frame onto the floor.

\subsection{Recovery of the Orientation}
\label{Kak-SubSec: orientation recovery}

In social dancing, the dancers always approximately face each other on
the same floor.  In that situation, their orientations on the floor
can be roughly estimated by the sizes and the positions of the regions
corresponding to each dancer in an image as is illustrated in
Figure \ref{Kak-Fig: estimating orientation}.

\begin{figure}
\centerline{\epsfxsize=7cm
\epsffile{Kakusho_figs/estimation.eps}}
\caption{Estimation of the orientation}
\label{Kak-Fig: estimating orientation}
\end{figure}

Using the information about the color of each dancer's costume, a
human region is segmented into two regions, which correspond to the
male dancer and the female dancer respectively.  We call these regions
the {\it male region} and the {\it female region} of an image.  Since
a fairly rough description is sufficient enough for the rotation angle
to narrow down the candidates of figures as shown in Table \ref{Kak-Tab:
figure list}, we describe the orientation of a pair by one of eight angles
with the interval of $45^\circ$.  This angle is determined by its two
components along with the depth from the camera and with the direction
orthogonal to the depth on the floor.  These two components can be
estimated from the appearance of the male region and the female region
of each image frame.  The depth component is determined from the
difference of the two regions in size.  The other component is
determined from the difference of the positions of the regions'
centroids along with the horizontal axis of the image plane.

\subsection{Recognizing Motion Elements by the Movement Directions and 
the Rotation Angles}
\label{Kak-SubSec: recognition by MD and RA}

The movement direction and the rotation angle of each motion element
is obtained by accumulating the difference in the positions and the
orientations of a pair of dancers over the motion element.  They are
represented by the codes shown in Figure \ref{Kak-Fig: eight codes}.

\begin{figure}
\begin{center}
\begin{tabular}{cc}
\epsfxsize=3cm
\epsffile{Kakusho_figs/8_directions.eps}&
\epsfxsize=3.5cm
\epsffile{Kakusho_figs/8_rotations.eps}\\
{\small (a)} & {\small (b)}
\end{tabular}
\caption{The codes to represent (a) movement directions and (b)
rotation angles}
\label{Kak-Fig: eight codes}
\end{center}
\end{figure}

Making use of the description with the parameter space $(\phi,\theta)$
whose axes denote the movement direction and the rotation angle,
figures are represented as points in the parameter space.  We call
them {\it rough figure models}.  The rough figure models for the waltz are
illustrated in Figure \ref{Kak-Fig: rough figure model}(a).  Each motion
element segmented from an image sequence is recognized by the
chessboard distance of its movement direction and rotation angle from
each rough figure model in the parameter space.  If the distance from
the model of a figure is smaller than a threshold $\Delta$, the figure
is accepted as a candidate of the correct figure(Figure \ref{Kak-Fig: rough
figure model}(b)).

\begin{figure}
\begin{center}
\begin{tabular}{cc}
\epsfxsize=5cm
\epsffile{Kakusho_figs/rough_models.eps}&
\epsfxsize=5cm
\epsffile{Kakusho_figs/distance_rough.eps}\\
{\small (a)}&{\small (b)}\\
\end{tabular}
\caption{(a) The rough figure models for waltz   (b) the area for a
motion element to be accepted as a figure}
\label{Kak-Fig: rough figure model}
\end{center}
\end{figure}

\subsection{Narrowing down Candidates by Relations among Figures}
\label{Kak-Sec: narrowing}

As is mentioned in Section \ref{Kak-SubSec: figure combination}, each
figure has to begin with a step taken by the foot different from that
used for taking the last step of the previous figure.  This relation
has to be satisfied between each pair of adjacent figures throughout a
movement of social dancing.  Thus, even if a figure is included in the
results of the above motion recognition for a motion element, it is
rejected unless the results of motion recognition from its previous and
next motion elements include at least one figure that satisfies this
relation.

\section{Motion Recognition by Pivoting Feet}
\label{Kak-Sec: recognition by pivoting feet}

\subsection{Recovering the Positions of Pivoting Feet}
\label{Kak-SubSec: pivoting feet recovery}

If more than one figure is still included in the result of motion
recognition described in section \ref{Kak-Sec: recognition by MD and RA}
for each motion element, the position of the male dancer's pivoting
foot is recovered to distinguish the most probable figure.  Since we
assume that the camera is fixed with no roll, the sole of a dancer's
foot in contact with the floor makes a horizontal edge appear in the
image, if it is visible.  Furthermore, a pivoting foot is stable on the
floor while the other foot is in motion.  Thus, the edge corresponding
to pivoting feet of dancers are extracted by {\it AND} operation
between horizontal edges extracted from two frames at the interval of
several frames.

The extracted edges includes both of the edges corresponding to the
pivoting feet of the male dancer and those of the female dancer.  They
are distinguished from each other based on the information about the
colors of the dancers' costumes, especially their shoes.  The
positions of the male dancer's pivoting feet are then recovered by
projecting the edges onto the floor in the same way as in Section
\ref{Kak-SubSec: position recovery}.

\subsection{Recognition by the Positions of the Male Dancer's Pivoting
Feet}
\label{Kak-SubSec: recognition by pivoting feet}

The positions of the male dancer's pivoting feet extracted from the
image sequence corresponding to a single motion element are matched to
the description of each figure model based on the positions of a male
dancer's pivoting feet.  The position of pivoting feet of each
model is first transformed so that the first step and the last step
coincide with the positions of pivoting feet recovered from the first
frame and the last frame of a motion element to be recognized.  Then
the positions of pivoting feet recovered from all the other image
frames are evaluated by their deviation from the nearest pivoting feet
of the model.  The figure with the smallest value for the sum of all
these deviations is selected from among the results of motion
recognition in Section \ref{Kak-Sec: recognition by MD and RA}.

\section{Experimental Results}
\label{Kak-Sec: results}

\subsection{Segmentation of a Movement}
\label{Kak-SubSec: segmentation results}

Figure \ref{Kak-Fig: segmentation results}(a) shows an example of the
acoustic signal of music.  Figure \ref{Kak-Fig: segmentation results}(b) is
the power of the signal in each small interval after removing hardware
noises with the range from $5500Hz$ to $9500Hz$.  Based on the power,
the beat shown in Figure \ref{Kak-Fig: segmentation results}(c) is obtained.
Figure \ref{Kak-Fig: segmentation results}(d) shows the normalized difference
between adjacent frames of the image sequence corresponding to the
acoustic signal in (a).  Using this difference together with the beat
in (c), we got the turnings of motion element as shown by the dots in
(d).

\begin{figure}
\begin{tabular}{cc}
\epsfxsize=5cm
\epsffile{Kakusho_figs/signal.eps}&
\epsfxsize=5cm
\epsffile{Kakusho_figs/power.eps}\\
{\small (a)} & {\small (b)}\\
\epsfxsize=5cm
\epsffile{Kakusho_figs/beat.eps}& 
\epsfxsize=5cm
\epsffile{Kakusho_figs/difference.eps}\\
{\small (c)} & {\small (d)}
\end{tabular}
\caption{Results of tracking beats and figures: (a) n acoustic
signal, (b) The power of (a) in each small interval of time, (c)
Extracted beats, (d) The normalized difference of an image sequence}
\label{Kak-Fig: segmentation results}
\end{figure}

\begin{table}
\begin{footnotesize}
\begin{center}
\caption{Evaluation of the segmentation results}
\label{Kak-Tab: segmentation results}
\begin{tabular}{|c|c|c|}\hline
     & Extraction rate & Hitting Rate\\  \hline
Waltz& 0.778 & 0.778\\ \hline
Tango& 0.667 & 0.400\\ \hline
Blues& 1.000 & 0.900\\ \hline
Quick Step& 0.750 & 0.545\\ \hline
Slow Fox Trot & 0.875 & 0.467\\ \hline
\end{tabular}
\end{center}
\end{footnotesize}
\end{table}

We define an {\it extraction rate} as the rate of the number of
correct frames to the number of figures, and an {\it hitting rate} as
the rate of the number of correct frames to the number of extracted
frames.  The extraction rates and the hitting rates for five acoustic
signals of music  (waltz, tango, blues, quick step and slow fox trot)
are shown in Table \ref{Kak-Tab: segmentation results}.  In these experiments,
we regard an extracted frame as a correct frame when its error to the
ground-truth is less than ten frames.

\subsection{Recognition of Figures by Movement Directions and Rotation
Angles}
\label{Kak-SubSec: recognition results by MD and RA}

Figure  \ref{Kak-Fig: image processing results}(c) is the human region in a
frame extracted by difference between an image frame of blues shown in
(a) and the background image in (b).  The positions of the pair of
dancers on the floor is obtained as shown in Figure \ref{Kak-Fig: position
and orientation results}(a).  The initial position and the final
position of the pair in a dance are denoted by the letters {\sf S} and
{\sf E}.  The camera is viewing the pair along with the direction from
the bottom to the top of this page.

\begin{figure}
\begin{center}
\begin{tabular}{cc}
\epsfxsize=5cm
\epsffile{Kakusho_figs/input.ps}&
\epsfxsize=5cm
\epsffile{Kakusho_figs/background.ps}\\
{\small (a)}&
{\small (b)}\\
\smallskip
\epsfxsize=5cm
\epsffile{Kakusho_figs/human.eps}&
\epsfxsize=5cm
\epsffile{Kakusho_figs/pivot_edge.eps}\\
{\small (c)}&
{\small (d)}
\end{tabular}
\caption{(a) An image frame of blues, (b) the background image, (c) the
human region extracted from (a) and (b),  (d) edges corresponding to
pivoting feet}
\label{Kak-Fig: image processing results}
\end{center}
\end{figure}

Figure \ref{Kak-Fig: position and orientation results}(b) shows the
orientations of the pair at each frame, where the values
$0,1,2,3,4,5,6,7$ respectively denotes how many times $45^\circ$ the
pair of dancers rotates counterclockwise from the orientation in which
the male dancer completely occludes the female dancer from the camera.
They are obtained from the differences between the male region and the
female region in position of the centroid along with the horizontal
axis of the image plane and in size (see Figure \ref{Kak-Fig: region and
centroid}.

\begin{figure}
\begin{center}
\begin{tabular}{cc}
\epsfxsize=6cm \epsffile{Kakusho_figs/position.eps}&
\epsfxsize=4cm \epsffile{Kakusho_figs/orientation2.ps}\\
{\small (a)}& {\small (b)}\\
\end{tabular}
\caption{(a) The positions and (b) the orientations of a pair}
\label{Kak-Fig: position and orientation results}
\end{center}
\end{figure}

\begin{figure}
\begin{center}
\begin{tabular}{cc}
\epsfxsize=4cm
\epsffile{Kakusho_figs/centroid2.ps}&
\epsfxsize=4cm
\epsffile{Kakusho_figs/region2.ps}\\
{\small (a)}& {\small (b)}\\
\end{tabular}
\caption{Differences between the male region and the female region (a)
in horizontal components of the centroid, and (b) in size}
\label{Kak-Fig: region and centroid}
\end{center}
\end{figure}

The figures recognized for each motion element based on the rotation
angle and the movement direction are shown in Table \ref{Kak-Tab:
recognition results}.  We employed $\Delta=2$ for the
experiments.

\begin{table}
\begin{footnotesize}
\begin{center}
\caption{Results of Recognizing Figures}
\label{Kak-Tab: recognition results}
\begin{tabular}{|c|l|}\hline
Motion Element & Recognized Figures\\ \hline
\#1 & FQT(0.18), FNT(0.31), CCH(0.41)\\ \hline
\#2 & BQT(0.16), BRT(0.21)\\ \hline
\#3 & FQT(--), \underline{CCH(--)}, \underline{FNT(--)}\\ \hline
\#4 & BNT(0.26), ST(0.17)\\ \hline
\#5 & BQT(0.23), \underline{BRT(--)}, BNT(0.22)\\ \hline
\#6 & FQT(--), \underline{CCH(--)}, \underline{FNT(--)}\\ \hline
\#7 & ST(--)\\ \hline  
\#8 & BQT(--), \underline{BRT(--)}\\ \hline
\end{tabular}
\end{center}
\end{footnotesize}
\end{table}

These figures are narrowed down by their relationships.  For example,
CCH, which ends with a step by the left foot, is removed from the
result for the third motion element because no figure that begins with
the right foot is included in the result for the fourth motion
element.  Similary, the underlined figures in Table \ref{Kak-Tab:
recognition results} are rejected as improper candidates.

\subsection{Recognition of Figures by Pivoting Feet}
\label{Kak-SubSec: recognition results by pivoting feet}

The edges corresponding to pivoting feet of the human region in
Figure \ref{Kak-Fig: image processing results}(c) are acquired as shown in
(d) by {\it AND} operation between horizontal edges in two frames with
the interval of 5 frames.

The positions of the male dancer's pivoting feet for the first motion
element are shown in Figure \ref{Kak-Fig: selection by pivoting feet}(a).
This result is matched to the models of FQT, FNT and CCH, which are
left as the candidates of the first figure in the result of the
process so far.  The models described with the positions of the male
dancer's pivoting feet are shown in (b), (c), (d).  The result of
matching is denoted by the value in the parentheses following names of
figures in Table \ref{Kak-Tab: recognition results}.

\begin{figure}
\begin{tabular}{cc}
\epsfxsize=2.5cm
\epsffile{Kakusho_figs/pivot_position.eps}&
\epsfxsize=3cm
\epsffile{Kakusho_figs/FQT.ps}\\
{\small (a)} & {\small (b)}\\
&\\
\epsfxsize=3cm
\epsffile{Kakusho_figs/FNT.ps}&
\epsfxsize=3cm
\epsffile{Kakusho_figs/CCH.ps}\\
{\small (c)} & {\small (d)}
\end{tabular}
\caption{Recognition of figures by the positions of the male dancer's
pivoting feet,  (a) recovered positions and models of (b) FQT, (c) FNT,
and (d) CCH.}
\label{Kak-Fig: selection by pivoting feet}
\end{figure}

As a result, we get the series of figures, FQT, BQT, FQT, ST, BNT,
FQT, ST, BQT for this image sequence.

We applied the method to 28 image sequences of different social
dancing including waltz, tango, blues, quick step and slow fox trot in
different settings for the costumes of dancers, the position of the
camera and the reflection of the floor.  These image sequences include
208 motion elements, and 178 elements are correctly recognized.

\section{Conclusions and Future Steps}
\label{Kak-Sec: conclusions}

In this chapter, we have shown it is possible to recognize movements
of multiple humans from a real image sequence by formulating the
problem from a task-oriented viewpoint and by employing data with
different modalities in the discussion on recognition of social
dancing.  Social dancing is a complicated movement by a pair of
humans, and it is difficult to acquire detailed descriptions of each
dancer's posture from a real image sequence based on an articulated
human body model.  But, assuming the task of recognizing figures in
the movement, we do not always need to obtain such detailed
descriptions of dancers for recognizing the figures of a dance.  In
many cases, it is possible to recognize figures only from their rough
description.  And, it is also possible to employ the knowledge about
the relations among figures.  From this task-oriented viewpoint, we
focused on the movement direction and the rotation angle of a pair of
dancers, and the positions of the male dancer's pivoting feet on the
floor for the descriptions to recognize figures.  We have shown that
those descriptions are extracted from a real image sequence by simple
and familiar image processing techniques, and that the figures in
various kinds of social dances can be recognized only from those
descriptions.

In addition, since social dancing is performed on the beat of music,
acoustic data of music is another source of useful information.  As an
approach to recognition of social dancing by employing data different
in modality, we proposed the method to segment the movement into
motion elements to be recognized as figures based on an image sequence
and an acoustic signal of music without recognizing each figure.

These methods still only deal with limited kinds of music and figures
of social dancing, but there is no other work that discusses
recognition of movements of multiple humans assuming both a real image
sequence and an acoustic signal as the input.  As the future steps of
this work, we are currently considering the following issues.

\medskip
\noindent {\it 1) Interaction between Auditory Data Processing and
Visual Data Processing}

In our method, a movement is segmented into motion elements with an
acoustic signal of music together with an image sequence.  But, the
process of beat tracking employs no image data.  Since no beat
tracking method can always extract the beat of music correctly, and the
movement of social dancing is performed on the beat, the result of
motion analysis should also be considered in the process of the beat
tracking.  In order to increase the performances of beat tracking and
motion analysis from our multimodal approach, the two processes have
to interact with each other efficiently.

\medskip
\noindent {\it 2) Top-down Processing}

The dataflow in the method proposed in this chapter is currently
completely bottom-up.  But, for taking the full advantage of the
task-oriented viewpoint, a top-down dataflow should be added to our
methods especially when we consider the process to extract more
detailed description of motions to disambiguate the result of motion
recognition for more generalized domain and formulation.  In general,
it is difficult to realize detailed description of complicated motions
correctly from real images without any preliminary information about
objects in motion.  But, after the class of possible motions is
narrowed down from the result of motion recognition with rough
descriptions, it would be possible to design and evoke motion analysis
that is specific for discriminating the motions in that class.  This
process of motion analysis is effective because it is able to employ
an assumption on the class of motions to be considered and to focus on
motion features which may not be suitable to classify all kinds of
motions but enough to discriminate motions in a limited class.  It is
also possible to realize {\it focus attention} by predicting
appearance of the features to be extracted from an image.

\medskip
\noindent {\it 3) Case-based Reasoning on Figure Combinations}

In addition to the relations among figures mentioned in Section
\ref{Kak-SubSec: figure combination}, there are a lot of typical
combinations frequently used for a movement of social dancing.  Since
those combinations are not always used in performing a dance, they
cannot be used as absolute constraints in motion recognition.  But, by
employing each of them as a typical {\it case} of a sequence of
figures to evaluate the probability of each figure, we can further
narrow down the possible figures for each motion element.  These cases
can also be used for predicting the figure that will be performed
next.  Predicting next figures would be very effective when we
consider a real time processing for motion recognition.  It would be
possible to realize this reasoning on the sequence of figures for
their evaluation and prediction by introducing the framework of {\it
case-based reasoning} or {\it case-based planning}~\cite{Kak-Hammond},
which is often discussed in the community of symbolic reasoning.

\section*{Acknowledgment}
This research is partly funded by Grant-in-Aid for Scientific Research
of the Ministry of Education, Science and Culture of Japan.  Hideaki
Uehara, Nobuyuki Matsuoka, Takeshi Suenaga, and Tomotaka Tatsumi helped
with programming and experiments.  Thanks also go to Atsushi Katsuma
and Kayo Suzuki for performing various kinds of social dances.

