%\documentstyle[cup7b,david,epsf]{cuphci}

%\begin{document}

%\bibliographystyle{plain}

\author[Christoph Maggioni and Bernhard K\"{a}mmerer]{Christoph Maggioni and 
Bernhard K\"{a}mmerer\\SIEMENS AG, Central Research and Development\\
Otto Hahn Ring 6, 81730 Muenchen, Germany\\
Tel.: (+49)-89-636-42573 / 45310\\
E-mail: Christoph.Maggioni@zfe.siemens.de, Bernhard.Kaemmerer@zfe.siemens.de}

\chapter{GestureComputer -- History, Design and Applications}

 
 
 
 
\begin{abstract} 
 
We describe a video-based system,  
called GestureComputer, for  
contactless man-machine interaction.  
The system is able to detect human  
hand gestures and head movement  
in real-time. The software runs on  
state-of-the-art workstations without  
the need for special hardware and  
requires only a moderate amount of  
the available computational power,  
thus leaving room for the "real" application. 
We discuss the motivation,  
design ideas, underlying algorithms,  
applications, and product ideas of the  
GestureComputer system. Usability  
tests show the benefits obtained. 
The present paper describes the  
system set-up and different  
approaches to grey-value and colour  
segmentation. We explain contour  
and feature computation and derive   
ways to object selection.  The  
segmentation of human head as well  
as the recognition of the hand and  
different gestures are outlined.  
Supporting approaches for system  
calibration, definition of tracking  
regions, and data filtering are  
considered. Realizations on different  
platforms are the basis for  
applications that find their way to  
future products. Included are areas in  
medicine, virtual reality,  
communication, and harsh  
environments. The paper closes with  
results of a usability study and an  
outlook to further improvements for  
extended functionality. 
\end{abstract}
 
\section{Introduction} 
 
 
\subsection{It all Started in the  
Bathtub} 
 
In autumn 1991 Christoph was lying  
in his bathtub and relaxing when his  
girlfriend came in. She asked how  
many chocolate-bars he would like to  
eat. For some reasons he did not  
want to talk and just showed her the  
hand gesture for the number two.  
Suddenly the idea came to him - why  
not building a vision-based computer  
system that is able to recognise  
human hand gestures. Christoph did  
some literature work.  Recognizing  
that he was not the first person with  
this idea, he convinced the colleges  
at SIEMENS, built a prototype system,  
and final got the approval for a  
research project. The rest of the story  
will be told on the remaining pages. 
  
\begin{figure}
\centerline{\epsfxsize=5cm \epsffile{Maggioni_figs/fig1.eps}}
\caption{(a) GestureComputer Set-up. (b) Human-human interaction}
\label{Mag-Fig.1and2}
\end{figure}
 
\subsection{Motivation} 
 
Many existing and potential domains  
of computer applications suffer from  
input and output devices that are not  
well suited for the task at hand. For  
example visualisations of three  
dimensional structures are difficult to  
manipulate and control by using  
keyboard or mouse commands.  
Careful investigation of human  
interactions soon reveals the  
important role of hand gestures for  
communicating with other people and  
for manipulating three-dimensional  
objects (Figure \ref{Mag-Fig.1and2}b). 
   
 
Hauptman \cite{Mag-Hau89} stated that there  
are intuitive, common principles for  
gesture communication forming a  
communication channel equally  
accessible to all computer users. The  
human hand is a very skilled input  
device able to coordinate many  
degrees of freedom at the same time.  
Thus it is naturally suited to control  
complex applications, which  
otherwise would have to be adjusted  
by a variety of control panels \cite{Mag-Fel92}.  
As a consequence, three dimensional  
hand gesture recognition will form a  
very natural and easy to use  
computer interface \cite{Mag-Fol87,Mag-Wol88}.  

\begin{figure}
\vspace{5cm}
\caption{(a) Manipulation.~~~(b) Holographic display}
\label{Mag-Fig.3and4}
\end{figure}

Besides the control of applications  
the visualisation of 3D-data becomes  
more and more an urgent challenge.  
Conventionally, stereo images are  
used to give a three-dimensional  
perception of an object rendered onto  
a two-dimensional screen. To supply  
the different images head mounted  
displays with two screens or special  
glasses (red-green or shutter)  
allowing a time-sequential  
presentation with just one display are  
used. Both techniques require the  
user to wear something in addition, to  
care for expensive and sensitive  
equipment and to cope with  
difficulties if he needs glasses by  
himself. 
A new approach to improve depth  
perception is called {\em virtual  
holography} \cite{G-Azarbayejani:93}. The position of  
the human head is tracked and the  
image on the computer screen is  
updated accordingly to the changes  
in line of sight. This creates the  
illusion of the screen being a window  
into a three-dimensional world where  
the user may �look behind" objects or  
�look around the corner". 
  
Those findings led us to focus  
research on novel input devices that  
utilise head and hands to control  
complex computer applications (Figure \ref{Mag-Fig.3and4}). 
Now this is not a unique  
idea since systems for hand-gesture  
recognition using the DataGlove  
\cite{Mag-Zim87} or the determination of the  
3D-position employing Polhemus  
sensors are well known. In contrast to  
those solutions we emphasised non-% 
immersive and non-obstructive  
devices that do not hinder users and  
therefore increase the acceptance for  
their deployment in every-day tasks.  
As a consequence the decision was  
to use video cameras that observe  
head and hand movements relieving  
users from cables, gloves, helmets,  
or even whole suits. 
Previous approaches to use a  
computer vision system as an input  
device for computer applications in 
clude the Videodesk \cite{G-krueger90}, the  
Mandala system \cite{Mag-Vin91}, the  
DigitalDesk \cite{Mag-Wel91} and GEST  
\cite{Mag-Seg92,G-rehg94b}. All these systems  
have the limitations of dealing with  
only two-dimensional gestures,  
requiring a controlled uniform image  
background, and using a lot of  
computational power and special  
hardware.  
 
A comprehensive and more recent  
overview on video-based gesture  
recognition systems can be found in  
\cite{G-Huang95}. A collection of relevant  
articles on the topic is available in  
\cite{G-afgr95}. Further reading on the  
GestureComputer project was  
published in \cite{Mag-Wir93,Mag-Sch93,Mag-Mag93a,Mag-Mag93b,Mag-
Ahm93,Mag-Mag94,Mag-Ahm93,Mag-Mag95a,Mag-Kammerer95}. 
 
We use the video based system as  
an additional input device to new or  
existing computer systems. From an  
industrial viewpoint any new input  
device has to be cheap, since it has  
to compete with existing low-cost  
solutions that already found their way  
into products and do more or less suit  
the needs. In addition, performance,  
functionality, and robustness have to  
clearly show advantages that allow  
the company to spend money for  
costly developments. 
With the advent of powerful  
multimedia workstations video input  
is available with no additional cost.  
This focuses our task on design and  
realisation of smart algorithms that  
reduce the enormous amount of  
video data down to handsome num
bers that can be processed in real-% 
time, even leaving the possibility to  
run applications on the same  
machine (Figure \ref{Mag-Fig.5}). 
  
\begin{figure}
\vspace{5cm}
\caption{Required data reduction}
\label{Mag-Fig.5}
\end{figure}

Taking all requirements into  
consideration we developed a  
computer system that is able to work  
in real-time on a standard workstation  
under noisy and changing  
environmental conditions, and that  
detects the three-dimensional  
position and orientation of the human  
hand as well as the position of the  
head.  
 
\subsection{A View on the System Set-up} 
 
Figure \ref{Mag-Fig.6} shows our proposed system  
structure that involves a classical  
feedback loop. Images of the user  
and his hand are acquired by CCD-% 
cameras. Using image processing  
techniques, the human hand is  
recognised. Its position and  
orientation in 3-D space are  
computed and optionally the  
bendings of the fingers are calculated  
as well. Gestures are recognised and  
the interaction of the user's hand with  
the application is determined. Finally,  
a three-dimensional model of the vir 
tual hand is rendered on the screen.  
  
\begin{figure}
\vspace{5cm}
\caption{GestureComputer overview}
\label{Mag-Fig.6}
\end{figure}

When moving the real hand, the user  
sees the corresponding movement of  
the virtual hand on the screen.  
Human hand eye co-ordination  
allows the user to control the system  
even if the image processing is  
somewhat inaccurate. 
Since our system operates in a  
feedback loop, timing is a crucial  
issue. If the total time needed for the  
whole loop is longer than 1/10 of a  
second the user will likely be  
confused and overcompensate,  
bringing the system to an unstable  
state \cite{Mag-War94}. 
The image processing system has to  
satisfy conflicting demands. On one  
hand it has to be reliable, stable and  
insensitive to changes in the image  
background and lighting conditions.  
On the other hand it has to be suffi
ciently fast. Since the CCD camera  
provides an enormous amount of  
data we have to use some ingenuity  
to arrive at a real-time system. 
The physical layout of the system  
and the position of the camera  
depend highly on the application. A  
natural place to put the camera is on  
top of the monitor facing the user,  
where it can also be used for video-% 
conference applications. However,  
for applications requiring extensive  
use this is not a very good set-up.  
Without support for the elbow, raising  
the hand all the time tires the arm  
(similar problems are encountered in  
DataGlove applications). The system  
configuration we use most  
frequently is to fix the camera above  
the monitor looking down onto the  
desk. The user can accomplish 2-D  
operations by moving the hand on  
the desktop surface, whereas 3-D  
gestures can be done by raising the  
hand. Thus we combine the benefits  
of the 2-D mouse with those of 3-D  
manipulation. Other possible set-ups  
include placing the camera on the  
ceiling, on a wall or even inside the  
keyboard looking upwards (see  
Section 3). For co-operative work a  
set-up where the monitor is mounted  
horizontally and the camera is  
looking down observing the hand  
movement and gestures of multiple  
users may be useful. 
 
\section{Approaches} 
 
   
\begin{figure}
\vspace{5cm}
\caption{(a) Grey-value image with marker. (b) Colour hand image}
\label{Mag-Fig.7and8}
\end{figure}

We have chosen two different  
approaches to detect the position and  
gestures of the human hand and  
head. Both share common building  
blocks that will be described in detail  
in the following chapters. 
 
 
The first one works on grey-level  
images and uses a cotton glove with  
an attached marker to speed up and  
simplify the image processing task  
(Figure \ref{Mag-Fig.7and8}a). The marker simplifies the  
image processing by providing a well-constrained object to be recognized  
and tracked for continuous hand  
position determination. 
The second approach takes a colour  
image as input. The image is  
segmented to find patches of human  
skin colour by using a very fast  
colour classifier (Figure \ref{Mag-Fig.7and8}b). 
 
   
Figure \ref{Mag-Fig.9} shows the building blocks of  
the GestureComputer system. They  
are used in different combinations  
and will be described in the next  
chapters. Using these blocks we are  
able to detect the marker in grey-% 
level images as well as human hand  
and head in colour images. 
 
  
\begin{figure}
\vspace{4cm}
\caption{Building Block}
\label{Mag-Fig.9}
\end{figure}

 
 
\subsection{Image Calibration} 
 
The image is transferred into  
computer memory by using standard  
computer frame-grabber hardware. 
The image acquisition system has to  
be calibrated to the geometrical set-% 
up, the lens distortion of the camera,  
and the lighting conditions.  
A special calibration pattern with  
black, white and coloured patches is  
either shown to the camera or  
projected onto the workspace by  
using a video projector. 
  
A feature extraction algorithm  
determines the position and colour  
values of the different patches.  
Calibration of the geometrical set-up  
and lens distortions is done by using  
Tsai's method \cite{Mag-Tsa87}. Frame-% 
grabber parameters, due to  
camera gain, shutter speed, white  
balance, and colour saturation, are  
adjusted using the colour values of  
specific patches for an incremental  
match to predefined values. 
 
 
\subsection{Creating a Binarised  
Image} 
 
To reduce the amount of data and to  
allow an efficient computation of  
image contours we first convert the  
image into a binary representation. In  
case of a grey-value image  
binarisation is straightforward. When  
using colour images we apply a  
colour classifier to do the  
binarisation. 

\begin{figure}
\vspace{5cm}
\caption{(a) Binarised grey-level input. (b) Skin colour
classification (hand)}
\label{Mag-Fig.10and11}
\end{figure}

The grey-level image of the glove  
with marker is binarised by applying a  
global threshold (Figure \ref{Mag-Fig.7and8}, Figure \ref{Mag-Fig.10and11}). 
The knowledge of the form of the  
marker pattern and the positional  
information computed in a later step  
is used to calculate an updated  
threshold for the initial binarisation  
value from the observed grey-values  
of the image. This is necessary due  
to both changing overall lighting  
conditions and local effects like  
shadowing the back of the hand  
when moving it away from the light  
sources.  

\subsubsection{Colour Based System}
  
\begin{figure}
\vspace{5cm}
\caption{Skin colour classification (hand)}
\label{Mag-Fig.12}
\end{figure}


In case of colour images we use a  
classifier that decides if an object has  
skin colour, thus performing the  
binarisation step. 
To improve speed we generate a  
look-up table that is addressed with  
the RGB-values of an image pixel.  
The values stored in the table-cells  
are the result of the colour  
classificator. We have built a tool that  
enables the user to interactively  
generate this table. In a real-time  
video window the user specifies  
positive and negative examples of  
image regions with skin and  
background colour. 
A positive/negative classificator is fed  
with this information, identifies  
regions in colour space that have  
skin colour and constructs the look-% 
up table \cite{Mag-Sch93}.  
Experiments have shown that the  
system is insensitive to moderate  
changes in lighting. The colour  
segmentation is valid for a wide  
range of human skin types without  
need for retraining the colour look-up  
table (Fig.\ref{Mag-Fig.10and11}b, Fig.\ref{Mag-Fig.12}).  
  
\begin{figure}
\vspace{5cm}
\caption{Skin colour classification (head)}
\label{Mag-Fig. 12}
\end{figure}

\subsubsection{Improvements Through a Background Image}

By using a background image the  
performance and quality of the above  
segmentation step can be greatly  
improved.  
When starting the system we copy  
the first image into a background  
store. Whenever the binarisation of a  
pixel has to be done the pixel is first  
compared to the corresponding  
background pixel. If they differ by a  
certain threshold the above  
binarisation is performed. Otherwise  
the pixel is regarded as belonging to  
the background and therefore non  
skin-coloured. 
The update of the background store  
is done by using information from the  
object recognition process. All  
objects that have been identified as  
skin-coloured but do not belong to  
head or hand regions are copied into  
the background store. The regions  
that have been hidden by head or  
hand and become visible when the  
user moves are copied into the  
background as well.  
 
 
\subsection{Finding Object Contours} 
 
The result of the above processing  
steps is a binarised image. The  
contours of the image objects are  
detected using a contour following  
algorithm similar to that of Pavlidis  
\cite{Mag-Pav82}. The algorithm scans the  
image line by line to find a jump in  
the pixel value, follows the contour  
and stores the chain-code in a list.  
Since all contours are closed, the  
process will eventually reach the  
starting point and scanning of the  
image continues. By using knowledge  
about the minimum size of the hand  
or head in the image, we can greatly  
speed up the scanning process by  
only sampling a grid of lines that is  
small enough to touch the desired  
objects in all cases. 
Suppose the interesting objects have  
a minimum height being 1/10 of the  
image height. The number of  
necessary pixel operations is  
computed as 1/10 times the image  
height for scanning a line plus the  
number of contour points. This is far  
below the total number of image  
pixels and is one key to the excellent  
performance of the algorithm. 
 
\subsection{Computing Moments as  
Object Features} 
 
The recognition of objects  
independent of their position, size,  
and orientation is an important goal  
in image processing and methods  
have been developed that are based  
on moment invariants \cite{Mag-Hu62,Mag-Li91}.  
The two-dimensional moments of  
order p, q of an area A computed in a  
discrete, binarised image with image  
coordinates x, y is defined as  
  %%%%
In specific m0,0 is the area of A and  
(m1,0 /m0,0 , m1,0 /m0,0) is the center of  
mass. The number of operations  
needed to compute the moments  
straightforward is proportional to the  
number of points in A. However,  
Green's theorem allows us to reduce  
the amount of calculations by an  
order of magnitude just by following  
the border of the area \cite{Mag-Li91}. Let A be  
a simple connected domain and B the  
boundary of A in the clockwise direc
tion, the moment calculation is  
equivalent to: 
  %%%%
 
We use the outline of each area  
computed in the contour following  
step to calculate the first three  
moments (m0,0,m1,0,m0,1 ) as well as  
some other object features.  
Note that the moments calculation  
step can be done together with the  
contour following thus improving  
algorithmic speed. 
 
 
\subsection{Object Recognition} 
 
\subsubsection{Marker Based Approach}

In case of a grey-level image we use  
a marker based approach to detect  
the human hand wearing a marked  
glove. The contours and moments  
computed in the previous step are  
used to detect the marker that can be  
characterised by the following: 
 
\begin{itemize}
\item There are two objects that are circular  
with approximately the same centers. 
\item The outer circle is white and the inner  
one is black. 
\item The circle size has to be in known up-% 
per and lower bounds. 
\item We know the ratio of circle sizes. 
\item We know the previous position of the  
marker in the image. 
\end{itemize}

\begin{figure} %%%%
\vspace{4cm}
\caption{Geometrical properties of the marker}
\label{Mag-Fig.13}
\end{figure}

The previous constraints are strong  
and misclassifications are very  
unlikely. Having detected the marker,  
we compute the parameters of  
position and orientation in space. The  
position in x and y direction is given  
by the center of mass derived from  
the moment calculation. The distance  
from the camera is a function of the  
known real marker size, camera  
parameters such as focal length, and  
the observed area of the marker in  
the image. 
The rotational parameters can be  
derived as follows.: the line between  
the centers of the two circles P1 and  
P2 of the marker gives us the rotation  
of the hand along the z axes. Due to  
rotation around the x and y axes, a  
marker circle in general appears as  
an ellipse (Figure \ref{Mag-Fig.13}). The major and  
minor axes M, N of an ellipse can be  
computed from the central moments  
and are used to determine the  
rotation around the x and y axes  
\cite{Mag-Zho89}. Note that we have  
ambiguities in the sign of the rotation  
around the x and y axes. In practice  
we can neglect this problem and  
constrain the hand rotations only to  
positive values. 

\subsection*{Colour Based Approach}

\subsubsection*{Detecting a Human Head}

The head detection algorithm  
computes the position of the human  
head in the image plane (x, y, z). We  
use the contour and moment data  
computed in the previous step as  
input to the colour based head  
detector. 
First, the contour data is searched for  
the object with the biggest area that  
fulfils certain size constrains and that  
matches the form of a human head  
(e.g., is elliptical). Sometimes it may  
happen that the head is split into  
several disconnected contour  
objects. This might happen in cases  
with spotlights on the skin, when the  
user wears glasses or has a beard.  
By using a connected components  
algorithm we merge the different  
parts belonging to the head object  
into one new object. 
The shape of the new head object is  
further analysed and the center of the  
forehead is determined by using a  
histogram approach that can deal  
with segmentation errors (Figure \ref{Mag-Fig.14}).  
The distance of the head from the  
camera is roughly estimated by  
calculating the width of the head  
area. 
  
\begin{figure}%%%%
\vspace{4cm}
\caption{Human head detection}
\label{Mag-Fig.14}
\end{figure}

\subsubsection*{ Detecting a Human Hand and its Gestures}

The hand recognition system is able  
to detect the position of the palm in  
the image plane (x, y), the distance to  
the camera (z) and six static hand  
gestures based on finger  
configurations. The contour data  
computed in the previous steps form  
the input of the hand, fingertip, and  
gesture recognition process. 
The contour data is searched for a  
hand shaped object that fulfils known  
size constrains. To remove jags, the  
contour data is smoothed by using a  
running average algorithm.  
The center of the hand contour is  
determined using the center of  
gravity and a region shrinking  
algorithm. Candidates for fingertips  
are defined as points that form a local  
maximum of distance from the center  
of the hand. To verify the fingertip  
points and to determine the correct  
position of the fingertip we search for  
the longest T-like structure starting at  
the fingertip (Figure \ref{Mag-Fig.15and16}a). 
  
For each fingertip point F all pairs of  
points (P1, P2) to the right and left  
side of the contour having the same  
distance to F are determined. The  
pair with the maximum value of d1 / d2  
where d1 is bigger than a certain  
value is selected as the T-shape that  
best describes the finger. D1 gives  
the length of the finger and d2 the  
width. 
To implement finger tracking over  
time all fingertip points are fed into a  
algorithm based on dynamic  
programming that tries to match the  
fingertips of time t to the fingertips of  
time t-1. Because of the small amount  
of fingertip points and the fact that  
the input is an ordered list dynamic  
programming can be used efficiently (Figure \ref{Mag-Fig.15and16}). 
  
\begin{figure}%%%%
\vspace{5cm}
\caption{Fingertip detection and tracking}
\label{Mag-Fig.15and16}
\end{figure}

Finally, the fingertips are matched to  
a model of the human hand and static  
hand gestures are recognised. These  
gestures are defined by the number  
of visible fingers, their orientation  
relative to each other and to the  
palm. Figure \ref{Mag-Fig.17} shows an overview of  
some of the gestures we are able to  
recognise.  
The surrounding rectangle of the  
head or hand region is determined  
and all object pixel outside this region  
are regarded as new background  
pixels used to update the background  
store. 
  
\begin{figure}%%%%
\vspace{5cm}
\caption{Recognisable hand gestures}
\label{Mag-Fig.17}
\end{figure}
 
\subsection{Tracking Regions} 
 
As we are processing real-time image  
sequences we can apply the knowledge  
from the current frame in order to predict  
features of consecutive frames. We use  
motion-estimation techniques to estimate  
the size and position of a region of in 
terest to search for the marker, hand or  
head in the next image. In practical  
experiments this reduces the amount of  
computations by a factor of five.  
 
\subsection{Data Filtering} 
 
In a final step, the position data of the  
head, the palm, and the fingertips are  
filtered by using running average and  
hysteresis filters.  
We have developed a new directional  
hysteresis filter. In contrast to well-% 
known one-dimensional filters with a  
constant hysteresis threshold we use  
a threshold that is weighted with the  
angular difference of consecutive 2-  
or 3- dimensional movement vectors.  
The main advantage of this filter is  
that small movements in similar  
directions are not suppressed.  
Sudden changes in direction that  
may occur in case of light flickering  
or segmentation errors are  
eliminated. 
Knowledge about the movement  
characteristics of human heads and  
hands are used to eliminate sudden  
jumps in the position data.  
 
 
\section{System Aspects and  
Implementation} 
 
 
\subsection{Structure of the  
Software} 
 
The GestureComputer system  
consists of three building blocks: a  
library to perform the hand and head  
recognition, the camera calibration  
and the colour classifier tools. 
The algorithms are implemented in C  
and can be easily configured to  
match the specific demand. The data  
abstraction of the frame-grabber  
interface makes it relatively easy to  
move to a new platform. Currently, a  
port to C++ that will speed up the  
development of new algorithms and  
portability is under the way. 
An Application Programmers  
Interface exists that can be easily  
used. To get the head  
position, for example, the programmer has to call a  
function get-head-position(data). 
All the initialisation of the software is  
hidden and occurs when the function  
is called for the first time. 
 
\subsection{Implementation Platforms  
and Performance} 
 
We have ported the  
GestureComputer software to  
different computer platforms with  
different types of frame grabbers.  
Platforms are SUN, Silicon Graphics  
Indy, SIEMENS RW 410, PC (Win  
3.1 and Win 95) and Apple  
Macintosh.  
Figure \ref{Mag-Fig.18} shows the average  
performance of the head tracking  
algorithm integrated into the Virtual  
Cameraman application (see section  
4.5) on different platforms. 
 
\begin{table}
\begin{tabular}{llll}
Computer & Image & MHz & Frames\\
         & Size  &     & per sec\\
\hline \hline
SGI Indy, Irix 5.3, built in grabber &
640x480 &
150 &
20 \\
\hline Sun Sparc 10, SunOS 4.1, S2200 grabber &
512x512 & 
65 &
15\\
\hline PC - Pentium,  Win 3.1, Spea Showtime &
320x240 &
90 &
16 \\
\hline PC - Pentium, Win 95, Matrox Meteor &
320x240 &
120 &
25 \\
\hline Laptop 486DX2,  Win 3.1, PCMCIA grabber &
120x90 &
75 &
8 \\
\hline Mac Performa 630,  MAC-OS 7.5.2, built in grabber &
320x240 &
66 &
10 \\
\hline 
\end{tabular}
\caption{Performance of HeadTracking} %%%%
\label{Mag-Fig.18}
\end{table}
 
\section{Application Areas and  
Prototypes} 
 
 
\subsection{User Interface} 
 
Virtual reality systems are one of the  
many application areas for video
based hand gesture recognition and  
head tracking. We avoided building 
an immersive virtual reality system  
that aims at the best possible  
simulation of an 3D-world. Such  
systems use complex devices like  
head mounted displays and data  
gloves, but do not allow the user to  
interact with the �real� office world. In  
contrast our goal is to develop  
systems, sometimes called �Desktop  
VR�, that benefit from three
dimensional representations of virtual  
environments but use standard  
computer monitors and non
obstructive input devices. 

Hand Control %%%%

We currently use the hand  
recognition system to navigate  
through three-dimensional worlds  
and to manipulate three-dimensional  
objects. The observer is modelled by  
a virtual body whose position and  
rotation determines his point of view  
and a virtual hand that is used to  
manipulate objects.  
 
  
 
\begin{figure}
\vspace{5cm} %%%%
\caption{Control volume for hand input}
\label{Mag-Fig.19}
\end{figure}

The camera is mounted above the  
table monitoring a certain volume in  
space to the right, left or front of the  
computer screen. When the user is  
performing tasks requiring 2-D input  
the hand can be moved on the  
tabletop like in a conventional mouse  
interface or in Krueger's system  
\cite{Mag-Kru83}. When the user raises the  
hand to make a complex gesture or to  
move objects in 3D space, the 3-D  
mode is automatically entered. As  
long as the real hand is in the inner  
part of the control volume, it controls  
the movement of the virtual hand, but  
when the hand enters the border  
region, the observer's viewpoint  
moves in the corresponding direction.  
We have designed several virtual worlds  
as examples of possible applications. Figure \ref{Mag-Fig.20}
 shows a simulation of our lab at  
SIEMENS, Munich, where one can fly  
around and move objects. The other  
example (Figure \ref{Mag-Fig.21}) deals with a  
marshalling-yard with animated trains.  
The user can fly around, touch objects  
like cars and switches and thus get  
information about them, or switch the  
points by touching them with a special  
gesture and thereby causes the trains to  
change tracks. 
   
\begin{figure}
\vspace{4cm} %%%%
\caption{SIEMENS lab}
\label{Mag-Fig.20}
\end{figure}

  
\begin{figure}
\vspace{4cm} %%%%
\caption{Marshalling-Yard example}
\label{Mag-Fig.21}
\end{figure}

\subsubsection{Head Control}

The head tracking system is used to  
control the viewpoint to three-% 
dimensional graphics on a screen  
forming a virtual holographic system.  
In this set-up a second camera is  
mounted on top of the computer  
screen facing the user.  

\begin{figure}
\vspace{4cm}  %%%%
\caption{Virtual holography - geometry (a) and CT display (b)}
\label{Mag-Fig.22and23}
\end{figure}

The virtual 3D object is situated in  
the center of the solid computer  
screen. Whenever the user changes  
the head position his line of sight is  
changed. This angle alpha is  
measured by the head tracking  
system and the 3D graphic is  
rerendered accordingly to the new  
line of sight (Figure \ref{Mag-Fig.22and23}). If the update  
rate of the system is fast enough  
(more than 10 frames/sec) the user  
has the impression of a real 3D  
object existing inside the volume of  
the monitor. The effect is similar to a  
well-known experience in real life:  
Whenever an object cannot be  
moved we move ourselves around  
the object and our brain forms the 3D  
representation. 
In order to move around the object  
we introduce a scale factor between  
the head movement angle alpha and  
the virtual object rotation beta. 
 
\subsection{Medicine} 
 
Suitable application areas for optical  
gesture recognition systems can also  
be found for diagnosis in medical  
treatment. Doctors will be able to  
observe three-dimensional computer  
tomographic scans of the human  
body, manipulate the viewpoint,  
perform virtual cuts or zooming on  
specific organs by using simple to  
learn hand gestures. Interactive  
planning of surgeries (i.e. in minimal  
invasive surgery) might also benefit  
from a gesture recognition interface. 
  

A potential application using the head  
tracking input device is a substitution  
for the backlit display nowadays used  
by doctors to view x-rays (Fig.\ref{Mag-Fig.22and23}).  
Three dimensional data coming from  
various sources like computer to
mographs, magnetic resonance or  
ultrasound devices are continuously  
becoming more common in medicine.  
We plan to mount a high resolution  
computer screen on the wall to  
display this data. The viewpoint into  
these 3D visualisations can be  
changed very easily by moving the  
head left/right and up/and down. A  
few additional buttons on the display  
or a speech recognition system will  
allow the doctor to stop the head-% 
tracking and to proceed to the next  
data set. 
 
\subsection{Advertisement} 
 
A variation of the above virtual  
holographic system for medicine can  
be used in product placement and  
advertisement (Figure \ref{Mag-Fig.24}). New  
products that only exist as computer  
graphics can be shown to the public  
in a nice way.  
For use in a kind of virtual museum,  
pieces of art will be photographed  
from all sides, compressed (JPEG)  
and stored on disk. The visitor of the  
holographic museum is able to view  
the pieces from many viewpoints by  
moving around the screen. 
 
  
\begin{figure}
\vspace{5cm} %%%%
\caption{Product placement terminal}
\label{Mag-Fig.24}
\end{figure}

 
\subsection{Harsh Environments } 
 
Another potential for gestural control  
can be found wherever an awkward  
physical environment hampers the  
operation of complex systems.  
Gloves or oily hands make using a  
keyboard or touching screens tricky.  
Being able to select functions or  
parameters by pointing instead of  
touching or changing displayed  
information by moving the head could  
improve operations in industrial  
plants. During medical operations  
touchless video-based input devices  
can help to maintain a highly sterile  
environment. 
 
\subsection{Videoconferencing} 
 
Videoconferencing applications suffer  
from two problems: available network  
bandwidth and screen space on the  
computer screen. In a lot of desktop  
videoconferences the participants  
only want to see the faces of their  
partners. Thus a camera with a tele  
lens is often used limiting the mobility  
of the user. Using a wide angle lens  
allows the users to move in front of  
the screen but a lot of unnecessary  
image background is transferred and  
occupies screen space on the op
posite monitor. This is especially true  
in the case of a video conference  
with more than two partners and  
shared applications. 
We have developed a system called  
�Virtual Cameraman" that is based on  
our head-tracking technique and  
overcomes the above problems. We  
are using a camera with a wide angle  
lens but transmit and display only the  
image area that is occupied by the  
users head. This frame smoothly  
follows the users head movements  
thus forming a kind of virtual tele  
camera.  
Figure \ref{Mag-Fig.25} shows the geometrical  
relationship between screen size, full  
video size and the size of the head  
image. Using the Virtual Cameraman  
technique and a standard set-up  
(camera on top the screen) we can  
reduce the amount of data to be  
transferred up to a factor of ten  
without the need for compression.  
 
  
\begin{figure}
\vspace{5cm} %%%%
\caption{Virtual Cameraman}
\label{Mag-Fig.25}
\end{figure}
 
 
\subsection{The Tele Tourist: Looking through the telescope} 
 
For the Telecom fair '95 in Geneva,  
Switzerland, we realised an  
advanced combination of the  
GestureComputer and ATM  
telephone link.  
The GestureComputer served as the  
local viewing station with an intuitive  
user-interface for the selection of  
different camera sites (Figure \ref{Mag-Fig.26}).  
There were six video channels  
available, four from locally stored  
panoramic views, one from a video  
camera positioned at the fair and  
connected by a direct link, and one  
from a camera located on top of a  
building in Lugano, connected via an  
ATM link. The stored images showed  
famous places in Switzerland like the  
Matterhorn, Lac Leman, Uri, or Zurich. 
 
  
\begin{figure}
\vspace{5.5cm} %%%%
\caption{Tele-Tourist application at Telecom 1995}
\label{Mag-Fig.26}
\end{figure}

Both live cameras could be adjusted by high speed servos whereas  
the view at stored images was a window moved electronically over  
the panorama. The GestureComputer system determined the  
position of the visitor's head and selected the viewing direction in  
accordance. Therefore the visitor had the illusion of looking  
through a telescope at landscapes far away. Because of the high  
speed of the position determination and the fast servos, movement  
was smooth and perception very natural. 
Selection of video inputs was even more advanced: simply  
pointing to one of six tiny plastic cameras mounted on a paper map  
of Switzerland switched the system to the respective video  
channel. The pointing gesture was actively recognised and other  
gestures like moving the flat or closed hand over the map were  
suppressed.  
This set-up showed that a purely gesture based control is feasible,  
fast, and ergonomic, and people got used to it in short time. 
 
\subsection{Virtual Touchscreen} 
 
Interfaces to technical systems commonly rely on keyboards,  
switches, mice, or buttons. We propose an extension to  
Wellners \cite{Mag-Wel91} approach to use video projection and  
loudspeaker for the output of information together with speech and  
video based gesture recognition for input. 
The projected display realizes a Graphical User Interface on any  
useful physical surface. The GUI may be designed like a standard  
window-based display or may be tailored to the specific needs of  
the system to be controlled. In this way the projected display  
replaces conventional monitors (CRT, LCD). 
Commands, abstract information, or text may be inputted by  
speech recognition. Special noise reduction approaches help to  
use the system even under difficult environmental conditions. 
In analogy with a touch screen, the input of analog values,  
positions, and selections is realised by pointing with the hand on  
the projected GUI. Through this approach any input hardware like  
keyboard, mouse, traditional touch screen, or digitising tablet can  
be replaced. Pointing gestures and positioning within the GUI is  
recognised with the help of the GestureComputer system. 
The combination of the above mentioned components realises a  
system that is visible only during operation, allocating working  
space with task-oriented assignment just during usage. Because  
the surface under the projection area has just to meet minimal  
requirements regarding reflectivity and shape, any material may be  
chosen from a broad variety of possibilities. Therefore, completely  
new and innovative designs for control can be realised. 
The combination of projection display, speech input, and gesture  
recognition forms an advanced solution for future systems with  
applications e.g., for the "office at the sofa table", in medicine, or for  
Info Kiosks. 
Compared to physical interface modules positioned at the working  
areas following benefits are obvious: 
 
\begin{enumerate}
\item 	no permanent space requirements  
\item  	display and font size easily adjustable by zoom (compares to very  
limited and expensive screen sizes) 
\item  	sterile (e.g., if projection surface consists of high-grade steel) 
\item  	safe against vandalism 
\item  	broad variety of suitable surfaces: (office: desktop with paper;  
household: kitchen table, sofa table; hospital: area besides operative  
field, production: conveyor belt, Info Kiosk: high-grade steel; ...). 
\end{enumerate}

\subsubsection{VT - Office Application}

Often desktops are crowded with electronic equipment like  
telephone, organiser, or computer. The Virtual Touchscreen offers  
the possibility to combine all those functionalities within one  
system. The user interface will be projected on the desktop only if  
requested, leaving documents and other equipment in  
use where they are. Through a verbal command, an incoming phone  
call, or a special gesture, the system is activated. The necessary  
modules may be attached to a lever system (or integrated in a  
desktop lamp) allowing a free choice of placing the GUI on the  
desktop. Because of the large projection area more complex, but  
nevertheless usable interfaces can be realised. 
  
\begin{figure}
\vspace{5cm} %%%%
\caption{Virtual Touchscreen in the office}
\label{Mag-Fig.27}
\end{figure}


\subsubsection{VT - Public Information Terminals}

Public terminals have to be protected against vandalism and  
environmental influences (dust, moisture, ...). This includes the  
protection of the monitor against dirt and damage as well as the  
protection of input devices like mouse or keyboard against theft.  
Through the Virtual Touchscreen approach those influences can  
be minimised. Every part of the system may be mounted under the  
ceiling, unreachable for visitors and optimally protected. Only the  
projection surface has to be made from resistive material that can  
be easily cleaned. 
  
\begin{figure}
\vspace{6cm} %%%%
\caption{Public information terminal}
\label{Mag-Fig.28}
\end{figure}



\subsubsection{VT - Medicine}

During diagnosis and treatment the control of technical equipment  
hinders doctors and raises problems because of sterility. Besides  
that, continuing change of focus between the treatment area and  
the display reduces concentration. 
Both aspects are solved if the interaction area of the Virtual  
Touchscreen is projected next to the treatment area. Suitable  
surfaces for projection may be the skin of the patient, a bright cloth,  
or a steel plate. Using his voice, the doctor activates different modi  
like system control or display options. A pointing gesture at  
projected sliders then enables e.g., to modify contrast or radiation  
power or the selection of sub-areas for focusing. 
  
\begin{figure}
\vspace{5cm} %%%%
\caption{Sterile display and input device}
\label{Mag-Fig.29}
\end{figure}
 
\section{Usability Study} 
 
We performed a usability study on the Virtual Holography aspect of  
the system together with Professor Al Badre from Georgia Tech  
University, USA. 
The overall purpose of the study was to investigate the  
comparative effectiveness of manipulating objects in a non-% 
immersive 3-D space using various controls. In particular, our goal  
was to evaluate the use of the head to control rotation of 3-D  
objects depicted on a computer display in comparison with a direct  
manipulation control device ordinarily used with 2-D surfaces such  
as the mouse as well as with non-direct manipulation devices such  
as the Wheels to affect rotation and viewpoint changing.  
The design for the experiment was a simple 3-way comparison  
between using the mouse only, the combination of mouse and  
head, or the combination of mouse and wheels to control  
movement and viewpoint changing. 
Participants were volunteers from SIEMENS Munich as well as  
students, ages 14 to 18, from an international school at Munich. A  
total of 58 participants were used. 
  
\begin{figure}
\vspace{5cm} %%%%
\caption{Task to accomplish}
\label{Mag-Fig.30}
\end{figure}
 
\subsection{Experiments} 
 
Participants were asked to perform a matching task in a 3D  
environment. Shaped toy-block like objects were presented to the  
user. These objects had to be fitted to matching holes of plate-% 
shaped objects distributed in the environment. Thus the users had  
to make viewpoint changes in order to discover the matching plate.  
Finally, the users placed the object above the correct plate by using  
the mouse. Figure \ref{Mag-Fig.31} is a depiction of the 3D image that the  
participants had to rotate in order to match the object in the lower  
left corner of the picture. 
In both experiments, each participant was assigned to one of three  
main groups. All groups had to place the object by using a mouse  
as a pointing device. Group one was required to control the  
viewpoint changing of the 3-D image with head movement; group 2  
had to control viewpoint changing by using the mouse and thus  
had to toggle between viewpoint change mode and object placing  
mode by pressing a button; group 3 used the mouse to place  
objects and a set of wheels to control viewpoint changing (Fig.\ref{Mag-Fig.31}). 
  
\begin{figure}
\vspace{4cm} %%%%
\caption{Set-up of the usability study}
\label{Mag-Fig.31}
\end{figure}
 
\subsection{Usability Test Results} 
 
In terms of time to perform the task, the school data shows that the  
HeadTracking is the fastest input device for the task. The mouse  
and the wheel are both about 50 % worse (Figure \ref{Mag-Fig.32}).  
In the adults' study, the data shows that the HeadTracking is the  
fastest input device for the task at hand. The mouse comes next,  
with about a 50 %. The wheel leads to the worst performance, being  
about twice as slow as the head tracking. 
 
\begin{table}
\begin{tabular}{|llll|}
\hline
Groups & Count & Average & Variance \\
\hline School participants: & & &\\
\hline
Head  & 19 &95  &  823 \\
Mouse & 18 &141 & 4294 \\
Wheel & 18 &137 & 4501 \\
\hline SIEMENS participants: & & &\\
\hline
Head  & 23 &90  &  784 \\
Mouse & 18 & 133 & 1938\\
Wheel & 6  & 176 & 4167 \\
\hline
\end{tabular}
\caption{Time used to complete the task} %%%%
\label{Mag-Fig.32}
\end{table}

The results clearly show that changing the viewpoint into a 3-D  
scene by using HeadTracking as an input device reduces the  
problem solving time compared to other input devices. 
These results are applicable in most domains of three-dimensional  
interaction. 
 
\section{Summary.  GestureComputer: An idea becomes reality. } 
 
We presented approaches for efficient image analysis to achieve  
real-time gesture recognition and head tracking. Binarisation of  
grey level or colour images and contour extraction serve as basic  
modules for high level object identification. Knowledge about head  
and hand geometries is used for detailed examinations that lead to  
the determination of face areas even under difficult conditions or to  
the extraction of fingertips for gesture recognition. 
The power and robustness of the realised algorithms allow a cost  
efficient implementation on state-of-the-art hardware, opening the  
way to real life applications. Head and hand movement, pointing,  
grabbing - the GestureComputer brings human interaction to  
technical systems. Applications have been identified that make  
usage of the new possibilities and benefit from intuitiveness,  
freedom, and improved usability through the GestureComputer.  
Extensive usability tests have scientifically proven the  
GestureComputer concept. 
The algorithms will be improved to better deal with changing  
lighting conditions and to ease the calibration process. New  
features like eye and mouth detection will be integrated to allow  
more accurate head position determination. Dynamic gesture hand  
gesture recognition will be added in the near future if requested by  
application domains. 
The current redesign phase yields professional software quality for  
product integration. Realization of various parts for  
marketing purposes modules to integrate products are on their  
way to the SIEMENS operating groups. 
 
 
\section*{Acknowledgement} 
 
We want to thank all people that contributed to the  
GestureComputer project: 
Subutai Ahmad, Daniel Goryn, Holger Kattner, Uwe Kusch,  
Christoph Pomper, Hans Rottger, Rolf Schuster, Sven Schroter  
and Brigitte Wirtz. 
 
 

