Results

**Figure:** Frames on the left show video and 2-D blobs from one camera in the stereo pair. Frames on the right show corresponding configurations of the dynamic model at that instant in time.
$\begin{figure}\centerline{\psfig{figure=figs/skel10.ps,width=70mm}} \centerline{\psfig{figure=figs/skel08.ps,width=70mm}} \end{figure}$
$\begin{figure}\centerline{\psfig{figure=figs/skel10.ps,width=70mm}} \centerline{\psfig{figure=figs/skel08.ps,width=70mm}} \end{figure}$

The dynamic skeleton model currently includes the upper body and arms. Figure 7 shows the real-time response to various target postures. The model interpolates those portions of the body state that are not measured directly, such as the upper body and elbow orientation, by use of the model's intrinsic dynamics and the behavior (control) model. The model also rejects noise that is inconsistent with the dynamic model. Table 2 compares RMS noise in the dynamic model output with noise in the underlying feature tracker. The ``line following'' test measures error from the best-fit line to data produced by constraining the users hand to move along a linear trajectory. The ``rotational jitter'' measures error to a smoothed version of data obtained by smooth motions of the user's hand through a rotation.

Table: Comparison of RMS tracking error for tracking with and without feedback.

	tracker		dynamic model
line following	1.4	cm	0.9	cm
rotational jitter	2.2	deg	0.6	deg

**Figure:** **Top:** frames showing self-occlusion during crossing. **Left:** tracking results without feedback. **Right:** correct tracking when feedback is enabled.

$\begin{figure}\centerline{\psfig{figure=figs/crossmovie190.ps,width=33mm} \psf... ...sfig{figure=figs/logmancross.ps,width=60mm,height=35.5mm} \hfill} \end{figure}$

It can be seen that Figure 8 illustrates another advantage of feedback from higher-level models to the low-level vision system. Without feedback, the 2-D tracker fails if there is even partial self-occlusion from a single camera's perspective. With feedback, information from the dynamic model can be used to resolve ambiguity during 2-D tracking.

**Figure:** Users sharing the workspace. Physical constraints stabilize the 2-D tracker with respect to competing targets.
$\begin{figure}\centerline{\psfig{figure=figs/multi.ps,width=65mm}} \end{figure}$

The model predictions also stabilize tracking by providing constraints that help the tracking algorithm reject distractions in the environment. The addition of another person to the scene, as in Figure 9, produces many patches in the image that are similar to the target blobs. Without high-level model knowledge, the 2-D tracker can only reject these distractions based on some assumptions about the temporal stability of blobs. With the addition of high-level feedback, however, the 2-D tracker now has information about the physical constraints of the underlying system. Consequently, it is generally not distracted by competing targets (such as other people).

**Figure:** The left hand is stationary on the table. The right hand occludes the face. (top-left) Illustration of real motion (top-right) the system without feedback fails. (bottom-left) using an observer to provide feedback prevents the failure (bottom-right) Kalman filtered feedback also prevents the failure.
$\begin{figure} \centerline{ \psfig{figure=figs/occlude.eps,height=3.0in} \psf... ...o.ps,height=3.0in} \psfig{figure=figs/kalmano.ps,height=3.0in} } \end{figure}$

**Figure:** The left and right hands moving in a circular motion, repeatedly occluding each other. (top-left) Illustration of real motion (top-right) the system without feedback fails. (bottom-left) using an observer to provide feedback helps, but hand tracking still often fails. (bottom-right) Kalman filtered feedback prevents the failures.
$\begin{figure} \centerline{ \psfig{figure=figs/rumba.eps,height=3.0in} \psfig... ...r.ps,height=3.0in} \psfig{figure=figs/kalmanr.ps,height=3.0in} } \end{figure}$

Figure 10 illustrates a simple occlusion example, where a hand can occlude the face from the viewpoint of one or both cameras. Without feedback to guide the low-level vision systems, tracking fails. The top-right graph in Figure 10 shows this case. The ambiguity caused by the occlusion caused one camera to mislabel the head and tight hand. This correspondence error causes stereo estimation to fail, resulting in an erroneous head position (second cluster of blue data points) and an erroneous path for the right hand (red data points).

The bottom two plots in Figure 10 show correct tracking when feedback is enabled, either with the observer-based system, or with the Kalman filters.

Figure 11 illustrates a more complex occlusion example, where the hands are moving in circular paths, repeatedly occluding each other simultaneously in both cameras. Without feedback, again, tracking fails. Even though the head is never occluded, the system fails badly enough to require several re-initializations. Ambiguities encountered in this process lead to the head being mislabeled in several instances. The scattering of blue (head position) data points in the top-right graph in Figure 11 is a result of these errors.

The bottom-left graph in Figure 11 shows the observer-based feedback system not faring much better. Compared to the previous case, there are very few head tracking errors. the feedback system has, at least, eliminated the need for constant, bottom-up re-initializations. The observer just can't keep up with the frequent occlusions. It seems to need more time to stabilize its estimates between occlusions.

The bottom-right plot in Figure 11 show correct tracking with the Kalman filter based feedback system. The track paths aren't entirely smooth. They should be smooth given that the actual physical process behind the data was smooth. These aberrations are probably caused by the significant increase in measurement noise associated with occlusions.