This paper describes a real-time, fully-dynamic, 3-D person tracking system that is able to tolerate full (temporary) occlusions and whose performance is substantially unaffected by the presence of multiple people. The system is driven by 2-D blob features observed in two or more cameras [1,22] and by behavior models that estimate control signals. These features and controls are then probabilistically integrated into a fully-dynamic 3-D skeletal model, which in turn drives the 2-D feature tracking process by setting appropriate prior probabilities. The intrinsic state of the skeletal model is also used by the behavior module to choose the appropriate control strategy.
The feedback between 3-D model and 2-D image features is an extended Kalman filter. One unusual aspect of our approach is that the filter directly couples raw pixel measurements with an articulated dynamic model of the human skeleton. Previous attempts at person tracking have utilized a generic set of image features (e.g., edges, optical flow) that were computed as a preprocessing step, without consideration of the task to be accomplished. In this aspect our system is similar to that of Dickmanns in automobile control [4], and we believe that we can obtain similar advantages in efficiency and stability though this direct coupling. A second unusual aspect is that the Kalman filter goes beyond passive physics of the body by incorporating various patterns of control (which we call `behaviors') that are learned from observing humans while they perform various tasks.
This paper will illustrate the structure of the behavior system with some simple examples in Section 3. We will then briefly discuss the formulation of our 3-D skeletal model in Section 4, followed by an explanation of how to drive that model from 2-D probabilistic measurements, how to 2-D observations and feedback relate to that model in Section 5. Finally, we will report on experiments showing an increase in 3-D tracking accuracy, insensitivity to temporary occlusion, and the ability to handle multiple people in Section 7.