This paper describes a real-time, fully-dynamic, 3-D person tracking system that is able to tolerate full (temporary) occlusions and whose performance is substantially unaffected by the presence of multiple people. The system is driven using 2-D blob features observed in two or more cameras [2,22]. These features are then probabilistically integrated into a fully-dynamic 3-D skeletal model, which in turn drives the 2-D feature tracking process by providing 2-D projections of the 3-D model predicitions.
The feedback between 3-D model and 2-D image features is a recursive framework similar to an extended Kalman filter. Previous attempts at person tracking have utilized generic, mid-level image features (such as edges) that are computed as a preprocessing step, without consideration of the task to be accomplished. Our application is unusual, because the framework directly couples raw pixel measurements with an articulated, dynamic model of the human skeleton: no part of the system is blindly feed-forward. In this aspect our system is similar to that of Dickmanns in automobile control [6], and we obtain similar advantages in efficiency and stability though this direct coupling.
This paper will begin by describing our formulation for driving a 3-D skeletal model from 2-D probabilistic measurements, and then address the problem of incorporating feedback from the 3-D model to the 2-D feature finding process. Finally, we will report on experiments showing an increase in 3-D tracking accuracy, insensitivity to temporary occlusion, and the ability to handle multiple people.