Conclusions

We have presented an integrated system for detecting, modeling and tracking faces in real-time. The system uses detection to automatically initialize a tracking system and to re-initialize upon failure. The tracking system uses a feedback approach to stabilize 2D correlation-based trackers by recovering structure from motion and constraining structure with learned 3D facial geometry. Adaptive Kalman filtering is used to weight features by determining a mapping between 2D spatial tracking accuracy and textures and correlation residuals. The system achieves greater stability under 3D variations, occlusion and local feature failure since a global estimation framework links the individual trackers by acquiring the underlying 3D structure of the face. The system is demonstrated on live video sequences where it tracks large out-of-plane rotations stably.

We are currently investigating more sophisticated representations of the 3D model of facial structure to better constrain the structure from motion problem. In particular, it is possible to place the model's structural parameters (i.e. the coefficients of the eigenspace) directly into the EKF as parameters in its internal state vector. This would replace the current estimation and post-processing of point-wise depth structure. The linearization in the EKF would be performed on our eigenspace of 3D heads directly and would be used to form the Jacobians for the estimation of internal state.