Feeding Forward SfM to High Level Vision

We can also feed forward the SfM results. Recall that SfM recovers 3D pose (or motion) as well as 3D structure. In the above, the 16 3D points recovered are not detailed enough to generate a 3D graphical model of the face (adding points is too slow, with complexity $\approx O(N^3)$). Instead, 3D pose is fed forward into a module that unwarps the face into a standard mug-shot pose for a second stage where a fast, linear 3D shape estimator can be used (see Jebara, Russel and Pentland [33]), to compute a full 3D model. The result is shown in Figure 14 which is a full 3D model computed in real-time from facial images of the user seen in Figure 11.

Figure: Feeding Forward for Detailed 3D Model Recovery