3D Facial Pose and Directional Illumination Normalization

We begin by considering a set of candidate anchor points for the facial features (eyes, nose and mouth). These may be detected in a variety of configurations. The loci of these feature points gives an estimate of the pose of a face. Unfortunately, not all faces will be facing the camera in a canonical frontal view and this prevents us from using techniques such as eigenspace analysis where correspondance is important. We thus propose to warp a detected face into frontal view using a 3D model of a head.

A 3D range data model of an average human face is formed off-line from a database of range data and is depicted in Figure 5(a). Several Cyberware range models were averaged in 3D to obtain this average head. The eyes, nose and mouth were located on the models and used to align them via a 3D mapping and a vertical stretch into a standard pose. This alignment was done by manually selecting the 4 points and then using a least-squares iterative fit of the 3D anchor points.

Using the computed average 3D model, a Weak-Perspective-3-Points [1] computation can then be used to align its eyes and nose to the ones found in a 2D image. The model is also iteratively deformed by a vertical stretch so that its mouth is also properly aligned with the mouth in the 2D image as shown in Figure 5(c).

  

Figure: Normalizing with 3D Warping and Histogram Fitting

Once the optimal 4-point alignment is found, the 2D image's intensity data is mapped onto the 3D structure that now overlaps it. Thus, the 3D mesh is 'coated' with the appropriate intensity values of the underlying 2D image. This coated 3d model is shown in Figure 5(d). If parts of the 3D structure are occluded due to excessive rotation, we use symmetry to mirror the face intensities across the midline of the 3D structure. The 3D structure is then rotated into a normalized frontal view and projected to form a segmented, mug-shot image of the face. Thus, we generate a frontal, colour mug-shot of the individual from the original image and 4 anchor points corresponding to facial features.

Each side of this new 2D face undergoes histogram fitting to normalize its illumination [11]. Two transfer functions are computed: one for mapping the left half of the face to a desired histogram (i.e. a histogram of a well-illuminated face) and the other for mapping the right half of the face. A weighted mixture of these transfer functions is used as we traverse from the left side of the face to the right side, smoothly removing directional shading of the face. Furthermore, the generation of the transfer functions is windowed to avoid facial hair and head hair so that illumination normalization does not over-brighten mug-shots of bearded men and so on. The fully normalized face is shown in Figure 5(e).