INTRO: ----- The broad goal: The goal of the face recognition software is to take an image that contains a person's face, to project the image onto a set of eigenfaces and produce a set of coefficients, to record the coefficients for that image, and to perform recognition by comparing the coefficients from one image to those from others. For information on the directory structure please read DIRINFO. BEGINNING THE LOOP: ------------------ You have an image on disk and you want to process it to produce its set of coefficients. FaceRecInterface contains the necessary code to take an image and compute the eigencoefficients for recognition. /*--------------------------------------------------------------------------- Program Name: faceRecInterface.cc Purpose: The idea of the software is to get an image, normalize it, and project it onto an eigenface basis in order to compute a description of the image. Author: Tanzeem Choudhury Baback Moghaddam -----------------------------------------------------------------------------*/ /*--------------------------------------------------------------------------- Function Name: loadEigen() Purpose: function for loading eigenvectors into memory. Author: Tanzeem Choudhury, Baback Moghaddam ---------------------------------------------------------------------------*/ This function loads the image to memory. Bytes per pixel identifies whether the image is raw chars (bytesPixel =1) or raw floats (btyesPixel=4). Alll the images provided are raw chars. /*--------------------------------------------------------------------------- Function Name: loadKnownFaces() Purpose: function for loading coefficients of known faces into memory. Author: Tanzeem Choudhury ---------------------------------------------------------------------------*/ This function simply loads the eigencoefficients of known faces off the disk from the directory knwon_dir and returns them in the data structure, kfaces. /*--------------------------------------------------------------------------- Function Name: loadEigen() Purpose: function for loading eigenvectors into memory. Author: Tanzeem Choudhury, Baback Moghaddam ---------------------------------------------------------------------------*/ This function simply loads the eigenvectors off the disk from the directory eigdir and returns them in the data structure, eigimages. Neigframe_in is the number of eigenimages to load. /*--------------------------------------------------------------------------- Function Name: loadTemplates() Purpose: function for loading BF format files into memory Author: Tanzeem Choudhury ---------------------------------------------------------------------------*/ This routine loads data from BF format files: dmf%dt%.bf from the datapath and a feature model matrix (in BF) format as well as a geometry_musigma.bf, model.bf and ,ask.bf files /*---------------------------------------------------------------------- FUNCTION: scalewarp() AUTHOR: Dave Becker This function, scalewarp, scales and translates an image so as to place the (x,y) point at (x_dest,y_dest). ------------------------------------------------------------------------*/ The image is now scaled and translated so that the head is in the proper place. However, the face might be tilted or not perfectly aligned. /*---------------------------------------------------------------------- FUNCTION: eigsearch() AUTHOR: Baback Moghaddam, Tanzeem Choudhury ------------------------------------------------------------------------ Local feature search by eigentemplates This routine uses data read in earlier from the following files stored in datapath: 1. features.bf which is an N-by-2 matrix defining the size (row,col) of each of the N templates. 2. template[n].bf where n=1...N is the eigenvector ROW matrix where the 1st row is the mean feature, and the remaining rows are the principal eigenvectors 1:M. NOTE: The eigenvectors are stored in column-order (as in MATLAB) 3. variances.bf which is an N-by-M matrix, where each row represents the rank-ordered eigenvalues associated with that feature 4. windows.bf 5. umap_musigma[n].bf 6. spatial_musigma[n].bf ---------------------------------------------------------------------- NOTE: In this version if unitmap'ing is selected it will use the Mahalanobis mean/stddev in addition to the standard distance computation. ------------------------------------------------------------------------*/ Eigsearch() is picking out _candidate_ feature locations, rather than picking _the_ feature locations (again, see main.ps for more detail). These candidate feature locations are returned to ml_detect(), which picks from between these candidates fthe most likely locations of features. /*---------------------------------------------------------------------- FUNCTION: ml_detect() AUTHOR: Baback Moghaddam, Tanzeem Choudhury ------------------------------------------------------------------------ Feature Geometry Validation using Maximum Likelihood Minimization This routine uses data from BF format files: dmf%dt%.bf from the templates and a feature model matrix (in BF) format as well as a geometry_musigma.bf file And dumps the coods of best feature matches in output data structure ---------------------------------------------------------------------- */ Ml_detect(), basically, compares the geometry of candidate feature locations to the rough geometry that it is known features should lie in, and gives a score corresponding to how well the features match that geometry. /*--------------------EIG -> Load eigentemplate data ------------------*/ variance, unit map, and spatial information (see main.ps). /*--------------------EIG -> windows stuff -----------------------*/ The window.bf file controls how big of an area eigsearch() searches in when looking for features. In other words, eigsearch is handed a head location. It knows that the left eye is sort of off to the left and above the head center, so rather than search the whole image for the left eye, eigsearch() just wants to search in an area around there. This windows.bf file contains the size of the window to use for that search. Now that the feature locations are known, the image is warped again so that the features are perfectly normalized. That is, the eyes are set to be in given locations (the interocular distance is normalized, that is), and the rest of the image is tilted so that the mouths all lie along the same line (the mouth is set to be directly below the midpoint of the two eye locations). This is taken care of in modelwarp_mc() : /*---------------------------------------------------------------------- FUNCTION: modelwarp_mc() AUTHOR: Baback Moghaddam, Tanzeem Choudhury ------------------------------------------------------------------------ This contains code to warp images according to the affine transform between the feature locations indicated and those in the model data structure (from the model.bf file) ---------------------------------------------------------------------- */ The final bit of processing that occurs is that a mask is applied to the image. At this point, the image has been warped so that the face lies in a particular place, with the eyes and mouth also so normalized. The masking is a step to get rid of the effect of the background and of hair. So, another image is simply dot-multiplied to the image being processed. The image (the mask) being multiplied by our image is set to be one where faces are and zero where the background is, with a little bit of smoothing in between. The two images are multiplied, with the result being an image that has a face in the middle and black around the edges. The code to the do the masking is included in modelwarp_mc(). The processed image is now ready to be projected onto the eigenvectors, so the image is sent to the function computeCoeffs(): /*--------------------------------------------------------------------------- FUNCTION: computeCoeffs() Purpose: Given an input image eigenvectors, produces coefficients by projection. -----------------------------------------------------------------------------*/ This function simply takes the processed image, and sees how much of each eigenface it is made up of (see main.ps for the details). Once these coefficients have been computed, there are two things that can be done. Either the image just processed is meant to be enrolled into the database, or the image just processed is meant to be compared to people already enrolled in the database and recognized. /*--------------------------------------------------------------------------- FUNCTION: writeCoeffs() Purpose: Writes the coefficients for an image into a file -----------------------------------------------------------------------------*/ This procedure simply takes the coefficients and writes them out into a file with the same name as the image filename in the directory of known faces (the database). If the image was not meant to be enrolled, but recognized instead, then neighbor() is called instead of writeCoeffs(). /*--------------------------------------------------------------------------- FUNCTION: neighbor() Purpose: function for taking coefficients and figuring out which among known faces has closest set of coefficients. Author: Dave Becker Wasiuddin Wahid Tanzeem Choudhury -----------------------------------------------------------------------------*/ Neighbor() takes the set of known faces (this is loaded earlier via loadKnownFaces()) and the new set of coefficients and compares them. It sees which known face has coefficients most similar to the newly computed coefficients. It does this basically by finding the Euclidean distance between two sets of coefficients via find_distance() and then maintaining a linked-list stack of faces that is ordered in terms of distance (this is done via add_match(), which simply inserts a new face into the stack depending on the distance. firstCoeff and lastCOeff simply defines the coefficients to use. /*--------------------------------------------------------------------------- FUNCTION: print_match() Purpose: function for printing the n closest match Author: Dave Becker Wasiuddin Wahid Tanzeem Choudhury This function prints out the first n matches to the new set of coefficients and dumps it out to a file. -----------------------------------------------------------------------------*/ Eigsearch(), and ml_detect() are assuming 128 by 128 image. The maskfile included with this version of the software is 128 by 128. DO_RIGID is used the second time the image gets scalewarped. That is, after the features have been located, the image is warped so that the eyes are aligned and the mouth is in the right place. This can either be done with a rigid (rotation and translation) warp (fixing DO_RIGID_PT1 and DO_RIGID_PT_2 in particular locations) or with an affine warp. DO_RIGID sets which to do: 1 for rigid, 0 for affine. X_DEST and Y_DEST is the x,y coordinates where you would like the head center to be located after the image is fully normalized. The current numbers, 59 and 72, are a function of the size of the image. If you change image size, you should change X_DEST and Y_DEST accordingly. ========================================================================= /*--------------------------------------------------------------------------- Program Name: klt.c Purpose: Performs the Discrete Karhunen-Loeve Transform to compute the set of eigenimages for a set of images. Author: Baback Moghaddam -----------------------------------------------------------------------------*/ This is the code for calculating eignevector given training images. If you need to recalculate the eigenvectors, this is the code to use. The input images should be training normalized training images. All the preprocessing for computing coefficients and computing eigenvectors is the same. You probably don't need use this right now. There are some functions missing, I can either send it to you or put it in the directory shortly. I should mention that the eigenvectors were calculated from a large set of face images and probably will be the best ones to use. That's it! I hope this clears things up for you, but do please feel free to contact me with any questions. Tanzeem Choudhury tanzeem@media.mit.edu