Feature Modeling

After the feature extraction stage comes the modeling stage -- in this stage we would like to capture the relationships among feature movements over time in expressive gestures. For example, an eye-blink always consists of an open eye gradually closing and then opening again -- there is an explicit temporal pattern in the eyelid motion. There are many such cases where there is a clear temporal pattern, e.g during the raising of eyebrows or looking towards some direction etc. We need to capture these expressive patterns to build models for a person's facial expressions. However, a brow raise or an eye-blink does not provide enough information about emotional expressions -- it is the combinations and the temporal relationships between the short time expressions that can explain what a person is trying to convey through his/her facial expressions.

Our low level modeling step uses Hidden Markov Models (HMMs) to model expressions that occur consistently across people and which can be extracted from unconstrained data. For eyes, these expressions are blinks, raising and lowering of eyebrows, looking in different directions etc. Once these low-level expressions can be detected reliably we can model high level expressions which can be described as a structured combination of the low-level expressions.