Errors in classification and feature tracking can lead to instability in the model. Special care must be taken to make sure the model remains valid. One way to accomplish this is to reconcile the individual blob models with domain-specific prior knowledge. For instance, some parameters (e.g., color of a person's hand) are expected to be stable and to stay fairly close to the prior distribution, some are expected to be stable but have weak priors (e.g., shirt color) and others are both expected to change quickly and have weak priors (e.g., hand position).
Intelligently chosen prior knowledge can turn a class into a very solid feature tracker. For instance, classes intended to follow flesh are good candidates for assertive prior knowledge, because people's normalized skin color is surprisingly constant across race and tan. There is some variation with very different lighting conditions (e.g., sun versus fluorescent), so a small library of skin classes is required to handle different illumination conditions.