A framework is presented for recovering the 3D structure and visual appearance of a human head from sparse data obtained from a real-time tracking system. An eigenvector decomposition of CyberWare-scanned heads is used to code incoming information. Modular eigenspaces are used to decorrelate eigenfeatures (eyes, nose, and mouth) from the rest of the head data. We observe that the modular eigenspace encoding often does not perform as well as a single eigenspace, and offer reasons for this based on experimental evidence.