next up previous
Next: EM and Conditional Likelihood Up: Maximum Conditional Likelihood via Previous: Maximum Conditional Likelihood via

Introduction

Conditional densities have played an important role in statistics and their merits over joint density models have been debated. Advantages in feature selection, robustness and limited resource allocation have been studied. Ultimately, tasks such as regression and classification reduce to the evaluation of a conditional density.

However, popularity of maximum joint likelihood and EM techniques remains strong in part due to their elegance and convergence properties. Thus, many conditional problems are solved by first estimating joint models then conditioning them. This results in concise solutions such as the Nadarya-Watson estimator [2], Xu's mixture of experts [7], and Amari's em-neural networks [1]. However, direct conditional density approaches [2,4] can offer solutions with higher conditional likelihood on test data than their joint counter-parts.


  
Figure: Average Joint (x,y)vs. Conditional (y|x) Likelihood Visualization
\begin{figure}\center
\begin{tabular}[b]{cc}
\epsfxsize=1.2in
\epsfbox{visual...
... ~~~ L^c_a=-2.4$\space &
(b) $L_b=-5.2 ~~~ L^c_b=-1.8$ \end{tabular}\end{figure}

Popat [6] describes a simple visualization example where 4 clusters must be fit with 2 Gaussian models as in Figure 1. Here, the model in (a) has a superior joint likelihood (La > Lb) and hence a better p(x,y)solution. However, when the models are conditioned to estimate p(y|x), model (b) is superior ( Lbc>Lac). Model (a) yields a poor unimodal conditional density in y and (b) yields a bi-modal conditional density. It is therefore of interest to directly optimize conditional models using conditional likelihood. We introduce the CEM (Conditional Expectation Maximization) algorithm for this purpose and apply it to the case of Gaussian mixture models.


next up previous
Next: EM and Conditional Likelihood Up: Maximum Conditional Likelihood via Previous: Maximum Conditional Likelihood via
Tony Jebara
2000-03-20