Gaussian Mixture Model (GMM)

A Gaussian mixture model (GMM) is a probabilistic model that represents a population as a mixture of multiple Gaussian distributions, each with its own parameters (mean, variance, and mixing weight).

Log-likelihood

For $n$ i.i.d. datapoints, the observed log-likelihood is

$$\ell (\theta )=\sum _{i=1}^n\log \left(\sum _{k=1}^K{\pi }_k\mathcal{N}(x_i\mid {\mu }_k,{\sigma }_k^2)\right).$$

• This $\log \sum$ form is hard to maximize directly.

Resources

• https://scikit-learn.org/stable/modules/mixture.html

How to think about this

Mixture models as generalize k-means clustering to incorporate information about the covariance structure of the data as well as the centers of the latent Gaussians.