# Gaussian Mixtures & EM

> EM alternates soft assignments (E-step) and parameter re-estimation (M-step); the likelihood never decreases.

Source: https://www.mlfoundry.in/learn/gaussian-mixtures-em  
Section: ML Algorithms › Unsupervised · Clustering  
Level: advanced · 18 min read · updated 2026-10-07

**A Gaussian mixture model (GMM):** says the data were drawn from $K$ Gaussian distributions, each with its own mean $\mu_k$, covariance $\Sigma_k$ and weight $\pi_k$. **Expectation–maximization (EM)** fits it by alternating two steps. The E-step softly assigns each point to the components. The M-step re-estimates each component from its weighted points.

- **Task:** Soft clustering and density estimation (unsupervised)
- **Input → output:** $n$ points in $\mathbb R^d$ → responsibilities $\gamma_{ik}$ (the probability that point $i$ came from component $k$) and a density $p(x)$ you can evaluate anywhere
- **Learns:** for each of $K$ components: weight $\pi_k$, mean $\mu_k$, covariance $\Sigma_k$
- **Training:** EM, $O(nKd^2)$ per iteration plus $O(Kd^3)$ to factor the covariances; usually tens to hundreds of iterations, several restarts
- **Prediction:** responsibilities by Bayes' rule, $O(Kd^2)$ per point
- **Key hyperparameters:** number of components $K$; covariance type (full, tied, diagonal, spherical); initialization and number of restarts; covariance floor (`reg_covar`); tolerance
- **Assumes:** each cluster is roughly Gaussian (an ellipsoid), and points are independent

## The problem

k-means puts every point into exactly one cluster, and every cluster boundary is the perpendicular bisector between two centroids. Hidden in that rule are three assumptions: clusters are round, about equally wide, and every point belongs fully to one of them. Real data often break all three. One group may be tight and another diffuse. Clusters can be elongated or tilted. And a point that sits between two groups really is ambiguous.

We also often want more than labels. We want a **density** $p(x)$ that says how typical a new point is, so we can flag anomalies, sample synthetic data, or compare models by likelihood. A GMM provides both: soft memberships, and a proper probability model that can be fitted by maximum likelihood.

> **Where the siblings fit:** A GMM is a **model-based** clusterer: each cluster is a parametric ellipsoid. If your clusters have arbitrary shapes (rings, moons, filaments) or there is a lot of noise, use the density-based methods instead: **DBSCAN**, **HDBSCAN** and **OPTICS** grow clusters from dense neighbourhoods. **Mean shift** climbs to the modes of a non-parametric kernel density. **Spectral clustering** cuts a similarity graph. All of these give hard labels. The GMM is the one that gives you calibrated probabilities and a density formula.

## The idea

Tell a story about how each point was generated. First roll a $K$-sided die with probabilities $\pi_1,\dots,\pi_K$ to pick a component $z$. Then draw the point from that component's Gaussian, $\mathcal N(\mu_z,\Sigma_z)$. We see the points, but the die rolls $z$ are **hidden**.

This is a chicken-and-egg problem. If we knew every $z_i$, fitting would be trivial: take the mean and covariance of each group, and use group sizes for the weights. If we knew the parameters, Bayes' rule would tell us how likely each $z_i$ is. EM breaks the circle by alternating. Guess the parameters. Compute **soft** memberships. Refit the parameters as if those memberships were fractional counts. Repeat. Each round provably increases the likelihood until it levels off.

**Vocabulary**

- **Component:** one Gaussian $\mathcal N(\mu_k,\Sigma_k)$ in the mixture; it plays the role of a cluster
- **Mixing weight:** $\pi_k\ge 0$, $\sum_k\pi_k=1$: the prior probability that a point comes from component $k$
- **Latent variable:** $z_i\in\{1..K\}$, the unobserved component that generated $x_i$
- **Responsibility:** $\gamma_{ik}=P(z_i=k\mid x_i)$, the posterior probability that component $k$ generated $x_i$. Each row sums to 1
- **Soft count:** $N_k=\sum_i\gamma_{ik}$, the effective number of points that component $k$ owns. It need not be an integer
- **Log-likelihood:** $\log L=\sum_i\log p(x_i)$, the objective EM climbs
- **E-step / M-step:** Expectation: compute responsibilities from the current parameters. Maximization: refit the parameters from the responsibilities
- **Covariance type:** how much shape each $\Sigma_k$ may have: **full** (any ellipsoid), **diagonal** (axis-aligned), **spherical** ($\sigma_k^2I$), **tied** (all components share one $\Sigma$)
- **Degenerate component:** a component that collapses onto one point (or onto a lower-dimensional subspace). Its $|\Sigma_k|\to0$ and the likelihood $\to\infty$

## The math

**The model.** The density of a point is a weighted sum of Gaussian bumps:

$$
p(x)=\sum_{k=1}^{K}\pi_k\,\mathcal N(x\mid\mu_k,\Sigma_k),\qquad \mathcal N(x\mid\mu,\Sigma)=\frac{\exp\!\big(-\tfrac12(x-\mu)^\top\Sigma^{-1}(x-\mu)\big)}{(2\pi)^{d/2}\,|\Sigma|^{1/2}}
$$

- $K$: number of components
- $\pi_k$: mixing weight of component $k$; $\pi_k\ge0$, $\sum_k\pi_k=1$
- $\mu_k$: mean of component $k$ (a $d$-vector)
- $\Sigma_k$: covariance of component $k$ ($d\times d$, symmetric positive definite)
- $|\Sigma|$: determinant of Σ; it measures the volume of the ellipsoid
- $(x-\mu)^\top\Sigma^{-1}(x-\mu)$: squared Mahalanobis distance: distance measured in units of the component's own spread

**The objective.** Maximum likelihood over $n$ independent points:

$$
\log L(\theta)=\sum_{i=1}^{n}\log\sum_{k=1}^{K}\pi_k\,\mathcal N(x_i\mid\mu_k,\Sigma_k)
$$

- $\theta$: all parameters $\{\pi_k,\mu_k,\Sigma_k\}_{k=1}^K$
- $x_i$: the $i$-th data point

The **log of a sum** is the obstacle. With one Gaussian the log cancels the exponential and you get a closed form: the sample mean and covariance. Here the log sits outside the sum over $k$, the parameters of all components are coupled, and setting the gradient to zero does not give a closed form.

**E-step: responsibilities.** Bayes' rule gives the posterior over the hidden component, with prior $\pi_k$ and likelihood $\mathcal N(x_i\mid\mu_k,\Sigma_k)$:

$$
\gamma_{ik}=P(z_i=k\mid x_i)=\frac{\pi_k\,\mathcal N(x_i\mid\mu_k,\Sigma_k)}{\sum_{j=1}^{K}\pi_j\,\mathcal N(x_i\mid\mu_j,\Sigma_j)}
$$

- $\gamma_{ik}$: responsibility of component $k$ for point $i$, in $[0,1]$; $\sum_k\gamma_{ik}=1$
- $z_i$: the hidden component label of $x_i$

**Deriving the M-step.** Differentiate $\log L$ with respect to $\mu_k$. The derivative of $\log\sum_j(\cdot)$ is the $k$-th term divided by the whole sum, and that ratio is exactly $\gamma_{ik}$:

$$
\frac{\partial\log L}{\partial\mu_k}=\sum_{i}\underbrace{\frac{\pi_k\mathcal N(x_i\mid\mu_k,\Sigma_k)}{\sum_j\pi_j\mathcal N(x_i\mid\mu_j,\Sigma_j)}}_{\gamma_{ik}}\;\Sigma_k^{-1}(x_i-\mu_k)=0\;\;\Longrightarrow\;\;\mu_k=\frac{\sum_i\gamma_{ik}\,x_i}{\sum_i\gamma_{ik}}
$$

- $\Sigma_k^{-1}(x_i-\mu_k)$: gradient of $\log\mathcal N(x_i\mid\mu_k,\Sigma_k)$ with respect to $\mu_k$

This is not a closed form, because $\gamma_{ik}$ itself depends on $\mu_k$. But if we **freeze** $\gamma$ at its current value, the equation is a weighted average. Doing the same for $\Sigma_k$, and for $\pi_k$ with a Lagrange multiplier for $\sum_k\pi_k=1$, gives the full M-step:

$$
N_k=\sum_i\gamma_{ik},\qquad \pi_k=\frac{N_k}{n},\qquad \mu_k=\frac{1}{N_k}\sum_i\gamma_{ik}\,x_i,\qquad \Sigma_k=\frac{1}{N_k}\sum_i\gamma_{ik}\,(x_i-\mu_k)(x_i-\mu_k)^\top+\varepsilon I
$$

- $N_k$: soft count: how many points component k effectively owns
- $n$: number of data points
- $\varepsilon I$: small covariance floor added for numerical safety (reg_covar in scikit-learn)

These are the ordinary mean, covariance and frequency, with each point counted $\gamma_{ik}$ times instead of 0 or 1 times.

**Why the likelihood never decreases.** Take any distribution $q_i(k)$ over components for each point. Jensen's inequality ($\log$ is concave) gives a lower bound on each term:

$$
\log p(x_i)=\log\sum_k q_i(k)\,\frac{\pi_k\mathcal N(x_i\mid\mu_k,\Sigma_k)}{q_i(k)}\;\ge\;\sum_k q_i(k)\log\frac{\pi_k\mathcal N(x_i\mid\mu_k,\Sigma_k)}{q_i(k)}
$$

- $q_i(k)$: any probability distribution over the $K$ components for point $i$
- $\ge$: Jensen's inequality; equality holds exactly when $q_i(k)=\gamma_{ik}$

The **E-step** sets $q_i=\gamma_i$, which makes the bound touch $\log L$ at the current parameters. The **M-step** maximizes the bound over $\theta$. Here the log has moved *inside* the sum, so the maximization splits into one weighted Gaussian fit per component, which is the closed form above. The bound went up, and $\log L$ is at least as large as the bound. So $\log L(\theta^{t+1})\ge\log L(\theta^{t})$. EM therefore climbs monotonically to a **local** maximum or saddle point. It is not guaranteed to find the global one.

**Hyperparameters, mathematically.**

- **K:** More components can never lower the best achievable $\log L$, because a $(K{+}1)$-mixture can copy any $K$-mixture. So $\log L$ alone cannot choose $K$. Penalize it: $\text{BIC}=-2\log L+p\log n$, where $p$ is the number of free parameters. Lower BIC is better.
- **Covariance type:** Free parameters with $K$ components in $d$ dimensions: weights $K-1$, means $Kd$, plus covariances **full** $K\,d(d+1)/2$, **diagonal** $Kd$, **spherical** $K$, **tied** $d(d+1)/2$. With $d=50$ and $K=10$, full covariances need 12,750 covariance numbers; diagonal ones need 500.
- **Floor ε (reg_covar):** $\mathcal N$ peaks at $(2\pi)^{-d/2}|\Sigma|^{-1/2}$. Put $\mu_k$ on a data point and shrink $\Sigma_k\to0$, and that point's density goes to $\infty$. So the likelihood is **unbounded**, and the global "maximum" is degenerate. Adding $\varepsilon I$ caps each peak: in 1-D the cap is $1/\sqrt{2\pi\varepsilon}$, which is 398.9 for $\varepsilon=10^{-6}$ and 1.26 for $\varepsilon=0.1$.
- **Initialization:** EM converges to the local optimum whose basin contains the starting point. Common starts are k-means centroids (scikit-learn's default), k-means++ seeding, or random responsibilities. Run several restarts (n_init) and keep the highest $\log L$.
- **Tolerance:** stop when $\Delta\log L$ (or the per-point average) drops below tol. Too loose and you stop on a plateau; too tight and you waste iterations.

> **k-means is the zero-variance limit of EM:** Fix every covariance to $\sigma^2I$ and every weight to $1/K$. Then $\gamma_{ik}\propto\exp(-\lVert x_i-\mu_k\rVert^2/2\sigma^2)$. As $\sigma\to0$ the nearest mean takes all the weight, so responsibilities become 0 or 1. The M-step becomes "average the assigned points". That is exactly Lloyd's k-means. A GMM is what you get when you let each cluster keep its own spread, shape and size, and let memberships stay fractional.

## Worked example: twelve numbers, two components

Twelve values on a line, made of a tight group near 2, a wider group near 7, and one ambiguous point at 4.6. We fit $K=2$ in 1-D, so $\Sigma_k$ is a variance $\sigma_k^2$. Start from $\mu=(3,6)$, $\sigma_1^2=\sigma_2^2=1$, $\pi=(\tfrac12,\tfrac12)$. The table shows the first E-step. With equal weights the $\pi$'s cancel, so $\gamma_{i1}=\mathcal N_1/(\mathcal N_1+\mathcal N_2)$.

Data and the first E-step: densities $\mathcal N(x\mid3,1)$ and $\mathcal N(x\mid6,1)$, then responsibilities $\gamma_{i1},\gamma_{i2}$. $\mathcal N(x\mid\mu,1)=0.3989\,e^{-(x-\mu)^2/2}$. Highlighted: $x=4.6$, the only point the initial guess is genuinely unsure about.

| i | x | density, comp. 1 | density, comp. 2 | P(comp. 1 \| x) | P(comp. 2 \| x) |
| --- | --- | --- | --- | --- | --- |
| 1 | 1.2 | 0.0790 | 0.0000 | 1.000 | 0.000 |
| 2 | 1.6 | 0.1497 | 0.0000 | 1.000 | 0.000 |
| 3 | 1.9 | 0.2179 | 0.0001 | 1.000 | 0.000 |
| 4 | 2.1 | 0.2661 | 0.0002 | 0.999 | 0.001 |
| 5 | 2.4 | 0.3332 | 0.0006 | 0.998 | 0.002 |
| 6 | 2.8 | 0.3910 | 0.0024 | 0.994 | 0.006 |
| 7 | 4.6 | 0.1109 | 0.1497 | 0.426 | 0.574 |
| 8 | 6.2 | 0.0024 | 0.3910 | 0.006 | 0.994 |
| 9 | 6.8 | 0.0003 | 0.2897 | 0.001 | 0.999 |
| 10 | 7.1 | 0.0001 | 0.2179 | 0.000 | 1.000 |
| 11 | 7.5 | 0.0000 | 0.1295 | 0.000 | 1.000 |
| 12 | 8.3 | 0.0000 | 0.0283 | 0.000 | 1.000 |

**Iteration 1 by hand**

1. **Initial log-likelihood.** $\log L=\sum_i\log\big(\tfrac12\mathcal N(x_i\mid3,1)+\tfrac12\mathcal N(x_i\mid6,1)\big)=-28.279$.
2. **E-step for $x=4.6$.** $\mathcal N(4.6\mid3,1)=0.3989\,e^{-1.6^2/2}=0.3989\times0.2780=0.1109$ and $\mathcal N(4.6\mid6,1)=0.3989\,e^{-1.4^2/2}=0.3989\times0.3753=0.1497$. So $\gamma_{7,1}=0.1109/(0.1109+0.1497)=0.426$. The point is 1.4 from $\mu_2$ but 1.6 from $\mu_1$, so it leans slightly right.
3. **Soft counts.** $N_1=\sum_i\gamma_{i1}=6.424$ and $N_2=5.576$. The six left points contribute almost 1 each to $N_1$, and $x=4.6$ adds its 0.426.
4. **Weights.** $\pi_1=6.424/12=0.535$ and $\pi_2=0.465$.
5. **Means.** $\sum_i\gamma_{i1}x_i=13.982$, so $\mu_1=13.982/6.424=2.177$. $\sum_i\gamma_{i2}x_i=38.518$, so $\mu_2=38.518/5.576=6.908$. Both means jump toward their groups.
6. **Variances.** Using the new means, $\sum_i\gamma_{i1}(x_i-\mu_1)^2=4.437$, so $\sigma_1^2=4.437/6.424=0.691$. Likewise $\sigma_2^2=6.068/5.576=1.088$. The left group is tighter, and its variance has already shrunk.
7. **Check.** The new log-likelihood is $-23.364$, up from $-28.279$, as the theory guarantees.

Repeating the two steps, the ambiguous point is quickly handed to the wide right component. The narrow left Gaussian says 4.6 is very unlikely, while the wide right one finds it plausible.

Parameters at the start of each iteration (iteration 0 is the initial guess). The last-but-one column is $\gamma_{7,1}$, the left component's responsibility for $x=4.6$. EM stops after 6 iterations, when $\Delta\log L<10^{-6}$.

| iter | weight 1 | mean 1 | mean 2 | var 1 | var 2 | P(comp. 1 \| 4.6) | log L |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 0 | 0.500 | 3.000 | 6.000 | 1.000 | 1.000 | 0.426 | −28.279 |
| 1 | 0.535 | 2.177 | 6.908 | 0.691 | 1.088 | 0.192 | −23.364 |
| 2 | 0.516 | 2.081 | 6.821 | 0.465 | 1.223 | 0.014 | −22.715 |
| 3 | 0.501 | 2.006 | 6.754 | 0.285 | 1.332 | 0.000 | −22.286 |
| 4 | 0.500 | 1.999 | 6.747 | 0.270 | 1.347 | 0.000 | −22.280 |
| 6 | 0.500 | 1.999 | 6.746 | 0.270 | 1.349 | 0.000 | −22.280 |

**Classifying a new point, x* = 4.0**

1. **Fitted model.** $\pi=(0.500,0.500)$, $\mu=(1.999,6.746)$, $\sigma^2=(0.270,1.349)$, so $\sigma=(0.519,1.162)$.
2. **Distances in standard deviations.** $x^*$ is $2.00$ from $\mu_1$, which is $3.85\,\sigma_1$. It is $2.75$ from $\mu_2$, which is only $2.36\,\sigma_2$.
3. **Weighted densities.** $\pi_1\mathcal N(4\mid\mu_1,\sigma_1^2)=0.00023$ and $\pi_2\mathcal N(4\mid\mu_2,\sigma_2^2)=0.01050$.
4. **Responsibilities.** $\gamma=(0.021,\,0.979)$: component 2 claims it with 97.9% probability. k-means on the same data finds centres 2.00 and 6.75, puts its boundary at the midpoint 4.375, and would assign $x^*=4.0$ to the **left** cluster.
5. **Density.** $p(4.0)=0.0107$, so $-\log p=4.53$. That is a usable anomaly score: a point at 4.6 scores 3.47, and one at 5.0 scores 2.89.
6. **Decision boundary.** $\gamma_1=\gamma_2$ at $x=3.566$, not at the midpoint of the means. Because the variances differ, the boundary is quadratic, and there is a second crossing far out at $x=-1.94$, where the wide component's tails take over again.

## See it run

> Figure: EM on the twelve worked-example points. Step through the E-steps (points and γ bars recolour, log L stays flat) and M-steps (curves refit, log L rises). From μ = (3, 6), log L climbs from −28.279 to −22.280 in 6 iterations, and x = 4.6 goes from 0.43/0.57 to fully right. Drag a ▲ mean to restart EM from a new guess: start μ₁ at 8.5 instead and EM settles in a worse local optimum (log L −23.596), where a wide left component claims 4.6. Drag the ◆ probe to read γ and −log p for any x: at x* = 4.0 the converged model gives γ = (0.02, 0.98). Switch to K = 3 (start μ = 3, 4.5, 6) and one component collapses onto the lone point 4.6. Its variance hits the 1e-6 floor and log L jumps to −14.948. With a floor of 0.1 it stops at σ² = 0.10 and log L = −20.878.

The step-through below fits three elongated, tilted clusters in 2-D with full covariances, which the 1-D example cannot show. Each M-step stretches and rotates its ellipse to match the weighted scatter of its points.

> Figure: Full-covariance EM in 2-D. Blended colours are soft memberships, and each M-step refits every ellipse to its weighted points. Log-likelihood rises from −37.7 to 89.4 and never falls.

## The algorithm, step by step

**EM for a Gaussian mixture**

1. **Initialize** $\{\pi_k,\mu_k,\Sigma_k\}$, usually from k-means: the centroids, the covariance of each k-means cluster, and the cluster sizes divided by $n$.
2. **E-step.** For every point and component, compute $\log\pi_k+\log\mathcal N(x_i\mid\mu_k,\Sigma_k)$. Normalize each row with **log-sum-exp** to get $\gamma_{ik}$, and keep $\log p(x_i)$ to evaluate $\log L$.
3. **M-step.** $N_k=\sum_i\gamma_{ik}$; $\pi_k=N_k/n$; $\mu_k=\sum_i\gamma_{ik}x_i/N_k$; $\Sigma_k=\sum_i\gamma_{ik}(x_i-\mu_k)(x_i-\mu_k)^\top/N_k+\varepsilon I$, or its diagonal, spherical or tied version.
4. **Converge?** Stop when $\log L$ improves by less than tol, or after max_iter iterations. If a component's $N_k$ falls near 0, re-seed it.
5. **Restart** $n_{\text{init}}$ times from different seeds and keep the run with the highest $\log L$.
6. **Use** $\gamma_{ik}$ as soft labels, $\arg\max_k\gamma_{ik}$ as hard labels, and $-\log p(x)$ as an outlier score.

**Complexity.** Each iteration costs $O(nKd^2)$ to evaluate the Mahalanobis distances with full covariances ($O(nKd)$ for diagonal), plus $O(Kd^3)$ to Cholesky-factor the covariances. Memory is $O(nK)$ for the responsibilities plus $O(Kd^2)$ for the covariances. Convergence is linear and can be slow when components overlap heavily. Then EM may need hundreds of iterations, where k-means would stop after a few.

## Variants and practical tuning

Covariance types. Params counts are for K components in d dimensions (the covariance part only).

| covariance_type | Each cluster is… | Covariance params | Use when |
| --- | --- | --- | --- |
| full | any rotated ellipsoid | K·d(d+1)/2 | low d, plenty of points per cluster |
| tied | the same ellipsoid, shifted | d(d+1)/2 | clusters differ only in location (this is LDA's assumption) |
| diag | axis-aligned ellipsoid | K·d | higher d, features roughly uncorrelated within a cluster |
| spherical | a ball with its own radius | K | a k-means-like model with soft labels and unequal sizes |

**Other variants**

- **Choosing K:** fit $K=1..K_{\max}$ and take the lowest **BIC** ($-2\log L+p\log n$). It is consistent for density estimation, but tends to over-split non-Gaussian clusters. AIC ($-2\log L+2p$) penalizes less and picks more components.
- **Bayesian GMM:** put a Dirichlet (or Dirichlet-process) prior on the weights and fit by variational inference (`BayesianGaussianMixture`). Unneeded components get weights near 0, so you can set K generously and let the data switch components off.
- **Hard / classification EM:** replace $\gamma$ by a one-hot argmax in the E-step. It is faster and gives crisp labels, but it no longer maximizes the likelihood of the mixture. With spherical, equal covariances it is k-means.
- **MAP-EM:** add a prior (for example inverse-Wishart) on $\Sigma_k$. The M-step becomes a shrunk covariance, a principled version of reg_covar.
- **Online / mini-batch EM:** update running sums of $\gamma$, $\gamma x$ and $\gamma xx^\top$ from batches, for streams or very large $n$.
- **EM beyond Gaussians:** the same E/M recipe fits mixtures of Bernoullis or multinomials (text, binary data), hidden Markov models (Baum–Welch), and missing-data problems.

> **Collapsing components, and BIC that rewards them:** On the twelve worked-example points with the default `reg_covar=1e-6`, BIC for K = 1, 2, 3 is 61.38, 56.99 and **49.77**. It picks K = 3, because the third component collapses onto the single point 4.6 with variance about 1e-6. That buys a huge likelihood that the parameter penalty cannot offset. With `reg_covar=0.1` the BICs become 61.38, **57.27** and 61.64, and K = 2 wins. When a component has $N_k \approx 1$ or a near-zero determinant, treat it as an artefact. Raise the floor, use diagonal or tied covariances, or switch to a Bayesian GMM.

> **Scale your features:** Full covariances adapt to each feature's scale, but diagonal and spherical ones, and the k-means initialization, do not. A feature measured in thousands will dominate the initial centroids and the spherical distances. Standardize first unless the units are already comparable.

> **Label switching and local optima:** Component numbers are arbitrary: two runs can find the same fit with the labels permuted, so never compare $\mu_1$ across runs without matching components. Different starts can also land on genuinely different local optima, such as two components sharing one cluster while another component covers two clusters. On the worked example, starting from $\mu=(8.5,6)$ instead of $(3,6)$ converges to $\mu=(7.172,2.367)$, $\sigma^2=(0.515,1.054)$ with $\log L=-23.596$, worse than $-22.280$, because the left component stays wide and keeps 4.6. Always use several restarts (n_init) and keep the highest $\log L$.

## In code

```python
import numpy as np

def log_gauss(X, mu, S):
  """log N(x | mu, S) for every row of X."""
  d = X.shape[1]
  L = np.linalg.cholesky(S)
  z = np.linalg.solve(L, (X - mu).T)                 # whitened residuals
  return -0.5 * (z**2).sum(0) - np.log(np.diag(L)).sum() - 0.5 * d * np.log(2 * np.pi)

def gmm_em(X, mu, S, pi, iters=200, tol=1e-6, reg=1e-6):
  n, d = X.shape
  K = len(pi)
  prev = -np.inf
  for it in range(iters):
      # E-step: responsibilities via log-sum-exp
      logw = np.stack([np.log(pi[k]) + log_gauss(X, mu[k], S[k]) for k in range(K)], 1)
      m = logw.max(1, keepdims=True)
      logp = m[:, 0] + np.log(np.exp(logw - m).sum(1))
      ll = logp.sum()
      g = np.exp(logw - logp[:, None])
      if ll - prev < tol:
          break
      prev = ll
      # M-step: weighted counts, means, covariances
      Nk = g.sum(0)
      pi = Nk / n
      mu = (g.T @ X) / Nk[:, None]
      S = np.stack([((g[:, k, None] * (X - mu[k])).T @ (X - mu[k])) / Nk[k] + reg * np.eye(d)
                    for k in range(K)])
  return pi, mu, S, g, ll, it

X = np.array([1.2, 1.6, 1.9, 2.1, 2.4, 2.8, 4.6, 6.2, 6.8, 7.1, 7.5, 8.3])[:, None]
pi, mu, S, g, ll, it = gmm_em(X, mu=np.array([[3.0], [6.0]]),
                            S=np.array([[[1.0]], [[1.0]]]), pi=np.array([0.5, 0.5]))
print("iterations:", it)
print("pi:", pi.round(3), "mu:", mu.ravel().round(3), "var:", S.ravel().round(3))
print("log-likelihood:", round(ll, 3))
print("gamma(left) for x=4.6:", g[6, 0].round(4))
# iterations: 6
# pi: [0.5 0.5] mu: [1.999 6.746] var: [0.27  1.349]
# log-likelihood: -22.28
# gamma(left) for x=4.6: 0.0
```

From scratch: full covariances, log-sum-exp E-step, run on the worked example from μ = (3, 6).

```python
import numpy as np
from sklearn.mixture import GaussianMixture

X = np.array([1.2, 1.6, 1.9, 2.1, 2.4, 2.8, 4.6, 6.2, 6.8, 7.1, 7.5, 8.3])[:, None]

gmm = GaussianMixture(n_components=2, covariance_type="full",
                    means_init=[[3.0], [6.0]], random_state=0).fit(X)
print("pi:", gmm.weights_.round(3), "mu:", gmm.means_.ravel().round(3),
    "var:", gmm.covariances_.ravel().round(3))
print("log-likelihood:", round(gmm.score(X) * len(X), 3))   # score() is the per-point mean
print("P(component | x=4.0):", gmm.predict_proba([[4.0]]).round(3))
print("-log p(4.0):", round(-gmm.score_samples([[4.0]])[0], 3))

# Choosing K: BIC with the default tiny reg_covar vs a sensible one
for reg in (1e-6, 0.1):
  bics = [GaussianMixture(k, n_init=10, reg_covar=reg, random_state=0).fit(X).bic(X)
          for k in (1, 2, 3)]
  print(f"reg_covar={reg}: BIC K=1,2,3 =", np.round(bics, 2))

# pi: [0.5 0.5] mu: [1.999 6.746] var: [0.27  1.349]
# log-likelihood: -22.28
# P(component | x=4.0): [[0.021 0.979]]
# -log p(4.0): 4.535
# reg_covar=1e-06: BIC K=1,2,3 = [61.38 56.99 49.77]
# reg_covar=0.1: BIC K=1,2,3 = [61.38 57.27 61.64]
```

scikit-learn on the same data: soft labels, density scores, and choosing K by BIC.

> **Reading scikit-learn's numbers:** `score(X)` returns the **average** log-likelihood per point; multiply by n to get the total $\log L$ used in this article. `bic(X)` uses the total. `predict` is the argmax of `predict_proba`, and `sample(n)` draws new points from the fitted mixture.

**Strengths**

- Clusters are roughly elliptical but overlap, are tilted, or differ in size and spread.
- You need calibrated soft memberships, or an explicit density for anomaly scores ($-\log p(x)$), sampling or likelihood-based model comparison.
- You want a principled way to choose $K$ (BIC) and a model you can write down and inspect: weights, means and covariances.
- Moderate dimensions (or diagonal covariances) with enough points per component to estimate its covariance.

**Limitations**

- Clusters have arbitrary shapes (rings, moons, filaments): use DBSCAN, HDBSCAN or spectral clustering.
- High $d$ with few points: a full covariance has $d(d+1)/2$ parameters per component. Use diagonal or tied covariances, or reduce dimension (PCA) first.
- Heavy noise or outliers: every point must be explained by some Gaussian, so outliers inflate covariances. Add a uniform noise component, or use HDBSCAN.
- Very large n where k-means' speed and simplicity are enough, or where you only need hard labels.

- **Speaker verification:** GMM–UBM systems model each speaker's acoustic (MFCC) features as a mixture adapted from a universal background GMM. This was the standard before neural embeddings.
- **Background subtraction:** Each pixel's colour over time is modelled by a small GMM (Stauffer–Grimson). Pixels that fit no background component are foreground.
- **Anomaly and fraud detection:** Fit a GMM to normal transactions or sensor readings, and flag points whose $-\log p(x)$ exceeds a threshold set on validation data.
- **Science: separating populations:** Flow-cytometry cell types, stellar populations in colour–magnitude space, gene-expression subtypes. These are overlapping groups where soft membership is the honest answer.

**Question:** Two components have weights π = (0.2, 0.8). At a point x, their densities are 𝒩₁(x) = 0.3 and 𝒩₂(x) = 0.1. What is γ₁, the responsibility of component 1?

- A. 0.75
- B. 0.43
- C. 0.20
- D. 0.06

**Answer:** B. 0.43 — $\gamma_1=\dfrac{0.2\times0.3}{0.2\times0.3+0.8\times0.1}=\dfrac{0.06}{0.14}=0.43$. Component 1 fits the point three times better, but its prior is four times smaller.

**Question:** What does EM guarantee at every iteration?

- A. It reaches the global maximum of the likelihood
- B. The log-likelihood does not decrease
- C. Every responsibility moves closer to 0 or 1
- D. Each component keeps the same number of points

**Answer:** B. The log-likelihood does not decrease — The E-step makes the Jensen lower bound tight at the current parameters, and the M-step maximizes that bound. So the log-likelihood can only rise or stay level. It reaches a local optimum, not necessarily the global one.

**Question:** A GMM with K = 3 full-covariance components in d = 10 dimensions has how many free parameters?

- A. 63
- B. 165
- C. 197
- D. 330

**Answer:** C. 197 — Weights: $K-1=2$. Means: $Kd=30$. Covariances: $K\,d(d+1)/2=3\times55=165$. Total $2+30+165=197$. This is the $p$ in BIC.

**Question:** On the 12 worked-example points with reg_covar = 1e-6, BIC prefers K = 3 (49.77) over K = 2 (56.99). Why?

- A. There really are three clusters
- B. BIC penalizes parameters too heavily
- C. The third component collapses onto the single point 4.6, so its density there, and the likelihood, become enormous
- D. EM did not converge for K = 2

**Answer:** C. The third component collapses onto the single point 4.6, so its density there, and the likelihood, become enormous — A component with mean 4.6 and variance near 1e-6 puts a density of hundreds on that point. The likelihood gain outweighs the BIC penalty of 3 extra parameters. With reg_covar = 0.1 the spike is capped and K = 2 wins (57.27 vs 61.64).

**Key points**

- A GMM models $p(x)=\sum_k\pi_k\mathcal N(x\mid\mu_k,\Sigma_k)$. Its clusters are ellipsoids with their own size, shape and weight, and memberships are probabilities.
- E-step: responsibilities $\gamma_{ik}$ by Bayes' rule. M-step: the weighted frequency, mean and covariance, with each point counted $\gamma_{ik}$ times.
- EM maximizes a Jensen lower bound that is tight after each E-step, so the log-likelihood never decreases. It finds a local optimum, so use several restarts.
- The likelihood is unbounded, because a component can collapse onto one point. Keep a covariance floor (reg_covar) and distrust components with tiny N_k.
- Choose K with BIC, and choose the covariance type by dimension and data size. k-means is the hard, spherical, zero-variance special case.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
