Gaussian Mixtures & EM

EM alternates soft assignments (E-step) and parameter re-estimation (M-step); the likelihood never decreases.

18 min read1637 wordsadvanced9 viewsupdated 2026-10-07

Definition

A Gaussian mixture model (GMM) says the data were drawn from KK Gaussian distributions, each with its own mean μk\mu_k, covariance Σk\Sigma_k and weight πk\pi_k. Expectation–maximization (EM) fits it by alternating two steps. The E-step softly assigns each point to the components. The M-step re-estimates each component from its weighted points.

At a glance

Task
Soft clustering and density estimation (unsupervised)
Input → output
nn points in Rd\mathbb R^d → responsibilities γik\gamma_{ik} (the probability that point ii came from component kk) and a density p(x)p(x) you can evaluate anywhere
Learns
for each of KK components: weight πk\pi_k, mean μk\mu_k, covariance Σk\Sigma_k
Training
EM, O(nKd2)O(nKd^2) per iteration plus O(Kd3)O(Kd^3) to factor the covariances; usually tens to hundreds of iterations, several restarts
Prediction
responsibilities by Bayes' rule, O(Kd2)O(Kd^2) per point
Key hyperparameters
number of components KK; covariance type (full, tied, diagonal, spherical); initialization and number of restarts; covariance floor (reg_covar); tolerance
Assumes
each cluster is roughly Gaussian (an ellipsoid), and points are independent

The problem

k-means puts every point into exactly one cluster, and every cluster boundary is the perpendicular bisector between two centroids. Hidden in that rule are three assumptions: clusters are round, about equally wide, and every point belongs fully to one of them. Real data often break all three. One group may be tight and another diffuse. Clusters can be elongated or tilted. And a point that sits between two groups really is ambiguous.

We also often want more than labels. We want a density p(x)p(x) that says how typical a new point is, so we can flag anomalies, sample synthetic data, or compare models by likelihood. A GMM provides both: soft memberships, and a proper probability model that can be fitted by maximum likelihood.

The idea

Tell a story about how each point was generated. First roll a KK-sided die with probabilities π1,…,πK\pi_1,\dots,\pi_K to pick a component zz. Then draw the point from that component's Gaussian, N(μz,Σz)\mathcal N(\mu_z,\Sigma_z). We see the points, but the die rolls zz are hidden.

This is a chicken-and-egg problem. If we knew every ziz_i, fitting would be trivial: take the mean and covariance of each group, and use group sizes for the weights. If we knew the parameters, Bayes' rule would tell us how likely each ziz_i is. EM breaks the circle by alternating. Guess the parameters. Compute soft memberships. Refit the parameters as if those memberships were fractional counts. Repeat. Each round provably increases the likelihood until it levels off.

Vocabulary

Component
one Gaussian N(μk,Σk)\mathcal N(\mu_k,\Sigma_k) in the mixture; it plays the role of a cluster
Mixing weight
πk≥0\pi_k\ge 0, ∑kπk=1\sum_k\pi_k=1: the prior probability that a point comes from component kk
Latent variable
zi∈{1..K}z_i\in\{1..K\}, the unobserved component that generated xix_i
Responsibility
γik=P(zi=k∣xi)\gamma_{ik}=P(z_i=k\mid x_i), the posterior probability that component kk generated xix_i. Each row sums to 1
Soft count
Nk=∑iγikN_k=\sum_i\gamma_{ik}, the effective number of points that component kk owns. It need not be an integer
Log-likelihood
log⁡L=∑ilog⁡p(xi)\log L=\sum_i\log p(x_i), the objective EM climbs
E-step / M-step
Expectation: compute responsibilities from the current parameters. Maximization: refit the parameters from the responsibilities
Covariance type
how much shape each Σk\Sigma_k may have: full (any ellipsoid), diagonal (axis-aligned), spherical (σk2I\sigma_k^2I), tied (all components share one Σ\Sigma)
Degenerate component
a component that collapses onto one point (or onto a lower-dimensional subspace). Its ∣Σk∣→0|\Sigma_k|\to0 and the likelihood →∞\to\infty

The math

The model. The density of a point is a weighted sum of Gaussian bumps:

p(x)=∑k=1Kπk N(x∣μk,Σk),N(x∣μ,Σ)=exp⁡ ⁣(−12(x−μ)⊤Σ−1(x−μ))(2π)d/2 ∣Σ∣1/2p(x)=\sum_{k=1}^{K}\pi_k\,\mathcal N(x\mid\mu_k,\Sigma_k),\qquad \mathcal N(x\mid\mu,\Sigma)=\frac{\exp\!\big(-\tfrac12(x-\mu)^\top\Sigma^{-1}(x-\mu)\big)}{(2\pi)^{d/2}\,|\Sigma|^{1/2}}
KK
number of components
πk\pi_k
mixing weight of component kk; πk≥0\pi_k\ge0, ∑kπk=1\sum_k\pi_k=1
μk\mu_k
mean of component kk (a dd-vector)
Σk\Sigma_k
covariance of component kk (d×dd\times d, symmetric positive definite)
∣Σ∣|\Sigma|
determinant of Σ; it measures the volume of the ellipsoid
(x−μ)⊤Σ−1(x−μ)(x-\mu)^\top\Sigma^{-1}(x-\mu)
squared Mahalanobis distance: distance measured in units of the component's own spread

The objective. Maximum likelihood over nn independent points:

log⁡L(θ)=∑i=1nlog⁡∑k=1Kπk N(xi∣μk,Σk)\log L(\theta)=\sum_{i=1}^{n}\log\sum_{k=1}^{K}\pi_k\,\mathcal N(x_i\mid\mu_k,\Sigma_k)
θ\theta
all parameters {πk,μk,Σk}k=1K\{\pi_k,\mu_k,\Sigma_k\}_{k=1}^K
xix_i
the ii-th data point

The log of a sum is the obstacle. With one Gaussian the log cancels the exponential and you get a closed form: the sample mean and covariance. Here the log sits outside the sum over kk, the parameters of all components are coupled, and setting the gradient to zero does not give a closed form.

E-step: responsibilities. Bayes' rule gives the posterior over the hidden component, with prior πk\pi_k and likelihood N(xi∣μk,Σk)\mathcal N(x_i\mid\mu_k,\Sigma_k):

γik=P(zi=k∣xi)=πk N(xi∣μk,Σk)∑j=1Kπj N(xi∣μj,Σj)\gamma_{ik}=P(z_i=k\mid x_i)=\frac{\pi_k\,\mathcal N(x_i\mid\mu_k,\Sigma_k)}{\sum_{j=1}^{K}\pi_j\,\mathcal N(x_i\mid\mu_j,\Sigma_j)}
γik\gamma_{ik}
responsibility of component kk for point ii, in [0,1][0,1]; ∑kγik=1\sum_k\gamma_{ik}=1
ziz_i
the hidden component label of xix_i

Deriving the M-step. Differentiate log⁡L\log L with respect to μk\mu_k. The derivative of log⁡∑j(⋅)\log\sum_j(\cdot) is the kk-th term divided by the whole sum, and that ratio is exactly γik\gamma_{ik}:

∂log⁡L∂μk=∑iπkN(xi∣μk,Σk)∑jπjN(xi∣μj,Σj)⏟γik  Σk−1(xi−μk)=0    ⟹    μk=∑iγik xi∑iγik\frac{\partial\log L}{\partial\mu_k}=\sum_{i}\underbrace{\frac{\pi_k\mathcal N(x_i\mid\mu_k,\Sigma_k)}{\sum_j\pi_j\mathcal N(x_i\mid\mu_j,\Sigma_j)}}_{\gamma_{ik}}\;\Sigma_k^{-1}(x_i-\mu_k)=0\;\;\Longrightarrow\;\;\mu_k=\frac{\sum_i\gamma_{ik}\,x_i}{\sum_i\gamma_{ik}}
Σk−1(xi−μk)\Sigma_k^{-1}(x_i-\mu_k)
gradient of log⁡N(xi∣μk,Σk)\log\mathcal N(x_i\mid\mu_k,\Sigma_k) with respect to μk\mu_k

This is not a closed form, because γik\gamma_{ik} itself depends on μk\mu_k. But if we freeze γ\gamma at its current value, the equation is a weighted average. Doing the same for Σk\Sigma_k, and for πk\pi_k with a Lagrange multiplier for ∑kπk=1\sum_k\pi_k=1, gives the full M-step:

Nk=∑iγik,πk=Nkn,μk=1Nk∑iγik xi,Σk=1Nk∑iγik (xi−μk)(xi−μk)⊤+εIN_k=\sum_i\gamma_{ik},\qquad \pi_k=\frac{N_k}{n},\qquad \mu_k=\frac{1}{N_k}\sum_i\gamma_{ik}\,x_i,\qquad \Sigma_k=\frac{1}{N_k}\sum_i\gamma_{ik}\,(x_i-\mu_k)(x_i-\mu_k)^\top+\varepsilon I
NkN_k
soft count: how many points component k effectively owns
nn
number of data points
εI\varepsilon I
small covariance floor added for numerical safety (reg_covar in scikit-learn)

These are the ordinary mean, covariance and frequency, with each point counted γik\gamma_{ik} times instead of 0 or 1 times.

Why the likelihood never decreases. Take any distribution qi(k)q_i(k) over components for each point. Jensen's inequality (log⁡\log is concave) gives a lower bound on each term:

log⁡p(xi)=log⁡∑kqi(k) πkN(xi∣μk,Σk)qi(k)  ≥  ∑kqi(k)log⁡πkN(xi∣μk,Σk)qi(k)\log p(x_i)=\log\sum_k q_i(k)\,\frac{\pi_k\mathcal N(x_i\mid\mu_k,\Sigma_k)}{q_i(k)}\;\ge\;\sum_k q_i(k)\log\frac{\pi_k\mathcal N(x_i\mid\mu_k,\Sigma_k)}{q_i(k)}
qi(k)q_i(k)
any probability distribution over the KK components for point ii
≥\ge
Jensen's inequality; equality holds exactly when qi(k)=γikq_i(k)=\gamma_{ik}

The E-step sets qi=γiq_i=\gamma_i, which makes the bound touch log⁡L\log L at the current parameters. The M-step maximizes the bound over θ\theta. Here the log has moved inside the sum, so the maximization splits into one weighted Gaussian fit per component, which is the closed form above. The bound went up, and log⁡L\log L is at least as large as the bound. So log⁡L(θt+1)≥log⁡L(θt)\log L(\theta^{t+1})\ge\log L(\theta^{t}). EM therefore climbs monotonically to a local maximum or saddle point. It is not guaranteed to find the global one.

Hyperparameters, mathematically.

At a glance

K
More components can never lower the best achievable log⁡L\log L, because a (K+1)(K{+}1)-mixture can copy any KK-mixture. So log⁡L\log L alone cannot choose KK. Penalize it: BIC=−2log⁡L+plog⁡n\text{BIC}=-2\log L+p\log n, where pp is the number of free parameters. Lower BIC is better.
Covariance type
Free parameters with KK components in dd dimensions: weights K−1K-1, means KdKd, plus covariances full K d(d+1)/2K\,d(d+1)/2, diagonal KdKd, spherical KK, tied d(d+1)/2d(d+1)/2. With d=50d=50 and K=10K=10, full covariances need 12,750 covariance numbers; diagonal ones need 500.
Floor ε (reg_covar)
N\mathcal N peaks at (2π)−d/2∣Σ∣−1/2(2\pi)^{-d/2}|\Sigma|^{-1/2}. Put μk\mu_k on a data point and shrink Σk→0\Sigma_k\to0, and that point's density goes to ∞\infty. So the likelihood is unbounded, and the global "maximum" is degenerate. Adding εI\varepsilon I caps each peak: in 1-D the cap is 1/2πε1/\sqrt{2\pi\varepsilon}, which is 398.9 for ε=10−6\varepsilon=10^{-6} and 1.26 for ε=0.1\varepsilon=0.1.
Initialization
EM converges to the local optimum whose basin contains the starting point. Common starts are k-means centroids (scikit-learn's default), k-means++ seeding, or random responsibilities. Run several restarts (n_init) and keep the highest log⁡L\log L.
Tolerance
stop when Δlog⁡L\Delta\log L (or the per-point average) drops below tol. Too loose and you stop on a plateau; too tight and you waste iterations.

Worked example: twelve numbers, two components

Twelve values on a line, made of a tight group near 2, a wider group near 7, and one ambiguous point at 4.6. We fit K=2K=2 in 1-D, so Σk\Sigma_k is a variance σk2\sigma_k^2. Start from μ=(3,6)\mu=(3,6), σ12=σ22=1\sigma_1^2=\sigma_2^2=1, π=(12,12)\pi=(\tfrac12,\tfrac12). The table shows the first E-step. With equal weights the π\pi's cancel, so γi1=N1/(N1+N2)\gamma_{i1}=\mathcal N_1/(\mathcal N_1+\mathcal N_2).

ixdensity, comp. 1density, comp. 2P(comp. 1 | x)P(comp. 2 | x)
11.20.07900.00001.0000.000
21.60.14970.00001.0000.000
31.90.21790.00011.0000.000
42.10.26610.00020.9990.001
52.40.33320.00060.9980.002
62.80.39100.00240.9940.006
74.60.11090.14970.4260.574
86.20.00240.39100.0060.994
96.80.00030.28970.0010.999
107.10.00010.21790.0001.000
117.50.00000.12950.0001.000
128.30.00000.02830.0001.000
Data and the first E-step: densities N(x∣3,1)\mathcal N(x\mid3,1) and N(x∣6,1)\mathcal N(x\mid6,1), then responsibilities γi1,γi2\gamma_{i1},\gamma_{i2}. N(x∣μ,1)=0.3989 e−(x−μ)2/2\mathcal N(x\mid\mu,1)=0.3989\,e^{-(x-\mu)^2/2}. Highlighted: x=4.6x=4.6, the only point the initial guess is genuinely unsure about.

Iteration 1 by hand

  1. 1Initial log-likelihood. log⁡L=∑ilog⁡(12N(xi∣3,1)+12N(xi∣6,1))=−28.279\log L=\sum_i\log\big(\tfrac12\mathcal N(x_i\mid3,1)+\tfrac12\mathcal N(x_i\mid6,1)\big)=-28.279.
  2. 2E-step for x=4.6x=4.6. N(4.6∣3,1)=0.3989 e−1.62/2=0.3989×0.2780=0.1109\mathcal N(4.6\mid3,1)=0.3989\,e^{-1.6^2/2}=0.3989\times0.2780=0.1109 and N(4.6∣6,1)=0.3989 e−1.42/2=0.3989×0.3753=0.1497\mathcal N(4.6\mid6,1)=0.3989\,e^{-1.4^2/2}=0.3989\times0.3753=0.1497. So γ7,1=0.1109/(0.1109+0.1497)=0.426\gamma_{7,1}=0.1109/(0.1109+0.1497)=0.426. The point is 1.4 from μ2\mu_2 but 1.6 from μ1\mu_1, so it leans slightly right.
  3. 3Soft counts. N1=∑iγi1=6.424N_1=\sum_i\gamma_{i1}=6.424 and N2=5.576N_2=5.576. The six left points contribute almost 1 each to N1N_1, and x=4.6x=4.6 adds its 0.426.
  4. 4Weights. π1=6.424/12=0.535\pi_1=6.424/12=0.535 and π2=0.465\pi_2=0.465.
  5. 5Means. ∑iγi1xi=13.982\sum_i\gamma_{i1}x_i=13.982, so μ1=13.982/6.424=2.177\mu_1=13.982/6.424=2.177. ∑iγi2xi=38.518\sum_i\gamma_{i2}x_i=38.518, so μ2=38.518/5.576=6.908\mu_2=38.518/5.576=6.908. Both means jump toward their groups.
  6. 6Variances. Using the new means, ∑iγi1(xi−μ1)2=4.437\sum_i\gamma_{i1}(x_i-\mu_1)^2=4.437, so σ12=4.437/6.424=0.691\sigma_1^2=4.437/6.424=0.691. Likewise σ22=6.068/5.576=1.088\sigma_2^2=6.068/5.576=1.088. The left group is tighter, and its variance has already shrunk.
  7. 7Check. The new log-likelihood is −23.364-23.364, up from −28.279-28.279, as the theory guarantees.

Repeating the two steps, the ambiguous point is quickly handed to the wide right component. The narrow left Gaussian says 4.6 is very unlikely, while the wide right one finds it plausible.

iterweight 1mean 1mean 2var 1var 2P(comp. 1 | 4.6)log L
00.5003.0006.0001.0001.0000.426−28.279
10.5352.1776.9080.6911.0880.192−23.364
20.5162.0816.8210.4651.2230.014−22.715
30.5012.0066.7540.2851.3320.000−22.286
40.5001.9996.7470.2701.3470.000−22.280
60.5001.9996.7460.2701.3490.000−22.280
Parameters at the start of each iteration (iteration 0 is the initial guess). The last-but-one column is γ7,1\gamma_{7,1}, the left component's responsibility for x=4.6x=4.6. EM stops after 6 iterations, when Δlog⁡L<10−6\Delta\log L<10^{-6}.

Classifying a new point, x* = 4.0

  1. 1Fitted model. π=(0.500,0.500)\pi=(0.500,0.500), μ=(1.999,6.746)\mu=(1.999,6.746), σ2=(0.270,1.349)\sigma^2=(0.270,1.349), so σ=(0.519,1.162)\sigma=(0.519,1.162).
  2. 2Distances in standard deviations. x∗x^* is 2.002.00 from μ1\mu_1, which is 3.85 σ13.85\,\sigma_1. It is 2.752.75 from μ2\mu_2, which is only 2.36 σ22.36\,\sigma_2.
  3. 3Weighted densities. π1N(4∣μ1,σ12)=0.00023\pi_1\mathcal N(4\mid\mu_1,\sigma_1^2)=0.00023 and π2N(4∣μ2,σ22)=0.01050\pi_2\mathcal N(4\mid\mu_2,\sigma_2^2)=0.01050.
  4. 4Responsibilities. γ=(0.021, 0.979)\gamma=(0.021,\,0.979): component 2 claims it with 97.9% probability. k-means on the same data finds centres 2.00 and 6.75, puts its boundary at the midpoint 4.375, and would assign x∗=4.0x^*=4.0 to the left cluster.
  5. 5Density. p(4.0)=0.0107p(4.0)=0.0107, so −log⁡p=4.53-\log p=4.53. That is a usable anomaly score: a point at 4.6 scores 3.47, and one at 5.0 scores 2.89.
  6. 6Decision boundary. γ1=γ2\gamma_1=\gamma_2 at x=3.566x=3.566, not at the midpoint of the means. Because the variances differ, the boundary is quadratic, and there is a second crossing far out at x=−1.94x=-1.94, where the wide component's tails take over again.

See it run

▸ EM on 12 points · 1-Dinit · frame 1/14
πₖ𝒩(x | μₖ, σₖ²) · - - mixture p(x) · drag ▲ means, ◆ probe012345678910γᵢₖ — each point's membership, stacked to 1log L-22.28-28.28frames → (flat on E-steps, up on M-steps, never down)
  1. 1init μₖ, σₖ² = 1, πₖ = 1/K
  2. 2repeat
  3. 3 E: γᵢₖ ∝ πₖ 𝒩(xᵢ | μₖ, σₖ²)
  4. 4 M: Nₖ = Σᵢ γᵢₖ, πₖ = Nₖ / n
  5. 5 μₖ = Σᵢ γᵢₖ xᵢ / Nₖ
  6. 6 σₖ² = Σᵢ γᵢₖ(xᵢ−μₖ)²/Nₖ + ε
  7. 7until Δ log L < 1e-6
  8. k=1 π 0.50 · μ 3.00 · σ² 1.00
    k=2 π 0.50 · μ 6.00 · σ² 1.00
    log L -28.279
    ◆ x* = 4.0 → γ —
    p(x*) = 0.1480 · −log p = 1.91

Initial guess: μ = (3.00, 6.00), every σ² = 1, every π = 1/2. Points are grey — no responsibilities yet. log L = -28.279.

Kfloor ε
fig.EM on the twelve worked-example points. Step through the E-steps (points and γ bars recolour, log L stays flat) and M-steps (curves refit, log L rises). From μ = (3, 6), log L climbs from −28.279 to −22.280 in 6 iterations, and x = 4.6 goes from 0.43/0.57 to fully right. Drag a ▲ mean to restart EM from a new guess: start μ₁ at 8.5 instead and EM settles in a worse local optimum (log L −23.596), where a wide left component claims 4.6. Drag the ◆ probe to read γ and −log p for any x: at x* = 4.0 the converged model gives γ = (0.02, 0.98). Switch to K = 3 (start μ = 3, 4.5, 6) and one component collapses onto the lone point 4.6. Its variance hits the 1e-6 floor and log L jumps to −14.948. With a floor of 0.1 it stops at σ² = 0.10 and log L = −20.878.

The step-through below fits three elongated, tilted clusters in 2-D with full covariances, which the 1-D example cannot show. Each M-step stretches and rotates its ellipse to match the weighted scatter of its points.

▸ Gaussian mixture · EMstep 1/17
log-likelihood, never falls
iteration 0log-lik -37.71π = 0.33 / 0.33 / 0.33
  1. 1init μₖ, Σₖ, πₖ
  2. 2repeat:
  3. 3 E: γᵢₖ ∝ πₖ 𝒩(xᵢ | μₖ, Σₖ)
  4. 4 M: Nₖ ← Σᵢ γᵢₖ, πₖ ← Nₖ/n
  5. 5 μₖ ← Σᵢ γᵢₖ xᵢ / Nₖ
  6. 6 Σₖ ← Σᵢ γᵢₖ (xᵢ−μₖ)(·)ᵀ / Nₖ
  7. 7until log-lik stops rising

Start: three means on data points (two inside the elongated cluster), Σₖ = 0.02·I, πₖ = 1/3. Log-likelihood -37.71.

fig.Full-covariance EM in 2-D. Blended colours are soft memberships, and each M-step refits every ellipse to its weighted points. Log-likelihood rises from −37.7 to 89.4 and never falls.

The algorithm, step by step

EM for a Gaussian mixture

  1. 1Initialize {πk,μk,Σk}\{\pi_k,\mu_k,\Sigma_k\}, usually from k-means: the centroids, the covariance of each k-means cluster, and the cluster sizes divided by nn.
  2. 2E-step. For every point and component, compute log⁡πk+log⁡N(xi∣μk,Σk)\log\pi_k+\log\mathcal N(x_i\mid\mu_k,\Sigma_k). Normalize each row with log-sum-exp to get γik\gamma_{ik}, and keep log⁡p(xi)\log p(x_i) to evaluate log⁡L\log L.
  3. 3M-step. Nk=∑iγikN_k=\sum_i\gamma_{ik}; πk=Nk/n\pi_k=N_k/n; μk=∑iγikxi/Nk\mu_k=\sum_i\gamma_{ik}x_i/N_k; Σk=∑iγik(xi−μk)(xi−μk)⊤/Nk+εI\Sigma_k=\sum_i\gamma_{ik}(x_i-\mu_k)(x_i-\mu_k)^\top/N_k+\varepsilon I, or its diagonal, spherical or tied version.
  4. 4Converge? Stop when log⁡L\log L improves by less than tol, or after max_iter iterations. If a component's NkN_k falls near 0, re-seed it.
  5. 5Restart ninitn_{\text{init}} times from different seeds and keep the run with the highest log⁡L\log L.
  6. 6Use γik\gamma_{ik} as soft labels, arg⁡max⁡kγik\arg\max_k\gamma_{ik} as hard labels, and −log⁡p(x)-\log p(x) as an outlier score.

Complexity. Each iteration costs O(nKd2)O(nKd^2) to evaluate the Mahalanobis distances with full covariances (O(nKd)O(nKd) for diagonal), plus O(Kd3)O(Kd^3) to Cholesky-factor the covariances. Memory is O(nK)O(nK) for the responsibilities plus O(Kd2)O(Kd^2) for the covariances. Convergence is linear and can be slow when components overlap heavily. Then EM may need hundreds of iterations, where k-means would stop after a few.

Variants and practical tuning

covariance_typeEach cluster is…Covariance paramsUse when
fullany rotated ellipsoidK·d(d+1)/2low d, plenty of points per cluster
tiedthe same ellipsoid, shiftedd(d+1)/2clusters differ only in location (this is LDA's assumption)
diagaxis-aligned ellipsoidK·dhigher d, features roughly uncorrelated within a cluster
sphericala ball with its own radiusKa k-means-like model with soft labels and unequal sizes
Covariance types. Params counts are for K components in d dimensions (the covariance part only).

Other variants

Choosing K
fit K=1..Kmax⁡K=1..K_{\max} and take the lowest BIC (−2log⁡L+plog⁡n-2\log L+p\log n). It is consistent for density estimation, but tends to over-split non-Gaussian clusters. AIC (−2log⁡L+2p-2\log L+2p) penalizes less and picks more components.
Bayesian GMM
put a Dirichlet (or Dirichlet-process) prior on the weights and fit by variational inference (BayesianGaussianMixture). Unneeded components get weights near 0, so you can set K generously and let the data switch components off.
Hard / classification EM
replace γ\gamma by a one-hot argmax in the E-step. It is faster and gives crisp labels, but it no longer maximizes the likelihood of the mixture. With spherical, equal covariances it is k-means.
MAP-EM
add a prior (for example inverse-Wishart) on Σk\Sigma_k. The M-step becomes a shrunk covariance, a principled version of reg_covar.
Online / mini-batch EM
update running sums of γ\gamma, γx\gamma x and γxx⊤\gamma xx^\top from batches, for streams or very large nn.
EM beyond Gaussians
the same E/M recipe fits mixtures of Bernoullis or multinomials (text, binary data), hidden Markov models (Baum–Welch), and missing-data problems.

In code

python
1import numpy as np
2
3def log_gauss(X, mu, S):
4 """log N(x | mu, S) for every row of X."""
5 d = X.shape[1]
6 L = np.linalg.cholesky(S)
7 z = np.linalg.solve(L, (X - mu).T) # whitened residuals
8 return -0.5 * (z**2).sum(0) - np.log(np.diag(L)).sum() - 0.5 * d * np.log(2 * np.pi)
9
10def gmm_em(X, mu, S, pi, iters=200, tol=1e-6, reg=1e-6):
11 n, d = X.shape
12 K = len(pi)
13 prev = -np.inf
14 for it in range(iters):
15 # E-step: responsibilities via log-sum-exp
16 logw = np.stack([np.log(pi[k]) + log_gauss(X, mu[k], S[k]) for k in range(K)], 1)
17 m = logw.max(1, keepdims=True)
18 logp = m[:, 0] + np.log(np.exp(logw - m).sum(1))
19 ll = logp.sum()
20 g = np.exp(logw - logp[:, None])
21 if ll - prev < tol:
22 break
23 prev = ll
24 # M-step: weighted counts, means, covariances
25 Nk = g.sum(0)
26 pi = Nk / n
27 mu = (g.T @ X) / Nk[:, None]
28 S = np.stack([((g[:, k, None] * (X - mu[k])).T @ (X - mu[k])) / Nk[k] + reg * np.eye(d)
29 for k in range(K)])
30 return pi, mu, S, g, ll, it
31
32X = np.array([1.2, 1.6, 1.9, 2.1, 2.4, 2.8, 4.6, 6.2, 6.8, 7.1, 7.5, 8.3])[:, None]
33pi, mu, S, g, ll, it = gmm_em(X, mu=np.array([[3.0], [6.0]]),
34 S=np.array([[[1.0]], [[1.0]]]), pi=np.array([0.5, 0.5]))
35print("iterations:", it)
36print("pi:", pi.round(3), "mu:", mu.ravel().round(3), "var:", S.ravel().round(3))
37print("log-likelihood:", round(ll, 3))
38print("gamma(left) for x=4.6:", g[6, 0].round(4))
39# iterations: 6
40# pi: [0.5 0.5] mu: [1.999 6.746] var: [0.27 1.349]
41# log-likelihood: -22.28
42# gamma(left) for x=4.6: 0.0
From scratch: full covariances, log-sum-exp E-step, run on the worked example from μ = (3, 6).
python
1import numpy as np
2from sklearn.mixture import GaussianMixture
3
4X = np.array([1.2, 1.6, 1.9, 2.1, 2.4, 2.8, 4.6, 6.2, 6.8, 7.1, 7.5, 8.3])[:, None]
5
6gmm = GaussianMixture(n_components=2, covariance_type="full",
7 means_init=[[3.0], [6.0]], random_state=0).fit(X)
8print("pi:", gmm.weights_.round(3), "mu:", gmm.means_.ravel().round(3),
9 "var:", gmm.covariances_.ravel().round(3))
10print("log-likelihood:", round(gmm.score(X) * len(X), 3)) # score() is the per-point mean
11print("P(component | x=4.0):", gmm.predict_proba([[4.0]]).round(3))
12print("-log p(4.0):", round(-gmm.score_samples([[4.0]])[0], 3))
13
14# Choosing K: BIC with the default tiny reg_covar vs a sensible one
15for reg in (1e-6, 0.1):
16 bics = [GaussianMixture(k, n_init=10, reg_covar=reg, random_state=0).fit(X).bic(X)
17 for k in (1, 2, 3)]
18 print(f"reg_covar={reg}: BIC K=1,2,3 =", np.round(bics, 2))
19
20# pi: [0.5 0.5] mu: [1.999 6.746] var: [0.27 1.349]
21# log-likelihood: -22.28
22# P(component | x=4.0): [[0.021 0.979]]
23# -log p(4.0): 4.535
24# reg_covar=1e-06: BIC K=1,2,3 = [61.38 56.99 49.77]
25# reg_covar=0.1: BIC K=1,2,3 = [61.38 57.27 61.64]
scikit-learn on the same data: soft labels, density scores, and choosing K by BIC.

Use when

  • +Clusters are roughly elliptical but overlap, are tilted, or differ in size and spread.
  • +You need calibrated soft memberships, or an explicit density for anomaly scores (−log⁡p(x)-\log p(x)), sampling or likelihood-based model comparison.
  • +You want a principled way to choose KK (BIC) and a model you can write down and inspect: weights, means and covariances.
  • +Moderate dimensions (or diagonal covariances) with enough points per component to estimate its covariance.

Avoid when

  • −Clusters have arbitrary shapes (rings, moons, filaments): use DBSCAN, HDBSCAN or spectral clustering.
  • −High dd with few points: a full covariance has d(d+1)/2d(d+1)/2 parameters per component. Use diagonal or tied covariances, or reduce dimension (PCA) first.
  • −Heavy noise or outliers: every point must be explained by some Gaussian, so outliers inflate covariances. Add a uniform noise component, or use HDBSCAN.
  • −Very large n where k-means' speed and simplicity are enough, or where you only need hard labels.

Where it's used

Speaker verification

GMM–UBM systems model each speaker's acoustic (MFCC) features as a mixture adapted from a universal background GMM. This was the standard before neural embeddings.

Background subtraction

Each pixel's colour over time is modelled by a small GMM (Stauffer–Grimson). Pixels that fit no background component are foreground.

Anomaly and fraud detection

Fit a GMM to normal transactions or sensor readings, and flag points whose −log⁡p(x)-\log p(x) exceeds a threshold set on validation data.

Science: separating populations

Flow-cytometry cell types, stellar populations in colour–magnitude space, gene-expression subtypes. These are overlapping groups where soft membership is the honest answer.

Check yourself

Two components have weights π = (0.2, 0.8). At a point x, their densities are 𝒩₁(x) = 0.3 and 𝒩₂(x) = 0.1. What is γ₁, the responsibility of component 1?

Check yourself

What does EM guarantee at every iteration?

Check yourself

A GMM with K = 3 full-covariance components in d = 10 dimensions has how many free parameters?

Check yourself

On the 12 worked-example points with reg_covar = 1e-6, BIC prefers K = 3 (49.77) over K = 2 (56.99). Why?

Remember

  • ✓A GMM models p(x)=∑kπkN(x∣μk,Σk)p(x)=\sum_k\pi_k\mathcal N(x\mid\mu_k,\Sigma_k). Its clusters are ellipsoids with their own size, shape and weight, and memberships are probabilities.
  • ✓E-step: responsibilities γik\gamma_{ik} by Bayes' rule. M-step: the weighted frequency, mean and covariance, with each point counted γik\gamma_{ik} times.
  • ✓EM maximizes a Jensen lower bound that is tight after each E-step, so the log-likelihood never decreases. It finds a local optimum, so use several restarts.
  • ✓The likelihood is unbounded, because a component can collapse onto one point. Keep a covariance floor (reg_covar) and distrust components with tiny N_k.
  • ✓Choose K with BIC, and choose the covariance type by dimension and data size. k-means is the hard, spherical, zero-variance special case.
← back to ML Algorithms roadmap