Definition
A Gaussian mixture model (GMM) says the data were drawn from Gaussian distributions, each with its own mean , covariance and weight . Expectation–maximization (EM) fits it by alternating two steps. The E-step softly assigns each point to the components. The M-step re-estimates each component from its weighted points.
At a glance
- Task
- Soft clustering and density estimation (unsupervised)
- Input → output
- points in → responsibilities (the probability that point came from component ) and a density you can evaluate anywhere
- Learns
- for each of components: weight , mean , covariance
- Training
- EM, per iteration plus to factor the covariances; usually tens to hundreds of iterations, several restarts
- Prediction
- responsibilities by Bayes' rule, per point
- Key hyperparameters
- number of components ; covariance type (full, tied, diagonal, spherical); initialization and number of restarts; covariance floor (
reg_covar); tolerance - Assumes
- each cluster is roughly Gaussian (an ellipsoid), and points are independent
The problem
k-means puts every point into exactly one cluster, and every cluster boundary is the perpendicular bisector between two centroids. Hidden in that rule are three assumptions: clusters are round, about equally wide, and every point belongs fully to one of them. Real data often break all three. One group may be tight and another diffuse. Clusters can be elongated or tilted. And a point that sits between two groups really is ambiguous.
We also often want more than labels. We want a density that says how typical a new point is, so we can flag anomalies, sample synthetic data, or compare models by likelihood. A GMM provides both: soft memberships, and a proper probability model that can be fitted by maximum likelihood.
The idea
Tell a story about how each point was generated. First roll a -sided die with probabilities to pick a component . Then draw the point from that component's Gaussian, . We see the points, but the die rolls are hidden.
This is a chicken-and-egg problem. If we knew every , fitting would be trivial: take the mean and covariance of each group, and use group sizes for the weights. If we knew the parameters, Bayes' rule would tell us how likely each is. EM breaks the circle by alternating. Guess the parameters. Compute soft memberships. Refit the parameters as if those memberships were fractional counts. Repeat. Each round provably increases the likelihood until it levels off.
Vocabulary
- Component
- one Gaussian in the mixture; it plays the role of a cluster
- Mixing weight
- , : the prior probability that a point comes from component
- Latent variable
- , the unobserved component that generated
- Responsibility
- , the posterior probability that component generated . Each row sums to 1
- Soft count
- , the effective number of points that component owns. It need not be an integer
- Log-likelihood
- , the objective EM climbs
- E-step / M-step
- Expectation: compute responsibilities from the current parameters. Maximization: refit the parameters from the responsibilities
- Covariance type
- how much shape each may have: full (any ellipsoid), diagonal (axis-aligned), spherical (), tied (all components share one )
- Degenerate component
- a component that collapses onto one point (or onto a lower-dimensional subspace). Its and the likelihood
The math
The model. The density of a point is a weighted sum of Gaussian bumps:
- number of components
- mixing weight of component ; ,
- mean of component (a -vector)
- covariance of component (, symmetric positive definite)
- determinant of Σ; it measures the volume of the ellipsoid
- squared Mahalanobis distance: distance measured in units of the component's own spread
The objective. Maximum likelihood over independent points:
- all parameters
- the -th data point
The log of a sum is the obstacle. With one Gaussian the log cancels the exponential and you get a closed form: the sample mean and covariance. Here the log sits outside the sum over , the parameters of all components are coupled, and setting the gradient to zero does not give a closed form.
E-step: responsibilities. Bayes' rule gives the posterior over the hidden component, with prior and likelihood :
- responsibility of component for point , in ;
- the hidden component label of
Deriving the M-step. Differentiate with respect to . The derivative of is the -th term divided by the whole sum, and that ratio is exactly :
- gradient of with respect to
This is not a closed form, because itself depends on . But if we freeze at its current value, the equation is a weighted average. Doing the same for , and for with a Lagrange multiplier for , gives the full M-step:
- soft count: how many points component k effectively owns
- number of data points
- small covariance floor added for numerical safety (reg_covar in scikit-learn)
These are the ordinary mean, covariance and frequency, with each point counted times instead of 0 or 1 times.
Why the likelihood never decreases. Take any distribution over components for each point. Jensen's inequality ( is concave) gives a lower bound on each term:
- any probability distribution over the components for point
- Jensen's inequality; equality holds exactly when
The E-step sets , which makes the bound touch at the current parameters. The M-step maximizes the bound over . Here the log has moved inside the sum, so the maximization splits into one weighted Gaussian fit per component, which is the closed form above. The bound went up, and is at least as large as the bound. So . EM therefore climbs monotonically to a local maximum or saddle point. It is not guaranteed to find the global one.
Hyperparameters, mathematically.
At a glance
- K
- More components can never lower the best achievable , because a -mixture can copy any -mixture. So alone cannot choose . Penalize it: , where is the number of free parameters. Lower BIC is better.
- Covariance type
- Free parameters with components in dimensions: weights , means , plus covariances full , diagonal , spherical , tied . With and , full covariances need 12,750 covariance numbers; diagonal ones need 500.
- Floor ε (reg_covar)
- peaks at . Put on a data point and shrink , and that point's density goes to . So the likelihood is unbounded, and the global "maximum" is degenerate. Adding caps each peak: in 1-D the cap is , which is 398.9 for and 1.26 for .
- Initialization
- EM converges to the local optimum whose basin contains the starting point. Common starts are k-means centroids (scikit-learn's default), k-means++ seeding, or random responsibilities. Run several restarts (n_init) and keep the highest .
- Tolerance
- stop when (or the per-point average) drops below tol. Too loose and you stop on a plateau; too tight and you waste iterations.
Worked example: twelve numbers, two components
Twelve values on a line, made of a tight group near 2, a wider group near 7, and one ambiguous point at 4.6. We fit in 1-D, so is a variance . Start from , , . The table shows the first E-step. With equal weights the 's cancel, so .
| i | x | density, comp. 1 | density, comp. 2 | P(comp. 1 | x) | P(comp. 2 | x) |
|---|---|---|---|---|---|
| 1 | 1.2 | 0.0790 | 0.0000 | 1.000 | 0.000 |
| 2 | 1.6 | 0.1497 | 0.0000 | 1.000 | 0.000 |
| 3 | 1.9 | 0.2179 | 0.0001 | 1.000 | 0.000 |
| 4 | 2.1 | 0.2661 | 0.0002 | 0.999 | 0.001 |
| 5 | 2.4 | 0.3332 | 0.0006 | 0.998 | 0.002 |
| 6 | 2.8 | 0.3910 | 0.0024 | 0.994 | 0.006 |
| 7 | 4.6 | 0.1109 | 0.1497 | 0.426 | 0.574 |
| 8 | 6.2 | 0.0024 | 0.3910 | 0.006 | 0.994 |
| 9 | 6.8 | 0.0003 | 0.2897 | 0.001 | 0.999 |
| 10 | 7.1 | 0.0001 | 0.2179 | 0.000 | 1.000 |
| 11 | 7.5 | 0.0000 | 0.1295 | 0.000 | 1.000 |
| 12 | 8.3 | 0.0000 | 0.0283 | 0.000 | 1.000 |
Iteration 1 by hand
- 1Initial log-likelihood. .
- 2E-step for . and . So . The point is 1.4 from but 1.6 from , so it leans slightly right.
- 3Soft counts. and . The six left points contribute almost 1 each to , and adds its 0.426.
- 4Weights. and .
- 5Means. , so . , so . Both means jump toward their groups.
- 6Variances. Using the new means, , so . Likewise . The left group is tighter, and its variance has already shrunk.
- 7Check. The new log-likelihood is , up from , as the theory guarantees.
Repeating the two steps, the ambiguous point is quickly handed to the wide right component. The narrow left Gaussian says 4.6 is very unlikely, while the wide right one finds it plausible.
| iter | weight 1 | mean 1 | mean 2 | var 1 | var 2 | P(comp. 1 | 4.6) | log L |
|---|---|---|---|---|---|---|---|
| 0 | 0.500 | 3.000 | 6.000 | 1.000 | 1.000 | 0.426 | −28.279 |
| 1 | 0.535 | 2.177 | 6.908 | 0.691 | 1.088 | 0.192 | −23.364 |
| 2 | 0.516 | 2.081 | 6.821 | 0.465 | 1.223 | 0.014 | −22.715 |
| 3 | 0.501 | 2.006 | 6.754 | 0.285 | 1.332 | 0.000 | −22.286 |
| 4 | 0.500 | 1.999 | 6.747 | 0.270 | 1.347 | 0.000 | −22.280 |
| 6 | 0.500 | 1.999 | 6.746 | 0.270 | 1.349 | 0.000 | −22.280 |
Classifying a new point, x* = 4.0
- 1Fitted model. , , , so .
- 2Distances in standard deviations. is from , which is . It is from , which is only .
- 3Weighted densities. and .
- 4Responsibilities. : component 2 claims it with 97.9% probability. k-means on the same data finds centres 2.00 and 6.75, puts its boundary at the midpoint 4.375, and would assign to the left cluster.
- 5Density. , so . That is a usable anomaly score: a point at 4.6 scores 3.47, and one at 5.0 scores 2.89.
- 6Decision boundary. at , not at the midpoint of the means. Because the variances differ, the boundary is quadratic, and there is a second crossing far out at , where the wide component's tails take over again.
See it run
- 1init μₖ, σₖ² = 1, πₖ = 1/K
- 2repeat
- 3 E: γᵢₖ ∝ πₖ 𝒩(xᵢ | μₖ, σₖ²)
- 4 M: Nₖ = Σᵢ γᵢₖ, πₖ = Nₖ / n
- 5 μₖ = Σᵢ γᵢₖ xᵢ / Nₖ
- 6 σₖ² = Σᵢ γᵢₖ(xᵢ−μₖ)²/Nₖ + ε
- 7until Δ log L < 1e-6
- k=1 π 0.50 · μ 3.00 · σ² 1.00k=2 π 0.50 · μ 6.00 · σ² 1.00log L -28.279◆ x* = 4.0 → γ —
p(x*) = 0.1480 · −log p = 1.91
Initial guess: μ = (3.00, 6.00), every σ² = 1, every π = 1/2. Points are grey — no responsibilities yet. log L = -28.279.
The step-through below fits three elongated, tilted clusters in 2-D with full covariances, which the 1-D example cannot show. Each M-step stretches and rotates its ellipse to match the weighted scatter of its points.
- 1init μₖ, Σₖ, πₖ
- 2repeat:
- 3 E: γᵢₖ ∝ πₖ 𝒩(xᵢ | μₖ, Σₖ)
- 4 M: Nₖ ← Σᵢ γᵢₖ, πₖ ← Nₖ/n
- 5 μₖ ← Σᵢ γᵢₖ xᵢ / Nₖ
- 6 Σₖ ← Σᵢ γᵢₖ (xᵢ−μₖ)(·)ᵀ / Nₖ
- 7until log-lik stops rising
Start: three means on data points (two inside the elongated cluster), Σₖ = 0.02·I, πₖ = 1/3. Log-likelihood -37.71.
The algorithm, step by step
EM for a Gaussian mixture
- 1Initialize , usually from k-means: the centroids, the covariance of each k-means cluster, and the cluster sizes divided by .
- 2E-step. For every point and component, compute . Normalize each row with log-sum-exp to get , and keep to evaluate .
- 3M-step. ; ; ; , or its diagonal, spherical or tied version.
- 4Converge? Stop when improves by less than tol, or after max_iter iterations. If a component's falls near 0, re-seed it.
- 5Restart times from different seeds and keep the run with the highest .
- 6Use as soft labels, as hard labels, and as an outlier score.
Complexity. Each iteration costs to evaluate the Mahalanobis distances with full covariances ( for diagonal), plus to Cholesky-factor the covariances. Memory is for the responsibilities plus for the covariances. Convergence is linear and can be slow when components overlap heavily. Then EM may need hundreds of iterations, where k-means would stop after a few.
Variants and practical tuning
| covariance_type | Each cluster is… | Covariance params | Use when |
|---|---|---|---|
| full | any rotated ellipsoid | K·d(d+1)/2 | low d, plenty of points per cluster |
| tied | the same ellipsoid, shifted | d(d+1)/2 | clusters differ only in location (this is LDA's assumption) |
| diag | axis-aligned ellipsoid | K·d | higher d, features roughly uncorrelated within a cluster |
| spherical | a ball with its own radius | K | a k-means-like model with soft labels and unequal sizes |
Other variants
- Choosing K
- fit and take the lowest BIC (). It is consistent for density estimation, but tends to over-split non-Gaussian clusters. AIC () penalizes less and picks more components.
- Bayesian GMM
- put a Dirichlet (or Dirichlet-process) prior on the weights and fit by variational inference (
BayesianGaussianMixture). Unneeded components get weights near 0, so you can set K generously and let the data switch components off. - Hard / classification EM
- replace by a one-hot argmax in the E-step. It is faster and gives crisp labels, but it no longer maximizes the likelihood of the mixture. With spherical, equal covariances it is k-means.
- MAP-EM
- add a prior (for example inverse-Wishart) on . The M-step becomes a shrunk covariance, a principled version of reg_covar.
- Online / mini-batch EM
- update running sums of , and from batches, for streams or very large .
- EM beyond Gaussians
- the same E/M recipe fits mixtures of Bernoullis or multinomials (text, binary data), hidden Markov models (Baum–Welch), and missing-data problems.
In code
1import numpy as np23def log_gauss(X, mu, S):4 """log N(x | mu, S) for every row of X."""5 d = X.shape[1]6 L = np.linalg.cholesky(S)7 z = np.linalg.solve(L, (X - mu).T) # whitened residuals8 return -0.5 * (z**2).sum(0) - np.log(np.diag(L)).sum() - 0.5 * d * np.log(2 * np.pi)910def gmm_em(X, mu, S, pi, iters=200, tol=1e-6, reg=1e-6):11 n, d = X.shape12 K = len(pi)13 prev = -np.inf14 for it in range(iters):15 # E-step: responsibilities via log-sum-exp16 logw = np.stack([np.log(pi[k]) + log_gauss(X, mu[k], S[k]) for k in range(K)], 1)17 m = logw.max(1, keepdims=True)18 logp = m[:, 0] + np.log(np.exp(logw - m).sum(1))19 ll = logp.sum()20 g = np.exp(logw - logp[:, None])21 if ll - prev < tol:22 break23 prev = ll24 # M-step: weighted counts, means, covariances25 Nk = g.sum(0)26 pi = Nk / n27 mu = (g.T @ X) / Nk[:, None]28 S = np.stack([((g[:, k, None] * (X - mu[k])).T @ (X - mu[k])) / Nk[k] + reg * np.eye(d)29 for k in range(K)])30 return pi, mu, S, g, ll, it3132X = np.array([1.2, 1.6, 1.9, 2.1, 2.4, 2.8, 4.6, 6.2, 6.8, 7.1, 7.5, 8.3])[:, None]33pi, mu, S, g, ll, it = gmm_em(X, mu=np.array([[3.0], [6.0]]),34 S=np.array([[[1.0]], [[1.0]]]), pi=np.array([0.5, 0.5]))35print("iterations:", it)36print("pi:", pi.round(3), "mu:", mu.ravel().round(3), "var:", S.ravel().round(3))37print("log-likelihood:", round(ll, 3))38print("gamma(left) for x=4.6:", g[6, 0].round(4))39# iterations: 640# pi: [0.5 0.5] mu: [1.999 6.746] var: [0.27 1.349]41# log-likelihood: -22.2842# gamma(left) for x=4.6: 0.01import numpy as np2from sklearn.mixture import GaussianMixture34X = np.array([1.2, 1.6, 1.9, 2.1, 2.4, 2.8, 4.6, 6.2, 6.8, 7.1, 7.5, 8.3])[:, None]56gmm = GaussianMixture(n_components=2, covariance_type="full",7 means_init=[[3.0], [6.0]], random_state=0).fit(X)8print("pi:", gmm.weights_.round(3), "mu:", gmm.means_.ravel().round(3),9 "var:", gmm.covariances_.ravel().round(3))10print("log-likelihood:", round(gmm.score(X) * len(X), 3)) # score() is the per-point mean11print("P(component | x=4.0):", gmm.predict_proba([[4.0]]).round(3))12print("-log p(4.0):", round(-gmm.score_samples([[4.0]])[0], 3))1314# Choosing K: BIC with the default tiny reg_covar vs a sensible one15for reg in (1e-6, 0.1):16 bics = [GaussianMixture(k, n_init=10, reg_covar=reg, random_state=0).fit(X).bic(X)17 for k in (1, 2, 3)]18 print(f"reg_covar={reg}: BIC K=1,2,3 =", np.round(bics, 2))1920# pi: [0.5 0.5] mu: [1.999 6.746] var: [0.27 1.349]21# log-likelihood: -22.2822# P(component | x=4.0): [[0.021 0.979]]23# -log p(4.0): 4.53524# reg_covar=1e-06: BIC K=1,2,3 = [61.38 56.99 49.77]25# reg_covar=0.1: BIC K=1,2,3 = [61.38 57.27 61.64]Use when
- +Clusters are roughly elliptical but overlap, are tilted, or differ in size and spread.
- +You need calibrated soft memberships, or an explicit density for anomaly scores (), sampling or likelihood-based model comparison.
- +You want a principled way to choose (BIC) and a model you can write down and inspect: weights, means and covariances.
- +Moderate dimensions (or diagonal covariances) with enough points per component to estimate its covariance.
Avoid when
- −Clusters have arbitrary shapes (rings, moons, filaments): use DBSCAN, HDBSCAN or spectral clustering.
- −High with few points: a full covariance has parameters per component. Use diagonal or tied covariances, or reduce dimension (PCA) first.
- −Heavy noise or outliers: every point must be explained by some Gaussian, so outliers inflate covariances. Add a uniform noise component, or use HDBSCAN.
- −Very large n where k-means' speed and simplicity are enough, or where you only need hard labels.
Where it's used
Speaker verification
GMM–UBM systems model each speaker's acoustic (MFCC) features as a mixture adapted from a universal background GMM. This was the standard before neural embeddings.
Background subtraction
Each pixel's colour over time is modelled by a small GMM (Stauffer–Grimson). Pixels that fit no background component are foreground.
Anomaly and fraud detection
Fit a GMM to normal transactions or sensor readings, and flag points whose exceeds a threshold set on validation data.
Science: separating populations
Flow-cytometry cell types, stellar populations in colour–magnitude space, gene-expression subtypes. These are overlapping groups where soft membership is the honest answer.
Check yourself
Two components have weights π = (0.2, 0.8). At a point x, their densities are 𝒩₁(x) = 0.3 and 𝒩₂(x) = 0.1. What is γ₁, the responsibility of component 1?
Check yourself
What does EM guarantee at every iteration?
Check yourself
A GMM with K = 3 full-covariance components in d = 10 dimensions has how many free parameters?
Check yourself
On the 12 worked-example points with reg_covar = 1e-6, BIC prefers K = 3 (49.77) over K = 2 (56.99). Why?
Remember
- ✓A GMM models . Its clusters are ellipsoids with their own size, shape and weight, and memberships are probabilities.
- ✓E-step: responsibilities by Bayes' rule. M-step: the weighted frequency, mean and covariance, with each point counted times.
- ✓EM maximizes a Jensen lower bound that is tight after each E-step, so the log-likelihood never decreases. It finds a local optimum, so use several restarts.
- ✓The likelihood is unbounded, because a component can collapse onto one point. Keep a covariance floor (reg_covar) and distrust components with tiny N_k.
- ✓Choose K with BIC, and choose the covariance type by dimension and data size. k-means is the hard, spherical, zero-variance special case.