Linear discriminant analysis

Model each class as a Gaussian with a shared covariance; Bayes' rule then gives a linear decision boundary.

17 min read1775 wordsadvanced8 viewsupdated 2026-10-07

Definition

Linear discriminant analysis (LDA) is a generative classifier. It assumes the features of each class kk follow a multivariate Gaussian N(μk,Σ)\mathcal N(\mu_k,\Sigma) with its own mean but one covariance matrix shared by all classes, estimates the means, the shared covariance and the class priors in closed form, and classifies a new point with Bayes' rule. Because the covariance is shared, the quadratic terms cancel and every decision boundary is a straight line (a hyperplane).

At a glance

Task
Classification (binary or multi-class); the same fit also gives a supervised projection
Input → output
x∈Rdx\in\mathbb R^d → posterior P(y=k∣x)P(y=k\mid x) for every class, and the class with the largest one
Learns
class priors πk\pi_k, class means μk\mu_k, one pooled covariance Σ\Sigma — giving a weight vector ak=Σ−1μka_k=\Sigma^{-1}\mu_k and an offset ckc_k per class
Training
one pass for means and covariance, then one d×dd\times d solve: O(nd2+d3)O(nd^2+d^3); no iterations, no learning rate
Prediction
O(Kd)O(Kd) — one dot product per class
Hyperparameters
class priors, covariance shrinkage γ, solver (svd / lsqr / eigen); nothing else
Assumes
Gaussian classes with equal covariance; roughly linear separation; no extreme outliers

The problem

The simplest classifier anyone invents is nearest mean: average the training points of each class, then give a new point the label of the closest average. It fails in a specific, common way: it measures distance with a ruler that ignores the shape of the classes. If two features are correlated within each class — long fish are also wide fish — the clouds are tilted ellipses, and a point can be closer to the other class's centre in plain distance while sitting squarely on the long axis of its own class.

Logistic regression (see logistic-regression) fixes the boundary by fitting it directly, but it needs iterative optimisation, its coefficients run off to infinity on perfectly separable data, and with few examples per feature it is noisy. LDA takes the other route: write down a probability model of how each class generates its features, fit that model in closed form, and let Bayes' rule produce the boundary. The covariance matrix it estimates is exactly the ruler that nearest-mean was missing.

The idea

Picture each class as a Gaussian "cloud" — an ellipse of the same size, shape and orientation, just centred in a different place. To classify a point, ask which cloud makes it most likely, adjusted for how common each class is. Because the ellipses are identical copies, the place where two clouds are equally likely is a straight line. Measuring with the ellipse rather than with a circle is the Mahalanobis distance: LDA is nearest-mean classification after you "un-stretch" the space so each cloud becomes a circle.

Vocabulary

Class prior πₖ
probability of class kk before looking at xx; estimated as the class's share of the training set nk/nn_k/n
Class-conditional density
p(x∣y=k)p(x\mid y=k) — how the features of class kk are distributed; here N(μk,Σ)\mathcal N(\mu_k,\Sigma)
Pooled covariance Σ
one covariance matrix estimated from the deviations of every point from its own class mean; describes the common shape of all clouds
Discriminant function δₖ(x)
a score per class whose argmax is the Bayes decision; for LDA it is linear in xx
Mahalanobis distance
(x−μ)⊤Σ−1(x−μ)\sqrt{(x-\mu)^\top\Sigma^{-1}(x-\mu)} — distance measured in units of the cloud's own spread along each direction
Fisher direction w
Σ−1(μ1−μ0)\Sigma^{-1}(\mu_1-\mu_0): the line onto which the two classes, projected, are best separated relative to their spread
Shrinkage γ
blending the estimated covariance with a multiple of the identity, to keep it well-conditioned when data are scarce
Generative vs discriminative
LDA models p(x∣y)p(x\mid y) and p(y)p(y) and derives p(y∣x)p(y\mid x); logistic regression models p(y∣x)p(y\mid x) directly

The math

The model. Each class has a prior and a Gaussian density with a class-specific mean and a shared covariance:

P(y=k)=πk,p(x∣y=k)=1(2π)d/2∣Σ∣1/2exp⁡ ⁣(−12(x−μk)⊤Σ−1(x−μk))P(y=k)=\pi_k,\qquad p(x\mid y=k)=\frac{1}{(2\pi)^{d/2}|\Sigma|^{1/2}}\exp\!\Big(-\tfrac12(x-\mu_k)^\top\Sigma^{-1}(x-\mu_k)\Big)
πk\pi_k
prior probability of class kk, with ∑kπk=1\sum_k\pi_k=1
μk\mu_k
mean vector of class kk (dd entries)
Σ\Sigma
d×dd\times d covariance, the same for every class
∣Σ∣|\Sigma|
determinant of Σ (also shared, so it will cancel)
dd
number of features

Bayes' rule turns the model into a posterior. The denominator p(x)p(x) is the same for every class, so the most probable class maximises the numerator — or its logarithm:

P(y=k∣x)=πk p(x∣y=k)∑jπj p(x∣y=j),y^=arg⁡max⁡k  [log⁡πk+log⁡p(x∣y=k)]P(y=k\mid x)=\frac{\pi_k\,p(x\mid y=k)}{\sum_j \pi_j\,p(x\mid y=j)},\qquad \hat y=\arg\max_k\;\big[\log\pi_k+\log p(x\mid y=k)\big]
P(y=k∣x)P(y=k\mid x)
posterior probability of class kk after seeing xx
y^\hat y
predicted class

Why the boundary is linear. Take the log of the Gaussian and expand the quadratic form: −12(x−μk)⊤Σ−1(x−μk)=−12x⊤Σ−1x+x⊤Σ−1μk−12μk⊤Σ−1μk-\tfrac12(x-\mu_k)^\top\Sigma^{-1}(x-\mu_k)=-\tfrac12x^\top\Sigma^{-1}x+x^\top\Sigma^{-1}\mu_k-\tfrac12\mu_k^\top\Sigma^{-1}\mu_k. The constant −d2log⁡2π−12log⁡∣Σ∣-\tfrac d2\log 2\pi-\tfrac12\log|\Sigma| and the term −12x⊤Σ−1x-\tfrac12x^\top\Sigma^{-1}x are identical for every class because Σ\Sigma is shared, so they cannot change the argmax and we drop them. What is left is linear in xx:

δk(x)=x⊤Σ−1μk⏟ak  −  12μk⊤Σ−1μk+log⁡πk⏟ck\delta_k(x)=x^\top\underbrace{\Sigma^{-1}\mu_k}_{a_k}\;\underbrace{-\;\tfrac12\mu_k^\top\Sigma^{-1}\mu_k+\log\pi_k}_{c_k}
δk(x)\delta_k(x)
linear discriminant score of class kk; predict arg⁡max⁡kδk(x)\arg\max_k\delta_k(x)
aka_k
weight vector of class kk
ckc_k
offset of class kk: a penalty for a far-out mean plus the log prior

With two classes the boundary δ1(x)=δ0(x)\delta_1(x)=\delta_0(x) is a single hyperplane w⊤x+b=0w^\top x+b=0:

w=Σ−1(μ1−μ0),b=−12(μ0+μ1)⊤w+log⁡π1π0,P(y=1∣x)=σ(w⊤x+b)w=\Sigma^{-1}(\mu_1-\mu_0),\qquad b=-\tfrac12(\mu_0+\mu_1)^\top w+\log\frac{\pi_1}{\pi_0},\qquad P(y=1\mid x)=\sigma(w^\top x+b)
ww
normal vector of the boundary (the Fisher direction)
bb
offset: puts the boundary at the midpoint of the means, shifted by the prior log-odds
σ(z)\sigma(z)
logistic function 1/(1+e−z)1/(1+e^{-z})

Fitting. Maximum likelihood gives the obvious estimates. For the covariance we pool the deviations of every point from its own class mean and divide by n−Kn-K (the unbiased version, which is what most texts and scikit-learn's default solver use):

π^k=nkn,μ^k=1nk∑i: yi=kxi,Σ^=1n−K∑k=1K∑i: yi=k(xi−μ^k)(xi−μ^k)⊤=SWn−K\hat\pi_k=\frac{n_k}{n},\qquad \hat\mu_k=\frac1{n_k}\sum_{i:\,y_i=k}x_i,\qquad \hat\Sigma=\frac{1}{n-K}\sum_{k=1}^{K}\sum_{i:\,y_i=k}(x_i-\hat\mu_k)(x_i-\hat\mu_k)^\top=\frac{S_W}{n-K}
nkn_k
number of training points in class kk; n=∑knkn=\sum_k n_k
KK
number of classes
SWS_W
within-class scatter matrix: the double sum before dividing

The Mahalanobis view. Adding back the class-independent −12x⊤Σ−1x-\tfrac12x^\top\Sigma^{-1}x shows that LDA is a nearest-mean rule in a re-measured space:

y^=arg⁡min⁡k[12 Δk2(x)−log⁡πk],Δk2(x)=(x−μk)⊤Σ−1(x−μk)\hat y=\arg\min_k\Big[\tfrac12\,\Delta_k^2(x)-\log\pi_k\Big],\qquad \Delta_k^2(x)=(x-\mu_k)^\top\Sigma^{-1}(x-\mu_k)
Δk2(x)\Delta_k^2(x)
squared Mahalanobis distance from xx to the mean of class kk

If Σ=σ2I\Sigma=\sigma^2I the Mahalanobis distance is just Euclidean distance over σ\sigma, and LDA with equal priors is nearest mean. A general Σ\Sigma penalises a step along a direction in which the classes barely vary much more than a step along their long axis.

Fisher's derivation — same answer, no Gaussians. In 1936 Fisher asked a different question: which direction vv should we project onto so that the projected class means are far apart relative to the projected spread? Maximise the ratio of between-class to within-class scatter along vv:

J(v)=(v⊤(μ1−μ0))2v⊤SW vJ(v)=\frac{\big(v^\top(\mu_1-\mu_0)\big)^2}{v^\top S_W\,v}
J(v)J(v)
Fisher criterion: squared gap between projected means ÷ projected within-class scatter
vv
candidate projection direction (its length cancels)

Set the gradient to zero: ∇J=0⇒(v⊤SWv) (μ1−μ0) (v⊤(μ1−μ0))=(v⊤(μ1−μ0))2SWv\nabla J=0 \Rightarrow (v^\top S_W v)\,(\mu_1-\mu_0)\,\big(v^\top(\mu_1-\mu_0)\big)=\big(v^\top(\mu_1-\mu_0)\big)^2 S_W v. Both bracketed quantities are scalars, so SWv∝μ1−μ0S_W v\propto \mu_1-\mu_0, i.e. v∝SW−1(μ1−μ0)∝wv\propto S_W^{-1}(\mu_1-\mu_0)\propto w. Fisher's best projection is the normal of the Gaussian LDA boundary, and the maximum value is J∗=(μ1−μ0)⊤SW−1(μ1−μ0)J^\ast=(\mu_1-\mu_0)^\top S_W^{-1}(\mu_1-\mu_0). With KK classes the same idea yields up to K−1K-1 directions — that use of LDA as a supervised dimensionality reducer has its own article, lda-for-dimensionality-reduction.

Hyperparameter 1: priors. Priors appear only in bb, through log⁡(π1/π0)\log(\pi_1/\pi_0). Changing them leaves ww alone and slides the boundary parallel to itself by log⁡(π1/π0)/∥w∥\log(\pi_1/\pi_0)/\lVert w\rVert along ww, towards the rarer class. Set them to the class frequencies you expect at deployment, not necessarily those in the training sample.

Σ^(γ)=(1−γ) Σ^+γ tr⁡Σ^d I,0≤γ≤1\hat\Sigma(\gamma)=(1-\gamma)\,\hat\Sigma+\gamma\,\frac{\operatorname{tr}\hat\Sigma}{d}\,I,\qquad 0\le\gamma\le1
γ\gamma
shrinkage intensity
tr⁡Σ^/d\operatorname{tr}\hat\Sigma/d
average variance, so the target has the same overall scale
II
identity matrix

Hyperparameter 2: shrinkage. Σ^\hat\Sigma has d(d+1)/2d(d+1)/2 free entries. When dd approaches the number of points its smallest eigenvalues are badly underestimated, Σ^−1\hat\Sigma^{-1} blows those directions up, and ww chases noise; when d≥n−Kd\ge n-K the matrix is singular. Shrinking toward a scaled identity raises every eigenvalue toward the average, trading a little bias for a large drop in variance. At γ=0\gamma=0 we have plain LDA; at γ=1\gamma=1, Σ∝I\Sigma\propto I and LDA reduces to nearest mean (plus priors). The Ledoit–Wolf formula picks γ\gamma analytically; otherwise cross-validate it.

Worked example: salmon or sea bass?

A fish-sorting plant measures two scaled features of each fish on a conveyor: x1x_1 = lightness and x2x_2 = width. Ten labelled fish: five salmon (class 0) and five sea bass (class 1). Within each species, wider fish tend to be lighter — the features are correlated — which is exactly where nearest mean goes wrong.

Fishx₁ lightnessx₂ widthSpecies
112salmon (0)
232salmon (0)
334salmon (0)
436salmon (0)
556salmon (0)
641sea bass (1)
763sea bass (1)
865sea bass (1)
966sea bass (1)
1085sea bass (1)
Training data. Highlighted: the two fish that nearest-mean misclassifies but LDA gets right.

Fit LDA by hand, then classify a new fish at x = (5, 7)

  1. 1Priors. 5 of 10 fish in each class, so π^0=π^1=0.5\hat\pi_0=\hat\pi_1=0.5 and log⁡(π1/π0)=0\log(\pi_1/\pi_0)=0.
  2. 2Means. Salmon: μ^0=(1+3+3+3+55,2+2+4+6+65)=(3,4)\hat\mu_0=\big(\tfrac{1+3+3+3+5}{5},\tfrac{2+2+4+6+6}{5}\big)=(3,4). Sea bass: μ^1=(4+6+6+6+85,1+3+5+6+55)=(6,4)\hat\mu_1=\big(\tfrac{4+6+6+6+8}{5},\tfrac{1+3+5+6+5}{5}\big)=(6,4). The means differ only in lightness: μ1−μ0=(3,0)\mu_1-\mu_0=(3,0).
  3. 3Deviations from own mean. Salmon: (−2,−2),(0,−2),(0,0),(0,2),(2,2)(-2,-2),(0,-2),(0,0),(0,2),(2,2). Sea bass: (−2,−3),(0,−1),(0,1),(0,2),(2,1)(-2,-3),(0,-1),(0,1),(0,2),(2,1).
  4. 4Within-class scatter. Sum of x1x_1 squares: 8+8=168+8=16; cross products x1x2x_1x_2: 8+8=168+8=16; x2x_2 squares: 16+16=3216+16=32. So SW=(16161632)S_W=\begin{pmatrix}16&16\\16&32\end{pmatrix}.
  5. 5Pooled covariance. Divide by n−K=10−2=8n-K=10-2=8: Σ^=(2224)\hat\Sigma=\begin{pmatrix}2&2\\2&4\end{pmatrix}. The correlation is 2/2⋅4=0.7072/\sqrt{2\cdot4}=0.707 — strongly tilted clouds.
  6. 6Invert. det⁡Σ^=2⋅4−2⋅2=4\det\hat\Sigma=2\cdot4-2\cdot2=4, so Σ^−1=14(4−2−22)=(1−0.5−0.50.5)\hat\Sigma^{-1}=\tfrac14\begin{pmatrix}4&-2\\-2&2\end{pmatrix}=\begin{pmatrix}1&-0.5\\-0.5&0.5\end{pmatrix}.
  7. 7Weights. w=Σ^−1(μ1−μ0)=(1⋅3, −0.5⋅3)=(3, −1.5)w=\hat\Sigma^{-1}(\mu_1-\mu_0)=(1\cdot3,\,-0.5\cdot3)=(3,\,-1.5). Not (3,0)(3,0): the covariance tilts the boundary away from vertical.
  8. 8Offset. b=−12(μ0+μ1)⊤w=−12(9⋅3+8⋅(−1.5))=−12(27−12)=−7.5b=-\tfrac12(\mu_0+\mu_1)^\top w=-\tfrac12\big(9\cdot3+8\cdot(-1.5)\big)=-\tfrac12(27-12)=-7.5. Boundary: 3x1−1.5x2=7.53x_1-1.5x_2=7.5, i.e. x2=2x1−5x_2=2x_1-5.
  9. 9Per-class view. a0=Σ^−1μ0=(1,0.5)a_0=\hat\Sigma^{-1}\mu_0=(1,0.5), c0=−12μ0⊤a0+log⁡0.5=−2.5−0.693=−3.193c_0=-\tfrac12\mu_0^\top a_0+\log0.5=-2.5-0.693=-3.193; a1=Σ^−1μ1=(4,−1)a_1=\hat\Sigma^{-1}\mu_1=(4,-1), c1=−10−0.693=−10.693c_1=-10-0.693=-10.693. Subtracting gives back w=a1−a0=(3,−1.5)w=a_1-a_0=(3,-1.5) and b=c1−c0=−7.5b=c_1-c_0=-7.5.
  10. 10New fish x=(5,7)x=(5,7). δ0=5⋅1+7⋅0.5−3.193=5.307\delta_0=5\cdot1+7\cdot0.5-3.193=5.307 and δ1=5⋅4−7⋅1−10.693=2.307\delta_1=5\cdot4-7\cdot1-10.693=2.307. Score w⊤x+b=15−10.5−7.5=−3.0w^\top x+b=15-10.5-7.5=-3.0, so P(sea bass∣x)=σ(−3)=0.047P(\text{sea bass}\mid x)=\sigma(-3)=0.047 → salmon, with 95.3% probability.
  11. 11Compare with nearest mean. Squared Euclidean distances: 1313 to μ0\mu_0 and 1010 to μ1\mu_1 → nearest mean says sea bass. Squared Mahalanobis distances: Δ02=2.5\Delta_0^2=2.5, Δ12=8.5\Delta_1^2=8.5. The fish lies along the long diagonal axis of the salmon cloud, so in the cloud's own units it is much closer to the salmon; and indeed −12(8.5−2.5)=−3-\tfrac12(8.5-2.5)=-3 reproduces the score.
FishxSpeciesScore wᵀx + bP(bass | x)Euclid² to μ₀ / μ₁Mahalanobis² to μ₀ / μ₁Nearest mean
1(1, 2)salmon−7.50.0018 / 292.0 / 17.0salmon
2(3, 2)salmon−1.50.1824 / 132.0 / 5.0salmon
3(3, 4)salmon−4.50.0110 / 90.0 / 9.0salmon
4(3, 6)salmon−7.50.0014 / 132.0 / 17.0salmon
5(5, 6)salmon−1.50.1828 / 52.0 / 5.0sea bass ✗
6(4, 1)sea bass3.00.95310 / 138.5 / 2.5salmon ✗
7(6, 3)sea bass6.00.99810 / 112.5 / 0.5sea bass
8(6, 5)sea bass3.00.95310 / 16.5 / 0.5sea bass
9(6, 6)sea bass1.50.81813 / 45.0 / 2.0sea bass
10(8, 5)sea bass9.01.00026 / 520.5 / 2.5sea bass
Every training fish scored by LDA (w=(3,−1.5)w=(3,-1.5), b=−7.5b=-7.5) and by nearest mean. LDA separates all 10 (the closest fish sit at score ±1.5); nearest mean gets 8 of 10.

Fisher's ratio confirms it. Projected onto the mean difference (1,0)(1,0) — what nearest mean effectively uses — the classes have J=0.5625J=0.5625. Projected onto w∝(3,−1.5)w\propto(3,-1.5), J=1.125J=1.125, exactly the maximum J∗=(3,0) SW−1(3,0)⊤=9/8J^\ast=(3,0)\,S_W^{-1}(3,0)^\top=9/8. Turning the projection by 26.6° doubles the separation-to-spread ratio.

● salmon ● sea bass · shading = P(bass | x) · drag a fish or the ◆ queryw012345678910012345678x₁ lightness →x₂ width━ LDA · ┈ P = 0.1 / 0.9 · ellipses = shared Σ · ╌ nearest meanμ₀μ₁(5.0, 7.0) P(bass) = 0.047projection onto w · Fisher J = 1.125 · 0 on the wrong side← salmon sidesea bass side →
μ₀ = (3.00, 4.00) · μ₁ = (6.00, 4.00) · Σ̂ = [2.00 2.00; 2.00 4.00]
w = (3.00, -1.50) · b = -7.50 · training accuracy: LDA 10/10, nearest mean 8/10
query: δ₀ = 5.31, δ₁ = 2.31 → LDA says salmon · Mahalanobis² 2.50 / 8.50 · Euclidean² 13.00 / 10.00 → nearest mean says sea bass
fig.The worked example, live. Both classes share the same covariance ellipse (solid = 1, dashed = 2 Mahalanobis units); the white line is the LDA boundary x₂ = 2x₁ − 5 with normal w = (3, −1.5), the dashed rose line is nearest mean (x₁ = 4.5), and the strip below projects every fish onto w (Fisher J = 1.125, 0 on the wrong side) — switch to μ₁ − μ₀ to see J halve to 0.563 and two fish cross over. The query (5, 7) gets δ₀ = 5.31, δ₁ = 2.31, P(bass) = 0.047. Drag the prior to 0.80 and the boundary slides parallel (P(bass) rises to 0.166); push shrinkage γ to 1 and it rotates onto the nearest-mean line, which then calls the query a sea bass. Drag any fish to watch the means, the ellipse and w update.

The algorithm, step by step

LDA fit and predict

  1. 1Priors. πk←nk/n\pi_k\leftarrow n_k/n, or the deployment frequencies if you know them.
  2. 2Means. μk←\mu_k\leftarrow average of the class-kk rows — O(nd)O(nd).
  3. 3Pooled covariance. Subtract each row's own class mean, form SW=R⊤RS_W=R^\top R from the centred matrix RR, divide by n−Kn-K — O(nd2)O(nd^2). Optionally shrink: Σ←(1−γ)Σ+γ(tr⁡Σ/d)I\Sigma\leftarrow(1-\gamma)\Sigma+\gamma(\operatorname{tr}\Sigma/d)I.
  4. 4Solve, do not invert. Compute A=Σ−1[μ1,…,μK]A=\Sigma^{-1}[\mu_1,\dots,\mu_K] with one Cholesky / linear solve — O(d3+Kd2)O(d^3+Kd^2). (The SVD solver instead whitens the centred data and never forms Σ\Sigma, which is more stable when Σ\Sigma is near-singular.)
  5. 5Offsets. ck←−12μk⊤ak+log⁡πkc_k\leftarrow-\tfrac12\mu_k^\top a_k+\log\pi_k.
  6. 6Predict. Scores δk(x)=x⊤ak+ck\delta_k(x)=x^\top a_k+c_k — O(Kd)O(Kd); label =arg⁡max⁡kδk=\arg\max_k\delta_k; probabilities =softmax⁡k δk(x)=\operatorname{softmax}_k\,\delta_k(x).

Total training cost is O(nd2+d3)O(nd^2+d^3) with no iterations and nothing to converge: the same data always give the same model. Memory is O(d2)O(d^2) for the covariance. Prediction is as cheap as any linear model. For multi-class problems the boundaries between each pair of classes are hyperplanes, so the decision regions are convex polygons (polytopes in dd dimensions).

Variants and tuning

VariantCovariance modelBoundaryWhen to use it
Nearest mean (nearest centroid)Σ=σ2I\Sigma=\sigma^2 Ilinear, ⊥ to μ₁ − μ₀very high d, tiny n; features already decorrelated
Gaussian naive Bayesdiagonal, per classquadratic (axis-aligned)many weakly dependent features — see naive-bayes
LDAone full Σ, sharedlinearmoderate d, classes of similar shape
Shrinkage / regularised LDA(1−γ)Σ^+γσˉ2I(1-\gamma)\hat\Sigma+\gamma\bar\sigma^2Ilineardd comparable to nn; noisy or collinear features
QDAone full Σk\Sigma_k per classquadraticplenty of data, classes with different spreads — see quadratic-discriminant-analysis
Regularised DA (Friedman)blend of Σk\Sigma_k and pooled Σ\Sigmabetween linear and quadratictune the blend by cross-validation
Fisher / reduced-rank LDAshared Σproject to ≤ K − 1 dims firstvisualisation, preprocessing — see lda-for-dimensionality-reduction
The LDA family and its relatives.

Knobs that matter (scikit-learn names)

priors
changes only bb: shifts boundaries parallel by log⁡(π1/π0)/∥w∥\log(\pi_1/\pi_0)/\lVert w\rVert. In the worked example, priors 0.2/0.80.2/0.8 raise bb by log⁡4=1.386\log 4=1.386 to −6.114-6.114, and the query's P(bass) rises from 0.047 to 0.166.
shrinkage
None, a float γ in [0, 1], or "auto" (Ledoit–Wolf). Requires solver="lsqr" or "eigen". Rotates w from the LDA direction toward μ₁ − μ₀.
solver
"svd" (default; no covariance inversion, handles d > n, no shrinkage), "lsqr" (fast, supports shrinkage), "eigen" (supports shrinkage and transform).
n_components
only for transform(): number of discriminant directions, at most K − 1.

Shrinkage in action. Synthetic two-class data: dd features with correlation 0.6∣i−j∣0.6^{|i-j|} between features ii and jj, the class means differing by 0.8 in the first five features, 60 training points, 2,000 test points, averaged over 50 repeats. The Bayes-optimal accuracy is 0.739 for every dd.

d (features)LDA, no shrinkageLDA, shrinkage="auto"LDA, γ = 0.5Gaussian NBLogistic regression
100.6940.7050.7070.6950.700
500.5620.6540.6530.6360.633
1000.5790.6320.6250.6130.615
Test accuracy. Plain LDA collapses when d approaches n; Ledoit–Wolf shrinkage recovers most of the gap and beats both naive Bayes and logistic regression.

In code

python
1import numpy as np
2
3class LDA:
4 def __init__(self, priors=None, shrinkage=0.0):
5 self.priors, self.shrinkage = priors, shrinkage
6
7 def fit(self, X, y):
8 self.classes = np.unique(y)
9 n, d = X.shape
10 K = len(self.classes)
11 self.means = np.array([X[y == k].mean(axis=0) for k in self.classes])
12 # pooled within-class covariance, unbiased (divide by n - K)
13 R = X - self.means[np.searchsorted(self.classes, y)]
14 S = R.T @ R / (n - K)
15 g = self.shrinkage # shrink toward a scaled identity
16 S = (1 - g) * S + g * np.trace(S) / d * np.eye(d)
17 if self.priors is not None:
18 pri = np.asarray(self.priors, float)
19 else:
20 pri = np.bincount(np.searchsorted(self.classes, y)) / n
21 # delta_k(x) = x . a_k + c_k (linear in x)
22 self.A = np.linalg.solve(S, self.means.T) # columns a_k = S^-1 mu_k
23 self.c = -0.5 * np.sum(self.means.T * self.A, axis=0) + np.log(pri)
24 self.cov = S
25 return self
26
27 def decision(self, X):
28 return X @ self.A + self.c # (n, K) discriminant scores
29
30 def predict_proba(self, X):
31 s = self.decision(X)
32 s -= s.max(axis=1, keepdims=True) # softmax of the scores = posterior
33 p = np.exp(s)
34 return p / p.sum(axis=1, keepdims=True)
35
36 def predict(self, X):
37 return self.classes[self.decision(X).argmax(axis=1)]
38
39X = np.array([[1, 2], [3, 2], [3, 4], [3, 6], [5, 6], # salmon (0)
40 [4, 1], [6, 3], [6, 5], [6, 6], [8, 5]], float) # sea bass (1)
41y = np.array([0] * 5 + [1] * 5)
42m = LDA().fit(X, y)
43print("means", m.means.tolist())
44print("pooled cov", m.cov.tolist())
45print("a_k columns", m.A.T.tolist(), "c_k", np.round(m.c, 3).tolist())
46q = np.array([[5.0, 7.0]])
47print("scores", np.round(m.decision(q), 3).tolist(),
48 "P", np.round(m.predict_proba(q), 3).tolist(), "class", m.predict(q))
49print("train acc", (m.predict(X) == y).mean())
50print("shrinkage 1.0 ->", m.__class__(shrinkage=1.0).fit(X, y).predict(q))
51
52# means [[3.0, 4.0], [6.0, 4.0]]
53# pooled cov [[2.0, 2.0], [2.0, 4.0]]
54# a_k columns [[1.0, 0.5], [4.0, -1.0]] c_k [-3.193, -10.693]
55# scores [[5.307, 2.307]] P [[0.953, 0.047]] class [0]
56# train acc 1.0
57# shrinkage 1.0 -> [1]
From scratch in NumPy: multi-class LDA with priors and shrinkage. Prints the worked-example numbers.
python
1import numpy as np
2from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
3
4X = np.array([[1, 2], [3, 2], [3, 4], [3, 6], [5, 6],
5 [4, 1], [6, 3], [6, 5], [6, 6], [8, 5]], float)
6y = np.array([0] * 5 + [1] * 5) # 0 = salmon, 1 = sea bass
7q = np.array([[5.0, 7.0]])
8
9lda = LinearDiscriminantAnalysis(store_covariance=True).fit(X, y) # solver="svd"
10print("w =", lda.coef_.round(3), " b =", lda.intercept_.round(3))
11print("P(class | q) =", lda.predict_proba(q).round(3), "->", lda.predict(q))
12print("train accuracy =", lda.score(X, y))
13print("covariance_ (divides by n):", lda.covariance_.tolist())
14
15# Same boundary, different probabilities with the lsqr solver
16lsq = LinearDiscriminantAnalysis(solver="lsqr").fit(X, y)
17print("lsqr w =", lsq.coef_.round(3), " b =", lsq.intercept_.round(3), " P =", lsq.predict_proba(q).round(3))
18
19# Unequal priors shift the boundary; shrinkage pulls it toward nearest-mean
20print("priors .2/.8:", LinearDiscriminantAnalysis(priors=[0.2, 0.8]).fit(X, y).predict_proba(q).round(3))
21print("shrinkage=1:", LinearDiscriminantAnalysis(solver="lsqr", shrinkage=1.0).fit(X, y).predict(q))
22
23# w = [[ 3. -1.5]] b = [-7.5]
24# P(class | q) = [[0.953 0.047]] -> [0]
25# train accuracy = 1.0
26# covariance_ (divides by n): [[1.6, 1.6], [1.6, 3.2]]
27# lsqr w = [[ 3.75 -1.875]] b = [-9.375] P = [[0.977 0.023]]
28# priors .2/.8: [[0.834 0.166]]
29# shrinkage=1: [1]
scikit-learn on the same ten fish: the default solver reproduces w, b and the posterior exactly.

Use when

  • +Classes are roughly Gaussian with similar spreads, and features are correlated — the case LDA was built for.
  • +You have few samples per feature: with shrinkage, LDA is one of the strongest small-n classifiers (EEG/BCI, spectra, genomics).
  • +You want a fast, deterministic, closed-form baseline with calibrated-ish probabilities and no tuning loop.
  • +You also want a low-dimensional supervised projection for plotting — the same fit gives it.

Avoid when

  • −Classes have clearly different covariances (one tight, one diffuse): the optimal boundary curves — use QDA or a kernel method.
  • −The data have heavy tails or outliers: means and covariances are not robust, and one bad point can rotate w.
  • −Features are binary, counts or strongly skewed: the Gaussian model is wrong; logistic regression or naive Bayes fit better.
  • −The boundary is genuinely non-linear (rings, XOR): no linear method helps without feature engineering.

Where it's used

Brain–computer interfaces

Shrinkage LDA is the standard classifier for EEG event-related potentials (e.g. P300 spellers): hundreds of correlated features, a few hundred trials.

Spectroscopy and chemometrics

Classifying samples (olive-oil origin, bacteria, tablets) from near-infrared or Raman spectra, usually after PCA (PCA-LDA).

Credit scoring and bankruptcy prediction

Altman's Z-score (1968) is a linear discriminant on five financial ratios — one of the earliest production uses of the method.

Face recognition

Fisherfaces: PCA then LDA on pixel vectors, so the projection separates identities rather than lighting changes.

Check yourself

Two classes share Σ=(2224)\Sigma=\begin{pmatrix}2&2\\2&4\end{pmatrix}, with μ0=(3,4)\mu_0=(3,4) and μ1=(6,4)\mu_1=(6,4). What is the LDA weight vector ww?

Check yourself

In the worked example, the priors change from 0.5 / 0.5 to π(salmon) = 0.2, π(bass) = 0.8. What happens to the boundary?

Check yourself

With shrinkage γ=1\gamma=1 (and equal priors), LDA on the fish data has boundary…

Check yourself

Why does LDA produce a linear boundary while QDA produces a quadratic one?

Remember

  • ✓LDA models each class as N(μk,Σ)\mathcal N(\mu_k,\Sigma) with one shared covariance; Bayes' rule gives linear scores δk(x)=x⊤Σ−1μk−12μk⊤Σ−1μk+log⁡πk\delta_k(x)=x^\top\Sigma^{-1}\mu_k-\tfrac12\mu_k^\top\Sigma^{-1}\mu_k+\log\pi_k.
  • ✓Two classes: w=Σ−1(μ1−μ0)w=\Sigma^{-1}(\mu_1-\mu_0), b=−12(μ0+μ1)⊤w+log⁡(π1/π0)b=-\tfrac12(\mu_0+\mu_1)^\top w+\log(\pi_1/\pi_0), and P(y=1∣x)=σ(w⊤x+b)P(y=1\mid x)=\sigma(w^\top x+b) — the same form as logistic regression, fitted generatively.
  • ✓It is nearest-mean in Mahalanobis distance: the covariance tilts the boundary to respect correlated features (10/10 vs 8/10 on the fish).
  • ✓Fisher's criterion — maximise between-class over within-class scatter — gives the same direction without assuming Gaussians.
  • ✓Priors shift the boundary in parallel; shrinkage rotates it toward nearest mean and is essential when d is comparable to n.
  • ✓Closed form, O(nd² + d³), scale-invariant, but sensitive to outliers and to unequal class covariances (then use QDA).
← back to ML Algorithms roadmap