LDA for dimensionality reduction

Project onto the directions that maximize between-class over within-class scatter — at most K − 1 of them.

16 min read1489 wordscore6 viewsupdated 2026-10-07

Definition

Linear discriminant analysis (as a projection) is a supervised linear dimensionality reduction. It uses the class labels to find directions ww along which the class means sit far apart relative to the spread inside each class. These are Fisher's discriminant directions. Projecting dd-dimensional data onto the best q≤K−1q \le K-1 of them gives a low-dimensional representation that keeps as much class separation as a linear map can.

At a glance

Task
Supervised dimensionality reduction (feature extraction before a classifier, or plotting labelled data)
Input → output
labelled x∈Rdx\in\mathbb{R}^d → z=W⊤x∈Rqz=W^\top x\in\mathbb{R}^q with q≤min⁡(d, K−1)q\le\min(d,\,K-1)
Learns
a d×qd\times q projection matrix WW: the top generalized eigenvectors of SBw=λSWwS_B w=\lambda S_W w
Training
O(nd2)O(n d^2) to build the scatter matrices + O(d3)O(d^3) for the eigenproblem
Projection
O(dq)O(dq) per point, a single matrix multiply
Hyperparameters
n_components (≤ K − 1), solver (svd / eigen), shrinkage of the within-class covariance, class priors
Assumes
classes differ mainly in their means, with roughly the same covariance shape (the Gaussian, shared-covariance model). It works as a heuristic beyond that.

The problem

You have labelled data with many features and want a few new axes, to plot it in 2-D or to feed a small, well-conditioned feature set to a classifier. The obvious tool is PCA (see the PCA article), which keeps the directions of largest variance. But PCA never looks at the labels. Variance is often driven by something unrelated to the classes: overall size, lighting, a sensor that drifts. The direction that separates the classes can be a low-variance one. PCA then throws it away.

The worked example below makes this concrete. Two classes are elongated along the same diagonal and offset across it. PCA picks the diagonal, and on that axis the classes interleave completely: Fisher ratio J=0.048J=0.048, and 5 of 10 points end up nearer the other class's mean. The useful axis is almost perpendicular to it, at J=12.375J=12.375, with zero errors. We need a criterion that uses the labels.

The idea

Project every point onto a line ww and look at the 1-D result. Two numbers describe how well the classes separate there: the gap between the projected class means (we want this big) and the spread of each class around its own mean (we want this small). Fisher's 1936 proposal was to maximize their ratio. A big gap means nothing if each class is smeared across it, and a tight class means nothing if the means coincide.

Picture the two classes as two cigar-shaped clouds lying side by side. PCA aims along the cigars, because that is where the variance is. LDA aims across them: a short distance between the means, but measured in units of the cigars' thin width, which makes it a huge separation.

Vocabulary

Scatter matrix
an unnormalized covariance: ∑(x−μ)(x−μ)⊤\sum (x-\mu)(x-\mu)^\top. Dividing by a count gives a covariance, and that constant does not change the directions.
Within-class scatter S_W
how much points spread around their own class mean, summed over classes
Between-class scatter S_B
how much the class means spread around the overall mean, weighted by class size
Fisher ratio J(w)
between-class over within-class scatter after projecting onto ww; a signal-to-noise ratio for the classes
Discriminant direction (LD1, LD2, …)
a stationary point of JJ. LD1 is the best; each later one is the best that is uncorrelated with the earlier ones (within classes).
Whitening
a linear map that turns a covariance into the identity; here x↦SW−1/2xx\mapsto S_W^{-1/2}x makes every class cloud round
Rank of S_B
at most K−1K-1, so at most K−1K-1 directions with J>0J>0 exist; the rest carry no class information

The math

Take a unit direction ww. Each point becomes the scalar zi=w⊤xiz_i=w^\top x_i, and class kk's mean becomes w⊤μkw^\top\mu_k. Both the between-class and the within-class spread of these scalars turn out to be quadratic forms in ww:

SW=∑k=1K∑i∈Ck(xi−μk)(xi−μk)⊤,SB=∑k=1Knk(μk−μ)(μk−μ)⊤S_W=\sum_{k=1}^{K}\sum_{i\in C_k}(x_i-\mu_k)(x_i-\mu_k)^\top,\qquad S_B=\sum_{k=1}^{K}n_k(\mu_k-\mu)(\mu_k-\mu)^\top
Ck, nkC_k,\ n_k
the set of training points in class kk, and its size
μk\mu_k
mean of class kk; μ\mu is the overall mean
SWS_W
d×dd\times d within-class scatter (sum of per-class scatters)
SBS_B
d×dd\times d between-class scatter of the class means

For the projection, ∑i∈Ck(w⊤xi−w⊤μk)2=w⊤[∑i∈Ck(xi−μk)(xi−μk)⊤]w\sum_{i\in C_k}(w^\top x_i-w^\top\mu_k)^2=w^\top\big[\sum_{i\in C_k}(x_i-\mu_k)(x_i-\mu_k)^\top\big]w. Summing over classes gives w⊤SWww^\top S_W w as the projected within-class scatter, and the same algebra gives w⊤SBww^\top S_B w for the means. The total scatter splits exactly as ST=SW+SBS_T=S_W+S_B, an ANOVA decomposition.

J(w)=w⊤SB ww⊤SW wJ(w)=\frac{w^\top S_B\,w}{w^\top S_W\,w}
J(w)J(w)
Fisher criterion: projected between-class scatter over projected within-class scatter
ww
projection direction in Rd\mathbb{R}^d; only its direction matters since J(cw)=J(w)J(cw)=J(w)

Deriving the solution. Because JJ ignores the length of ww, we can fix the denominator and maximize the numerator: maximize w⊤SBww^\top S_B w subject to w⊤SWw=1w^\top S_W w=1. The Lagrangian is L=w⊤SBw−λ(w⊤SWw−1)\mathcal{L}=w^\top S_B w-\lambda(w^\top S_W w-1). Setting ∇wL=2SBw−2λSWw=0\nabla_w\mathcal{L}=2S_Bw-2\lambda S_Ww=0 gives

SB w=λ SW w⟺SW−1SB w=λ w,λ=J(w)S_B\,w=\lambda\,S_W\,w\quad\Longleftrightarrow\quad S_W^{-1}S_B\,w=\lambda\,w,\qquad \lambda=J(w)
λ\lambda
a generalized eigenvalue; multiplying the condition by w⊤w^\top shows λ=w⊤SBw/w⊤SWw=J(w)\lambda=w^\top S_Bw / w^\top S_Ww=J(w)

So every stationary point of JJ is a generalized eigenvector, and its eigenvalue is the Fisher ratio it achieves. LD1 is the eigenvector with the largest λ\lambda, LD2 the next, and so on. For qq directions at once, the matrix criterion tr⁡((W⊤SWW)−1W⊤SBW)\operatorname{tr}\big((W^\top S_W W)^{-1}W^\top S_B W\big) is maximized by the top qq eigenvectors. The resulting scores are uncorrelated within classes, W⊤SWWW^\top S_W W diagonal. Unlike PCA, the directions are generally not orthogonal in the original space.

K=2:SB=n1n2n(μ1−μ2)(μ1−μ2)⊤ ⇒ w⋆∝SW−1(μ1−μ2),J(w⋆)=n1n2n(μ1−μ2)⊤SW−1(μ1−μ2)K=2:\quad S_B=\tfrac{n_1n_2}{n}(\mu_1-\mu_2)(\mu_1-\mu_2)^\top\ \Rightarrow\ w^\star\propto S_W^{-1}(\mu_1-\mu_2),\quad J(w^\star)=\tfrac{n_1n_2}{n}(\mu_1-\mu_2)^\top S_W^{-1}(\mu_1-\mu_2)
w⋆w^\star
Fisher's linear discriminant; no eigen-solver needed, just one linear solve
(μ1−μ2)⊤SW−1(μ1−μ2)(\mu_1-\mu_2)^\top S_W^{-1}(\mu_1-\mu_2)
a (scaled) squared Mahalanobis distance between the class means

With two classes, SBwS_Bw always points along μ1−μ2\mu_1-\mu_2, whatever ww is. So SW−1SBw=λwS_W^{-1}S_Bw=\lambda w forces w∝SW−1(μ1−μ2)w\propto S_W^{-1}(\mu_1-\mu_2). The factor SW−1S_W^{-1} is what rotates the answer away from the naive "line joining the means": it turns down directions where the classes are wide and turns up directions where they are thin.

Why at most K−1K-1. SBS_B is a sum of KK rank-one matrices built from the vectors nk(μk−μ)n_k(\mu_k-\mu), and those vectors sum to zero (∑knkμk=nμ\sum_k n_k\mu_k=n\mu). They therefore span at most K−1K-1 dimensions, so rank⁡SB≤K−1\operatorname{rank}S_B\le K-1. Any ww orthogonal to that span has w⊤SBw=0w^\top S_Bw=0. At most K−1K-1 eigenvalues are non-zero, whatever the value of dd. Two classes give one axis; the 10 digits give 9.

SW(α)=(1−α) SW+α tr⁡SWd IS_W^{(\alpha)}=(1-\alpha)\,S_W+\alpha\,\frac{\operatorname{tr}S_W}{d}\,I
α∈[0,1]\alpha\in[0,1]
shrinkage: 0 = plain LDA, 1 = ignore correlations (equal spherical within-class spread)
tr⁡SWdI\tfrac{\operatorname{tr}S_W}{d}I
a scaled identity with the same average variance as SWS_W

Hyperparameters, mathematically. SW−1S_W^{-1} amplifies directions where the classes look thin. When nn is small compared with dd, some directions look thin only by chance: SWS_W has tiny or zero eigenvalues, and SW−1S_W^{-1} blows them up into spurious "perfect" discriminants. Shrinkage adds α tr⁡SWd\alpha\,\tfrac{\operatorname{tr}S_W}{d} to every eigenvalue, which caps the amplification. As α→1\alpha\to1, w⋆→μ1−μ2w^\star\to\mu_1-\mu_2, the plain mean difference. Priors reweight the class means in μ\mu and SBS_B. The number of components qq just truncates the eigenvalue list; λj/∑λ\lambda_j/\sum\lambda (scikit-learn's explained_variance_ratio_) tells you how much separating power each axis carries.

Worked example: two cigar-shaped classes

Ten points with two features. Class B is class A shifted by (+1,−2)(+1,-2), so both classes have exactly the same shape, a cloud stretched along the diagonal. A new point (3,4)(3,4) is to be projected and classified.

#x₁x₂class
112A
223A
333A
445A
555A
620B
731B
841B
953B
1063B
The data. Class B = class A + (1, −2). Highlighted: class A.

Fisher's discriminant by hand

  1. 1Means. μA=(3, 3.6)\mu_A=(3,\,3.6), μB=(4, 1.6)\mu_B=(4,\,1.6), overall μ=(3.5, 2.6)\mu=(3.5,\,2.6). Difference μA−μB=(−1, 2)\mu_A-\mu_B=(-1,\,2).
  2. 2Within-class scatter. Class A's deviations from μA\mu_A give SA=(10887.2)S_A=\begin{pmatrix}10&8\\8&7.2\end{pmatrix}. Class B is a shifted copy, so SB(class)=SAS_B^{(\text{class})}=S_A. Hence SW=(20161614.4)S_W=\begin{pmatrix}20&16\\16&14.4\end{pmatrix}.
  3. 3Between-class scatter. μA−μ=(−0.5,1)\mu_A-\mu=(-0.5,1) and μB−μ=(0.5,−1)\mu_B-\mu=(0.5,-1), so SB=5 (−0.5,1)(−0.5,1)⊤+5 (0.5,−1)(0.5,−1)⊤=(2.5−5−510)S_B=5\,(-0.5,1)(-0.5,1)^\top+5\,(0.5,-1)(0.5,-1)^\top=\begin{pmatrix}2.5&-5\\-5&10\end{pmatrix}. Its rows are proportional, so it has rank 1: one useful direction (K−1=1K-1=1).
  4. 4Invert SWS_W. det⁡SW=20⋅14.4−162=32\det S_W=20\cdot14.4-16^2=32, so SW−1=132(14.4−16−1620)=(0.45−0.5−0.50.625)S_W^{-1}=\tfrac{1}{32}\begin{pmatrix}14.4&-16\\-16&20\end{pmatrix}=\begin{pmatrix}0.45&-0.5\\-0.5&0.625\end{pmatrix}.
  5. 5Direction. w⋆=SW−1(μA−μB)=(−0.45−1, 0.5+1.25)=(−1.45, 1.75)w^\star=S_W^{-1}(\mu_A-\mu_B)=(-0.45-1,\ 0.5+1.25)=(-1.45,\,1.75). Its length is 2.2732.273, giving the unit vector w=(−0.638, 0.770)w=(-0.638,\,0.770), at an angle of 129.6∘129.6^\circ.
  6. 6Score it. J(w⋆)=5⋅510(μA−μB)⊤w⋆=2.5 (1.45+3.5)=12.375J(w^\star)=\tfrac{5\cdot5}{10}(\mu_A-\mu_B)^\top w^\star=2.5\,(1.45+3.5)=12.375. Along unit ww: between w⊤SBw=11.860w^\top S_Bw=11.860, within w⊤SWw=0.958w^\top S_Ww=0.958, ratio 12.37512.375. For comparison, the PCA axis (0.676,0.737)(0.676,0.737) at 47.5∘47.5^\circ scores only J=0.048J=0.048.
  7. 7Project. zi=w⊤xiz_i=w^\top x_i (table below). Class A lands on 0.40…1.300.40\ldots1.30 and class B on −1.78…−0.88-1.78\ldots-0.88, with no overlap. The projected means are 0.8580.858 and −1.320-1.320, and their midpoint is −0.231-0.231.
  8. 8New point. z=w⊤(3,4)=−0.638⋅3+0.770⋅4=1.166>−0.231z=w^\top(3,4)=-0.638\cdot3+0.770\cdot4=1.166>-0.231, nearer A's projected mean, so it is classified A.
#classLDA z = wᵀxPCA z = vᵀx
1A0.9022.150
2A1.0343.563
3A0.3964.239
4A1.2986.389
5A0.6607.065
6B−1.2761.352
7B−1.1442.765
8B−1.7823.441
9B−0.8805.591
10B−1.5186.267
meanA / B0.858 / −1.3204.681 / 3.883
Each point projected onto the LDA axis w = (−0.638, 0.770) and onto the PCA axis v = (0.676, 0.737). On the LDA axis the classes are separated by a gap; on the PCA axis they interleave.
original features (x₁, x₂) · drag ●new (3, 4)w1-D projection z = wᵀ(x − m)ABbetween wᵀS_Bw = 1.591within wᵀS_Ww = 32.900J(w) = 0.048 / max 12.375nearest-mean errors: 5/10J for every axis angle (as drawn in this view)012.40°45°90°135°180°PCALD1
w = (0.676, 0.737) · angle 47.5° · projected means A 4.681, B 3.883
S_W⁻¹S_B eigenvalues λ = 12.375, 0.000 · rank S_B = 1, so only K − 1 = 1 direction carries class information
new point (3, 4) → z = 4.976 → nearest projected mean: class A
fig.The worked example. Drag the handle w to rotate the projection axis. Each point drops onto it, the right panel shows the resulting 1-D data per class, and the bottom curve is J for every angle. The figure starts on the PCA axis (47.5°, J = 0.048, 5/10 nearest-mean errors); LD1 at 129.6° reaches the maximum J = 12.375 with 0 errors and sends the new point (3, 4) to z = 1.166, class A. "Whiten by S_W" animates the map S_W^−½: the dashed within-class ellipses become circles, and LD1 becomes the line joining the class means. "3 classes" adds class C = A + (3, 1): λ = 8.258 and 0.687 (92.3% / 7.7%), so two discriminants exist; LD2 is the minimum of J, and in the whitened view LD1 and LD2 are perpendicular.

Note what the figure's J-curve shows: in 2-D, J(θ)J(\theta) has exactly two stationary angles, its maximum (LD1) and its minimum. With two classes the minimum is 00. That is the axis perpendicular to μA−μB\mu_A-\mu_B, where both projected means coincide. With three classes the minimum is LD2, still positive, because the three means span two dimensions.

The algorithm, step by step

fit(X, y, q) and transform(X)

  1. 1Class statistics. For each class kk compute nkn_k and μk\mu_k; compute the overall mean μ\mu. Cost O(nd)O(nd).
  2. 2Scatter matrices. SW=∑k∑i∈Ck(xi−μk)(xi−μk)⊤S_W=\sum_k\sum_{i\in C_k}(x_i-\mu_k)(x_i-\mu_k)^\top and SB=∑knk(μk−μ)(μk−μ)⊤S_B=\sum_k n_k(\mu_k-\mu)(\mu_k-\mu)^\top. Cost O(nd2)O(nd^2).
  3. 3Regularize (optional). Replace SWS_W by the shrunk (1−α)SW+αtr⁡SWdI(1-\alpha)S_W+\alpha\tfrac{\operatorname{tr}S_W}{d}I if nn is not much larger than dd.
  4. 4Eigenproblem. Solve SBw=λSWwS_Bw=\lambda S_Ww, either directly (eigen solver) or by whitening with an SVD of the centred within-class data and then an SVD of the whitened class means (svd solver, which never forms SWS_W and copes with singular SWS_W). Cost O(d3)O(d^3).
  5. 5Keep q≤min⁡(d,K−1)q\le\min(d,K-1) eigenvectors with the largest λ\lambda as the columns of WW; report λj/∑λ\lambda_j/\sum\lambda.
  6. 6Transform. z=W⊤(x−μ)z=W^\top(x-\mu), one O(dq)O(dq) multiply per point. Downstream, use zz for plotting, k-NN, clustering or any classifier.

Total cost is O(nd2+d3)O(nd^2+d^3), the same order as PCA via the covariance matrix and far cheaper than t-SNE or UMAP. The KK class means are all that SBS_B needs, so the eigenproblem is effectively rank K−1K-1, and the svd solver exploits this.

Variants and practical tuning

PropertyPCALDA (Fisher)Kernel / nonlinear
Uses labelsnoyeskernel FDA: yes; t-SNE/UMAP: no
Objectivemax total variancemax between / within scattersame ratio in a feature space
Max componentsmin(n, d)K − 1K − 1 (kernel FDA)
Axes orthogonal?yesno (uncorrelated within classes)—
Feature scaling matters?yesno (affine invariant)yes (kernel bandwidth)
Can transform new pointsyesyeskernel FDA yes; t-SNE no
LDA versus its sibling projections. Only LDA uses the labels.

Knobs that matter (scikit-learn LinearDiscriminantAnalysis)

n_components
≤ min(d, K − 1). Inspect explained_variance_ratio_. On iris (4 features, 3 classes) the two axes carry 99.12% and 0.88%.
solver
svd (default; no shrinkage, robust to singular SWS_W), eigen (supports shrinkage and transform). lsqr supports shrinkage but cannot transform: it is a classifier-only solver.
shrinkage
'auto' uses the Ledoit–Wolf estimate of α; or set a float in [0, 1] chosen by cross-validation. Essential when n is not ≫ d.
priors
class weights in the overall mean and the Bayes decision rule; defaults to the class frequencies.
Kernel FDA
run the same ratio on kernel features (ϕ(x)\phi(x)) for curved class boundaries; scales as O(n3)O(n^3).
Heteroscedastic variants
when the classes have very different covariance shapes, Fisher's criterion is no longer optimal; HLDA or QDA-style models handle that.
PipelineAccuracy
PCA(1) → 5-NN0.920
LDA(1) → 5-NN0.967
Iris, 5-fold cross-validated accuracy of a 5-nearest-neighbour classifier after reducing 4 features to 1. The reducer is fitted inside each fold.

In code

python
1import numpy as np
2
3def lda_fit(X, y, n_components=None, shrinkage=0.0):
4 classes = np.unique(y)
5 d, K = X.shape[1], len(classes)
6 q = min(d, K - 1) if n_components is None else n_components
7 assert q <= min(d, K - 1), "LDA gives at most K-1 directions"
8 m = X.mean(axis=0)
9 Sw, Sb = np.zeros((d, d)), np.zeros((d, d))
10 for c in classes:
11 Xc = X[y == c]
12 mc = Xc.mean(axis=0)
13 Sw += (Xc - mc).T @ (Xc - mc) # within-class scatter
14 Sb += len(Xc) * np.outer(mc - m, mc - m) # between-class scatter
15 if shrinkage: # pull S_W toward a scaled identity
16 Sw = (1 - shrinkage) * Sw + shrinkage * np.trace(Sw) / d * np.eye(d)
17 evals, evecs = np.linalg.eig(np.linalg.solve(Sw, Sb)) # S_W^-1 S_B w = lambda w
18 order = np.argsort(evals.real)[::-1][:q]
19 W = evecs[:, order].real
20 W /= np.linalg.norm(W, axis=0) # unit-length columns
21 return W, evals.real[order], Sw, Sb
22
23def lda_transform(X, W):
24 return X @ W
25
26A = np.array([[1, 2], [2, 3], [3, 3], [4, 5], [5, 5]], float)
27B = A + [1, -2]
28X = np.vstack([A, B]); y = np.array([0] * 5 + [1] * 5)
29
30W, lam, Sw, Sb = lda_fit(X, y)
31w = W[:, 0] * np.sign(W[1, 0]) # fix the sign: point toward class A
32print("S_W =", Sw.tolist(), " S_B =", Sb.tolist())
33print("w =", w.round(3), " lambda = J(w) =", lam.round(3)) # [-0.638 0.77] [12.375]
34z = lda_transform(X, w)
35print("z(A) =", z[:5].round(3), " z(B) =", z[5:].round(3))
36mA, mB = z[:5].mean(), z[5:].mean()
37print(f"projected means {mA:.3f} {mB:.3f} threshold {(mA + mB) / 2:.3f}") # 0.858 -1.320 -0.231
38znew = np.array([3, 4]) @ w
39print(f"new point (3,4): z = {znew:.3f} ->", "A" if abs(znew - mA) < abs(znew - mB) else "B") # 1.166 -> A
40
41from sklearn.datasets import load_iris
42Xi, yi = load_iris(return_X_y=True)
43Wi, lami, _, _ = lda_fit(Xi, yi)
44print("iris: directions", Wi.shape[1], " eigenvalue share", (lami / lami.sum()).round(4)) # 2 [0.9912 0.0088]
From scratch with NumPy: scatter matrices, the eigenproblem, projection. Prints w = (−0.638, 0.770), J = 12.375 and classifies (3, 4) as A.
python
1import numpy as np
2from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
3from sklearn.decomposition import PCA
4
5A = np.array([[1, 2], [2, 3], [3, 3], [4, 5], [5, 5]], float)
6X = np.vstack([A, A + [1, -2]]); y = np.array(["A"] * 5 + ["B"] * 5)
7
8lda = LinearDiscriminantAnalysis(n_components=1).fit(X, y) # n_components <= K-1 = 1
9Z = lda.transform(X).ravel()
10print("scalings_:", lda.scalings_.ravel().round(3)) # [ 1.843 -2.225] = -1.271 * (-1.45, 1.75)
11print("z:", Z.round(3)) # A: -3.274 -3.655 -1.812 -4.418 -2.574 | B: 3.019 2.638 4.481 1.875 3.719
12print("explained_variance_ratio_:", lda.explained_variance_ratio_) # [1.]
13print("new (3,4):", lda.transform([[3, 4]]).ravel().round(3), lda.predict([[3, 4]])) # [-4.036] ['A']
14
15pca = PCA(n_components=1).fit(X)
16print("PCA axis:", pca.components_.ravel().round(3)) # [0.676 0.737] (labels ignored)
17
18s = np.sqrt(sum(((Z[y == c] - Z[y == c].mean()) ** 2).sum() for c in "AB") / (len(X) - 2))
19print("pooled within-class std of z:", round(s, 3)) # 1.0
20
21# Asking for more than K-1 components fails:
22# LinearDiscriminantAnalysis(n_components=2).fit(X, y)
23# ValueError: n_components cannot be larger than min(n_features, n_classes - 1).
scikit-learn on the same ten points. The axis matches up to sign and scale: sklearn scales scores so the pooled within-class standard deviation is 1, and centres them.

Use when

  • +You have labels and want a few axes that keep class separation, for plotting, or as compact features for k-NN, clustering or a small model.
  • +Classes differ mostly in their means and have similar spread: then LDA's projection is the optimal linear one.
  • +You need something fast, deterministic and able to transform new points, with no tuning beyond shrinkage.
  • +Features are on arbitrary scales: LDA is invariant to rescaling, so standardization is not required.

Avoid when

  • −You need more than K − 1 dimensions: with 2 classes you get exactly one axis.
  • −The classes differ in their covariance shape rather than their means (for example, the same centre but different spread): S_B ≈ 0 and LDA finds nothing.
  • −Classes are multimodal or curved (each class several blobs, rings): linear projections can't untangle them. Use kernel FDA or a nonlinear model.
  • −n ≲ d without shrinkage: the projection overfits to noise.
  • −There are no labels: use PCA, ICA or UMAP instead.

Where it's used

Face recognition (Fisherfaces)

PCA to a few hundred dimensions, then LDA to (number of people − 1) axes. This is robust to lighting changes that dominate the variance.

Brain–computer interfaces

Shrinkage LDA is the standard projection/classifier for EEG event-related potentials, where trials are few and channels × time samples are many.

Spectroscopy & chemometrics

Projecting hundreds of wavelengths onto a few discriminant axes to separate sample types (food origin, tissue, minerals), often after PCA (PCA-LDA).

Visualizing labelled data

A 2-D LDA plot of a 3+ class problem shows at a glance which classes are separable and which overlap, as a supervised companion to a PCA plot.

Check yourself

A dataset has 50 features and 10 classes. What is the largest useful n_components for LDA?

Check yourself

Two classes have within-class scatter S_W = diag(4, 1) and mean difference μ₁ − μ₂ = (2, 2). Fisher's direction is proportional to…

Check yourself

In the figure's 3-class data the generalized eigenvalues are 8.258 and 0.687. What share of the separating power does LD1 carry?

Check yourself

You multiply feature x₁ of the worked example by 100. What happens?

Remember

  • ✓LDA projects onto directions maximizing J(w)=w⊤SBw / w⊤SWwJ(w)=w^\top S_Bw\,/\,w^\top S_Ww: class means far apart, measured in units of within-class spread.
  • ✓Setting the gradient of the constrained problem to zero gives the generalized eigenproblem SBw=λSWwS_Bw=\lambda S_Ww, whose eigenvalue λ\lambda equals the Fisher ratio. For two classes, w∝SW−1(μ1−μ2)w\propto S_W^{-1}(\mu_1-\mu_2).
  • ✓rank⁡SB≤K−1\operatorname{rank}S_B\le K-1, so you get at most K−1K-1 discriminant axes, however many features you have.
  • ✓Equivalent view: whiten by S_W^−½, then do PCA on the class means. Hence LDA is invariant to feature scaling and its axes are not orthogonal in the original space.
  • ✓Shrink S_W (solver='eigen', shrinkage='auto') when n is not much bigger than d, and fit LDA inside cross-validation because it uses the labels.
← back to ML Algorithms roadmap