Definition
Linear discriminant analysis (as a projection) is a supervised linear dimensionality reduction. It uses the class labels to find directions along which the class means sit far apart relative to the spread inside each class. These are Fisher's discriminant directions. Projecting -dimensional data onto the best of them gives a low-dimensional representation that keeps as much class separation as a linear map can.
At a glance
- Task
- Supervised dimensionality reduction (feature extraction before a classifier, or plotting labelled data)
- Input → output
- labelled → with
- Learns
- a projection matrix : the top generalized eigenvectors of
- Training
- to build the scatter matrices + for the eigenproblem
- Projection
- per point, a single matrix multiply
- Hyperparameters
n_components(≤ K − 1), solver (svd/eigen),shrinkageof the within-class covariance, classpriors- Assumes
- classes differ mainly in their means, with roughly the same covariance shape (the Gaussian, shared-covariance model). It works as a heuristic beyond that.
The problem
You have labelled data with many features and want a few new axes, to plot it in 2-D or to feed a small, well-conditioned feature set to a classifier. The obvious tool is PCA (see the PCA article), which keeps the directions of largest variance. But PCA never looks at the labels. Variance is often driven by something unrelated to the classes: overall size, lighting, a sensor that drifts. The direction that separates the classes can be a low-variance one. PCA then throws it away.
The worked example below makes this concrete. Two classes are elongated along the same diagonal and offset across it. PCA picks the diagonal, and on that axis the classes interleave completely: Fisher ratio , and 5 of 10 points end up nearer the other class's mean. The useful axis is almost perpendicular to it, at , with zero errors. We need a criterion that uses the labels.
The idea
Project every point onto a line and look at the 1-D result. Two numbers describe how well the classes separate there: the gap between the projected class means (we want this big) and the spread of each class around its own mean (we want this small). Fisher's 1936 proposal was to maximize their ratio. A big gap means nothing if each class is smeared across it, and a tight class means nothing if the means coincide.
Picture the two classes as two cigar-shaped clouds lying side by side. PCA aims along the cigars, because that is where the variance is. LDA aims across them: a short distance between the means, but measured in units of the cigars' thin width, which makes it a huge separation.
Vocabulary
- Scatter matrix
- an unnormalized covariance: . Dividing by a count gives a covariance, and that constant does not change the directions.
- Within-class scatter S_W
- how much points spread around their own class mean, summed over classes
- Between-class scatter S_B
- how much the class means spread around the overall mean, weighted by class size
- Fisher ratio J(w)
- between-class over within-class scatter after projecting onto ; a signal-to-noise ratio for the classes
- Discriminant direction (LD1, LD2, …)
- a stationary point of . LD1 is the best; each later one is the best that is uncorrelated with the earlier ones (within classes).
- Whitening
- a linear map that turns a covariance into the identity; here makes every class cloud round
- Rank of S_B
- at most , so at most directions with exist; the rest carry no class information
The math
Take a unit direction . Each point becomes the scalar , and class 's mean becomes . Both the between-class and the within-class spread of these scalars turn out to be quadratic forms in :
- the set of training points in class , and its size
- mean of class ; is the overall mean
- within-class scatter (sum of per-class scatters)
- between-class scatter of the class means
For the projection, . Summing over classes gives as the projected within-class scatter, and the same algebra gives for the means. The total scatter splits exactly as , an ANOVA decomposition.
- Fisher criterion: projected between-class scatter over projected within-class scatter
- projection direction in ; only its direction matters since
Deriving the solution. Because ignores the length of , we can fix the denominator and maximize the numerator: maximize subject to . The Lagrangian is . Setting gives
- a generalized eigenvalue; multiplying the condition by shows
So every stationary point of is a generalized eigenvector, and its eigenvalue is the Fisher ratio it achieves. LD1 is the eigenvector with the largest , LD2 the next, and so on. For directions at once, the matrix criterion is maximized by the top eigenvectors. The resulting scores are uncorrelated within classes, diagonal. Unlike PCA, the directions are generally not orthogonal in the original space.
- Fisher's linear discriminant; no eigen-solver needed, just one linear solve
- a (scaled) squared Mahalanobis distance between the class means
With two classes, always points along , whatever is. So forces . The factor is what rotates the answer away from the naive "line joining the means": it turns down directions where the classes are wide and turns up directions where they are thin.
Why at most . is a sum of rank-one matrices built from the vectors , and those vectors sum to zero (). They therefore span at most dimensions, so . Any orthogonal to that span has . At most eigenvalues are non-zero, whatever the value of . Two classes give one axis; the 10 digits give 9.
- shrinkage: 0 = plain LDA, 1 = ignore correlations (equal spherical within-class spread)
- a scaled identity with the same average variance as
Hyperparameters, mathematically. amplifies directions where the classes look thin. When is small compared with , some directions look thin only by chance: has tiny or zero eigenvalues, and blows them up into spurious "perfect" discriminants. Shrinkage adds to every eigenvalue, which caps the amplification. As , , the plain mean difference. Priors reweight the class means in and . The number of components just truncates the eigenvalue list; (scikit-learn's explained_variance_ratio_) tells you how much separating power each axis carries.
Worked example: two cigar-shaped classes
Ten points with two features. Class B is class A shifted by , so both classes have exactly the same shape, a cloud stretched along the diagonal. A new point is to be projected and classified.
| # | x₁ | x₂ | class |
|---|---|---|---|
| 1 | 1 | 2 | A |
| 2 | 2 | 3 | A |
| 3 | 3 | 3 | A |
| 4 | 4 | 5 | A |
| 5 | 5 | 5 | A |
| 6 | 2 | 0 | B |
| 7 | 3 | 1 | B |
| 8 | 4 | 1 | B |
| 9 | 5 | 3 | B |
| 10 | 6 | 3 | B |
Fisher's discriminant by hand
- 1Means. , , overall . Difference .
- 2Within-class scatter. Class A's deviations from give . Class B is a shifted copy, so . Hence .
- 3Between-class scatter. and , so . Its rows are proportional, so it has rank 1: one useful direction ().
- 4Invert . , so .
- 5Direction. . Its length is , giving the unit vector , at an angle of .
- 6Score it. . Along unit : between , within , ratio . For comparison, the PCA axis at scores only .
- 7Project. (table below). Class A lands on and class B on , with no overlap. The projected means are and , and their midpoint is .
- 8New point. , nearer A's projected mean, so it is classified A.
| # | class | LDA z = wᵀx | PCA z = vᵀx |
|---|---|---|---|
| 1 | A | 0.902 | 2.150 |
| 2 | A | 1.034 | 3.563 |
| 3 | A | 0.396 | 4.239 |
| 4 | A | 1.298 | 6.389 |
| 5 | A | 0.660 | 7.065 |
| 6 | B | −1.276 | 1.352 |
| 7 | B | −1.144 | 2.765 |
| 8 | B | −1.782 | 3.441 |
| 9 | B | −0.880 | 5.591 |
| 10 | B | −1.518 | 6.267 |
| mean | A / B | 0.858 / −1.320 | 4.681 / 3.883 |
Note what the figure's J-curve shows: in 2-D, has exactly two stationary angles, its maximum (LD1) and its minimum. With two classes the minimum is . That is the axis perpendicular to , where both projected means coincide. With three classes the minimum is LD2, still positive, because the three means span two dimensions.
The algorithm, step by step
fit(X, y, q) and transform(X)
- 1Class statistics. For each class compute and ; compute the overall mean . Cost .
- 2Scatter matrices. and . Cost .
- 3Regularize (optional). Replace by the shrunk if is not much larger than .
- 4Eigenproblem. Solve , either directly (
eigensolver) or by whitening with an SVD of the centred within-class data and then an SVD of the whitened class means (svdsolver, which never forms and copes with singular ). Cost . - 5Keep eigenvectors with the largest as the columns of ; report .
- 6Transform. , one multiply per point. Downstream, use for plotting, k-NN, clustering or any classifier.
Total cost is , the same order as PCA via the covariance matrix and far cheaper than t-SNE or UMAP. The class means are all that needs, so the eigenproblem is effectively rank , and the svd solver exploits this.
Variants and practical tuning
| Property | PCA | LDA (Fisher) | Kernel / nonlinear |
|---|---|---|---|
| Uses labels | no | yes | kernel FDA: yes; t-SNE/UMAP: no |
| Objective | max total variance | max between / within scatter | same ratio in a feature space |
| Max components | min(n, d) | K − 1 | K − 1 (kernel FDA) |
| Axes orthogonal? | yes | no (uncorrelated within classes) | — |
| Feature scaling matters? | yes | no (affine invariant) | yes (kernel bandwidth) |
| Can transform new points | yes | yes | kernel FDA yes; t-SNE no |
Knobs that matter (scikit-learn LinearDiscriminantAnalysis)
- n_components
- ≤ min(d, K − 1). Inspect
explained_variance_ratio_. On iris (4 features, 3 classes) the two axes carry 99.12% and 0.88%. - solver
svd(default; no shrinkage, robust to singular ),eigen(supports shrinkage and transform).lsqrsupports shrinkage but cannot transform: it is a classifier-only solver.- shrinkage
'auto'uses the Ledoit–Wolf estimate of α; or set a float in [0, 1] chosen by cross-validation. Essential when n is not ≫ d.- priors
- class weights in the overall mean and the Bayes decision rule; defaults to the class frequencies.
- Kernel FDA
- run the same ratio on kernel features () for curved class boundaries; scales as .
- Heteroscedastic variants
- when the classes have very different covariance shapes, Fisher's criterion is no longer optimal; HLDA or QDA-style models handle that.
| Pipeline | Accuracy |
|---|---|
| PCA(1) → 5-NN | 0.920 |
| LDA(1) → 5-NN | 0.967 |
In code
1import numpy as np23def lda_fit(X, y, n_components=None, shrinkage=0.0):4 classes = np.unique(y)5 d, K = X.shape[1], len(classes)6 q = min(d, K - 1) if n_components is None else n_components7 assert q <= min(d, K - 1), "LDA gives at most K-1 directions"8 m = X.mean(axis=0)9 Sw, Sb = np.zeros((d, d)), np.zeros((d, d))10 for c in classes:11 Xc = X[y == c]12 mc = Xc.mean(axis=0)13 Sw += (Xc - mc).T @ (Xc - mc) # within-class scatter14 Sb += len(Xc) * np.outer(mc - m, mc - m) # between-class scatter15 if shrinkage: # pull S_W toward a scaled identity16 Sw = (1 - shrinkage) * Sw + shrinkage * np.trace(Sw) / d * np.eye(d)17 evals, evecs = np.linalg.eig(np.linalg.solve(Sw, Sb)) # S_W^-1 S_B w = lambda w18 order = np.argsort(evals.real)[::-1][:q]19 W = evecs[:, order].real20 W /= np.linalg.norm(W, axis=0) # unit-length columns21 return W, evals.real[order], Sw, Sb2223def lda_transform(X, W):24 return X @ W2526A = np.array([[1, 2], [2, 3], [3, 3], [4, 5], [5, 5]], float)27B = A + [1, -2]28X = np.vstack([A, B]); y = np.array([0] * 5 + [1] * 5)2930W, lam, Sw, Sb = lda_fit(X, y)31w = W[:, 0] * np.sign(W[1, 0]) # fix the sign: point toward class A32print("S_W =", Sw.tolist(), " S_B =", Sb.tolist())33print("w =", w.round(3), " lambda = J(w) =", lam.round(3)) # [-0.638 0.77] [12.375]34z = lda_transform(X, w)35print("z(A) =", z[:5].round(3), " z(B) =", z[5:].round(3))36mA, mB = z[:5].mean(), z[5:].mean()37print(f"projected means {mA:.3f} {mB:.3f} threshold {(mA + mB) / 2:.3f}") # 0.858 -1.320 -0.23138znew = np.array([3, 4]) @ w39print(f"new point (3,4): z = {znew:.3f} ->", "A" if abs(znew - mA) < abs(znew - mB) else "B") # 1.166 -> A4041from sklearn.datasets import load_iris42Xi, yi = load_iris(return_X_y=True)43Wi, lami, _, _ = lda_fit(Xi, yi)44print("iris: directions", Wi.shape[1], " eigenvalue share", (lami / lami.sum()).round(4)) # 2 [0.9912 0.0088]1import numpy as np2from sklearn.discriminant_analysis import LinearDiscriminantAnalysis3from sklearn.decomposition import PCA45A = np.array([[1, 2], [2, 3], [3, 3], [4, 5], [5, 5]], float)6X = np.vstack([A, A + [1, -2]]); y = np.array(["A"] * 5 + ["B"] * 5)78lda = LinearDiscriminantAnalysis(n_components=1).fit(X, y) # n_components <= K-1 = 19Z = lda.transform(X).ravel()10print("scalings_:", lda.scalings_.ravel().round(3)) # [ 1.843 -2.225] = -1.271 * (-1.45, 1.75)11print("z:", Z.round(3)) # A: -3.274 -3.655 -1.812 -4.418 -2.574 | B: 3.019 2.638 4.481 1.875 3.71912print("explained_variance_ratio_:", lda.explained_variance_ratio_) # [1.]13print("new (3,4):", lda.transform([[3, 4]]).ravel().round(3), lda.predict([[3, 4]])) # [-4.036] ['A']1415pca = PCA(n_components=1).fit(X)16print("PCA axis:", pca.components_.ravel().round(3)) # [0.676 0.737] (labels ignored)1718s = np.sqrt(sum(((Z[y == c] - Z[y == c].mean()) ** 2).sum() for c in "AB") / (len(X) - 2))19print("pooled within-class std of z:", round(s, 3)) # 1.02021# Asking for more than K-1 components fails:22# LinearDiscriminantAnalysis(n_components=2).fit(X, y)23# ValueError: n_components cannot be larger than min(n_features, n_classes - 1).Use when
- +You have labels and want a few axes that keep class separation, for plotting, or as compact features for k-NN, clustering or a small model.
- +Classes differ mostly in their means and have similar spread: then LDA's projection is the optimal linear one.
- +You need something fast, deterministic and able to transform new points, with no tuning beyond shrinkage.
- +Features are on arbitrary scales: LDA is invariant to rescaling, so standardization is not required.
Avoid when
- −You need more than K − 1 dimensions: with 2 classes you get exactly one axis.
- −The classes differ in their covariance shape rather than their means (for example, the same centre but different spread): S_B ≈ 0 and LDA finds nothing.
- −Classes are multimodal or curved (each class several blobs, rings): linear projections can't untangle them. Use kernel FDA or a nonlinear model.
- −n ≲ d without shrinkage: the projection overfits to noise.
- −There are no labels: use PCA, ICA or UMAP instead.
Where it's used
Face recognition (Fisherfaces)
PCA to a few hundred dimensions, then LDA to (number of people − 1) axes. This is robust to lighting changes that dominate the variance.
Brain–computer interfaces
Shrinkage LDA is the standard projection/classifier for EEG event-related potentials, where trials are few and channels × time samples are many.
Spectroscopy & chemometrics
Projecting hundreds of wavelengths onto a few discriminant axes to separate sample types (food origin, tissue, minerals), often after PCA (PCA-LDA).
Visualizing labelled data
A 2-D LDA plot of a 3+ class problem shows at a glance which classes are separable and which overlap, as a supervised companion to a PCA plot.
Check yourself
A dataset has 50 features and 10 classes. What is the largest useful n_components for LDA?
Check yourself
Two classes have within-class scatter S_W = diag(4, 1) and mean difference μ₁ − μ₂ = (2, 2). Fisher's direction is proportional to…
Check yourself
In the figure's 3-class data the generalized eigenvalues are 8.258 and 0.687. What share of the separating power does LD1 carry?
Check yourself
You multiply feature x₁ of the worked example by 100. What happens?
Remember
- ✓LDA projects onto directions maximizing : class means far apart, measured in units of within-class spread.
- ✓Setting the gradient of the constrained problem to zero gives the generalized eigenproblem , whose eigenvalue equals the Fisher ratio. For two classes, .
- ✓, so you get at most discriminant axes, however many features you have.
- ✓Equivalent view: whiten by S_W^−½, then do PCA on the class means. Hence LDA is invariant to feature scaling and its axes are not orthogonal in the original space.
- ✓Shrink S_W (solver='eigen', shrinkage='auto') when n is not much bigger than d, and fit LDA inside cross-validation because it uses the labels.