# LDA for dimensionality reduction

> Project onto the directions that maximize between-class over within-class scatter — at most K − 1 of them.

Source: https://www.mlfoundry.in/learn/lda-for-dimensionality-reduction  
Section: ML Algorithms › Unsupervised · Dimensionality Reduction  
Level: core · 16 min read · updated 2026-10-07

**Linear discriminant analysis (as a projection):** is a **supervised** linear dimensionality reduction. It uses the class labels to find directions $w$ along which the class means sit far apart *relative to* the spread inside each class. These are Fisher's discriminant directions. Projecting $d$-dimensional data onto the best $q \le K-1$ of them gives a low-dimensional representation that keeps as much class separation as a linear map can.

- **Task:** Supervised dimensionality reduction (feature extraction before a classifier, or plotting labelled data)
- **Input → output:** labelled $x\in\mathbb{R}^d$ → $z=W^\top x\in\mathbb{R}^q$ with $q\le\min(d,\,K-1)$
- **Learns:** a $d\times q$ projection matrix $W$: the top generalized eigenvectors of $S_B w=\lambda S_W w$
- **Training:** $O(n d^2)$ to build the scatter matrices + $O(d^3)$ for the eigenproblem
- **Projection:** $O(dq)$ per point, a single matrix multiply
- **Hyperparameters:** `n_components` (≤ K − 1), solver (`svd` / `eigen`), `shrinkage` of the within-class covariance, class `priors`
- **Assumes:** classes differ mainly in their **means**, with roughly the same covariance shape (the Gaussian, shared-covariance model). It works as a heuristic beyond that.

## The problem

You have labelled data with many features and want a few new axes, to plot it in 2-D or to feed a small, well-conditioned feature set to a classifier. The obvious tool is **PCA** (see the PCA article), which keeps the directions of **largest variance**. But PCA never looks at the labels. Variance is often driven by something unrelated to the classes: overall size, lighting, a sensor that drifts. The direction that separates the classes can be a *low*-variance one. PCA then throws it away.

The worked example below makes this concrete. Two classes are elongated along the same diagonal and offset across it. PCA picks the diagonal, and on that axis the classes interleave completely: Fisher ratio $J=0.048$, and 5 of 10 points end up nearer the other class's mean. The useful axis is almost perpendicular to it, at $J=12.375$, with zero errors. We need a criterion that uses the labels.

## The idea

Project every point onto a line $w$ and look at the 1-D result. Two numbers describe how well the classes separate there: the gap between the projected **class means** (we want this big) and the spread of each class **around its own mean** (we want this small). Fisher's 1936 proposal was to maximize their ratio. A big gap means nothing if each class is smeared across it, and a tight class means nothing if the means coincide.

Picture the two classes as two cigar-shaped clouds lying side by side. PCA aims along the cigars, because that is where the variance is. LDA aims across them: a short distance between the means, but measured in units of the cigars' thin width, which makes it a huge separation.

**Vocabulary**

- **Scatter matrix:** an unnormalized covariance: $\sum (x-\mu)(x-\mu)^\top$. Dividing by a count gives a covariance, and that constant does not change the directions.
- **Within-class scatter S_W:** how much points spread around **their own** class mean, summed over classes
- **Between-class scatter S_B:** how much the **class means** spread around the overall mean, weighted by class size
- **Fisher ratio J(w):** between-class over within-class scatter **after projecting onto** $w$; a signal-to-noise ratio for the classes
- **Discriminant direction (LD1, LD2, …):** a stationary point of $J$. LD1 is the best; each later one is the best that is uncorrelated with the earlier ones (within classes).
- **Whitening:** a linear map that turns a covariance into the identity; here $x\mapsto S_W^{-1/2}x$ makes every class cloud round
- **Rank of S_B:** at most $K-1$, so at most $K-1$ directions with $J>0$ exist; the rest carry no class information

## The math

Take a unit direction $w$. Each point becomes the scalar $z_i=w^\top x_i$, and class $k$'s mean becomes $w^\top\mu_k$. Both the between-class and the within-class spread of these scalars turn out to be quadratic forms in $w$:

$$
S_W=\sum_{k=1}^{K}\sum_{i\in C_k}(x_i-\mu_k)(x_i-\mu_k)^\top,\qquad S_B=\sum_{k=1}^{K}n_k(\mu_k-\mu)(\mu_k-\mu)^\top
$$

- $C_k,\ n_k$: the set of training points in class $k$, and its size
- $\mu_k$: mean of class $k$; $\mu$ is the overall mean
- $S_W$: $d\times d$ within-class scatter (sum of per-class scatters)
- $S_B$: $d\times d$ between-class scatter of the class means

For the projection, $\sum_{i\in C_k}(w^\top x_i-w^\top\mu_k)^2=w^\top\big[\sum_{i\in C_k}(x_i-\mu_k)(x_i-\mu_k)^\top\big]w$. Summing over classes gives $w^\top S_W w$ as the projected within-class scatter, and the same algebra gives $w^\top S_B w$ for the means. The total scatter splits exactly as $S_T=S_W+S_B$, an ANOVA decomposition.

$$
J(w)=\frac{w^\top S_B\,w}{w^\top S_W\,w}
$$

- $J(w)$: Fisher criterion: projected between-class scatter over projected within-class scatter
- $w$: projection direction in $\mathbb{R}^d$; only its direction matters since $J(cw)=J(w)$

**Deriving the solution.** Because $J$ ignores the length of $w$, we can fix the denominator and maximize the numerator: maximize $w^\top S_B w$ subject to $w^\top S_W w=1$. The Lagrangian is $\mathcal{L}=w^\top S_B w-\lambda(w^\top S_W w-1)$. Setting $\nabla_w\mathcal{L}=2S_Bw-2\lambda S_Ww=0$ gives

$$
S_B\,w=\lambda\,S_W\,w\quad\Longleftrightarrow\quad S_W^{-1}S_B\,w=\lambda\,w,\qquad \lambda=J(w)
$$

- $\lambda$: a generalized eigenvalue; multiplying the condition by $w^\top$ shows $\lambda=w^\top S_Bw / w^\top S_Ww=J(w)$

So every stationary point of $J$ is a **generalized eigenvector**, and its eigenvalue is the Fisher ratio it achieves. LD1 is the eigenvector with the largest $\lambda$, LD2 the next, and so on. For $q$ directions at once, the matrix criterion $\operatorname{tr}\big((W^\top S_W W)^{-1}W^\top S_B W\big)$ is maximized by the top $q$ eigenvectors. The resulting scores are uncorrelated within classes, $W^\top S_W W$ diagonal. Unlike PCA, the directions are generally **not** orthogonal in the original space.

$$
K=2:\quad S_B=\tfrac{n_1n_2}{n}(\mu_1-\mu_2)(\mu_1-\mu_2)^\top\ \Rightarrow\ w^\star\propto S_W^{-1}(\mu_1-\mu_2),\quad J(w^\star)=\tfrac{n_1n_2}{n}(\mu_1-\mu_2)^\top S_W^{-1}(\mu_1-\mu_2)
$$

- $w^\star$: Fisher's linear discriminant; no eigen-solver needed, just one linear solve
- $(\mu_1-\mu_2)^\top S_W^{-1}(\mu_1-\mu_2)$: a (scaled) squared Mahalanobis distance between the class means

With two classes, $S_Bw$ always points along $\mu_1-\mu_2$, whatever $w$ is. So $S_W^{-1}S_Bw=\lambda w$ forces $w\propto S_W^{-1}(\mu_1-\mu_2)$. The factor $S_W^{-1}$ is what rotates the answer away from the naive "line joining the means": it turns down directions where the classes are wide and turns up directions where they are thin.

**Why at most $K-1$.** $S_B$ is a sum of $K$ rank-one matrices built from the vectors $n_k(\mu_k-\mu)$, and those vectors sum to zero ($\sum_k n_k\mu_k=n\mu$). They therefore span at most $K-1$ dimensions, so $\operatorname{rank}S_B\le K-1$. Any $w$ orthogonal to that span has $w^\top S_Bw=0$. At most $K-1$ eigenvalues are non-zero, whatever the value of $d$. Two classes give one axis; the 10 digits give 9.

$$
S_W^{(\alpha)}=(1-\alpha)\,S_W+\alpha\,\frac{\operatorname{tr}S_W}{d}\,I
$$

- $\alpha\in[0,1]$: shrinkage: 0 = plain LDA, 1 = ignore correlations (equal spherical within-class spread)
- $\tfrac{\operatorname{tr}S_W}{d}I$: a scaled identity with the same average variance as $S_W$

**Hyperparameters, mathematically.** $S_W^{-1}$ amplifies directions where the classes look thin. When $n$ is small compared with $d$, some directions look thin only by chance: $S_W$ has tiny or zero eigenvalues, and $S_W^{-1}$ blows them up into spurious "perfect" discriminants. Shrinkage adds $\alpha\,\tfrac{\operatorname{tr}S_W}{d}$ to every eigenvalue, which caps the amplification. As $\alpha\to1$, $w^\star\to\mu_1-\mu_2$, the plain mean difference. **Priors** reweight the class means in $\mu$ and $S_B$. The number of components $q$ just truncates the eigenvalue list; $\lambda_j/\sum\lambda$ (scikit-learn's `explained_variance_ratio_`) tells you how much separating power each axis carries.

> **LDA = whiten, then PCA on the class means:** Substitute $u=S_W^{1/2}w$. The criterion becomes $J=u^\top(S_W^{-1/2}S_BS_W^{-1/2})u\,/\,u^\top u$, an ordinary Rayleigh quotient. So LDA (1) whitens the data with $S_W^{-1/2}$, making every class cloud round, and then (2) runs PCA on the $K$ whitened class means. Once the noise is round, the best axis is simply the one along which the means spread most. This is why LDA is unchanged by rescaling any feature, while PCA is not. In the figure, press **whiten** and LD1 lines up with the line joining the two class means.

## Worked example: two cigar-shaped classes

Ten points with two features. Class B is class A shifted by $(+1,-2)$, so both classes have exactly the same shape, a cloud stretched along the diagonal. A new point $(3,4)$ is to be projected and classified.

The data. Class B = class A + (1, −2). Highlighted: class A.

| # | x₁ | x₂ | class |
| --- | --- | --- | --- |
| 1 | 1 | 2 | A |
| 2 | 2 | 3 | A |
| 3 | 3 | 3 | A |
| 4 | 4 | 5 | A |
| 5 | 5 | 5 | A |
| 6 | 2 | 0 | B |
| 7 | 3 | 1 | B |
| 8 | 4 | 1 | B |
| 9 | 5 | 3 | B |
| 10 | 6 | 3 | B |

**Fisher's discriminant by hand**

1. **Means.** $\mu_A=(3,\,3.6)$, $\mu_B=(4,\,1.6)$, overall $\mu=(3.5,\,2.6)$. Difference $\mu_A-\mu_B=(-1,\,2)$.
2. **Within-class scatter.** Class A's deviations from $\mu_A$ give $S_A=\begin{pmatrix}10&8\\8&7.2\end{pmatrix}$. Class B is a shifted copy, so $S_B^{(\text{class})}=S_A$. Hence $S_W=\begin{pmatrix}20&16\\16&14.4\end{pmatrix}$.
3. **Between-class scatter.** $\mu_A-\mu=(-0.5,1)$ and $\mu_B-\mu=(0.5,-1)$, so $S_B=5\,(-0.5,1)(-0.5,1)^\top+5\,(0.5,-1)(0.5,-1)^\top=\begin{pmatrix}2.5&-5\\-5&10\end{pmatrix}$. Its rows are proportional, so it has rank 1: one useful direction ($K-1=1$).
4. **Invert $S_W$.** $\det S_W=20\cdot14.4-16^2=32$, so $S_W^{-1}=\tfrac{1}{32}\begin{pmatrix}14.4&-16\\-16&20\end{pmatrix}=\begin{pmatrix}0.45&-0.5\\-0.5&0.625\end{pmatrix}$.
5. **Direction.** $w^\star=S_W^{-1}(\mu_A-\mu_B)=(-0.45-1,\ 0.5+1.25)=(-1.45,\,1.75)$. Its length is $2.273$, giving the unit vector $w=(-0.638,\,0.770)$, at an angle of $129.6^\circ$.
6. **Score it.** $J(w^\star)=\tfrac{5\cdot5}{10}(\mu_A-\mu_B)^\top w^\star=2.5\,(1.45+3.5)=12.375$. Along unit $w$: between $w^\top S_Bw=11.860$, within $w^\top S_Ww=0.958$, ratio $12.375$. For comparison, the PCA axis $(0.676,0.737)$ at $47.5^\circ$ scores only $J=0.048$.
7. **Project.** $z_i=w^\top x_i$ (table below). Class A lands on $0.40\ldots1.30$ and class B on $-1.78\ldots-0.88$, with no overlap. The projected means are $0.858$ and $-1.320$, and their midpoint is $-0.231$.
8. **New point.** $z=w^\top(3,4)=-0.638\cdot3+0.770\cdot4=1.166>-0.231$, nearer A's projected mean, so it is classified **A**.

Each point projected onto the LDA axis w = (−0.638, 0.770) and onto the PCA axis v = (0.676, 0.737). On the LDA axis the classes are separated by a gap; on the PCA axis they interleave.

| # | class | LDA  z = wᵀx | PCA  z = vᵀx |
| --- | --- | --- | --- |
| 1 | A | 0.902 | 2.150 |
| 2 | A | 1.034 | 3.563 |
| 3 | A | 0.396 | 4.239 |
| 4 | A | 1.298 | 6.389 |
| 5 | A | 0.660 | 7.065 |
| 6 | B | −1.276 | 1.352 |
| 7 | B | −1.144 | 2.765 |
| 8 | B | −1.782 | 3.441 |
| 9 | B | −0.880 | 5.591 |
| 10 | B | −1.518 | 6.267 |
| mean | A / B | **0.858 / −1.320** | 4.681 / 3.883 |

> Figure: The worked example. Drag the handle w to rotate the projection axis. Each point drops onto it, the right panel shows the resulting 1-D data per class, and the bottom curve is J for every angle. The figure starts on the PCA axis (47.5°, J = 0.048, 5/10 nearest-mean errors); LD1 at 129.6° reaches the maximum J = 12.375 with 0 errors and sends the new point (3, 4) to z = 1.166, class A. "Whiten by S_W" animates the map S_W^−½: the dashed within-class ellipses become circles, and LD1 becomes the line joining the class means. "3 classes" adds class C = A + (3, 1): λ = 8.258 and 0.687 (92.3% / 7.7%), so two discriminants exist; LD2 is the minimum of J, and in the whitened view LD1 and LD2 are perpendicular.

Note what the figure's J-curve shows: in 2-D, $J(\theta)$ has exactly two stationary angles, its maximum (LD1) and its minimum. With two classes the minimum is $0$. That is the axis perpendicular to $\mu_A-\mu_B$, where both projected means coincide. With three classes the minimum is LD2, still positive, because the three means span two dimensions.

## The algorithm, step by step

**fit(X, y, q) and transform(X)**

1. **Class statistics.** For each class $k$ compute $n_k$ and $\mu_k$; compute the overall mean $\mu$. Cost $O(nd)$.
2. **Scatter matrices.** $S_W=\sum_k\sum_{i\in C_k}(x_i-\mu_k)(x_i-\mu_k)^\top$ and $S_B=\sum_k n_k(\mu_k-\mu)(\mu_k-\mu)^\top$. Cost $O(nd^2)$.
3. **Regularize (optional).** Replace $S_W$ by the shrunk $(1-\alpha)S_W+\alpha\tfrac{\operatorname{tr}S_W}{d}I$ if $n$ is not much larger than $d$.
4. **Eigenproblem.** Solve $S_Bw=\lambda S_Ww$, either directly (`eigen` solver) or by whitening with an SVD of the centred within-class data and then an SVD of the whitened class means (`svd` solver, which never forms $S_W$ and copes with singular $S_W$). Cost $O(d^3)$.
5. **Keep $q\le\min(d,K-1)$** eigenvectors with the largest $\lambda$ as the columns of $W$; report $\lambda_j/\sum\lambda$.
6. **Transform.** $z=W^\top(x-\mu)$, one $O(dq)$ multiply per point. Downstream, use $z$ for plotting, k-NN, clustering or any classifier.

Total cost is $O(nd^2+d^3)$, the same order as PCA via the covariance matrix and far cheaper than t-SNE or UMAP. The $K$ class means are all that $S_B$ needs, so the eigenproblem is effectively rank $K-1$, and the `svd` solver exploits this.

## Variants and practical tuning

LDA versus its sibling projections. Only LDA uses the labels.

| Property | PCA | LDA (Fisher) | Kernel / nonlinear |
| --- | --- | --- | --- |
| Uses labels | no | **yes** | kernel FDA: yes; t-SNE/UMAP: no |
| Objective | max total variance | **max between / within scatter** | same ratio in a feature space |
| Max components | min(n, d) | **K − 1** | K − 1 (kernel FDA) |
| Axes orthogonal? | yes | no (uncorrelated within classes) | — |
| Feature scaling matters? | yes | **no** (affine invariant) | yes (kernel bandwidth) |
| Can transform new points | yes | yes | kernel FDA yes; t-SNE no |

**Knobs that matter (scikit-learn LinearDiscriminantAnalysis)**

- **n_components:** ≤ min(d, K − 1). Inspect `explained_variance_ratio_`. On iris (4 features, 3 classes) the two axes carry 99.12% and 0.88%.
- **solver:** `svd` (default; no shrinkage, robust to singular $S_W$), `eigen` (supports shrinkage *and* transform). `lsqr` supports shrinkage but **cannot transform**: it is a classifier-only solver.
- **shrinkage:** `'auto'` uses the Ledoit–Wolf estimate of α; or set a float in [0, 1] chosen by cross-validation. Essential when n is not ≫ d.
- **priors:** class weights in the overall mean and the Bayes decision rule; defaults to the class frequencies.
- **Kernel FDA:** run the same ratio on kernel features ($\phi(x)$) for curved class boundaries; scales as $O(n^3)$.
- **Heteroscedastic variants:** when the classes have very different covariance shapes, Fisher's criterion is no longer optimal; HLDA or QDA-style models handle that.

Iris, 5-fold cross-validated accuracy of a 5-nearest-neighbour classifier after reducing 4 features to 1. The reducer is fitted inside each fold.

| Pipeline | Accuracy |
| --- | --- |
| PCA(1) → 5-NN | 0.920 |
| **LDA(1) → 5-NN** | **0.967** |

> **Small n, large d: LDA finds perfect noise:** With 60 training points and 100 features (5 informative, synthetic), plain LDA (`svd`) scored 0.917 on training data and only **0.636** on 940 held-out points. With `solver="eigen", shrinkage="auto"` the test accuracy rose to **0.745** (0.764 at `shrinkage=0.5`). When $n\lesssim d$, $S_W$ is singular or nearly so, and $S_W^{-1}$ amplifies directions where the classes look thin only by chance. Always shrink, or reduce with PCA first (the classic "PCA + LDA" pipeline of Fisherfaces).

> **Fitting LDA before the train/test split:** LDA uses the labels, so a projection fitted on all the data has already seen the test labels. Its 2-D plots look beautifully separated, and downstream scores are optimistic. Fit it inside a `Pipeline` so that each cross-validation fold learns its own $W$.

> **Projection vs classifier:** The same model is also a classifier. With Gaussian classes sharing one covariance, Bayes' rule gives linear boundaries, and nearest-projected-mean on all K − 1 axes reproduces it under equal priors. That probabilistic view, with posteriors and priors, is covered in the *Linear discriminant analysis* article. This article is about the projection.

## In code

```python
import numpy as np

def lda_fit(X, y, n_components=None, shrinkage=0.0):
    classes = np.unique(y)
    d, K = X.shape[1], len(classes)
    q = min(d, K - 1) if n_components is None else n_components
    assert q <= min(d, K - 1), "LDA gives at most K-1 directions"
    m = X.mean(axis=0)
    Sw, Sb = np.zeros((d, d)), np.zeros((d, d))
    for c in classes:
        Xc = X[y == c]
        mc = Xc.mean(axis=0)
        Sw += (Xc - mc).T @ (Xc - mc)                 # within-class scatter
        Sb += len(Xc) * np.outer(mc - m, mc - m)      # between-class scatter
    if shrinkage:                                      # pull S_W toward a scaled identity
        Sw = (1 - shrinkage) * Sw + shrinkage * np.trace(Sw) / d * np.eye(d)
    evals, evecs = np.linalg.eig(np.linalg.solve(Sw, Sb))   # S_W^-1 S_B w = lambda w
    order = np.argsort(evals.real)[::-1][:q]
    W = evecs[:, order].real
    W /= np.linalg.norm(W, axis=0)                     # unit-length columns
    return W, evals.real[order], Sw, Sb

def lda_transform(X, W):
    return X @ W

A = np.array([[1, 2], [2, 3], [3, 3], [4, 5], [5, 5]], float)
B = A + [1, -2]
X = np.vstack([A, B]); y = np.array([0] * 5 + [1] * 5)

W, lam, Sw, Sb = lda_fit(X, y)
w = W[:, 0] * np.sign(W[1, 0])                         # fix the sign: point toward class A
print("S_W =", Sw.tolist(), " S_B =", Sb.tolist())
print("w =", w.round(3), " lambda = J(w) =", lam.round(3))   # [-0.638 0.77] [12.375]
z = lda_transform(X, w)
print("z(A) =", z[:5].round(3), " z(B) =", z[5:].round(3))
mA, mB = z[:5].mean(), z[5:].mean()
print(f"projected means {mA:.3f} {mB:.3f}  threshold {(mA + mB) / 2:.3f}")  # 0.858 -1.320 -0.231
znew = np.array([3, 4]) @ w
print(f"new point (3,4): z = {znew:.3f} ->", "A" if abs(znew - mA) < abs(znew - mB) else "B")  # 1.166 -> A

from sklearn.datasets import load_iris
Xi, yi = load_iris(return_X_y=True)
Wi, lami, _, _ = lda_fit(Xi, yi)
print("iris: directions", Wi.shape[1], " eigenvalue share", (lami / lami.sum()).round(4))  # 2 [0.9912 0.0088]
```

From scratch with NumPy: scatter matrices, the eigenproblem, projection. Prints w = (−0.638, 0.770), J = 12.375 and classifies (3, 4) as A.

```python
import numpy as np
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.decomposition import PCA

A = np.array([[1, 2], [2, 3], [3, 3], [4, 5], [5, 5]], float)
X = np.vstack([A, A + [1, -2]]); y = np.array(["A"] * 5 + ["B"] * 5)

lda = LinearDiscriminantAnalysis(n_components=1).fit(X, y)   # n_components <= K-1 = 1
Z = lda.transform(X).ravel()
print("scalings_:", lda.scalings_.ravel().round(3))          # [ 1.843 -2.225]  = -1.271 * (-1.45, 1.75)
print("z:", Z.round(3))           # A: -3.274 -3.655 -1.812 -4.418 -2.574 | B: 3.019 2.638 4.481 1.875 3.719
print("explained_variance_ratio_:", lda.explained_variance_ratio_)        # [1.]
print("new (3,4):", lda.transform([[3, 4]]).ravel().round(3), lda.predict([[3, 4]]))  # [-4.036] ['A']

pca = PCA(n_components=1).fit(X)
print("PCA axis:", pca.components_.ravel().round(3))         # [0.676 0.737]  (labels ignored)

s = np.sqrt(sum(((Z[y == c] - Z[y == c].mean()) ** 2).sum() for c in "AB") / (len(X) - 2))
print("pooled within-class std of z:", round(s, 3))          # 1.0

# Asking for more than K-1 components fails:
# LinearDiscriminantAnalysis(n_components=2).fit(X, y)
# ValueError: n_components cannot be larger than min(n_features, n_classes - 1).
```

scikit-learn on the same ten points. The axis matches up to sign and scale: sklearn scales scores so the pooled within-class standard deviation is 1, and centres them.

**Strengths**

- You have labels and want a few axes that keep class separation, for plotting, or as compact features for k-NN, clustering or a small model.
- Classes differ mostly in their means and have similar spread: then LDA's projection is the optimal linear one.
- You need something fast, deterministic and able to transform new points, with no tuning beyond shrinkage.
- Features are on arbitrary scales: LDA is invariant to rescaling, so standardization is not required.

**Limitations**

- You need more than K − 1 dimensions: with 2 classes you get exactly one axis.
- The classes differ in their covariance shape rather than their means (for example, the same centre but different spread): S_B ≈ 0 and LDA finds nothing.
- Classes are multimodal or curved (each class several blobs, rings): linear projections can't untangle them. Use kernel FDA or a nonlinear model.
- n ≲ d without shrinkage: the projection overfits to noise.
- There are no labels: use PCA, ICA or UMAP instead.

- **Face recognition (Fisherfaces):** PCA to a few hundred dimensions, then LDA to (number of people − 1) axes. This is robust to lighting changes that dominate the variance.
- **Brain–computer interfaces:** Shrinkage LDA is the standard projection/classifier for EEG event-related potentials, where trials are few and channels × time samples are many.
- **Spectroscopy & chemometrics:** Projecting hundreds of wavelengths onto a few discriminant axes to separate sample types (food origin, tissue, minerals), often after PCA (PCA-LDA).
- **Visualizing labelled data:** A 2-D LDA plot of a 3+ class problem shows at a glance which classes are separable and which overlap, as a supervised companion to a PCA plot.

**Question:** A dataset has 50 features and 10 classes. What is the largest useful n_components for LDA?

- A. 50
- B. 10
- C. 9
- D. 2

**Answer:** C. 9 — $\operatorname{rank}S_B\le K-1=9$ because the 10 weighted mean deviations sum to zero; scikit-learn rejects anything above $\min(d,K-1)=9$.

**Question:** Two classes have within-class scatter S_W = diag(4, 1) and mean difference μ₁ − μ₂ = (2, 2). Fisher's direction is proportional to…

- A. (1, 1)
- B. (1, 4)
- C. (4, 1)
- D. (1, 2)

**Answer:** B. (1, 4) — $w\propto S_W^{-1}(\mu_1-\mu_2)=(2/4,\ 2/1)=(0.5,\,2)\propto(1,4)$. The thin direction $x_2$, with within-class scatter 1, gets four times the weight of the wide one.

**Question:** In the figure's 3-class data the generalized eigenvalues are 8.258 and 0.687. What share of the separating power does LD1 carry?

- A. 50.0%
- B. 76.8%
- C. 92.3%
- D. 100%

**Answer:** C. 92.3% — $8.258/(8.258+0.687)=0.923$. This is exactly scikit-learn's `explained_variance_ratio_`: the eigenvalues are the Fisher ratios of the axes.

**Question:** You multiply feature x₁ of the worked example by 100. What happens?

- A. Both PCA and LDA projections change
- B. PCA's axis swings to almost exactly x₁, while LDA's projected scores are unchanged
- C. LDA's axis swings to x₁, PCA's is unchanged
- D. Neither changes

**Answer:** B. PCA's axis swings to almost exactly x₁, while LDA's projected scores are unchanged — PCA chases variance, and its first axis became $(1.000,\,0.005)$. LDA is affine invariant: $w$ rescales to compensate, and scikit-learn's projected scores were identical to three decimals.

**Key points**

- LDA projects onto directions maximizing $J(w)=w^\top S_Bw\,/\,w^\top S_Ww$: class means far apart, measured in units of within-class spread.
- Setting the gradient of the constrained problem to zero gives the generalized eigenproblem $S_Bw=\lambda S_Ww$, whose eigenvalue $\lambda$ equals the Fisher ratio. For two classes, $w\propto S_W^{-1}(\mu_1-\mu_2)$.
- $\operatorname{rank}S_B\le K-1$, so you get at most $K-1$ discriminant axes, however many features you have.
- Equivalent view: whiten by S_W^−½, then do PCA on the class means. Hence LDA is invariant to feature scaling and its axes are not orthogonal in the original space.
- Shrink S_W (solver='eigen', shrinkage='auto') when n is not much bigger than d, and fit LDA inside cross-validation because it uses the labels.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
