# Linear discriminant analysis

> Model each class as a Gaussian with a shared covariance; Bayes' rule then gives a linear decision boundary.

Source: https://www.mlfoundry.in/learn/linear-discriminant-analysis  
Section: ML Algorithms › Supervised · Classification  
Level: advanced · 17 min read · updated 2026-10-07

**Linear discriminant analysis (LDA):** is a **generative** classifier. It assumes the features of each class $k$ follow a multivariate Gaussian $\mathcal N(\mu_k,\Sigma)$ with its own mean but **one covariance matrix shared by all classes**, estimates the means, the shared covariance and the class priors in closed form, and classifies a new point with Bayes' rule. Because the covariance is shared, the quadratic terms cancel and every decision boundary is a straight line (a hyperplane).

- **Task:** Classification (binary or multi-class); the same fit also gives a supervised projection
- **Input → output:** $x\in\mathbb R^d$ → posterior $P(y=k\mid x)$ for every class, and the class with the largest one
- **Learns:** class priors $\pi_k$, class means $\mu_k$, one pooled covariance $\Sigma$ — giving a weight vector $a_k=\Sigma^{-1}\mu_k$ and an offset $c_k$ per class
- **Training:** one pass for means and covariance, then one $d\times d$ solve: $O(nd^2+d^3)$; no iterations, no learning rate
- **Prediction:** $O(Kd)$ — one dot product per class
- **Hyperparameters:** class priors, covariance shrinkage γ, solver (svd / lsqr / eigen); nothing else
- **Assumes:** Gaussian classes with equal covariance; roughly linear separation; no extreme outliers

## The problem

The simplest classifier anyone invents is **nearest mean**: average the training points of each class, then give a new point the label of the closest average. It fails in a specific, common way: it measures distance with a ruler that ignores the **shape** of the classes. If two features are correlated within each class — long fish are also wide fish — the clouds are tilted ellipses, and a point can be closer to the *other* class's centre in plain distance while sitting squarely on the long axis of its own class.

Logistic regression (see **logistic-regression**) fixes the boundary by fitting it directly, but it needs iterative optimisation, its coefficients run off to infinity on perfectly separable data, and with few examples per feature it is noisy. LDA takes the other route: write down a probability model of how each class *generates* its features, fit that model in closed form, and let Bayes' rule produce the boundary. The covariance matrix it estimates is exactly the ruler that nearest-mean was missing.

## The idea

Picture each class as a Gaussian "cloud" — an ellipse of the same size, shape and orientation, just centred in a different place. To classify a point, ask which cloud makes it most likely, adjusted for how common each class is. Because the ellipses are identical copies, the place where two clouds are equally likely is a straight line. Measuring with the ellipse rather than with a circle is the **Mahalanobis distance**: LDA is nearest-mean classification after you "un-stretch" the space so each cloud becomes a circle.

**Vocabulary**

- **Class prior πₖ:** probability of class $k$ before looking at $x$; estimated as the class's share of the training set $n_k/n$
- **Class-conditional density:** $p(x\mid y=k)$ — how the features of class $k$ are distributed; here $\mathcal N(\mu_k,\Sigma)$
- **Pooled covariance Σ:** one covariance matrix estimated from the deviations of every point from *its own* class mean; describes the common shape of all clouds
- **Discriminant function δₖ(x):** a score per class whose argmax is the Bayes decision; for LDA it is linear in $x$
- **Mahalanobis distance:** $\sqrt{(x-\mu)^\top\Sigma^{-1}(x-\mu)}$ — distance measured in units of the cloud's own spread along each direction
- **Fisher direction w:** $\Sigma^{-1}(\mu_1-\mu_0)$: the line onto which the two classes, projected, are best separated relative to their spread
- **Shrinkage γ:** blending the estimated covariance with a multiple of the identity, to keep it well-conditioned when data are scarce
- **Generative vs discriminative:** LDA models $p(x\mid y)$ and $p(y)$ and derives $p(y\mid x)$; logistic regression models $p(y\mid x)$ directly

## The math

**The model.** Each class has a prior and a Gaussian density with a class-specific mean and a shared covariance:

$$
P(y=k)=\pi_k,\qquad p(x\mid y=k)=\frac{1}{(2\pi)^{d/2}|\Sigma|^{1/2}}\exp\!\Big(-\tfrac12(x-\mu_k)^\top\Sigma^{-1}(x-\mu_k)\Big)
$$

- $\pi_k$: prior probability of class $k$, with $\sum_k\pi_k=1$
- $\mu_k$: mean vector of class $k$ ($d$ entries)
- $\Sigma$: $d\times d$ covariance, the same for every class
- $|\Sigma|$: determinant of Σ (also shared, so it will cancel)
- $d$: number of features

**Bayes' rule** turns the model into a posterior. The denominator $p(x)$ is the same for every class, so the most probable class maximises the numerator — or its logarithm:

$$
P(y=k\mid x)=\frac{\pi_k\,p(x\mid y=k)}{\sum_j \pi_j\,p(x\mid y=j)},\qquad \hat y=\arg\max_k\;\big[\log\pi_k+\log p(x\mid y=k)\big]
$$

- $P(y=k\mid x)$: posterior probability of class $k$ after seeing $x$
- $\hat y$: predicted class

**Why the boundary is linear.** Take the log of the Gaussian and expand the quadratic form: $-\tfrac12(x-\mu_k)^\top\Sigma^{-1}(x-\mu_k)=-\tfrac12x^\top\Sigma^{-1}x+x^\top\Sigma^{-1}\mu_k-\tfrac12\mu_k^\top\Sigma^{-1}\mu_k$. The constant $-\tfrac d2\log 2\pi-\tfrac12\log|\Sigma|$ and the term $-\tfrac12x^\top\Sigma^{-1}x$ are **identical for every class** because $\Sigma$ is shared, so they cannot change the argmax and we drop them. What is left is linear in $x$:

$$
\delta_k(x)=x^\top\underbrace{\Sigma^{-1}\mu_k}_{a_k}\;\underbrace{-\;\tfrac12\mu_k^\top\Sigma^{-1}\mu_k+\log\pi_k}_{c_k}
$$

- $\delta_k(x)$: linear discriminant score of class $k$; predict $\arg\max_k\delta_k(x)$
- $a_k$: weight vector of class $k$
- $c_k$: offset of class $k$: a penalty for a far-out mean plus the log prior

With two classes the boundary $\delta_1(x)=\delta_0(x)$ is a single hyperplane $w^\top x+b=0$:

$$
w=\Sigma^{-1}(\mu_1-\mu_0),\qquad b=-\tfrac12(\mu_0+\mu_1)^\top w+\log\frac{\pi_1}{\pi_0},\qquad P(y=1\mid x)=\sigma(w^\top x+b)
$$

- $w$: normal vector of the boundary (the Fisher direction)
- $b$: offset: puts the boundary at the midpoint of the means, shifted by the prior log-odds
- $\sigma(z)$: logistic function $1/(1+e^{-z})$

> **LDA's posterior is a logistic regression:** The log-odds $\log\frac{P(y=1\mid x)}{P(y=0\mid x)}=\delta_1(x)-\delta_0(x)=w^\top x+b$ is linear, so the posterior has exactly the form logistic regression assumes. The two models share a hypothesis class and differ only in how they **fit** it: logistic regression maximises the conditional likelihood of the labels; LDA plugs in Gaussian estimates. When the Gaussian assumption holds, LDA's estimates use more information and are less noisy; when it fails (heavy tails, outliers, binary features), logistic regression is the safer fit.

**Fitting.** Maximum likelihood gives the obvious estimates. For the covariance we pool the deviations of every point from its own class mean and divide by $n-K$ (the unbiased version, which is what most texts and scikit-learn's default solver use):

$$
\hat\pi_k=\frac{n_k}{n},\qquad \hat\mu_k=\frac1{n_k}\sum_{i:\,y_i=k}x_i,\qquad \hat\Sigma=\frac{1}{n-K}\sum_{k=1}^{K}\sum_{i:\,y_i=k}(x_i-\hat\mu_k)(x_i-\hat\mu_k)^\top=\frac{S_W}{n-K}
$$

- $n_k$: number of training points in class $k$; $n=\sum_k n_k$
- $K$: number of classes
- $S_W$: within-class scatter matrix: the double sum before dividing

**The Mahalanobis view.** Adding back the class-independent $-\tfrac12x^\top\Sigma^{-1}x$ shows that LDA is a nearest-mean rule in a re-measured space:

$$
\hat y=\arg\min_k\Big[\tfrac12\,\Delta_k^2(x)-\log\pi_k\Big],\qquad \Delta_k^2(x)=(x-\mu_k)^\top\Sigma^{-1}(x-\mu_k)
$$

- $\Delta_k^2(x)$: squared Mahalanobis distance from $x$ to the mean of class $k$

If $\Sigma=\sigma^2I$ the Mahalanobis distance is just Euclidean distance over $\sigma$, and LDA with equal priors **is** nearest mean. A general $\Sigma$ penalises a step along a direction in which the classes barely vary much more than a step along their long axis.

**Fisher's derivation — same answer, no Gaussians.** In 1936 Fisher asked a different question: which direction $v$ should we project onto so that the projected class means are far apart *relative to* the projected spread? Maximise the ratio of between-class to within-class scatter along $v$:

$$
J(v)=\frac{\big(v^\top(\mu_1-\mu_0)\big)^2}{v^\top S_W\,v}
$$

- $J(v)$: Fisher criterion: squared gap between projected means ÷ projected within-class scatter
- $v$: candidate projection direction (its length cancels)

Set the gradient to zero: $\nabla J=0 \Rightarrow (v^\top S_W v)\,(\mu_1-\mu_0)\,\big(v^\top(\mu_1-\mu_0)\big)=\big(v^\top(\mu_1-\mu_0)\big)^2 S_W v$. Both bracketed quantities are scalars, so $S_W v\propto \mu_1-\mu_0$, i.e. $v\propto S_W^{-1}(\mu_1-\mu_0)\propto w$. Fisher's best projection is the normal of the Gaussian LDA boundary, and the maximum value is $J^\ast=(\mu_1-\mu_0)^\top S_W^{-1}(\mu_1-\mu_0)$. With $K$ classes the same idea yields up to $K-1$ directions — that use of LDA as a supervised dimensionality reducer has its own article, **lda-for-dimensionality-reduction**.

**Hyperparameter 1: priors.** Priors appear only in $b$, through $\log(\pi_1/\pi_0)$. Changing them leaves $w$ alone and slides the boundary **parallel to itself** by $\log(\pi_1/\pi_0)/\lVert w\rVert$ along $w$, towards the rarer class. Set them to the class frequencies you expect at deployment, not necessarily those in the training sample.

$$
\hat\Sigma(\gamma)=(1-\gamma)\,\hat\Sigma+\gamma\,\frac{\operatorname{tr}\hat\Sigma}{d}\,I,\qquad 0\le\gamma\le1
$$

- $\gamma$: shrinkage intensity
- $\operatorname{tr}\hat\Sigma/d$: average variance, so the target has the same overall scale
- $I$: identity matrix

**Hyperparameter 2: shrinkage.** $\hat\Sigma$ has $d(d+1)/2$ free entries. When $d$ approaches the number of points its smallest eigenvalues are badly underestimated, $\hat\Sigma^{-1}$ blows those directions up, and $w$ chases noise; when $d\ge n-K$ the matrix is singular. Shrinking toward a scaled identity raises every eigenvalue toward the average, trading a little bias for a large drop in variance. At $\gamma=0$ we have plain LDA; at $\gamma=1$, $\Sigma\propto I$ and LDA reduces to nearest mean (plus priors). The Ledoit–Wolf formula picks $\gamma$ analytically; otherwise cross-validate it.

## Worked example: salmon or sea bass?

A fish-sorting plant measures two scaled features of each fish on a conveyor: $x_1$ = **lightness** and $x_2$ = **width**. Ten labelled fish: five salmon (class 0) and five sea bass (class 1). Within each species, wider fish tend to be lighter — the features are correlated — which is exactly where nearest mean goes wrong.

Training data. Highlighted: the two fish that nearest-mean misclassifies but LDA gets right.

| Fish | x₁ lightness | x₂ width | Species |
| --- | --- | --- | --- |
| 1 | 1 | 2 | salmon (0) |
| 2 | 3 | 2 | salmon (0) |
| 3 | 3 | 4 | salmon (0) |
| 4 | 3 | 6 | salmon (0) |
| 5 | 5 | 6 | salmon (0) |
| 6 | 4 | 1 | sea bass (1) |
| 7 | 6 | 3 | sea bass (1) |
| 8 | 6 | 5 | sea bass (1) |
| 9 | 6 | 6 | sea bass (1) |
| 10 | 8 | 5 | sea bass (1) |

**Fit LDA by hand, then classify a new fish at x = (5, 7)**

1. **Priors.** 5 of 10 fish in each class, so $\hat\pi_0=\hat\pi_1=0.5$ and $\log(\pi_1/\pi_0)=0$.
2. **Means.** Salmon: $\hat\mu_0=\big(\tfrac{1+3+3+3+5}{5},\tfrac{2+2+4+6+6}{5}\big)=(3,4)$. Sea bass: $\hat\mu_1=\big(\tfrac{4+6+6+6+8}{5},\tfrac{1+3+5+6+5}{5}\big)=(6,4)$. The means differ only in lightness: $\mu_1-\mu_0=(3,0)$.
3. **Deviations from own mean.** Salmon: $(-2,-2),(0,-2),(0,0),(0,2),(2,2)$. Sea bass: $(-2,-3),(0,-1),(0,1),(0,2),(2,1)$.
4. **Within-class scatter.** Sum of $x_1$ squares: $8+8=16$; cross products $x_1x_2$: $8+8=16$; $x_2$ squares: $16+16=32$. So $S_W=\begin{pmatrix}16&16\\16&32\end{pmatrix}$.
5. **Pooled covariance.** Divide by $n-K=10-2=8$: $\hat\Sigma=\begin{pmatrix}2&2\\2&4\end{pmatrix}$. The correlation is $2/\sqrt{2\cdot4}=0.707$ — strongly tilted clouds.
6. **Invert.** $\det\hat\Sigma=2\cdot4-2\cdot2=4$, so $\hat\Sigma^{-1}=\tfrac14\begin{pmatrix}4&-2\\-2&2\end{pmatrix}=\begin{pmatrix}1&-0.5\\-0.5&0.5\end{pmatrix}$.
7. **Weights.** $w=\hat\Sigma^{-1}(\mu_1-\mu_0)=(1\cdot3,\,-0.5\cdot3)=(3,\,-1.5)$. Not $(3,0)$: the covariance **tilts** the boundary away from vertical.
8. **Offset.** $b=-\tfrac12(\mu_0+\mu_1)^\top w=-\tfrac12\big(9\cdot3+8\cdot(-1.5)\big)=-\tfrac12(27-12)=-7.5$. Boundary: $3x_1-1.5x_2=7.5$, i.e. $x_2=2x_1-5$.
9. **Per-class view.** $a_0=\hat\Sigma^{-1}\mu_0=(1,0.5)$, $c_0=-\tfrac12\mu_0^\top a_0+\log0.5=-2.5-0.693=-3.193$; $a_1=\hat\Sigma^{-1}\mu_1=(4,-1)$, $c_1=-10-0.693=-10.693$. Subtracting gives back $w=a_1-a_0=(3,-1.5)$ and $b=c_1-c_0=-7.5$.
10. **New fish $x=(5,7)$.** $\delta_0=5\cdot1+7\cdot0.5-3.193=5.307$ and $\delta_1=5\cdot4-7\cdot1-10.693=2.307$. Score $w^\top x+b=15-10.5-7.5=-3.0$, so $P(\text{sea bass}\mid x)=\sigma(-3)=0.047$ → **salmon**, with 95.3% probability.
11. **Compare with nearest mean.** Squared Euclidean distances: $13$ to $\mu_0$ and $10$ to $\mu_1$ → nearest mean says **sea bass**. Squared Mahalanobis distances: $\Delta_0^2=2.5$, $\Delta_1^2=8.5$. The fish lies along the long diagonal axis of the salmon cloud, so in the cloud's own units it is much closer to the salmon; and indeed $-\tfrac12(8.5-2.5)=-3$ reproduces the score.

Every training fish scored by LDA ($w=(3,-1.5)$, $b=-7.5$) and by nearest mean. LDA separates all 10 (the closest fish sit at score ±1.5); nearest mean gets 8 of 10.

| Fish | x | Species | Score wᵀx + b | P(bass \| x) | Euclid² to μ₀ / μ₁ | Mahalanobis² to μ₀ / μ₁ | Nearest mean |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 1 | (1, 2) | salmon | −7.5 | 0.001 | 8 / 29 | 2.0 / 17.0 | salmon |
| 2 | (3, 2) | salmon | −1.5 | 0.182 | 4 / 13 | 2.0 / 5.0 | salmon |
| 3 | (3, 4) | salmon | −4.5 | 0.011 | 0 / 9 | 0.0 / 9.0 | salmon |
| 4 | (3, 6) | salmon | −7.5 | 0.001 | 4 / 13 | 2.0 / 17.0 | salmon |
| 5 | (5, 6) | salmon | −1.5 | 0.182 | 8 / 5 | 2.0 / 5.0 | **sea bass ✗** |
| 6 | (4, 1) | sea bass | 3.0 | 0.953 | 10 / 13 | 8.5 / 2.5 | **salmon ✗** |
| 7 | (6, 3) | sea bass | 6.0 | 0.998 | 10 / 1 | 12.5 / 0.5 | sea bass |
| 8 | (6, 5) | sea bass | 3.0 | 0.953 | 10 / 1 | 6.5 / 0.5 | sea bass |
| 9 | (6, 6) | sea bass | 1.5 | 0.818 | 13 / 4 | 5.0 / 2.0 | sea bass |
| 10 | (8, 5) | sea bass | 9.0 | 1.000 | 26 / 5 | 20.5 / 2.5 | sea bass |

**Fisher's ratio confirms it.** Projected onto the mean difference $(1,0)$ — what nearest mean effectively uses — the classes have $J=0.5625$. Projected onto $w\propto(3,-1.5)$, $J=1.125$, exactly the maximum $J^\ast=(3,0)\,S_W^{-1}(3,0)^\top=9/8$. Turning the projection by 26.6° doubles the separation-to-spread ratio.

> Figure: The worked example, live. Both classes share the same covariance ellipse (solid = 1, dashed = 2 Mahalanobis units); the white line is the LDA boundary x₂ = 2x₁ − 5 with normal w = (3, −1.5), the dashed rose line is nearest mean (x₁ = 4.5), and the strip below projects every fish onto w (Fisher J = 1.125, 0 on the wrong side) — switch to μ₁ − μ₀ to see J halve to 0.563 and two fish cross over. The query (5, 7) gets δ₀ = 5.31, δ₁ = 2.31, P(bass) = 0.047. Drag the prior to 0.80 and the boundary slides parallel (P(bass) rises to 0.166); push shrinkage γ to 1 and it rotates onto the nearest-mean line, which then calls the query a sea bass. Drag any fish to watch the means, the ellipse and w update.

> **Why the boundary does not pass between the clouds the way you might draw it:** The boundary passes through the midpoint of the means, $(4.5,4)$, but it is **not** perpendicular to the segment joining them. It is perpendicular to $w=\Sigma^{-1}(\mu_1-\mu_0)$: it is the set of points that are equally far from both means **in Mahalanobis distance**, and moving along it never changes the ratio of the two class likelihoods. In this example it runs along slope 2, steeper than the long axis of the ellipses.

## The algorithm, step by step

**LDA fit and predict**

1. **Priors.** $\pi_k\leftarrow n_k/n$, or the deployment frequencies if you know them.
2. **Means.** $\mu_k\leftarrow$ average of the class-$k$ rows — $O(nd)$.
3. **Pooled covariance.** Subtract each row's own class mean, form $S_W=R^\top R$ from the centred matrix $R$, divide by $n-K$ — $O(nd^2)$. Optionally shrink: $\Sigma\leftarrow(1-\gamma)\Sigma+\gamma(\operatorname{tr}\Sigma/d)I$.
4. **Solve, do not invert.** Compute $A=\Sigma^{-1}[\mu_1,\dots,\mu_K]$ with one Cholesky / linear solve — $O(d^3+Kd^2)$. (The SVD solver instead whitens the centred data and never forms $\Sigma$, which is more stable when $\Sigma$ is near-singular.)
5. **Offsets.** $c_k\leftarrow-\tfrac12\mu_k^\top a_k+\log\pi_k$.
6. **Predict.** Scores $\delta_k(x)=x^\top a_k+c_k$ — $O(Kd)$; label $=\arg\max_k\delta_k$; probabilities $=\operatorname{softmax}_k\,\delta_k(x)$.

Total training cost is $O(nd^2+d^3)$ with **no iterations** and nothing to converge: the same data always give the same model. Memory is $O(d^2)$ for the covariance. Prediction is as cheap as any linear model. For multi-class problems the boundaries between each pair of classes are hyperplanes, so the decision regions are convex polygons (polytopes in $d$ dimensions).

## Variants and tuning

The LDA family and its relatives.

| Variant | Covariance model | Boundary | When to use it |
| --- | --- | --- | --- |
| Nearest mean (nearest centroid) | $\Sigma=\sigma^2 I$ | linear, ⊥ to μ₁ − μ₀ | very high d, tiny n; features already decorrelated |
| Gaussian naive Bayes | diagonal, per class | quadratic (axis-aligned) | many weakly dependent features — see **naive-bayes** |
| **LDA** | one full Σ, shared | **linear** | moderate d, classes of similar shape |
| Shrinkage / regularised LDA | $(1-\gamma)\hat\Sigma+\gamma\bar\sigma^2I$ | linear | $d$ comparable to $n$; noisy or collinear features |
| QDA | one full $\Sigma_k$ per class | quadratic | plenty of data, classes with different spreads — see **quadratic-discriminant-analysis** |
| Regularised DA (Friedman) | blend of $\Sigma_k$ and pooled $\Sigma$ | between linear and quadratic | tune the blend by cross-validation |
| Fisher / reduced-rank LDA | shared Σ | project to ≤ K − 1 dims first | visualisation, preprocessing — see **lda-for-dimensionality-reduction** |

**Knobs that matter (scikit-learn names)**

- **priors:** changes only $b$: shifts boundaries parallel by $\log(\pi_1/\pi_0)/\lVert w\rVert$. In the worked example, priors $0.2/0.8$ raise $b$ by $\log 4=1.386$ to $-6.114$, and the query's P(bass) rises from 0.047 to 0.166.
- **shrinkage:** None, a float γ in [0, 1], or "auto" (Ledoit–Wolf). Requires `solver="lsqr"` or `"eigen"`. Rotates w from the LDA direction toward μ₁ − μ₀.
- **solver:** `"svd"` (default; no covariance inversion, handles d > n, no shrinkage), `"lsqr"` (fast, supports shrinkage), `"eigen"` (supports shrinkage and transform).
- **n_components:** only for transform(): number of discriminant directions, at most K − 1.

**Shrinkage in action.** Synthetic two-class data: $d$ features with correlation $0.6^{|i-j|}$ between features $i$ and $j$, the class means differing by 0.8 in the first five features, **60 training points**, 2,000 test points, averaged over 50 repeats. The Bayes-optimal accuracy is 0.739 for every $d$.

Test accuracy. Plain LDA collapses when d approaches n; Ledoit–Wolf shrinkage recovers most of the gap and beats both naive Bayes and logistic regression.

| d (features) | LDA, no shrinkage | LDA, shrinkage="auto" | LDA, γ = 0.5 | Gaussian NB | Logistic regression |
| --- | --- | --- | --- | --- | --- |
| 10 | 0.694 | 0.705 | **0.707** | 0.695 | 0.700 |
| 50 | 0.562 | **0.654** | 0.653 | 0.636 | 0.633 |
| 100 | 0.579 | **0.632** | 0.625 | 0.613 | 0.615 |

> **Singular covariance when d is close to n:** The pooled covariance has rank at most $n-K$. With 60 points and 50 features the estimate is technically invertible but its smallest eigenvalues are tiny, and $\Sigma^{-1}$ multiplies noise in those directions — accuracy fell to 0.562 in the table above, barely better than a coin. Always try shrinkage (or PCA first) when the number of features is more than a small fraction of the number of samples.

## In code

```python
import numpy as np

class LDA:
    def __init__(self, priors=None, shrinkage=0.0):
        self.priors, self.shrinkage = priors, shrinkage

    def fit(self, X, y):
        self.classes = np.unique(y)
        n, d = X.shape
        K = len(self.classes)
        self.means = np.array([X[y == k].mean(axis=0) for k in self.classes])
        # pooled within-class covariance, unbiased (divide by n - K)
        R = X - self.means[np.searchsorted(self.classes, y)]
        S = R.T @ R / (n - K)
        g = self.shrinkage                       # shrink toward a scaled identity
        S = (1 - g) * S + g * np.trace(S) / d * np.eye(d)
        if self.priors is not None:
            pri = np.asarray(self.priors, float)
        else:
            pri = np.bincount(np.searchsorted(self.classes, y)) / n
        # delta_k(x) = x . a_k + c_k   (linear in x)
        self.A = np.linalg.solve(S, self.means.T)              # columns a_k = S^-1 mu_k
        self.c = -0.5 * np.sum(self.means.T * self.A, axis=0) + np.log(pri)
        self.cov = S
        return self

    def decision(self, X):
        return X @ self.A + self.c                # (n, K) discriminant scores

    def predict_proba(self, X):
        s = self.decision(X)
        s -= s.max(axis=1, keepdims=True)        # softmax of the scores = posterior
        p = np.exp(s)
        return p / p.sum(axis=1, keepdims=True)

    def predict(self, X):
        return self.classes[self.decision(X).argmax(axis=1)]

X = np.array([[1, 2], [3, 2], [3, 4], [3, 6], [5, 6],      # salmon (0)
              [4, 1], [6, 3], [6, 5], [6, 6], [8, 5]], float)  # sea bass (1)
y = np.array([0] * 5 + [1] * 5)
m = LDA().fit(X, y)
print("means", m.means.tolist())
print("pooled cov", m.cov.tolist())
print("a_k columns", m.A.T.tolist(), "c_k", np.round(m.c, 3).tolist())
q = np.array([[5.0, 7.0]])
print("scores", np.round(m.decision(q), 3).tolist(),
      "P", np.round(m.predict_proba(q), 3).tolist(), "class", m.predict(q))
print("train acc", (m.predict(X) == y).mean())
print("shrinkage 1.0 ->", m.__class__(shrinkage=1.0).fit(X, y).predict(q))

# means [[3.0, 4.0], [6.0, 4.0]]
# pooled cov [[2.0, 2.0], [2.0, 4.0]]
# a_k columns [[1.0, 0.5], [4.0, -1.0]] c_k [-3.193, -10.693]
# scores [[5.307, 2.307]] P [[0.953, 0.047]] class [0]
# train acc 1.0
# shrinkage 1.0 -> [1]
```

From scratch in NumPy: multi-class LDA with priors and shrinkage. Prints the worked-example numbers.

```python
import numpy as np
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis

X = np.array([[1, 2], [3, 2], [3, 4], [3, 6], [5, 6],
              [4, 1], [6, 3], [6, 5], [6, 6], [8, 5]], float)
y = np.array([0] * 5 + [1] * 5)          # 0 = salmon, 1 = sea bass
q = np.array([[5.0, 7.0]])

lda = LinearDiscriminantAnalysis(store_covariance=True).fit(X, y)   # solver="svd"
print("w =", lda.coef_.round(3), " b =", lda.intercept_.round(3))
print("P(class | q) =", lda.predict_proba(q).round(3), "->", lda.predict(q))
print("train accuracy =", lda.score(X, y))
print("covariance_ (divides by n):", lda.covariance_.tolist())

# Same boundary, different probabilities with the lsqr solver
lsq = LinearDiscriminantAnalysis(solver="lsqr").fit(X, y)
print("lsqr w =", lsq.coef_.round(3), " b =", lsq.intercept_.round(3), " P =", lsq.predict_proba(q).round(3))

# Unequal priors shift the boundary; shrinkage pulls it toward nearest-mean
print("priors .2/.8:", LinearDiscriminantAnalysis(priors=[0.2, 0.8]).fit(X, y).predict_proba(q).round(3))
print("shrinkage=1:", LinearDiscriminantAnalysis(solver="lsqr", shrinkage=1.0).fit(X, y).predict(q))

# w = [[ 3.  -1.5]]  b = [-7.5]
# P(class | q) = [[0.953 0.047]] -> [0]
# train accuracy = 1.0
# covariance_ (divides by n): [[1.6, 1.6], [1.6, 3.2]]
# lsqr w = [[ 3.75  -1.875]]  b = [-9.375]  P = [[0.977 0.023]]
# priors .2/.8: [[0.834 0.166]]
# shrinkage=1: [1]
```

scikit-learn on the same ten fish: the default solver reproduces w, b and the posterior exactly.

> **Solvers agree on the boundary, not on the probabilities:** The `"lsqr"` and `"eigen"` solvers (and the `covariance_` attribute) use the maximum-likelihood covariance, which divides by n = 10 instead of n − K = 8. That scales Σ by 0.8, so w and b grow by 1.25 to (3.75, −1.875) and −9.375. With equal priors the boundary is identical (−b/w₁ = 2.5 in both), but P(salmon | q) is 0.977 instead of 0.953. If you report LDA probabilities, say which estimator produced them — and calibrate on held-out data if they matter.

> **LDA stays finite where logistic regression does not:** These ten fish are linearly separable. Unregularised logistic regression (scikit-learn with $C=10^6$) responds by inflating its weights to $(18.7,-9.34)$ with intercept $-46.5$, about six times LDA's, pushing every training probability toward 0 or 1 — and with no penalty at all the weights diverge. LDA's closed form never notices separability: its weights are fixed by the means and covariance.

**Strengths**

- Classes are roughly Gaussian with similar spreads, and features are correlated — the case LDA was built for.
- You have few samples per feature: with shrinkage, LDA is one of the strongest small-n classifiers (EEG/BCI, spectra, genomics).
- You want a fast, deterministic, closed-form baseline with calibrated-ish probabilities and no tuning loop.
- You also want a low-dimensional supervised projection for plotting — the same fit gives it.

**Limitations**

- Classes have clearly different covariances (one tight, one diffuse): the optimal boundary curves — use QDA or a kernel method.
- The data have heavy tails or outliers: means and covariances are not robust, and one bad point can rotate w.
- Features are binary, counts or strongly skewed: the Gaussian model is wrong; logistic regression or naive Bayes fit better.
- The boundary is genuinely non-linear (rings, XOR): no linear method helps without feature engineering.

- **Brain–computer interfaces:** Shrinkage LDA is the standard classifier for EEG event-related potentials (e.g. P300 spellers): hundreds of correlated features, a few hundred trials.
- **Spectroscopy and chemometrics:** Classifying samples (olive-oil origin, bacteria, tablets) from near-infrared or Raman spectra, usually after PCA (PCA-LDA).
- **Credit scoring and bankruptcy prediction:** Altman's Z-score (1968) is a linear discriminant on five financial ratios — one of the earliest production uses of the method.
- **Face recognition:** Fisherfaces: PCA then LDA on pixel vectors, so the projection separates identities rather than lighting changes.

> **Scaling does not matter — but outliers do:** Unlike k-NN or SVMs, LDA is invariant to rescaling (or any invertible linear transform) of the features: $\Sigma^{-1}$ undoes it, and predictions are unchanged. What it is **not** robust to is outliers, because both $\hat\mu_k$ and $\hat\Sigma$ are plain averages. Inspect the data, or use a robust covariance estimate, before trusting $w$.

**Question:** Two classes share $\Sigma=\begin{pmatrix}2&2\\2&4\end{pmatrix}$, with $\mu_0=(3,4)$ and $\mu_1=(6,4)$. What is the LDA weight vector $w$?

- A. (3, 0)
- B. (3, −1.5)
- C. (1.5, 0.75)
- D. (6, −3)

**Answer:** B. (3, −1.5) — $\Sigma^{-1}=\begin{pmatrix}1&-0.5\\-0.5&0.5\end{pmatrix}$, so $w=\Sigma^{-1}(3,0)=(3,-1.5)$. The answer (3, 0) is the mean difference — what you would get if you ignored the covariance.

**Question:** In the worked example, the priors change from 0.5 / 0.5 to π(salmon) = 0.2, π(bass) = 0.8. What happens to the boundary?

- A. It rotates toward the salmon cloud
- B. It moves parallel to itself toward the salmon side; b rises by log 4 ≈ 1.386
- C. It does not move; priors only change the probabilities
- D. It becomes curved

**Answer:** B. It moves parallel to itself toward the salmon side; b rises by log 4 ≈ 1.386 — Priors enter only $b$ through $\log(\pi_1/\pi_0)=\log4=1.386$, so $b$ goes from $-7.5$ to $-6.114$ and $w$ is unchanged — a parallel shift of $1.386/\lVert w\rVert=0.413$ units, enlarging the sea-bass region.

**Question:** With shrinkage $\gamma=1$ (and equal priors), LDA on the fish data has boundary…

- A. $x_2=2x_1-5$, unchanged
- B. $x_1=4.5$, the nearest-mean boundary
- C. $x_2=4$
- D. undefined, because Σ is singular

**Answer:** B. $x_1=4.5$, the nearest-mean boundary — At $\gamma=1$, $\Sigma\propto I$, so $w\propto\mu_1-\mu_0=(3,0)$ and the boundary is the perpendicular bisector of the means: $x_1=4.5$. That rule misclassifies fish 5 and 6 and calls the query (5, 7) a sea bass.

**Question:** Why does LDA produce a linear boundary while QDA produces a quadratic one?

- A. LDA uses fewer training points
- B. With a shared $\Sigma$ the $x^\top\Sigma^{-1}x$ term is identical for every class and cancels
- C. LDA ignores the class priors
- D. LDA fits a straight line by gradient descent

**Answer:** B. With a shared $\Sigma$ the $x^\top\Sigma^{-1}x$ term is identical for every class and cancels — The log-density contains $-\tfrac12x^\top\Sigma_k^{-1}x$. If every class has the same $\Sigma$, that term (and $\log|\Sigma|$) is the same for all $k$ and drops out of the comparison, leaving a linear function of $x$. With class-specific $\Sigma_k$ it remains, and the boundary is quadratic.

**Key points**

- LDA models each class as $\mathcal N(\mu_k,\Sigma)$ with **one shared covariance**; Bayes' rule gives linear scores $\delta_k(x)=x^\top\Sigma^{-1}\mu_k-\tfrac12\mu_k^\top\Sigma^{-1}\mu_k+\log\pi_k$.
- Two classes: $w=\Sigma^{-1}(\mu_1-\mu_0)$, $b=-\tfrac12(\mu_0+\mu_1)^\top w+\log(\pi_1/\pi_0)$, and $P(y=1\mid x)=\sigma(w^\top x+b)$ — the same form as logistic regression, fitted generatively.
- It is nearest-mean in Mahalanobis distance: the covariance tilts the boundary to respect correlated features (10/10 vs 8/10 on the fish).
- Fisher's criterion — maximise between-class over within-class scatter — gives the same direction without assuming Gaussians.
- Priors shift the boundary in parallel; shrinkage rotates it toward nearest mean and is essential when d is comparable to n.
- Closed form, O(nd² + d³), scale-invariant, but sensitive to outliers and to unequal class covariances (then use QDA).

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
