Definition
Linear discriminant analysis (LDA) is a generative classifier. It assumes the features of each class follow a multivariate Gaussian with its own mean but one covariance matrix shared by all classes, estimates the means, the shared covariance and the class priors in closed form, and classifies a new point with Bayes' rule. Because the covariance is shared, the quadratic terms cancel and every decision boundary is a straight line (a hyperplane).
At a glance
- Task
- Classification (binary or multi-class); the same fit also gives a supervised projection
- Input → output
- → posterior for every class, and the class with the largest one
- Learns
- class priors , class means , one pooled covariance — giving a weight vector and an offset per class
- Training
- one pass for means and covariance, then one solve: ; no iterations, no learning rate
- Prediction
- — one dot product per class
- Hyperparameters
- class priors, covariance shrinkage γ, solver (svd / lsqr / eigen); nothing else
- Assumes
- Gaussian classes with equal covariance; roughly linear separation; no extreme outliers
The problem
The simplest classifier anyone invents is nearest mean: average the training points of each class, then give a new point the label of the closest average. It fails in a specific, common way: it measures distance with a ruler that ignores the shape of the classes. If two features are correlated within each class — long fish are also wide fish — the clouds are tilted ellipses, and a point can be closer to the other class's centre in plain distance while sitting squarely on the long axis of its own class.
Logistic regression (see logistic-regression) fixes the boundary by fitting it directly, but it needs iterative optimisation, its coefficients run off to infinity on perfectly separable data, and with few examples per feature it is noisy. LDA takes the other route: write down a probability model of how each class generates its features, fit that model in closed form, and let Bayes' rule produce the boundary. The covariance matrix it estimates is exactly the ruler that nearest-mean was missing.
The idea
Picture each class as a Gaussian "cloud" — an ellipse of the same size, shape and orientation, just centred in a different place. To classify a point, ask which cloud makes it most likely, adjusted for how common each class is. Because the ellipses are identical copies, the place where two clouds are equally likely is a straight line. Measuring with the ellipse rather than with a circle is the Mahalanobis distance: LDA is nearest-mean classification after you "un-stretch" the space so each cloud becomes a circle.
Vocabulary
- Class prior πₖ
- probability of class before looking at ; estimated as the class's share of the training set
- Class-conditional density
- — how the features of class are distributed; here
- Pooled covariance Σ
- one covariance matrix estimated from the deviations of every point from its own class mean; describes the common shape of all clouds
- Discriminant function δₖ(x)
- a score per class whose argmax is the Bayes decision; for LDA it is linear in
- Mahalanobis distance
- — distance measured in units of the cloud's own spread along each direction
- Fisher direction w
- : the line onto which the two classes, projected, are best separated relative to their spread
- Shrinkage γ
- blending the estimated covariance with a multiple of the identity, to keep it well-conditioned when data are scarce
- Generative vs discriminative
- LDA models and and derives ; logistic regression models directly
The math
The model. Each class has a prior and a Gaussian density with a class-specific mean and a shared covariance:
- prior probability of class , with
- mean vector of class ( entries)
- covariance, the same for every class
- determinant of Σ (also shared, so it will cancel)
- number of features
Bayes' rule turns the model into a posterior. The denominator is the same for every class, so the most probable class maximises the numerator — or its logarithm:
- posterior probability of class after seeing
- predicted class
Why the boundary is linear. Take the log of the Gaussian and expand the quadratic form: . The constant and the term are identical for every class because is shared, so they cannot change the argmax and we drop them. What is left is linear in :
- linear discriminant score of class ; predict
- weight vector of class
- offset of class : a penalty for a far-out mean plus the log prior
With two classes the boundary is a single hyperplane :
- normal vector of the boundary (the Fisher direction)
- offset: puts the boundary at the midpoint of the means, shifted by the prior log-odds
- logistic function
Fitting. Maximum likelihood gives the obvious estimates. For the covariance we pool the deviations of every point from its own class mean and divide by (the unbiased version, which is what most texts and scikit-learn's default solver use):
- number of training points in class ;
- number of classes
- within-class scatter matrix: the double sum before dividing
The Mahalanobis view. Adding back the class-independent shows that LDA is a nearest-mean rule in a re-measured space:
- squared Mahalanobis distance from to the mean of class
If the Mahalanobis distance is just Euclidean distance over , and LDA with equal priors is nearest mean. A general penalises a step along a direction in which the classes barely vary much more than a step along their long axis.
Fisher's derivation — same answer, no Gaussians. In 1936 Fisher asked a different question: which direction should we project onto so that the projected class means are far apart relative to the projected spread? Maximise the ratio of between-class to within-class scatter along :
- Fisher criterion: squared gap between projected means ÷ projected within-class scatter
- candidate projection direction (its length cancels)
Set the gradient to zero: . Both bracketed quantities are scalars, so , i.e. . Fisher's best projection is the normal of the Gaussian LDA boundary, and the maximum value is . With classes the same idea yields up to directions — that use of LDA as a supervised dimensionality reducer has its own article, lda-for-dimensionality-reduction.
Hyperparameter 1: priors. Priors appear only in , through . Changing them leaves alone and slides the boundary parallel to itself by along , towards the rarer class. Set them to the class frequencies you expect at deployment, not necessarily those in the training sample.
- shrinkage intensity
- average variance, so the target has the same overall scale
- identity matrix
Hyperparameter 2: shrinkage. has free entries. When approaches the number of points its smallest eigenvalues are badly underestimated, blows those directions up, and chases noise; when the matrix is singular. Shrinking toward a scaled identity raises every eigenvalue toward the average, trading a little bias for a large drop in variance. At we have plain LDA; at , and LDA reduces to nearest mean (plus priors). The Ledoit–Wolf formula picks analytically; otherwise cross-validate it.
Worked example: salmon or sea bass?
A fish-sorting plant measures two scaled features of each fish on a conveyor: = lightness and = width. Ten labelled fish: five salmon (class 0) and five sea bass (class 1). Within each species, wider fish tend to be lighter — the features are correlated — which is exactly where nearest mean goes wrong.
| Fish | x₁ lightness | x₂ width | Species |
|---|---|---|---|
| 1 | 1 | 2 | salmon (0) |
| 2 | 3 | 2 | salmon (0) |
| 3 | 3 | 4 | salmon (0) |
| 4 | 3 | 6 | salmon (0) |
| 5 | 5 | 6 | salmon (0) |
| 6 | 4 | 1 | sea bass (1) |
| 7 | 6 | 3 | sea bass (1) |
| 8 | 6 | 5 | sea bass (1) |
| 9 | 6 | 6 | sea bass (1) |
| 10 | 8 | 5 | sea bass (1) |
Fit LDA by hand, then classify a new fish at x = (5, 7)
- 1Priors. 5 of 10 fish in each class, so and .
- 2Means. Salmon: . Sea bass: . The means differ only in lightness: .
- 3Deviations from own mean. Salmon: . Sea bass: .
- 4Within-class scatter. Sum of squares: ; cross products : ; squares: . So .
- 5Pooled covariance. Divide by : . The correlation is — strongly tilted clouds.
- 6Invert. , so .
- 7Weights. . Not : the covariance tilts the boundary away from vertical.
- 8Offset. . Boundary: , i.e. .
- 9Per-class view. , ; , . Subtracting gives back and .
- 10New fish . and . Score , so → salmon, with 95.3% probability.
- 11Compare with nearest mean. Squared Euclidean distances: to and to → nearest mean says sea bass. Squared Mahalanobis distances: , . The fish lies along the long diagonal axis of the salmon cloud, so in the cloud's own units it is much closer to the salmon; and indeed reproduces the score.
| Fish | x | Species | Score wᵀx + b | P(bass | x) | Euclid² to μ₀ / μ₁ | Mahalanobis² to μ₀ / μ₁ | Nearest mean |
|---|---|---|---|---|---|---|---|
| 1 | (1, 2) | salmon | −7.5 | 0.001 | 8 / 29 | 2.0 / 17.0 | salmon |
| 2 | (3, 2) | salmon | −1.5 | 0.182 | 4 / 13 | 2.0 / 5.0 | salmon |
| 3 | (3, 4) | salmon | −4.5 | 0.011 | 0 / 9 | 0.0 / 9.0 | salmon |
| 4 | (3, 6) | salmon | −7.5 | 0.001 | 4 / 13 | 2.0 / 17.0 | salmon |
| 5 | (5, 6) | salmon | −1.5 | 0.182 | 8 / 5 | 2.0 / 5.0 | sea bass ✗ |
| 6 | (4, 1) | sea bass | 3.0 | 0.953 | 10 / 13 | 8.5 / 2.5 | salmon ✗ |
| 7 | (6, 3) | sea bass | 6.0 | 0.998 | 10 / 1 | 12.5 / 0.5 | sea bass |
| 8 | (6, 5) | sea bass | 3.0 | 0.953 | 10 / 1 | 6.5 / 0.5 | sea bass |
| 9 | (6, 6) | sea bass | 1.5 | 0.818 | 13 / 4 | 5.0 / 2.0 | sea bass |
| 10 | (8, 5) | sea bass | 9.0 | 1.000 | 26 / 5 | 20.5 / 2.5 | sea bass |
Fisher's ratio confirms it. Projected onto the mean difference — what nearest mean effectively uses — the classes have . Projected onto , , exactly the maximum . Turning the projection by 26.6° doubles the separation-to-spread ratio.
The algorithm, step by step
LDA fit and predict
- 1Priors. , or the deployment frequencies if you know them.
- 2Means. average of the class- rows — .
- 3Pooled covariance. Subtract each row's own class mean, form from the centred matrix , divide by — . Optionally shrink: .
- 4Solve, do not invert. Compute with one Cholesky / linear solve — . (The SVD solver instead whitens the centred data and never forms , which is more stable when is near-singular.)
- 5Offsets. .
- 6Predict. Scores — ; label ; probabilities .
Total training cost is with no iterations and nothing to converge: the same data always give the same model. Memory is for the covariance. Prediction is as cheap as any linear model. For multi-class problems the boundaries between each pair of classes are hyperplanes, so the decision regions are convex polygons (polytopes in dimensions).
Variants and tuning
| Variant | Covariance model | Boundary | When to use it |
|---|---|---|---|
| Nearest mean (nearest centroid) | linear, ⊥ to μ₁ − μ₀ | very high d, tiny n; features already decorrelated | |
| Gaussian naive Bayes | diagonal, per class | quadratic (axis-aligned) | many weakly dependent features — see naive-bayes |
| LDA | one full Σ, shared | linear | moderate d, classes of similar shape |
| Shrinkage / regularised LDA | linear | comparable to ; noisy or collinear features | |
| QDA | one full per class | quadratic | plenty of data, classes with different spreads — see quadratic-discriminant-analysis |
| Regularised DA (Friedman) | blend of and pooled | between linear and quadratic | tune the blend by cross-validation |
| Fisher / reduced-rank LDA | shared Σ | project to ≤ K − 1 dims first | visualisation, preprocessing — see lda-for-dimensionality-reduction |
Knobs that matter (scikit-learn names)
- priors
- changes only : shifts boundaries parallel by . In the worked example, priors raise by to , and the query's P(bass) rises from 0.047 to 0.166.
- shrinkage
- None, a float γ in [0, 1], or "auto" (Ledoit–Wolf). Requires
solver="lsqr"or"eigen". Rotates w from the LDA direction toward μ₁ − μ₀. - solver
"svd"(default; no covariance inversion, handles d > n, no shrinkage),"lsqr"(fast, supports shrinkage),"eigen"(supports shrinkage and transform).- n_components
- only for transform(): number of discriminant directions, at most K − 1.
Shrinkage in action. Synthetic two-class data: features with correlation between features and , the class means differing by 0.8 in the first five features, 60 training points, 2,000 test points, averaged over 50 repeats. The Bayes-optimal accuracy is 0.739 for every .
| d (features) | LDA, no shrinkage | LDA, shrinkage="auto" | LDA, γ = 0.5 | Gaussian NB | Logistic regression |
|---|---|---|---|---|---|
| 10 | 0.694 | 0.705 | 0.707 | 0.695 | 0.700 |
| 50 | 0.562 | 0.654 | 0.653 | 0.636 | 0.633 |
| 100 | 0.579 | 0.632 | 0.625 | 0.613 | 0.615 |
In code
1import numpy as np23class LDA:4 def __init__(self, priors=None, shrinkage=0.0):5 self.priors, self.shrinkage = priors, shrinkage67 def fit(self, X, y):8 self.classes = np.unique(y)9 n, d = X.shape10 K = len(self.classes)11 self.means = np.array([X[y == k].mean(axis=0) for k in self.classes])12 # pooled within-class covariance, unbiased (divide by n - K)13 R = X - self.means[np.searchsorted(self.classes, y)]14 S = R.T @ R / (n - K)15 g = self.shrinkage # shrink toward a scaled identity16 S = (1 - g) * S + g * np.trace(S) / d * np.eye(d)17 if self.priors is not None:18 pri = np.asarray(self.priors, float)19 else:20 pri = np.bincount(np.searchsorted(self.classes, y)) / n21 # delta_k(x) = x . a_k + c_k (linear in x)22 self.A = np.linalg.solve(S, self.means.T) # columns a_k = S^-1 mu_k23 self.c = -0.5 * np.sum(self.means.T * self.A, axis=0) + np.log(pri)24 self.cov = S25 return self2627 def decision(self, X):28 return X @ self.A + self.c # (n, K) discriminant scores2930 def predict_proba(self, X):31 s = self.decision(X)32 s -= s.max(axis=1, keepdims=True) # softmax of the scores = posterior33 p = np.exp(s)34 return p / p.sum(axis=1, keepdims=True)3536 def predict(self, X):37 return self.classes[self.decision(X).argmax(axis=1)]3839X = np.array([[1, 2], [3, 2], [3, 4], [3, 6], [5, 6], # salmon (0)40 [4, 1], [6, 3], [6, 5], [6, 6], [8, 5]], float) # sea bass (1)41y = np.array([0] * 5 + [1] * 5)42m = LDA().fit(X, y)43print("means", m.means.tolist())44print("pooled cov", m.cov.tolist())45print("a_k columns", m.A.T.tolist(), "c_k", np.round(m.c, 3).tolist())46q = np.array([[5.0, 7.0]])47print("scores", np.round(m.decision(q), 3).tolist(),48 "P", np.round(m.predict_proba(q), 3).tolist(), "class", m.predict(q))49print("train acc", (m.predict(X) == y).mean())50print("shrinkage 1.0 ->", m.__class__(shrinkage=1.0).fit(X, y).predict(q))5152# means [[3.0, 4.0], [6.0, 4.0]]53# pooled cov [[2.0, 2.0], [2.0, 4.0]]54# a_k columns [[1.0, 0.5], [4.0, -1.0]] c_k [-3.193, -10.693]55# scores [[5.307, 2.307]] P [[0.953, 0.047]] class [0]56# train acc 1.057# shrinkage 1.0 -> [1]1import numpy as np2from sklearn.discriminant_analysis import LinearDiscriminantAnalysis34X = np.array([[1, 2], [3, 2], [3, 4], [3, 6], [5, 6],5 [4, 1], [6, 3], [6, 5], [6, 6], [8, 5]], float)6y = np.array([0] * 5 + [1] * 5) # 0 = salmon, 1 = sea bass7q = np.array([[5.0, 7.0]])89lda = LinearDiscriminantAnalysis(store_covariance=True).fit(X, y) # solver="svd"10print("w =", lda.coef_.round(3), " b =", lda.intercept_.round(3))11print("P(class | q) =", lda.predict_proba(q).round(3), "->", lda.predict(q))12print("train accuracy =", lda.score(X, y))13print("covariance_ (divides by n):", lda.covariance_.tolist())1415# Same boundary, different probabilities with the lsqr solver16lsq = LinearDiscriminantAnalysis(solver="lsqr").fit(X, y)17print("lsqr w =", lsq.coef_.round(3), " b =", lsq.intercept_.round(3), " P =", lsq.predict_proba(q).round(3))1819# Unequal priors shift the boundary; shrinkage pulls it toward nearest-mean20print("priors .2/.8:", LinearDiscriminantAnalysis(priors=[0.2, 0.8]).fit(X, y).predict_proba(q).round(3))21print("shrinkage=1:", LinearDiscriminantAnalysis(solver="lsqr", shrinkage=1.0).fit(X, y).predict(q))2223# w = [[ 3. -1.5]] b = [-7.5]24# P(class | q) = [[0.953 0.047]] -> [0]25# train accuracy = 1.026# covariance_ (divides by n): [[1.6, 1.6], [1.6, 3.2]]27# lsqr w = [[ 3.75 -1.875]] b = [-9.375] P = [[0.977 0.023]]28# priors .2/.8: [[0.834 0.166]]29# shrinkage=1: [1]Use when
- +Classes are roughly Gaussian with similar spreads, and features are correlated — the case LDA was built for.
- +You have few samples per feature: with shrinkage, LDA is one of the strongest small-n classifiers (EEG/BCI, spectra, genomics).
- +You want a fast, deterministic, closed-form baseline with calibrated-ish probabilities and no tuning loop.
- +You also want a low-dimensional supervised projection for plotting — the same fit gives it.
Avoid when
- −Classes have clearly different covariances (one tight, one diffuse): the optimal boundary curves — use QDA or a kernel method.
- −The data have heavy tails or outliers: means and covariances are not robust, and one bad point can rotate w.
- −Features are binary, counts or strongly skewed: the Gaussian model is wrong; logistic regression or naive Bayes fit better.
- −The boundary is genuinely non-linear (rings, XOR): no linear method helps without feature engineering.
Where it's used
Brain–computer interfaces
Shrinkage LDA is the standard classifier for EEG event-related potentials (e.g. P300 spellers): hundreds of correlated features, a few hundred trials.
Spectroscopy and chemometrics
Classifying samples (olive-oil origin, bacteria, tablets) from near-infrared or Raman spectra, usually after PCA (PCA-LDA).
Credit scoring and bankruptcy prediction
Altman's Z-score (1968) is a linear discriminant on five financial ratios — one of the earliest production uses of the method.
Face recognition
Fisherfaces: PCA then LDA on pixel vectors, so the projection separates identities rather than lighting changes.
Check yourself
Two classes share , with and . What is the LDA weight vector ?
Check yourself
In the worked example, the priors change from 0.5 / 0.5 to π(salmon) = 0.2, π(bass) = 0.8. What happens to the boundary?
Check yourself
With shrinkage (and equal priors), LDA on the fish data has boundary…
Check yourself
Why does LDA produce a linear boundary while QDA produces a quadratic one?
Remember
- ✓LDA models each class as with one shared covariance; Bayes' rule gives linear scores .
- ✓Two classes: , , and — the same form as logistic regression, fitted generatively.
- ✓It is nearest-mean in Mahalanobis distance: the covariance tilts the boundary to respect correlated features (10/10 vs 8/10 on the fish).
- ✓Fisher's criterion — maximise between-class over within-class scatter — gives the same direction without assuming Gaussians.
- ✓Priors shift the boundary in parallel; shrinkage rotates it toward nearest mean and is essential when d is comparable to n.
- ✓Closed form, O(nd² + d³), scale-invariant, but sensitive to outliers and to unequal class covariances (then use QDA).