Quadratic discriminant analysis

Give each class its own Gaussian covariance; the Bayes decision boundary becomes quadratic.

16 min read1424 wordsadvanced7 viewsupdated 2026-10-07

Definition

Quadratic discriminant analysis (QDA) is a generative classifier that models each class kk as a multivariate Gaussian N(μk,Σk)\mathcal N(\mu_k,\Sigma_k) with its own mean and its own covariance matrix, then classifies a point by Bayes' rule. Because the covariances differ, the log-posterior is quadratic in xx, and so are the decision boundaries: ellipses, parabolas or hyperbolas instead of straight lines.

At a glance

Task
Classification (binary or multiclass), with calibrated-ish class probabilities
Input → output
x∈Rdx\in\mathbb R^d → P(y=k∣x)P(y=k\mid x) for every class, and the arg-max class
Learns
per class: a prior πk\pi_k, a mean μk\mu_k (dd numbers) and a covariance Σk\Sigma_k (d(d+1)/2d(d+1)/2 numbers)
Training
closed form, no iterations: O(nd2+Kd3)O(n d^2 + K d^3) (scatter matrices plus one inverse / determinant per class)
Prediction
O(Kd2)O(K d^2) per point: one Mahalanobis distance per class
Hyperparameters
reg_param (shrink each covariance toward the identity), class priors
Assumes
within each class the features are jointly Gaussian; classes may differ in spread, shape and orientation

The problem

Linear classifiers — logistic regression, the perceptron, and its closest relative linear discriminant analysis — can only draw a straight line (a hyperplane) between classes. That is the right shape when the classes differ only in where they sit. It is the wrong shape when they differ in how spread out they are.

Think of banknotes. Genuine notes come off one press, so their ink density and paper thickness cluster tightly. Forgeries come from many workshops and scatter widely — some too thick, some too thin, some too dark, some too light — on every side of the genuine cluster. No line separates "tight blob" from "wide cloud around it". You need a closed curve around the genuine notes. LDA cannot produce one, because it assumes every class shares a single covariance matrix; QDA drops exactly that assumption.

The idea

Fit one Gaussian "bell" per class — its centre, its spread along each direction, and its tilt. To classify a new point, ask each bell how likely am I to have produced this point?, weight the answer by how common the class is, and pick the largest. A tight class gives a very high density near its centre but drops off fast; a wide class gives a low, flat density everywhere. So a point near the tight centre goes to the tight class, and a point even moderately far from it goes to the wide one — the boundary wraps around the tight class.

Vocabulary

Generative classifier
models how each class generates data, p(x∣y=k)p(x\mid y=k), and gets P(y∣x)P(y\mid x) by Bayes' rule (contrast: logistic regression models P(y∣x)P(y\mid x) directly)
Prior
πk=P(y=k)\pi_k=P(y=k), the class frequency before seeing xx; estimated as nk/nn_k/n
Class-conditional density
p(x∣y=k)p(x\mid y=k); in QDA a Gaussian N(μk,Σk)\mathcal N(\mu_k,\Sigma_k)
Covariance matrix
Σk\Sigma_k: variances of each feature on the diagonal, covariances off it. It sets the size, shape and tilt of the class's ellipse
Mahalanobis distance
Δk(x)=(x−μk)⊤Σk−1(x−μk)\Delta_k(x)=\sqrt{(x-\mu_k)^\top\Sigma_k^{-1}(x-\mu_k)}: distance from μk\mu_k measured in that class's standard deviations, along its own axes
Discriminant function
δk(x)\delta_k(x): a score equal to log⁡πkp(x∣k)\log \pi_k p(x\mid k) up to a constant shared by all classes; predict arg⁡max⁡kδk(x)\arg\max_k \delta_k(x)
Conic section
the curve where a quadratic in two variables is zero: ellipse, parabola or hyperbola (a line is a degenerate case)
Regularized QDA
shrinking each Σk\Sigma_k toward a simpler matrix (the identity, or the pooled LDA covariance) to fight overfitting

The math

The generative model. Each label is drawn from the priors, then the features are drawn from that class's Gaussian:

y∼Categorical(π1,…,πK),x∣y=k  ∼  N(μk,Σk),p(x∣k)=1(2π)d/2∣Σk∣1/2exp⁡ ⁣(−12(x−μk)⊤Σk−1(x−μk))y\sim\text{Categorical}(\pi_1,\dots,\pi_K),\qquad x\mid y=k\;\sim\;\mathcal N(\mu_k,\Sigma_k),\qquad p(x\mid k)=\frac{1}{(2\pi)^{d/2}|\Sigma_k|^{1/2}}\exp\!\Big(-\tfrac12(x-\mu_k)^\top\Sigma_k^{-1}(x-\mu_k)\Big)
KK
number of classes
dd
number of features
πk\pi_k
prior probability of class kk, ∑kπk=1\sum_k\pi_k=1
μk\mu_k
mean vector of class kk (d×1d\times1)
Σk\Sigma_k
covariance matrix of class kk (d×dd\times d, symmetric positive definite)
∣Σk∣|\Sigma_k|
determinant of Σk\Sigma_k: the squared "volume" of the class's ellipse

From Bayes' rule to a score. The posterior is P(y=k∣x)=πkp(x∣k)/∑jπjp(x∣j)P(y=k\mid x)=\pi_k p(x\mid k)/\sum_j \pi_j p(x\mid j). The denominator is the same for every class, and the logarithm is increasing, so the arg-max class is the one maximizing log⁡πk+log⁡p(x∣k)\log\pi_k+\log p(x\mid k). Take the log of the Gaussian and drop −d2log⁡2π-\tfrac d2\log 2\pi, which is the same for every class:

δk(x)=−12log⁡∣Σk∣  −  12 (x−μk)⊤Σk−1(x−μk)  +  log⁡πk\delta_k(x)=-\tfrac12\log|\Sigma_k|\;-\;\tfrac12\,(x-\mu_k)^\top\Sigma_k^{-1}(x-\mu_k)\;+\;\log\pi_k
δk(x)\delta_k(x)
quadratic discriminant score of class k at x; predict the class with the largest score
−12log⁡∣Σk∣-\tfrac12\log|\Sigma_k|
volume penalty: a wide class spreads its probability thinly, so it pays for its size
−12Δk2(x)-\tfrac12\Delta_k^2(x)
fit term: half the squared Mahalanobis distance to the class mean
log⁡πk\log\pi_k
prior term: common classes get a head start

Those three terms are the whole algorithm. The volume penalty is what lets a tight class win near its centre even against a nearby wide class, and it is exactly the term that LDA throws away (with a shared Σ\Sigma, log⁡∣Σ∣\log|\Sigma| is the same for everyone). Posteriors come back from the scores with a softmax, P(y=k∣x)=eδk(x)/∑jeδj(x)P(y=k\mid x)=e^{\delta_k(x)}/\sum_j e^{\delta_j(x)}; for two classes that is σ(δA−δB)\sigma(\delta_A-\delta_B).

Why the boundary is quadratic. Expand δk\delta_k: −12x⊤Σk−1x+x⊤Σk−1μk+ck-\tfrac12 x^\top\Sigma_k^{-1}x+x^\top\Sigma_k^{-1}\mu_k+c_k. The boundary between classes jj and kk is where the scores tie:

δj(x)−δk(x)=x⊤(−12(Σj−1−Σk−1))⏟Q x  +  (Σj−1μj−Σk−1μk) ⁣⊤⏟w⊤x  +  c  =  0\delta_j(x)-\delta_k(x)=x^\top\underbrace{\Big(-\tfrac12\big(\Sigma_j^{-1}-\Sigma_k^{-1}\big)\Big)}_{Q}\,x\;+\;\underbrace{\big(\Sigma_j^{-1}\mu_j-\Sigma_k^{-1}\mu_k\big)^{\!\top}}_{w^\top}x\;+\;c\;=\;0
QQ
d×dd\times d matrix of quadratic coefficients; zero exactly when Σj=Σk\Sigma_j=\Sigma_k
ww
linear coefficients
cc
constant: −12(μj⊤Σj−1μj−μk⊤Σk−1μk)−12log⁡∣Σj∣∣Σk∣+log⁡πjπk-\tfrac12(\mu_j^\top\Sigma_j^{-1}\mu_j-\mu_k^\top\Sigma_k^{-1}\mu_k)-\tfrac12\log\tfrac{|\Sigma_j|}{|\Sigma_k|}+\log\tfrac{\pi_j}{\pi_k}

That is a quadric surface. In 2-D, the sign of det⁡Q\det Q tells you its shape: det⁡Q>0\det Q>0 gives an ellipse (one class's region is a closed island), det⁡Q<0\det Q<0 a hyperbola (the regions interleave in an X shape), det⁡Q≈0\det Q\approx 0 a parabola. If all classes share one covariance, Q=0Q=0 and the boundary collapses to the hyperplane of LDA — the full derivation of that special case is in the Linear discriminant analysis article. If instead each Σk\Sigma_k is forced to be diagonal, QDA becomes Gaussian Naive Bayes.

Fitting = counting and averaging. Maximizing the joint log-likelihood ∑ilog⁡πyip(xi∣yi)\sum_i \log \pi_{y_i}p(x_i\mid y_i) separates into one independent problem per class, and each has the familiar Gaussian closed form:

π^k=nkn,μ^k=1nk∑i: yi=kxi,Σ^k=1nk−1∑i: yi=k(xi−μ^k)(xi−μ^k)⊤\hat\pi_k=\frac{n_k}{n},\qquad \hat\mu_k=\frac1{n_k}\sum_{i:\,y_i=k}x_i,\qquad \hat\Sigma_k=\frac1{n_k-1}\sum_{i:\,y_i=k}(x_i-\hat\mu_k)(x_i-\hat\mu_k)^\top
nkn_k
number of training points in class kk; n=∑knkn=\sum_k n_k
nk−1n_k-1
unbiased denominator (scikit-learn's choice); the exact MLE uses nkn_k

There is no gradient descent and no learning rate: set ∂/∂μk=0\partial/\partial\mu_k=0 and ∂/∂Σk−1=0\partial/\partial\Sigma_k^{-1}=0 in the Gaussian log-likelihood and these averages fall out. The catch is the parameter count: each Σk\Sigma_k has d(d+1)/2d(d+1)/2 free entries, and Σ^k\hat\Sigma_k is singular (not invertible) whenever nk≤dn_k\le d.

Σ^k(λ)=(1−λ) Σ^k+λ I,0≤λ≤1\hat\Sigma_k(\lambda)=(1-\lambda)\,\hat\Sigma_k+\lambda\,I,\qquad 0\le\lambda\le1
λ\lambda
reg_param in scikit-learn
II
identity matrix

The one hyperparameter. Shrinking toward II adds λ\lambda to every eigenvalue's share, so the smallest eigenvalues — the ones estimated from too little data, which make Σk−1\Sigma_k^{-1} explode — are lifted away from zero. At λ=0\lambda=0 you get plain QDA; at λ=1\lambda=1 every class has Σk=I\Sigma_k=I, the volume penalties are all 0, and δk=−12∥x−μk∥2+log⁡πk\delta_k=-\tfrac12\|x-\mu_k\|^2+\log\pi_k: a nearest-centroid classifier with a prior bonus. Because the target is II, features must be on comparable scales for λ>0\lambda>0 to make sense. Friedman's regularized discriminant analysis (RDA) also blends toward the pooled covariance, αΣ^k+(1−α)Σ^\alpha\hat\Sigma_k+(1-\alpha)\hat\Sigma, sliding between QDA and LDA.

Worked example: genuine or forged?

Eleven banknotes with two scaled measurements, x1x_1 = ink density and x2x_2 = paper thickness. Five are genuine (class A) and sit tightly together; six are forgeries (class B) and scatter around them.

Notex₁ (ink)x₂ (paper)Class
134A genuine
254A genuine
343A genuine
445A genuine
544A genuine
623B forged
789B forged
838B forged
974B forged
1052B forged
11510B forged
Training data. Highlighted: the 5 genuine notes. A new note x = (5, 5) arrives at the end.

Fit QDA by hand, then classify x = (5, 5)

  1. 1Priors. π^A=5/11\hat\pi_A=5/11, π^B=6/11\hat\pi_B=6/11, so log⁡π^A=−0.788\log\hat\pi_A=-0.788 and log⁡π^B=−0.606\log\hat\pi_B=-0.606.
  2. 2Means. μ^A=(4,4)\hat\mu_A=(4,4); μ^B=(306,366)=(5,6)\hat\mu_B=\big(\tfrac{30}{6},\tfrac{36}{6}\big)=(5,6). The means are only about 2.2 units apart — position alone barely separates the classes.
  3. 3Covariance of A. Deviations from (4,4)(4,4): (−1,0),(1,0),(0,−1),(0,1),(0,0)(-1,0),(1,0),(0,-1),(0,1),(0,0). Sums of squares 2 and 2, cross-products 0, divided by nA−1=4n_A-1=4: Σ^A=(0.5000.5)\hat\Sigma_A=\begin{pmatrix}0.5&0\\0&0.5\end{pmatrix}, ∣Σ^A∣=0.25|\hat\Sigma_A|=0.25, Σ^A−1=2I\hat\Sigma_A^{-1}=2I.
  4. 4Covariance of B. Deviations from (5,6)(5,6): (−3,−3),(3,3),(−2,2),(2,−2),(0,−4),(0,4)(-3,-3),(3,3),(-2,2),(2,-2),(0,-4),(0,4). Sums: ∑d12=26\sum d_1^2=26, ∑d22=58\sum d_2^2=58, ∑d1d2=10\sum d_1d_2=10; divide by nB−1=5n_B-1=5: Σ^B=(5.22211.6)\hat\Sigma_B=\begin{pmatrix}5.2&2\\2&11.6\end{pmatrix}, ∣Σ^B∣=5.2⋅11.6−22=56.32|\hat\Sigma_B|=5.2\cdot11.6-2^2=56.32, Σ^B−1=156.32(11.6−2−25.2)=(0.206−0.036−0.0360.092)\hat\Sigma_B^{-1}=\tfrac1{56.32}\begin{pmatrix}11.6&-2\\-2&5.2\end{pmatrix}=\begin{pmatrix}0.206&-0.036\\-0.036&0.092\end{pmatrix}.
  5. 5Volume penalties. −12log⁡0.25=+0.693-\tfrac12\log 0.25=+0.693 for A; −12log⁡56.32=−2.016-\tfrac12\log 56.32=-2.016 for B. B's ellipse has about 15 times the area of A's, and pays for it.
  6. 6Mahalanobis distances to x = (5, 5). For A, x−μA=(1,1)x-\mu_A=(1,1) and ΔA2=2(12+12)=4\Delta_A^2=2(1^2+1^2)=4 — the note is ΔA=2\Delta_A=2 standard deviations from the genuine centre. For B, x−μB=(0,−1)x-\mu_B=(0,-1) and ΔB2=0.092\Delta_B^2=0.092 — only 0.300.30 of B's standard deviations away.
  7. 7Scores. δA=0.693−12(4)−0.788=−2.095\delta_A=0.693-\tfrac12(4)-0.788=-2.095; δB=−2.016−12(0.092)−0.606=−2.668\delta_B=-2.016-\tfrac12(0.092)-0.606=-2.668.
  8. 8Decision. δA−δB=0.573>0\delta_A-\delta_B=0.573>0, so QDA says genuine, with P(A∣x)=σ(0.573)=0.639P(A\mid x)=\sigma(0.573)=0.639. Note that x is closer to B in Mahalanobis terms; A wins only because its tight Gaussian is so much denser near its centre.
Class−½ log|Σ|−½ Δ²log πδ(x)P(class | x)
A genuine+0.693−2.000−0.788−2.0950.639
B forged−2.016−0.046−0.606−2.6680.361
The three terms of the discriminant at x = (5, 5). The volume penalty decides it.

The boundary. Plugging the fitted parameters into δA−δB=0\delta_A-\delta_B=0 gives −0.897x12−0.036x1x2−0.954x22+7.183x1+7.624x2−26.303=0-0.897x_1^2-0.036x_1x_2-0.954x_2^2+7.183x_1+7.624x_2-26.303=0. Its quadratic part has det⁡Q=0.855>0\det Q=0.855>0, so it is an ellipse, centred at about (3.93,3.92)(3.93, 3.92): a closed island of "genuine" around class A, with "forged" everywhere outside. All 11 training notes land on the correct side.

Compare LDA. Pooling the two scatter matrices, Σ^=111−2(28101060)\hat\Sigma=\tfrac{1}{11-2}\begin{pmatrix}28&10\\10&60\end{pmatrix}, forces one shared shape. The boundary becomes a straight line, LDA misclassifies 4 of the 11 training notes (7/11 = 0.636 accuracy), and for x = (5, 5) it says forged with P(A∣x)=0.426P(A\mid x)=0.426.

00224466881010x = (5.0, 5.0)QDAA genuineB forgedclass meanquery x (drag)boundary1σ, 2σ ellipsesring through xmisclassifiedboundary: ellipsetrain acc 11/11Mahalanobis of xto A: 2.00σto B: 0.30σP(A | x) = 0.639predict A genuine
classμΣ (a b; b c)−½log|Σ|−½Δ²log πδ(x)
A (n=5)(4.00, 4.00)0.50 0.00; 0.00 0.500.693-2.000-0.788-2.095
B (n=6)(5.00, 6.00)5.20 2.00; 2.00 11.60-2.016-0.046-0.606-2.668
δ_A − δ_B = 0.573 → P(A|x) = σ(0.573) = 0.639
fig.The worked example, live. Shading is P(genuine | x); solid ellipses are each class's 1σ and 2σ contours; the white curve is the QDA boundary — an ellipse around class A. The dashed rings pass through the query x = (5, 5): it sits on A's 2.00σ ring but B's 0.30σ ring, yet QDA predicts genuine with P = 0.639, 11/11 on training. Switch to LDA: one pooled shape, a straight boundary, 4 red-ringed errors and P = 0.426. Raise reg toward 1: at 0.50 accuracy falls to 10/11 (P = 0.578), at 1.00 to 9/11 (P = 0.336). Drag any note or the query; drag the forgeries into a tilted line and the boundary turns into a hyperbola.

The algorithm, step by step

QDA

  1. 1Fit, for each class kk: count nkn_k and set log⁡πk=log⁡(nk/n)\log\pi_k=\log(n_k/n); average the points to get μk\mu_k; form the centred matrix Xk−μkX_k-\mu_k and its covariance Σk\Sigma_k.
  2. 2Regularize (optional): Σk←(1−λ)Σk+λI\Sigma_k\leftarrow(1-\lambda)\Sigma_k+\lambda I with λ\lambda = reg_param.
  3. 3Factorize each Σk\Sigma_k once (Cholesky or eigendecomposition) to get log⁡∣Σk∣\log|\Sigma_k| stably and a way to apply Σk−1\Sigma_k^{-1} without forming an explicit inverse.
  4. 4Predict: for a new xx, compute δk(x)=−12log⁡∣Σk∣−12Δk2(x)+log⁡πk\delta_k(x)=-\tfrac12\log|\Sigma_k|-\tfrac12\Delta_k^2(x)+\log\pi_k for every class.
  5. 5Output arg⁡max⁡kδk(x)\arg\max_k\delta_k(x), and probabilities by a softmax over the scores (subtract the max score first to avoid overflow).

Complexity

Training time
O(nd2)O(nd^2) to build the KK covariance matrices, plus O(Kd3)O(Kd^3) to factorize them
Prediction time
O(Kd2)O(Kd^2) per point (one triangular solve or eigen-projection per class)
Memory / parameters
K(d+d(d+1)2)+(K−1)K\big(d+\tfrac{d(d+1)}2\big)+(K-1) numbers. With d=10d=10, K=3K=3: 165 covariance entries (LDA needs 55) and 197 parameters in total
Data needed
each class needs nk>dn_k>d points just for Σ^k\hat\Sigma_k to be invertible, and many more for it to be accurate

Variants and tuning

ModelCovariancesBoundaryCov. params (d features, K classes)
Nearest centroidΣk=I\Sigma_k=Ilinear (perpendicular bisectors)0
Gaussian Naive BayesΣk\Sigma_k diagonal, per classquadratic, axis-alignedKdKd
LDAone shared Σ\Sigmalineard(d+1)/2d(d+1)/2
Regularized DA (Friedman)αΣk+(1−α)Σ\alpha\Sigma_k+(1-\alpha)\Sigma, then shrink toward IIbetween linear and quadratictuned by α, γ
QDAfull Σk\Sigma_k per classquadratic (any conic)Kd(d+1)/2Kd(d+1)/2
Gaussian mixture per classseveral Gaussians per classpiecewise quadratic, very flexiblemany more
The Gaussian-classifier family: same recipe, different constraint on the covariances.

Practical tuning

reg_param
Start at 0 when every class has many more points than features; otherwise search a grid such as 0, 0.01, 0.1, 0.3, 0.5 with cross-validated log-loss. Standardize features first, because the shrinkage target is the identity.
priors
Defaults to class frequencies. If the deployment class balance differs from training (e.g. forgeries are rarer in the wild), pass the real priors — only the log π term changes, no refit of means or covariances is needed.
Too many features
Reduce dimension first (PCA, feature selection), use diagonal covariances (Naive Bayes), or fall back to LDA.
Non-Gaussian classes
Transform skewed features (log, Box–Cox), or model each class with a Gaussian mixture instead of one Gaussian.

In code

python
1import numpy as np
2
3class QDA:
4 def __init__(self, reg_param=0.0):
5 self.reg = reg_param
6
7 def fit(self, X, y):
8 self.classes = np.unique(y)
9 self.mu, self.inv, self.logdet, self.logprior = [], [], [], []
10 for k in self.classes:
11 Xk = X[y == k]
12 mu = Xk.mean(axis=0)
13 S = np.cov(Xk, rowvar=False, ddof=1) # unbiased, like sklearn
14 S = (1 - self.reg) * S + self.reg * np.eye(X.shape[1])
15 sign, logdet = np.linalg.slogdet(S) # stable log|S|
16 self.mu.append(mu)
17 self.inv.append(np.linalg.inv(S))
18 self.logdet.append(logdet)
19 self.logprior.append(np.log(len(Xk) / len(X)))
20 return self
21
22 def decision(self, X): # (n, K) discriminant scores
23 out = []
24 for mu, P, ld, lp in zip(self.mu, self.inv, self.logdet, self.logprior):
25 D = X - mu
26 m2 = np.einsum("ij,jk,ik->i", D, P, D) # squared Mahalanobis
27 out.append(-0.5 * ld - 0.5 * m2 + lp)
28 return np.column_stack(out)
29
30 def predict_proba(self, X):
31 d = self.decision(X)
32 e = np.exp(d - d.max(axis=1, keepdims=True)) # softmax over classes
33 return e / e.sum(axis=1, keepdims=True)
34
35 def predict(self, X):
36 return self.classes[self.decision(X).argmax(axis=1)]
37
38X = np.array([[3,4],[5,4],[4,3],[4,5],[4,4],
39 [2,3],[8,9],[3,8],[7,4],[5,2],[5,10]], float)
40y = np.array([0]*5 + [1]*6) # 0 = genuine, 1 = forged
41m = QDA().fit(X, y)
42print(m.decision(np.array([[5., 5.]])).round(3)) # [[-2.095 -2.668]]
43print(m.predict_proba(np.array([[5., 5.]])).round(3)) # [[0.639 0.361]]
44print((m.predict(X) == y).mean()) # 1.0
From scratch with NumPy: fit, discriminant scores, softmax probabilities. Prints the worked-example numbers.
python
1import numpy as np
2from sklearn.discriminant_analysis import (
3 LinearDiscriminantAnalysis, QuadraticDiscriminantAnalysis)
4
5X = np.array([[3,4],[5,4],[4,3],[4,5],[4,4],
6 [2,3],[8,9],[3,8],[7,4],[5,2],[5,10]], float)
7y = np.array([0]*5 + [1]*6)
8x_new = [[5., 5.]]
9
10qda = QuadraticDiscriminantAnalysis(store_covariance=True).fit(X, y)
11print(qda.means_) # [[4. 4.] [5. 6.]]
12print(qda.covariance_[1]) # [[ 5.2 2. ] [ 2. 11.6]]
13print(qda.predict_proba(x_new).round(3)) # [[0.639 0.361]] -> genuine
14print(qda.score(X, y)) # 1.0
15
16lda = LinearDiscriminantAnalysis().fit(X, y)
17print(lda.predict_proba(x_new).round(3)) # [[0.426 0.574]] -> forged
18print(round(lda.score(X, y), 3)) # 0.636 (7 of 11)
19
20for r in [0.0, 0.5, 1.0]: # shrink each covariance toward I
21 m = QuadraticDiscriminantAnalysis(reg_param=r).fit(X, y)
22 print(r, round(m.score(X, y), 3), m.predict_proba(x_new)[0, 0].round(3))
23# 0.0 1.0 0.639
24# 0.5 0.909 0.578
25# 1.0 0.818 0.336
scikit-learn on the same notes: QDA vs LDA, and the effect of reg_param.

Use when

  • +Classes differ in spread, shape or orientation — not just in location — and a curved boundary is needed.
  • +You have plenty of samples per class relative to the number of features (roughly nk≫d2/2n_k\gg d^2/2).
  • +You want a fast, closed-form, no-tuning baseline that also gives class probabilities and an interpretable per-class model.
  • +Priors may change after deployment: they enter only through log⁡πk\log\pi_k, so you can adjust them without retraining.

Avoid when

  • −Many features and few samples: Kd(d+1)/2Kd(d+1)/2 covariance entries overfit, or are singular outright — prefer LDA, Naive Bayes or regularized QDA.
  • −Classes are clearly non-Gaussian: multimodal, heavy-tailed or with categorical features.
  • −Outliers are common: one extreme point inflates a class covariance and reshapes the whole boundary.
  • −Probabilities must be well calibrated: when the Gaussian assumption is off, QDA's posteriors are often overconfident.

Where it's used

Fraud and counterfeit screening

Genuine items or transactions form tight clusters; fraudulent ones vary widely — exactly the 'island inside a cloud' shape QDA draws.

Remote sensing

Classifying land cover from multispectral pixels, where classes such as water and forest have very different spectral covariances; Gaussian maximum-likelihood classifiers are a long-standing standard.

Biomedical signals

EEG/BCI and spectroscopy classification on a few well-chosen features, where class variability itself is diagnostic.

Speech and audio

Per-class full-covariance Gaussians (and mixtures of them) over acoustic features — the building block of classic GMM-based speaker and phoneme models.

Check yourself

Two classes have the same mean 0 in one dimension, variances 1 and 4, and equal priors. Where is the QDA decision boundary?

Check yourself

With d = 10 features and K = 3 classes, how many covariance entries does QDA estimate, compared with LDA?

Check yourself

In the worked example, x = (5, 5) is 2.00σ from the genuine mean but only 0.30σ from the forged mean. Why does QDA still predict genuine?

Check yourself

What does QDA with reg_param = 1 reduce to?

Remember

  • ✓QDA models each class as N(μk,Σk)\mathcal N(\mu_k,\Sigma_k) with its own covariance and classifies by Bayes' rule.
  • ✓Score: δk(x)=−12log⁡∣Σk∣−12Δk2(x)+log⁡πk\delta_k(x)=-\tfrac12\log|\Sigma_k|-\tfrac12\Delta_k^2(x)+\log\pi_k — volume penalty, Mahalanobis fit, prior.
  • ✓Different covariances make the boundary quadratic (ellipse, parabola, hyperbola); equal covariances cancel the quadratic term and give LDA.
  • ✓Fitting is closed form — counts, means and per-class covariances — but needs K d(d+1)/2K\,d(d+1)/2 covariance entries, so it overfits or goes singular with few samples per class.
  • ✓reg_param shrinks each covariance toward the identity; at 1 it becomes nearest-centroid. Tune it by cross-validation on standardized features.
← back to ML Algorithms roadmap