Definition
Quadratic discriminant analysis (QDA) is a generative classifier that models each class as a multivariate Gaussian with its own mean and its own covariance matrix, then classifies a point by Bayes' rule. Because the covariances differ, the log-posterior is quadratic in , and so are the decision boundaries: ellipses, parabolas or hyperbolas instead of straight lines.
At a glance
- Task
- Classification (binary or multiclass), with calibrated-ish class probabilities
- Input → output
- → for every class, and the arg-max class
- Learns
- per class: a prior , a mean ( numbers) and a covariance ( numbers)
- Training
- closed form, no iterations: (scatter matrices plus one inverse / determinant per class)
- Prediction
- per point: one Mahalanobis distance per class
- Hyperparameters
reg_param(shrink each covariance toward the identity), class priors- Assumes
- within each class the features are jointly Gaussian; classes may differ in spread, shape and orientation
The problem
Linear classifiers — logistic regression, the perceptron, and its closest relative linear discriminant analysis — can only draw a straight line (a hyperplane) between classes. That is the right shape when the classes differ only in where they sit. It is the wrong shape when they differ in how spread out they are.
Think of banknotes. Genuine notes come off one press, so their ink density and paper thickness cluster tightly. Forgeries come from many workshops and scatter widely — some too thick, some too thin, some too dark, some too light — on every side of the genuine cluster. No line separates "tight blob" from "wide cloud around it". You need a closed curve around the genuine notes. LDA cannot produce one, because it assumes every class shares a single covariance matrix; QDA drops exactly that assumption.
The idea
Fit one Gaussian "bell" per class — its centre, its spread along each direction, and its tilt. To classify a new point, ask each bell how likely am I to have produced this point?, weight the answer by how common the class is, and pick the largest. A tight class gives a very high density near its centre but drops off fast; a wide class gives a low, flat density everywhere. So a point near the tight centre goes to the tight class, and a point even moderately far from it goes to the wide one — the boundary wraps around the tight class.
Vocabulary
- Generative classifier
- models how each class generates data, , and gets by Bayes' rule (contrast: logistic regression models directly)
- Prior
- , the class frequency before seeing ; estimated as
- Class-conditional density
- ; in QDA a Gaussian
- Covariance matrix
- : variances of each feature on the diagonal, covariances off it. It sets the size, shape and tilt of the class's ellipse
- Mahalanobis distance
- : distance from measured in that class's standard deviations, along its own axes
- Discriminant function
- : a score equal to up to a constant shared by all classes; predict
- Conic section
- the curve where a quadratic in two variables is zero: ellipse, parabola or hyperbola (a line is a degenerate case)
- Regularized QDA
- shrinking each toward a simpler matrix (the identity, or the pooled LDA covariance) to fight overfitting
The math
The generative model. Each label is drawn from the priors, then the features are drawn from that class's Gaussian:
- number of classes
- number of features
- prior probability of class ,
- mean vector of class ()
- covariance matrix of class (, symmetric positive definite)
- determinant of : the squared "volume" of the class's ellipse
From Bayes' rule to a score. The posterior is . The denominator is the same for every class, and the logarithm is increasing, so the arg-max class is the one maximizing . Take the log of the Gaussian and drop , which is the same for every class:
- quadratic discriminant score of class k at x; predict the class with the largest score
- volume penalty: a wide class spreads its probability thinly, so it pays for its size
- fit term: half the squared Mahalanobis distance to the class mean
- prior term: common classes get a head start
Those three terms are the whole algorithm. The volume penalty is what lets a tight class win near its centre even against a nearby wide class, and it is exactly the term that LDA throws away (with a shared , is the same for everyone). Posteriors come back from the scores with a softmax, ; for two classes that is .
Why the boundary is quadratic. Expand : . The boundary between classes and is where the scores tie:
- matrix of quadratic coefficients; zero exactly when
- linear coefficients
- constant:
That is a quadric surface. In 2-D, the sign of tells you its shape: gives an ellipse (one class's region is a closed island), a hyperbola (the regions interleave in an X shape), a parabola. If all classes share one covariance, and the boundary collapses to the hyperplane of LDA — the full derivation of that special case is in the Linear discriminant analysis article. If instead each is forced to be diagonal, QDA becomes Gaussian Naive Bayes.
Fitting = counting and averaging. Maximizing the joint log-likelihood separates into one independent problem per class, and each has the familiar Gaussian closed form:
- number of training points in class ;
- unbiased denominator (scikit-learn's choice); the exact MLE uses
There is no gradient descent and no learning rate: set and in the Gaussian log-likelihood and these averages fall out. The catch is the parameter count: each has free entries, and is singular (not invertible) whenever .
reg_paramin scikit-learn- identity matrix
The one hyperparameter. Shrinking toward adds to every eigenvalue's share, so the smallest eigenvalues — the ones estimated from too little data, which make explode — are lifted away from zero. At you get plain QDA; at every class has , the volume penalties are all 0, and : a nearest-centroid classifier with a prior bonus. Because the target is , features must be on comparable scales for to make sense. Friedman's regularized discriminant analysis (RDA) also blends toward the pooled covariance, , sliding between QDA and LDA.
Worked example: genuine or forged?
Eleven banknotes with two scaled measurements, = ink density and = paper thickness. Five are genuine (class A) and sit tightly together; six are forgeries (class B) and scatter around them.
| Note | x₁ (ink) | x₂ (paper) | Class |
|---|---|---|---|
| 1 | 3 | 4 | A genuine |
| 2 | 5 | 4 | A genuine |
| 3 | 4 | 3 | A genuine |
| 4 | 4 | 5 | A genuine |
| 5 | 4 | 4 | A genuine |
| 6 | 2 | 3 | B forged |
| 7 | 8 | 9 | B forged |
| 8 | 3 | 8 | B forged |
| 9 | 7 | 4 | B forged |
| 10 | 5 | 2 | B forged |
| 11 | 5 | 10 | B forged |
Fit QDA by hand, then classify x = (5, 5)
- 1Priors. , , so and .
- 2Means. ; . The means are only about 2.2 units apart — position alone barely separates the classes.
- 3Covariance of A. Deviations from : . Sums of squares 2 and 2, cross-products 0, divided by : , , .
- 4Covariance of B. Deviations from : . Sums: , , ; divide by : , , .
- 5Volume penalties. for A; for B. B's ellipse has about 15 times the area of A's, and pays for it.
- 6Mahalanobis distances to x = (5, 5). For A, and — the note is standard deviations from the genuine centre. For B, and — only of B's standard deviations away.
- 7Scores. ; .
- 8Decision. , so QDA says genuine, with . Note that x is closer to B in Mahalanobis terms; A wins only because its tight Gaussian is so much denser near its centre.
| Class | −½ log|Σ| | −½ Δ² | log π | δ(x) | P(class | x) |
|---|---|---|---|---|---|
| A genuine | +0.693 | −2.000 | −0.788 | −2.095 | 0.639 |
| B forged | −2.016 | −0.046 | −0.606 | −2.668 | 0.361 |
The boundary. Plugging the fitted parameters into gives . Its quadratic part has , so it is an ellipse, centred at about : a closed island of "genuine" around class A, with "forged" everywhere outside. All 11 training notes land on the correct side.
Compare LDA. Pooling the two scatter matrices, , forces one shared shape. The boundary becomes a straight line, LDA misclassifies 4 of the 11 training notes (7/11 = 0.636 accuracy), and for x = (5, 5) it says forged with .
| class | μ | Σ (a b; b c) | −½log|Σ| | −½Δ² | log π | δ(x) |
|---|---|---|---|---|---|---|
| A (n=5) | (4.00, 4.00) | 0.50 0.00; 0.00 0.50 | 0.693 | -2.000 | -0.788 | -2.095 |
| B (n=6) | (5.00, 6.00) | 5.20 2.00; 2.00 11.60 | -2.016 | -0.046 | -0.606 | -2.668 |
The algorithm, step by step
QDA
- 1Fit, for each class : count and set ; average the points to get ; form the centred matrix and its covariance .
- 2Regularize (optional): with =
reg_param. - 3Factorize each once (Cholesky or eigendecomposition) to get stably and a way to apply without forming an explicit inverse.
- 4Predict: for a new , compute for every class.
- 5Output , and probabilities by a softmax over the scores (subtract the max score first to avoid overflow).
Complexity
- Training time
- to build the covariance matrices, plus to factorize them
- Prediction time
- per point (one triangular solve or eigen-projection per class)
- Memory / parameters
- numbers. With , : 165 covariance entries (LDA needs 55) and 197 parameters in total
- Data needed
- each class needs points just for to be invertible, and many more for it to be accurate
Variants and tuning
| Model | Covariances | Boundary | Cov. params (d features, K classes) |
|---|---|---|---|
| Nearest centroid | linear (perpendicular bisectors) | 0 | |
| Gaussian Naive Bayes | diagonal, per class | quadratic, axis-aligned | |
| LDA | one shared | linear | |
| Regularized DA (Friedman) | , then shrink toward | between linear and quadratic | tuned by α, γ |
| QDA | full per class | quadratic (any conic) | |
| Gaussian mixture per class | several Gaussians per class | piecewise quadratic, very flexible | many more |
Practical tuning
- reg_param
- Start at 0 when every class has many more points than features; otherwise search a grid such as 0, 0.01, 0.1, 0.3, 0.5 with cross-validated log-loss. Standardize features first, because the shrinkage target is the identity.
- priors
- Defaults to class frequencies. If the deployment class balance differs from training (e.g. forgeries are rarer in the wild), pass the real priors — only the log π term changes, no refit of means or covariances is needed.
- Too many features
- Reduce dimension first (PCA, feature selection), use diagonal covariances (Naive Bayes), or fall back to LDA.
- Non-Gaussian classes
- Transform skewed features (log, Box–Cox), or model each class with a Gaussian mixture instead of one Gaussian.
In code
1import numpy as np23class QDA:4 def __init__(self, reg_param=0.0):5 self.reg = reg_param67 def fit(self, X, y):8 self.classes = np.unique(y)9 self.mu, self.inv, self.logdet, self.logprior = [], [], [], []10 for k in self.classes:11 Xk = X[y == k]12 mu = Xk.mean(axis=0)13 S = np.cov(Xk, rowvar=False, ddof=1) # unbiased, like sklearn14 S = (1 - self.reg) * S + self.reg * np.eye(X.shape[1])15 sign, logdet = np.linalg.slogdet(S) # stable log|S|16 self.mu.append(mu)17 self.inv.append(np.linalg.inv(S))18 self.logdet.append(logdet)19 self.logprior.append(np.log(len(Xk) / len(X)))20 return self2122 def decision(self, X): # (n, K) discriminant scores23 out = []24 for mu, P, ld, lp in zip(self.mu, self.inv, self.logdet, self.logprior):25 D = X - mu26 m2 = np.einsum("ij,jk,ik->i", D, P, D) # squared Mahalanobis27 out.append(-0.5 * ld - 0.5 * m2 + lp)28 return np.column_stack(out)2930 def predict_proba(self, X):31 d = self.decision(X)32 e = np.exp(d - d.max(axis=1, keepdims=True)) # softmax over classes33 return e / e.sum(axis=1, keepdims=True)3435 def predict(self, X):36 return self.classes[self.decision(X).argmax(axis=1)]3738X = np.array([[3,4],[5,4],[4,3],[4,5],[4,4],39 [2,3],[8,9],[3,8],[7,4],[5,2],[5,10]], float)40y = np.array([0]*5 + [1]*6) # 0 = genuine, 1 = forged41m = QDA().fit(X, y)42print(m.decision(np.array([[5., 5.]])).round(3)) # [[-2.095 -2.668]]43print(m.predict_proba(np.array([[5., 5.]])).round(3)) # [[0.639 0.361]]44print((m.predict(X) == y).mean()) # 1.01import numpy as np2from sklearn.discriminant_analysis import (3 LinearDiscriminantAnalysis, QuadraticDiscriminantAnalysis)45X = np.array([[3,4],[5,4],[4,3],[4,5],[4,4],6 [2,3],[8,9],[3,8],[7,4],[5,2],[5,10]], float)7y = np.array([0]*5 + [1]*6)8x_new = [[5., 5.]]910qda = QuadraticDiscriminantAnalysis(store_covariance=True).fit(X, y)11print(qda.means_) # [[4. 4.] [5. 6.]]12print(qda.covariance_[1]) # [[ 5.2 2. ] [ 2. 11.6]]13print(qda.predict_proba(x_new).round(3)) # [[0.639 0.361]] -> genuine14print(qda.score(X, y)) # 1.01516lda = LinearDiscriminantAnalysis().fit(X, y)17print(lda.predict_proba(x_new).round(3)) # [[0.426 0.574]] -> forged18print(round(lda.score(X, y), 3)) # 0.636 (7 of 11)1920for r in [0.0, 0.5, 1.0]: # shrink each covariance toward I21 m = QuadraticDiscriminantAnalysis(reg_param=r).fit(X, y)22 print(r, round(m.score(X, y), 3), m.predict_proba(x_new)[0, 0].round(3))23# 0.0 1.0 0.63924# 0.5 0.909 0.57825# 1.0 0.818 0.336Use when
- +Classes differ in spread, shape or orientation — not just in location — and a curved boundary is needed.
- +You have plenty of samples per class relative to the number of features (roughly ).
- +You want a fast, closed-form, no-tuning baseline that also gives class probabilities and an interpretable per-class model.
- +Priors may change after deployment: they enter only through , so you can adjust them without retraining.
Avoid when
- −Many features and few samples: covariance entries overfit, or are singular outright — prefer LDA, Naive Bayes or regularized QDA.
- −Classes are clearly non-Gaussian: multimodal, heavy-tailed or with categorical features.
- −Outliers are common: one extreme point inflates a class covariance and reshapes the whole boundary.
- −Probabilities must be well calibrated: when the Gaussian assumption is off, QDA's posteriors are often overconfident.
Where it's used
Fraud and counterfeit screening
Genuine items or transactions form tight clusters; fraudulent ones vary widely — exactly the 'island inside a cloud' shape QDA draws.
Remote sensing
Classifying land cover from multispectral pixels, where classes such as water and forest have very different spectral covariances; Gaussian maximum-likelihood classifiers are a long-standing standard.
Biomedical signals
EEG/BCI and spectroscopy classification on a few well-chosen features, where class variability itself is diagnostic.
Speech and audio
Per-class full-covariance Gaussians (and mixtures of them) over acoustic features — the building block of classic GMM-based speaker and phoneme models.
Check yourself
Two classes have the same mean 0 in one dimension, variances 1 and 4, and equal priors. Where is the QDA decision boundary?
Check yourself
With d = 10 features and K = 3 classes, how many covariance entries does QDA estimate, compared with LDA?
Check yourself
In the worked example, x = (5, 5) is 2.00σ from the genuine mean but only 0.30σ from the forged mean. Why does QDA still predict genuine?
Check yourself
What does QDA with reg_param = 1 reduce to?
Remember
- ✓QDA models each class as with its own covariance and classifies by Bayes' rule.
- ✓Score: — volume penalty, Mahalanobis fit, prior.
- ✓Different covariances make the boundary quadratic (ellipse, parabola, hyperbola); equal covariances cancel the quadratic term and give LDA.
- ✓Fitting is closed form — counts, means and per-class covariances — but needs covariance entries, so it overfits or goes singular with few samples per class.
- ✓
reg_paramshrinks each covariance toward the identity; at 1 it becomes nearest-centroid. Tune it by cross-validation on standardized features.