# Quadratic discriminant analysis

> Give each class its own Gaussian covariance; the Bayes decision boundary becomes quadratic.

Source: https://www.mlfoundry.in/learn/quadratic-discriminant-analysis  
Section: ML Algorithms › Supervised · Classification  
Level: advanced · 16 min read · updated 2026-10-07

**Quadratic discriminant analysis (QDA):** is a generative classifier that models each class $k$ as a multivariate Gaussian $\mathcal N(\mu_k,\Sigma_k)$ with **its own mean and its own covariance matrix**, then classifies a point by Bayes' rule. Because the covariances differ, the log-posterior is quadratic in $x$, and so are the decision boundaries: ellipses, parabolas or hyperbolas instead of straight lines.

- **Task:** Classification (binary or multiclass), with calibrated-ish class probabilities
- **Input → output:** $x\in\mathbb R^d$ → $P(y=k\mid x)$ for every class, and the arg-max class
- **Learns:** per class: a prior $\pi_k$, a mean $\mu_k$ ($d$ numbers) and a covariance $\Sigma_k$ ($d(d+1)/2$ numbers)
- **Training:** closed form, no iterations: $O(n d^2 + K d^3)$ (scatter matrices plus one inverse / determinant per class)
- **Prediction:** $O(K d^2)$ per point: one Mahalanobis distance per class
- **Hyperparameters:** `reg_param` (shrink each covariance toward the identity), class priors
- **Assumes:** within each class the features are jointly Gaussian; classes may differ in spread, shape and orientation

## The problem

Linear classifiers — logistic regression, the perceptron, and its closest relative **linear discriminant analysis** — can only draw a straight line (a hyperplane) between classes. That is the right shape when the classes differ only in *where* they sit. It is the wrong shape when they differ in *how spread out* they are.

Think of banknotes. Genuine notes come off one press, so their ink density and paper thickness cluster tightly. Forgeries come from many workshops and scatter widely — some too thick, some too thin, some too dark, some too light — *on every side* of the genuine cluster. No line separates "tight blob" from "wide cloud around it". You need a closed curve around the genuine notes. LDA cannot produce one, because it assumes every class shares a single covariance matrix; QDA drops exactly that assumption.

## The idea

Fit one Gaussian "bell" per class — its centre, its spread along each direction, and its tilt. To classify a new point, ask each bell *how likely am I to have produced this point?*, weight the answer by how common the class is, and pick the largest. A tight class gives a very high density near its centre but drops off fast; a wide class gives a low, flat density everywhere. So a point near the tight centre goes to the tight class, and a point even moderately far from it goes to the wide one — the boundary wraps around the tight class.

**Vocabulary**

- **Generative classifier:** models how each class *generates* data, $p(x\mid y=k)$, and gets $P(y\mid x)$ by Bayes' rule (contrast: logistic regression models $P(y\mid x)$ directly)
- **Prior:** $\pi_k=P(y=k)$, the class frequency before seeing $x$; estimated as $n_k/n$
- **Class-conditional density:** $p(x\mid y=k)$; in QDA a Gaussian $\mathcal N(\mu_k,\Sigma_k)$
- **Covariance matrix:** $\Sigma_k$: variances of each feature on the diagonal, covariances off it. It sets the size, shape and tilt of the class's ellipse
- **Mahalanobis distance:** $\Delta_k(x)=\sqrt{(x-\mu_k)^\top\Sigma_k^{-1}(x-\mu_k)}$: distance from $\mu_k$ measured in that class's standard deviations, along its own axes
- **Discriminant function:** $\delta_k(x)$: a score equal to $\log \pi_k p(x\mid k)$ up to a constant shared by all classes; predict $\arg\max_k \delta_k(x)$
- **Conic section:** the curve where a quadratic in two variables is zero: ellipse, parabola or hyperbola (a line is a degenerate case)
- **Regularized QDA:** shrinking each $\Sigma_k$ toward a simpler matrix (the identity, or the pooled LDA covariance) to fight overfitting

## The math

**The generative model.** Each label is drawn from the priors, then the features are drawn from that class's Gaussian:

$$
y\sim\text{Categorical}(\pi_1,\dots,\pi_K),\qquad x\mid y=k\;\sim\;\mathcal N(\mu_k,\Sigma_k),\qquad p(x\mid k)=\frac{1}{(2\pi)^{d/2}|\Sigma_k|^{1/2}}\exp\!\Big(-\tfrac12(x-\mu_k)^\top\Sigma_k^{-1}(x-\mu_k)\Big)
$$

- $K$: number of classes
- $d$: number of features
- $\pi_k$: prior probability of class $k$, $\sum_k\pi_k=1$
- $\mu_k$: mean vector of class $k$ ($d\times1$)
- $\Sigma_k$: covariance matrix of class $k$ ($d\times d$, symmetric positive definite)
- $|\Sigma_k|$: determinant of $\Sigma_k$: the squared "volume" of the class's ellipse

**From Bayes' rule to a score.** The posterior is $P(y=k\mid x)=\pi_k p(x\mid k)/\sum_j \pi_j p(x\mid j)$. The denominator is the same for every class, and the logarithm is increasing, so the arg-max class is the one maximizing $\log\pi_k+\log p(x\mid k)$. Take the log of the Gaussian and drop $-\tfrac d2\log 2\pi$, which is the same for every class:

$$
\delta_k(x)=-\tfrac12\log|\Sigma_k|\;-\;\tfrac12\,(x-\mu_k)^\top\Sigma_k^{-1}(x-\mu_k)\;+\;\log\pi_k
$$

- $\delta_k(x)$: quadratic discriminant score of class k at x; predict the class with the largest score
- $-\tfrac12\log|\Sigma_k|$: volume penalty: a wide class spreads its probability thinly, so it pays for its size
- $-\tfrac12\Delta_k^2(x)$: fit term: half the squared Mahalanobis distance to the class mean
- $\log\pi_k$: prior term: common classes get a head start

Those three terms are the whole algorithm. The *volume penalty* is what lets a tight class win near its centre even against a nearby wide class, and it is exactly the term that LDA throws away (with a shared $\Sigma$, $\log|\Sigma|$ is the same for everyone). Posteriors come back from the scores with a softmax, $P(y=k\mid x)=e^{\delta_k(x)}/\sum_j e^{\delta_j(x)}$; for two classes that is $\sigma(\delta_A-\delta_B)$.

**Why the boundary is quadratic.** Expand $\delta_k$: $-\tfrac12 x^\top\Sigma_k^{-1}x+x^\top\Sigma_k^{-1}\mu_k+c_k$. The boundary between classes $j$ and $k$ is where the scores tie:

$$
\delta_j(x)-\delta_k(x)=x^\top\underbrace{\Big(-\tfrac12\big(\Sigma_j^{-1}-\Sigma_k^{-1}\big)\Big)}_{Q}\,x\;+\;\underbrace{\big(\Sigma_j^{-1}\mu_j-\Sigma_k^{-1}\mu_k\big)^{\!\top}}_{w^\top}x\;+\;c\;=\;0
$$

- $Q$: $d\times d$ matrix of quadratic coefficients; zero exactly when $\Sigma_j=\Sigma_k$
- $w$: linear coefficients
- $c$: constant: $-\tfrac12(\mu_j^\top\Sigma_j^{-1}\mu_j-\mu_k^\top\Sigma_k^{-1}\mu_k)-\tfrac12\log\tfrac{|\Sigma_j|}{|\Sigma_k|}+\log\tfrac{\pi_j}{\pi_k}$

That is a quadric surface. In 2-D, the sign of $\det Q$ tells you its shape: $\det Q>0$ gives an **ellipse** (one class's region is a closed island), $\det Q<0$ a **hyperbola** (the regions interleave in an X shape), $\det Q\approx 0$ a **parabola**. If all classes share one covariance, $Q=0$ and the boundary collapses to the hyperplane of LDA — the full derivation of that special case is in the *Linear discriminant analysis* article. If instead each $\Sigma_k$ is forced to be diagonal, QDA becomes Gaussian *Naive Bayes*.

**Fitting = counting and averaging.** Maximizing the joint log-likelihood $\sum_i \log \pi_{y_i}p(x_i\mid y_i)$ separates into one independent problem per class, and each has the familiar Gaussian closed form:

$$
\hat\pi_k=\frac{n_k}{n},\qquad \hat\mu_k=\frac1{n_k}\sum_{i:\,y_i=k}x_i,\qquad \hat\Sigma_k=\frac1{n_k-1}\sum_{i:\,y_i=k}(x_i-\hat\mu_k)(x_i-\hat\mu_k)^\top
$$

- $n_k$: number of training points in class $k$; $n=\sum_k n_k$
- $n_k-1$: unbiased denominator (scikit-learn's choice); the exact MLE uses $n_k$

There is no gradient descent and no learning rate: set $\partial/\partial\mu_k=0$ and $\partial/\partial\Sigma_k^{-1}=0$ in the Gaussian log-likelihood and these averages fall out. The catch is the parameter count: each $\Sigma_k$ has $d(d+1)/2$ free entries, and $\hat\Sigma_k$ is singular (not invertible) whenever $n_k\le d$.

$$
\hat\Sigma_k(\lambda)=(1-\lambda)\,\hat\Sigma_k+\lambda\,I,\qquad 0\le\lambda\le1
$$

- $\lambda$: `reg_param` in scikit-learn
- $I$: identity matrix

**The one hyperparameter.** Shrinking toward $I$ adds $\lambda$ to every eigenvalue's share, so the smallest eigenvalues — the ones estimated from too little data, which make $\Sigma_k^{-1}$ explode — are lifted away from zero. At $\lambda=0$ you get plain QDA; at $\lambda=1$ every class has $\Sigma_k=I$, the volume penalties are all 0, and $\delta_k=-\tfrac12\|x-\mu_k\|^2+\log\pi_k$: a **nearest-centroid** classifier with a prior bonus. Because the target is $I$, features must be on comparable scales for $\lambda>0$ to make sense. Friedman's *regularized discriminant analysis* (RDA) also blends toward the pooled covariance, $\alpha\hat\Sigma_k+(1-\alpha)\hat\Sigma$, sliding between QDA and LDA.

## Worked example: genuine or forged?

Eleven banknotes with two scaled measurements, $x_1$ = ink density and $x_2$ = paper thickness. Five are genuine (class A) and sit tightly together; six are forgeries (class B) and scatter around them.

Training data. Highlighted: the 5 genuine notes. A new note x = (5, 5) arrives at the end.

| Note | x₁ (ink) | x₂ (paper) | Class |
| --- | --- | --- | --- |
| 1 | 3 | 4 | A genuine |
| 2 | 5 | 4 | A genuine |
| 3 | 4 | 3 | A genuine |
| 4 | 4 | 5 | A genuine |
| 5 | 4 | 4 | A genuine |
| 6 | 2 | 3 | B forged |
| 7 | 8 | 9 | B forged |
| 8 | 3 | 8 | B forged |
| 9 | 7 | 4 | B forged |
| 10 | 5 | 2 | B forged |
| 11 | 5 | 10 | B forged |

**Fit QDA by hand, then classify x = (5, 5)**

1. **Priors.** $\hat\pi_A=5/11$, $\hat\pi_B=6/11$, so $\log\hat\pi_A=-0.788$ and $\log\hat\pi_B=-0.606$.
2. **Means.** $\hat\mu_A=(4,4)$; $\hat\mu_B=\big(\tfrac{30}{6},\tfrac{36}{6}\big)=(5,6)$. The means are only about 2.2 units apart — position alone barely separates the classes.
3. **Covariance of A.** Deviations from $(4,4)$: $(-1,0),(1,0),(0,-1),(0,1),(0,0)$. Sums of squares 2 and 2, cross-products 0, divided by $n_A-1=4$: $\hat\Sigma_A=\begin{pmatrix}0.5&0\\0&0.5\end{pmatrix}$, $|\hat\Sigma_A|=0.25$, $\hat\Sigma_A^{-1}=2I$.
4. **Covariance of B.** Deviations from $(5,6)$: $(-3,-3),(3,3),(-2,2),(2,-2),(0,-4),(0,4)$. Sums: $\sum d_1^2=26$, $\sum d_2^2=58$, $\sum d_1d_2=10$; divide by $n_B-1=5$: $\hat\Sigma_B=\begin{pmatrix}5.2&2\\2&11.6\end{pmatrix}$, $|\hat\Sigma_B|=5.2\cdot11.6-2^2=56.32$, $\hat\Sigma_B^{-1}=\tfrac1{56.32}\begin{pmatrix}11.6&-2\\-2&5.2\end{pmatrix}=\begin{pmatrix}0.206&-0.036\\-0.036&0.092\end{pmatrix}$.
5. **Volume penalties.** $-\tfrac12\log 0.25=+0.693$ for A; $-\tfrac12\log 56.32=-2.016$ for B. B's ellipse has about 15 times the area of A's, and pays for it.
6. **Mahalanobis distances to x = (5, 5).** For A, $x-\mu_A=(1,1)$ and $\Delta_A^2=2(1^2+1^2)=4$ — the note is $\Delta_A=2$ standard deviations from the genuine centre. For B, $x-\mu_B=(0,-1)$ and $\Delta_B^2=0.092$ — only $0.30$ of B's standard deviations away.
7. **Scores.** $\delta_A=0.693-\tfrac12(4)-0.788=-2.095$; $\delta_B=-2.016-\tfrac12(0.092)-0.606=-2.668$.
8. **Decision.** $\delta_A-\delta_B=0.573>0$, so QDA says **genuine**, with $P(A\mid x)=\sigma(0.573)=0.639$. Note that x is *closer* to B in Mahalanobis terms; A wins only because its tight Gaussian is so much denser near its centre.

The three terms of the discriminant at x = (5, 5). The volume penalty decides it.

| Class | −½ log\|Σ\| | −½ Δ² | log π | δ(x) | P(class \| x) |
| --- | --- | --- | --- | --- | --- |
| **A genuine** | +0.693 | −2.000 | −0.788 | **−2.095** | **0.639** |
| B forged | −2.016 | −0.046 | −0.606 | −2.668 | 0.361 |

**The boundary.** Plugging the fitted parameters into $\delta_A-\delta_B=0$ gives $-0.897x_1^2-0.036x_1x_2-0.954x_2^2+7.183x_1+7.624x_2-26.303=0$. Its quadratic part has $\det Q=0.855>0$, so it is an **ellipse**, centred at about $(3.93, 3.92)$: a closed island of "genuine" around class A, with "forged" everywhere outside. All 11 training notes land on the correct side.

**Compare LDA.** Pooling the two scatter matrices, $\hat\Sigma=\tfrac{1}{11-2}\begin{pmatrix}28&10\\10&60\end{pmatrix}$, forces one shared shape. The boundary becomes a straight line, LDA misclassifies 4 of the 11 training notes (7/11 = 0.636 accuracy), and for x = (5, 5) it says *forged* with $P(A\mid x)=0.426$.

> Figure: The worked example, live. Shading is P(genuine | x); solid ellipses are each class's 1σ and 2σ contours; the white curve is the QDA boundary — an ellipse around class A. The dashed rings pass through the query x = (5, 5): it sits on A's 2.00σ ring but B's 0.30σ ring, yet QDA predicts genuine with P = 0.639, 11/11 on training. Switch to LDA: one pooled shape, a straight boundary, 4 red-ringed errors and P = 0.426. Raise reg toward 1: at 0.50 accuracy falls to 10/11 (P = 0.578), at 1.00 to 9/11 (P = 0.336). Drag any note or the query; drag the forgeries into a tilted line and the boundary turns into a hyperbola.

> **QDA can separate classes with the same mean:** Move the means of the two classes on top of each other and every linear method — LDA, logistic regression, the perceptron — is left guessing, because a hyperplane through the shared centre splits both classes in half. QDA still works: the classes differ in $\Sigma_k$, and the volume and Mahalanobis terms use that difference directly. It is a *variance* detector as much as a *location* detector.

## The algorithm, step by step

**QDA**

1. **Fit, for each class $k$:** count $n_k$ and set $\log\pi_k=\log(n_k/n)$; average the points to get $\mu_k$; form the centred matrix $X_k-\mu_k$ and its covariance $\Sigma_k$.
2. **Regularize** (optional): $\Sigma_k\leftarrow(1-\lambda)\Sigma_k+\lambda I$ with $\lambda$ = `reg_param`.
3. **Factorize** each $\Sigma_k$ once (Cholesky or eigendecomposition) to get $\log|\Sigma_k|$ stably and a way to apply $\Sigma_k^{-1}$ without forming an explicit inverse.
4. **Predict:** for a new $x$, compute $\delta_k(x)=-\tfrac12\log|\Sigma_k|-\tfrac12\Delta_k^2(x)+\log\pi_k$ for every class.
5. **Output** $\arg\max_k\delta_k(x)$, and probabilities by a softmax over the scores (subtract the max score first to avoid overflow).

**Complexity**

- **Training time:** $O(nd^2)$ to build the $K$ covariance matrices, plus $O(Kd^3)$ to factorize them
- **Prediction time:** $O(Kd^2)$ per point (one triangular solve or eigen-projection per class)
- **Memory / parameters:** $K\big(d+\tfrac{d(d+1)}2\big)+(K-1)$ numbers. With $d=10$, $K=3$: 165 covariance entries (LDA needs 55) and 197 parameters in total
- **Data needed:** each class needs $n_k>d$ points just for $\hat\Sigma_k$ to be invertible, and many more for it to be accurate

## Variants and tuning

The Gaussian-classifier family: same recipe, different constraint on the covariances.

| Model | Covariances | Boundary | Cov. params (d features, K classes) |
| --- | --- | --- | --- |
| Nearest centroid | $\Sigma_k=I$ | linear (perpendicular bisectors) | 0 |
| Gaussian Naive Bayes | $\Sigma_k$ diagonal, per class | quadratic, axis-aligned | $Kd$ |
| LDA | one shared $\Sigma$ | linear | $d(d+1)/2$ |
| Regularized DA (Friedman) | $\alpha\Sigma_k+(1-\alpha)\Sigma$, then shrink toward $I$ | between linear and quadratic | tuned by α, γ |
| **QDA** | full $\Sigma_k$ per class | quadratic (any conic) | $Kd(d+1)/2$ |
| Gaussian mixture per class | several Gaussians per class | piecewise quadratic, very flexible | many more |

**Practical tuning**

- **reg_param:** Start at 0 when every class has many more points than features; otherwise search a grid such as 0, 0.01, 0.1, 0.3, 0.5 with cross-validated log-loss. Standardize features first, because the shrinkage target is the identity.
- **priors:** Defaults to class frequencies. If the deployment class balance differs from training (e.g. forgeries are rarer in the wild), pass the real priors — only the log π term changes, no refit of means or covariances is needed.
- **Too many features:** Reduce dimension first (PCA, feature selection), use diagonal covariances (Naive Bayes), or fall back to LDA.
- **Non-Gaussian classes:** Transform skewed features (log, Box–Cox), or model each class with a Gaussian mixture instead of one Gaussian.

> **Singular covariance with few points per class:** If a class has $n_k\le d$ samples — or any feature is constant or an exact linear combination of others within a class — $\hat\Sigma_k$ has a zero eigenvalue, $\log|\Sigma_k|=-\infty$ and the inverse does not exist. scikit-learn warns `Variables are collinear` and the predictions become garbage. Drop or combine features, or set `reg_param` > 0.

## In code

```python
import numpy as np

class QDA:
    def __init__(self, reg_param=0.0):
        self.reg = reg_param

    def fit(self, X, y):
        self.classes = np.unique(y)
        self.mu, self.inv, self.logdet, self.logprior = [], [], [], []
        for k in self.classes:
            Xk = X[y == k]
            mu = Xk.mean(axis=0)
            S = np.cov(Xk, rowvar=False, ddof=1)               # unbiased, like sklearn
            S = (1 - self.reg) * S + self.reg * np.eye(X.shape[1])
            sign, logdet = np.linalg.slogdet(S)                # stable log|S|
            self.mu.append(mu)
            self.inv.append(np.linalg.inv(S))
            self.logdet.append(logdet)
            self.logprior.append(np.log(len(Xk) / len(X)))
        return self

    def decision(self, X):                                     # (n, K) discriminant scores
        out = []
        for mu, P, ld, lp in zip(self.mu, self.inv, self.logdet, self.logprior):
            D = X - mu
            m2 = np.einsum("ij,jk,ik->i", D, P, D)           # squared Mahalanobis
            out.append(-0.5 * ld - 0.5 * m2 + lp)
        return np.column_stack(out)

    def predict_proba(self, X):
        d = self.decision(X)
        e = np.exp(d - d.max(axis=1, keepdims=True))           # softmax over classes
        return e / e.sum(axis=1, keepdims=True)

    def predict(self, X):
        return self.classes[self.decision(X).argmax(axis=1)]

X = np.array([[3,4],[5,4],[4,3],[4,5],[4,4],
              [2,3],[8,9],[3,8],[7,4],[5,2],[5,10]], float)
y = np.array([0]*5 + [1]*6)                                    # 0 = genuine, 1 = forged
m = QDA().fit(X, y)
print(m.decision(np.array([[5., 5.]])).round(3))               # [[-2.095 -2.668]]
print(m.predict_proba(np.array([[5., 5.]])).round(3))          # [[0.639 0.361]]
print((m.predict(X) == y).mean())                              # 1.0
```

From scratch with NumPy: fit, discriminant scores, softmax probabilities. Prints the worked-example numbers.

```python
import numpy as np
from sklearn.discriminant_analysis import (
    LinearDiscriminantAnalysis, QuadraticDiscriminantAnalysis)

X = np.array([[3,4],[5,4],[4,3],[4,5],[4,4],
              [2,3],[8,9],[3,8],[7,4],[5,2],[5,10]], float)
y = np.array([0]*5 + [1]*6)
x_new = [[5., 5.]]

qda = QuadraticDiscriminantAnalysis(store_covariance=True).fit(X, y)
print(qda.means_)                         # [[4. 4.] [5. 6.]]
print(qda.covariance_[1])                 # [[ 5.2  2. ] [ 2.  11.6]]
print(qda.predict_proba(x_new).round(3))  # [[0.639 0.361]] -> genuine
print(qda.score(X, y))                    # 1.0

lda = LinearDiscriminantAnalysis().fit(X, y)
print(lda.predict_proba(x_new).round(3))  # [[0.426 0.574]] -> forged
print(round(lda.score(X, y), 3))          # 0.636  (7 of 11)

for r in [0.0, 0.5, 1.0]:                 # shrink each covariance toward I
    m = QuadraticDiscriminantAnalysis(reg_param=r).fit(X, y)
    print(r, round(m.score(X, y), 3), m.predict_proba(x_new)[0, 0].round(3))
# 0.0 1.0 0.639
# 0.5 0.909 0.578
# 1.0 0.818 0.336
```

scikit-learn on the same notes: QDA vs LDA, and the effect of reg_param.

**Strengths**

- Classes differ in spread, shape or orientation — not just in location — and a curved boundary is needed.
- You have plenty of samples per class relative to the number of features (roughly $n_k\gg d^2/2$).
- You want a fast, closed-form, no-tuning baseline that also gives class probabilities and an interpretable per-class model.
- Priors may change after deployment: they enter only through $\log\pi_k$, so you can adjust them without retraining.

**Limitations**

- Many features and few samples: $Kd(d+1)/2$ covariance entries overfit, or are singular outright — prefer LDA, Naive Bayes or regularized QDA.
- Classes are clearly non-Gaussian: multimodal, heavy-tailed or with categorical features.
- Outliers are common: one extreme point inflates a class covariance and reshapes the whole boundary.
- Probabilities must be well calibrated: when the Gaussian assumption is off, QDA's posteriors are often overconfident.

- **Fraud and counterfeit screening:** Genuine items or transactions form tight clusters; fraudulent ones vary widely — exactly the 'island inside a cloud' shape QDA draws.
- **Remote sensing:** Classifying land cover from multispectral pixels, where classes such as water and forest have very different spectral covariances; Gaussian maximum-likelihood classifiers are a long-standing standard.
- **Biomedical signals:** EEG/BCI and spectroscopy classification on a few well-chosen features, where class variability itself is diagnostic.
- **Speech and audio:** Per-class full-covariance Gaussians (and mixtures of them) over acoustic features — the building block of classic GMM-based speaker and phoneme models.

> **An outlier reshapes the whole boundary:** Covariance is a sum of squared deviations, so a single far-away point in a small class can double its variance along one direction, shrink its volume penalty's advantage and rotate its ellipse. Drag one forgery far into a corner in the figure above and watch the island change. Inspect class-wise outliers before fitting, or use a robust covariance estimate (e.g. Minimum Covariance Determinant) for $\Sigma_k$.

> **QDA vs logistic regression with squared features:** Logistic regression on the features $x_i$, $x_i^2$ and $x_ix_j$ can represent the same quadratic boundaries. The difference is how they are fitted: QDA trusts the Gaussian model and estimates everything in closed form (data-efficient when the model is right), while logistic regression optimizes the conditional likelihood directly (more robust when it is wrong). See the *Logistic regression* article for that side.

**Question:** Two classes have the same mean 0 in one dimension, variances 1 and 4, and equal priors. Where is the QDA decision boundary?

- A. There is none — the classes cannot be separated
- B. At x = 0
- C. At |x| ≈ 1.36: inside goes to the variance-1 class
- D. At |x| = 2.5

**Answer:** C. At |x| ≈ 1.36: inside goes to the variance-1 class — Set $-\tfrac12\log1-\tfrac{x^2}2=-\tfrac12\log4-\tfrac{x^2}8$, so $\tfrac38x^2=\log2$ and $|x|=\sqrt{8\log2/3}=1.360$. Points closer to 0 than that go to the narrow class, the rest to the wide one — a boundary LDA could never find, because the means coincide.

**Question:** With d = 10 features and K = 3 classes, how many covariance entries does QDA estimate, compared with LDA?

- A. 30 vs 10
- B. 165 vs 55
- C. 300 vs 100
- D. 55 vs 55

**Answer:** B. 165 vs 55 — A symmetric $10\times10$ matrix has $10\cdot11/2=55$ free entries. QDA fits one per class ($3\times55=165$); LDA fits one shared matrix (55).

**Question:** In the worked example, x = (5, 5) is 2.00σ from the genuine mean but only 0.30σ from the forged mean. Why does QDA still predict genuine?

- A. Because the genuine class has the larger prior
- B. Because the forged class's volume penalty −½ log|Σ_B| = −2.016 is far below the genuine class's +0.693
- C. Because QDA uses Euclidean, not Mahalanobis, distance
- D. It doesn't — QDA predicts forged

**Answer:** B. Because the forged class's volume penalty −½ log|Σ_B| = −2.016 is far below the genuine class's +0.693 — The fit terms favour B ($-0.046$ vs $-2.000$) and so does the prior ($-0.606$ vs $-0.788$), but the volume penalties differ by $2.709$ in A's favour. Net: $\delta_A-\delta_B=0.573$, $P(A\mid x)=0.639$.

**Question:** What does QDA with reg_param = 1 reduce to?

- A. LDA
- B. Gaussian Naive Bayes
- C. A nearest-centroid rule (Euclidean distance) plus a log-prior bonus
- D. A model that always predicts the majority class

**Answer:** C. A nearest-centroid rule (Euclidean distance) plus a log-prior bonus — With $\lambda=1$ every $\Sigma_k=I$: $\log|I|=0$ and $\Delta_k^2=\|x-\mu_k\|^2$, so $\delta_k=-\tfrac12\|x-\mu_k\|^2+\log\pi_k$. In the example this drops training accuracy from 11/11 to 9/11.

**Key points**

- QDA models each class as $\mathcal N(\mu_k,\Sigma_k)$ with its own covariance and classifies by Bayes' rule.
- Score: $\delta_k(x)=-\tfrac12\log|\Sigma_k|-\tfrac12\Delta_k^2(x)+\log\pi_k$ — volume penalty, Mahalanobis fit, prior.
- Different covariances make the boundary quadratic (ellipse, parabola, hyperbola); equal covariances cancel the quadratic term and give LDA.
- Fitting is closed form — counts, means and per-class covariances — but needs $K\,d(d+1)/2$ covariance entries, so it overfits or goes singular with few samples per class.
- `reg_param` shrinks each covariance toward the identity; at 1 it becomes nearest-centroid. Tune it by cross-validation on standardized features.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
