Definition
The bias–variance decomposition splits the expected squared error of a learning algorithm at a point , averaged over random training sets and over the noise, into three parts: bias² (how far the average fitted model is from the truth), variance (how much the fitted model moves from one training set to the next) and irreducible noise (what no model can predict).
At a glance
- Applies to
- squared-error regression exactly; classification losses have analogous but messier decompositions
- Random over
- the training set (and the test noise), with the test input held fixed
- Bias²
- : systematic error, underfitting
- Variance
- : sensitivity to the sample, overfitting
- Noise
- : a floor no model beats
- Knobs
- polynomial degree, k in kNN, tree depth, regularization λ, number of features, ensemble size, training-set size n
- Diagnosed by
- training vs validation error, validation curves, learning curves
The intuition: four dartboards
Imagine you could rerun your whole pipeline on many fresh training sets from the same source, and each time throw one dart: the trained model's prediction at a fixed test input. The bullseye is the true value . Two different things can go wrong.
| Dartboard | Low variance (tight cluster) | High variance (scattered) |
|---|---|---|
| Low bias (centred on the bullseye) | the goal: right on average, and every time | right on average, but any single model can be far off |
| High bias (centred off the bullseye) | consistently wrong in the same way | wrong on average and unreliable |
Bias is the distance from the centre of the cluster to the bullseye. It comes from assumptions the model cannot escape: a straight line fitted to a sine wave misses the curve no matter how much data you give it. That is underfitting: training and test error are both high.
Variance is the spread of the cluster. A very flexible model, such as a degree-9 polynomial through 10 points, bends to follow whatever noise happened to land in its training set, so the curves from different sets disagree wildly. That is overfitting: training error is tiny, test error is large.
Making a model more flexible usually moves bias down and variance up. The trade-off is choosing the flexibility where their sum is smallest. The noise adds a fixed amount on top of every model.
Setup and notation
Data come from with , , noise independent of everything else. A learning algorithm maps a training set to a function . Fix a test input and draw a fresh test label , independent of . We want the expected prediction error at , averaged over both sources of randomness:
- true regression function,
- prediction of the model trained on training set (random, because is)
- the average prediction over all possible training sets: the centre of the dart cluster
- test-label noise, mean 0, variance , independent of
Deriving the decomposition
The whole proof is "add and subtract the right thing, expand the square, and show the cross terms have mean zero". We do it twice: once to peel off the noise, once to split the rest into bias and variance. Write for .
Step 1: separate the noise
- 1Add and subtract :
- 2Square:
- 3Take expectations. . The cross term factorizes because the test noise is independent of the training set and hence of :
- 4So : noise plus the mean squared error of the estimator as an estimate of the number .
Step 2: split the estimator's error
- 1Add and subtract the average prediction :
- 2Square:
- 3The first factor of the cross term, , is a constant (no randomness left in it), so it comes out of the expectation:
- 4What remains: , which is bias² plus variance.
- squared distance from the average model to the truth; zero if the model class can represent f and the method is unbiased
- spread of the fitted models around their own average; zero for a model that ignores the data
- irreducible error: even the true has expected squared error
Two facts make the cross terms vanish: the test noise is independent of the training set, and is the mean of , so deviations from it average to zero. All three terms are non-negative, so no model can beat . To get the error over the whole input distribution, average over :
See it: many training sets, one picture
The figure below runs the thought experiment literally. The true function is on , with noise . Each training set has inputs on a fixed, equally spaced grid ; only the noise is redrawn. For each of 30 training sets the model is refitted, and every fit is drawn. The bias, variance and total on the right are exact (computed in closed form, see below), averaged over 201 test points.
Worked example: polynomial degree 1, 3 and 9
Take the setting from the figure with training points, , and fit least-squares polynomials of degree 1, 3 and 9. Two ways to get the numbers. Monte Carlo: draw 20,000 training sets, fit each one, then measure the spread and the offset of the 20,000 predictions at every test point. Closed form: least squares is linear in , so the mean and variance of follow from one matrix (next section). Both agree:
| Degree | Bias² | Variance | Noise σ² | Total (exact) | MC bias² / var | MC direct test MSE | Verdict |
|---|---|---|---|---|---|---|---|
| 1 | 0.2235 | 0.0164 | 0.09 | 0.3299 | 0.2238 / 0.0163 | 0.3299 | underfits: bias dominates |
| 3 | 0.0066 | 0.0299 | 0.09 | 0.1265 | 0.0066 / 0.0298 | 0.1264 | best of the three |
| 9 | < 0.0001 | 0.5874 | 0.09 | 0.6774 | 0.0001 / 0.5768 | 0.6673 | overfits: variance dominates |
Reading the table: the line has bias² 0.22, so even with infinite data its error would stay near . The degree-9 polynomial is essentially unbiased (on average it reproduces the sine to within ) yet its variance is 0.59, 6.5 times the noise. The cubic pays a small bias (0.007) for a small variance (0.03) and wins with a total of 0.127, only 0.037 above the floor . Sweeping every degree, the minimum is at degree 3.
| Test point | Degree | f(x) | Mean fit f̄(x) | Bias² | Variance |
|---|---|---|---|---|---|
| x = 0.25 (peak) | 1 | 1.000 | 0.337 | 0.4393 | 0.0145 |
| x = 0.25 | 3 | 1.000 | 0.961 | 0.0015 | 0.0298 |
| x = 0.25 | 9 | 1.000 | 1.000 | 0.0000 | 0.0866 |
| x = 0.95 (edge) | 1 | −0.309 | −0.607 | 0.0888 | 0.0269 |
| x = 0.95 | 3 | −0.309 | −0.402 | 0.0086 | 0.0397 |
| x = 0.95 | 9 | −0.309 | −0.309 | 0.0000 | 3.9322 |
Two curiosities show up when you sweep all degrees. Degree 2 has exactly the same bias as degree 1 (0.2235), and degree 4 the same as degree 3: is antisymmetric about and the design is symmetric, so the added even term has expected coefficient 0 and only adds variance. And the training error of degree 9 is exactly 0 (10 coefficients, 10 points), while its test error is 0.68, the worst in the table.
| n | Degree 3: bias² | Degree 3: variance | Degree 3: total | Degree 9: variance | Degree 9: total |
|---|---|---|---|---|---|
| 10 | 0.0066 | 0.0299 | 0.1265 | 0.5874 | 0.6774 |
| 15 | 0.0056 | 0.0209 | 0.1165 | 0.0638 | 0.1538 |
| 25 | 0.0050 | 0.0132 | 0.1081 | 0.0329 | 0.1229 |
| 50 | 0.0047 | 0.0069 | 0.1016 | 0.0166 | 0.1066 |
At the cubic beats degree 9 by a factor of five; at the gap is under 5%. This is why "the right complexity" is not a property of the problem alone: it depends on how much data you have. The best degree drifts upward as grows.
Closed forms: linear smoothers
Least squares, ridge and kNN all predict with a fixed linear combination of the training labels. With the inputs held fixed (only the noise is random), write
- weight the method puts on training label when predicting at (depends on the inputs only)
- ; independent noise makes variances add with squared weights
The mean follows from and linearity; the variance from for independent noise. This is exactly how the figure and the "exact" column above are computed. Three cases are worth working out by hand.
Least squares: variance = σ² p / n
For a model with features (here polynomial coefficients) and design matrix , the fitted values at the training inputs are with the hat matrix . is symmetric and idempotent (), so the squared weights in row sum to . Averaging over the training inputs:
- hat matrix, projection onto the column space of Φ
- trace of a rank-p projection; the number of parameters
Every extra parameter costs of in-sample variance, whether or not it is needed. Checked numerically for : degree 1, 3, 9 give 0.018, 0.036, 0.090, exactly . Away from the training inputs variance is larger (0.59 rather than 0.09 for degree 9), because polynomials extrapolate badly between and beyond the outer points.
kNN regression: variance = σ² / k
kNN regression averages the labels of the training points nearest to , so on those neighbours and 0 elsewhere:
- indices of the training inputs closest to
- variance: squared weights of , times , exact for any
The variance is exact and does not depend on . The bias grows with because the neighbourhood reaches further from . In 1-D with spacing and smooth, a Taylor expansion gives bias for an interior point. So the bias grows like while the variance falls like . Small is the flexible end of the dial.
| k | Bias² | Variance = σ²/k | Total |
|---|---|---|---|
| 1 | 0.0007 | 0.0900 | 0.1807 |
| 3 | 0.0009 | 0.0300 | 0.1209 |
| 5 | 0.0027 | 0.0180 | 0.1107 |
| 10 | 0.0180 | 0.0090 | 0.1170 |
| 20 | 0.0659 | 0.0045 | 0.1604 |
| 50 (= n) | 0.4975 | 0.0018 | 0.5893 |
At every prediction is the global mean; its bias² is the variance of over the grid, close to . The minimum sits at –6 (total 0.109 at ).
Ridge: variance falls as λ grows
Ridge regression fits . Take the SVD with singular values . The fitted values are with : ridge shrinks the component of along each direction by . With , the in-sample terms are
- singular values and left singular vectors of the design matrix
- , the fraction of signal along that ridge throws away
- the part of f the features cannot represent at all; no λ fixes it
- , effective number of parameters
Derivation in one line each: , and scales the component by and keeps the part outside the column space. , whose trace is . As goes from 0 to , every falls from 1 to 0: variance decreases monotonically and bias increases monotonically. At this is the OLS result . Directions with small (poorly determined by the data) are shrunk first, which is where most of the variance lived.
| λ | df(λ) | In-sample bias² | In-sample variance | Test-grid total |
|---|---|---|---|---|
| 0 (OLS) | 10.00 | 0.0000 | 0.0900 | 0.6774 |
| 0.01 | 9.55 | 0.0000 | 0.0829 | 0.4134 |
| 0.1 | 8.26 | 0.0012 | 0.0669 | 0.1825 |
| 0.3 | 7.15 | 0.0082 | 0.0533 | 0.1530 |
| 1 | 5.42 | 0.0497 | 0.0345 | 0.1724 |
| 10 | 1.97 | 0.3038 | 0.0069 | 0.4278 |
A modest penalty cuts the degree-9 test error from 0.68 to 0.15, close to the cubic's 0.127, without touching the degree. Regularization is a second, continuous complexity dial. See the L1/L2 regularization article for the geometry.
A modern caveat: double descent
The U-shaped total-error curve is the classical picture for models whose capacity is below the number of samples. Large neural networks, random-feature models and even plain least squares behave differently once the parameter count passes . Test error rises to a peak at the interpolation threshold (where the fit is forced through every point and is extremely unstable, as degree 9 at is above), and then falls again as grows further. With more parameters than points there are infinitely many interpolating solutions. Gradient descent and the minimum-norm solver pick the smoothest one, an implicit regularization that lowers variance.
Diagnosing bias and variance in practice
On real data you cannot redraw training sets or see , so bias and variance are never measured directly. You read them off two error curves instead: training error (an optimistic proxy that ignores variance) and validation error (an honest estimate of ).
Reading the curves
- Validation curve
- error vs a complexity knob at fixed n. Training error falls steadily; validation error is U-shaped. Left of the minimum is the high-bias region, right of it the high-variance region.
- Learning curve
- error vs training-set size n at fixed complexity. Training error rises with n (harder to memorize), validation error falls; they converge to the model's bias² + σ².
- High bias
- both curves plateau early, close together, at an unacceptably high level. More data will not help.
- High variance
- a large gap between training and validation error that narrows slowly as n grows. More data will help.
| Symptom | Diagnosis | Remedies | Side effect |
|---|---|---|---|
| Train and validation error both high, close together | high bias (underfitting) | more flexible model (higher degree, deeper trees, smaller k), more or better features, interactions, less regularization, train longer / boosting | variance rises |
| Train error low, validation error much higher | high variance (overfitting) | more training data, stronger regularization (ridge, lasso, dropout, weight decay), simpler model, feature selection, bagging / random forests, early stopping, data augmentation | bias rises (except more data and bagging, which mostly leave it alone) |
| Both low and close | near the noise floor | stop. Further gains need less noisy labels or new informative features | none |
In code
1import numpy as np23f = lambda x: np.sin(2 * np.pi * x)4sigma, n, R = 0.3, 10, 20_0005x_train = np.linspace(0, 1, n) # fixed design: only the noise is redrawn6x_test = np.linspace(0, 1, 201)7rng = np.random.default_rng(1)89def fit_predict(y, degree):10 coef = np.polynomial.legendre.legfit(2 * x_train - 1, y, degree) # least squares11 return np.polynomial.legendre.legval(2 * x_test - 1, coef)1213for degree in [1, 3, 9]:14 Y = f(x_train) + sigma * rng.standard_normal((R, n)) # R training sets15 F = np.array([fit_predict(y, degree) for y in Y]) # R x 201 predictions16 bias2 = ((F.mean(0) - f(x_test)) ** 2).mean()17 var = F.var(0).mean()18 y_new = f(x_test) + sigma * rng.standard_normal(F.shape) # fresh test noise19 mse = ((y_new - F) ** 2).mean()20 print(f"degree {degree}: bias^2 {bias2:.4f} var {var:.4f} "21 f"sum {bias2 + var + sigma**2:.4f} direct test MSE {mse:.4f}")2223# Closed form: every fit is linear in y, f_hat(x_test) = L @ y24def smoother(degree):25 P = np.polynomial.legendre.legvander(2 * x_train - 1, degree)26 G = np.polynomial.legendre.legvander(2 * x_test - 1, degree)27 return G @ np.linalg.pinv(P)2829for degree in [1, 3, 9]:30 L = smoother(degree)31 bias2 = ((L @ f(x_train) - f(x_test)) ** 2).mean()32 var = sigma**2 * (L**2).sum(1).mean()33 print(f"degree {degree}: exact bias^2 {bias2:.4f} var {var:.4f} total {bias2 + var + sigma**2:.4f}")1import numpy as np2from sklearn.pipeline import make_pipeline3from sklearn.preprocessing import PolynomialFeatures, StandardScaler4from sklearn.linear_model import LinearRegression5from sklearn.model_selection import validation_curve, learning_curve, KFold67rng = np.random.default_rng(0)8X = rng.uniform(0, 1, size=(60, 1))9y = np.sin(2 * np.pi * X[:, 0]) + rng.normal(0, 0.3, size=60)1011model = make_pipeline(PolynomialFeatures(include_bias=False), StandardScaler(), LinearRegression())12cv = KFold(5, shuffle=True, random_state=0)13degrees = np.arange(1, 13)1415train, val = validation_curve(model, X, y, param_name="polynomialfeatures__degree",16 param_range=degrees, cv=cv, scoring="neg_mean_squared_error")17for d, tr, va in zip(degrees, -train.mean(1), -val.mean(1)):18 print(f"degree {d:2d} train MSE {tr:.3f} val MSE {va:.3f}")1920for d in [1, 9]: # learning curves: high bias vs high variance21 model.set_params(polynomialfeatures__degree=d)22 sizes, tr, va = learning_curve(model, X, y, train_sizes=[12, 24, 36, 48],23 cv=cv, scoring="neg_mean_squared_error")24 print(d, sizes, (-tr.mean(1)).round(3), (-va.mean(1)).round(3))Use when
- +Choosing model complexity: it explains why validation error is U-shaped and where to look for the minimum.
- +Deciding what to try next: more data and bagging help high variance; features and flexibility help high bias. The decomposition tells you which.
- +Understanding ensembles and regularization: bagging, random forests, ridge, dropout and early stopping are all variance reduction; boosting is bias reduction.
Avoid when
- −It is exact only for squared error. 0–1 loss has no clean additive split, and bias and variance can interact (more variance can even help a biased classifier).
- −You cannot compute it on real data: it needs the true f and many independent training sets. Use validation and learning curves instead.
- −Parameter count is a poor complexity measure for overparameterized models (double descent). Use effective degrees of freedom or norms instead.
Where it's used
Hyperparameter tuning
Every validation curve over depth, k, λ, degree or number of epochs is a bias–variance sweep.
Data-collection decisions
A learning curve that is still falling says more data will pay; a flat one says spend the effort on features or a richer model.
Ensemble design
Bag low-bias, high-variance learners (deep trees); boost high-bias, low-variance ones (stumps).
Small-data science
With few samples, strong priors and regularization (Bayesian or ridge models) trade a little bias for a large cut in variance.
Check yourself
Training MSE is 0.02 and validation MSE is 0.35; the label noise is believed to be about 0.05. What should you try first?
Check yourself
In the derivation, why does the cross term vanish?
Check yourself
kNN regression with noise variance . What is the variance of the prediction at any point when ?
Check yourself
You double the training-set size and keep the same polynomial degree. What happens, in expectation?
Remember
- ✓Expected squared error at = bias² + variance + . Derive it by adding and subtracting , then . The cross terms vanish by independence of the test noise and by the definition of .
- ✓Bias is the error of the average model (underfitting); variance is the spread across training sets (overfitting); σ² is a floor no model beats.
- ✓Closed forms for linear smoothers: least squares has in-sample variance , kNN has variance exactly with bias growing in , and ridge trades variance for bias monotonically as grows.
- ✓On sin(2πx) with n = 10, σ = 0.3: degree 1 totals 0.330 (bias), degree 9 totals 0.677 (variance), degree 3 is best at 0.127.
- ✓More data and bagging cut variance without adding bias; diagnose with validation and learning curves, never with training error.
- ✓Double descent: past the interpolation threshold, parameter count stops measuring complexity, but the decomposition itself still holds.