# Bias–variance in deep networks

> Error = bias² + variance + noise, but very large networks break the classic U-curve: past the interpolation point, test error can fall again (double descent).

Source: https://www.mlfoundry.in/learn/bias-variance-in-deep-networks  
Section: Deep Learning › Deep Learning Fundamentals  
Level: core · 8 min read · updated 2026-10-09

**Double descent:** is the shape test error takes as model size grows past the point where the model can fit the training set exactly: error first follows the classical U, peaks at the **interpolation threshold** (parameters $\approx$ training points), then **falls again** as the model keeps growing.

## The decomposition, briefly

Train the same architecture on many independent training sets and average. The expected squared error at an input $x$ splits into three parts.

$$
\mathbb{E}\big[(y-\hat f(x))^2\big]=\underbrace{\big(\mathbb{E}[\hat f(x)]-f(x)\big)^2}_{\text{bias}^2}+\underbrace{\mathbb{E}\big[(\hat f(x)-\mathbb{E}[\hat f(x)])^2\big]}_{\text{variance}}+\sigma^2
$$

- $f(x)$: the true function; labels are $y=f(x)+\varepsilon$
- $\hat f(x)$: the trained model's prediction (random through the training set and, for networks, the initialization and data order)
- $\mathbb{E}$: average over those sources of randomness
- $\sigma^2$: variance of the label noise $\varepsilon$: irreducible

The classical story, told in full in the ML article on the bias–variance trade-off, says more capacity lowers bias and raises variance, so test error is U-shaped and you should pick the size at the bottom of the U.

Deep learning contradicts the advice, not the identity. Modern networks have far more parameters than training examples, reach (near) zero training loss, and still generalize well. Zhang et al. (2017) showed the same networks can even memorize randomly relabelled images. Parameter count alone clearly does not set the variance.

## Three regimes

| Regime | Capacity vs data | Training error | What dominates |
| --- | --- | --- | --- |
| Under-parameterized | $p < n$ | > 0 | bias at small $p$, variance as $p$ approaches $n$ |
| Interpolation threshold | $p \approx n$ | $\approx 0$, just barely | variance: exactly one fit through the points, and it is wild |
| Over-parameterized | $p \gg n$ | 0 | neither: many interpolants exist, and training picks a smooth one |

At $p=n$ there is (generically) exactly one way to pass through every noisy point. The model has no slack: to hit each label it needs huge, finely balanced weights, and small changes in the data swing the fit wildly. That is a variance spike.

For $p>n$ there are infinitely many interpolating solutions. Which one you get depends on the **training algorithm**, and the usual ones prefer small, smooth solutions.

## Implicit regularization: the minimum-norm solution

For linear least squares in features $\Phi\in\mathbb{R}^{n\times p}$ with $p>n$, gradient descent started at $\theta=0$ never leaves the row space of $\Phi$ (every gradient $\Phi^\top(\Phi\theta-y)$ lies in it). It therefore converges to the interpolant with the smallest norm.

$$
\hat\theta=\arg\min_{\theta}\ \|\theta\|_2\ \ \text{s.t.}\ \ \Phi\theta=y\quad\Longrightarrow\quad\hat\theta=\Phi^{+}y=\Phi^\top(\Phi\Phi^\top)^{-1}y
$$

- $\Phi$: feature matrix: $n$ training points, $p$ features
- $\Phi^{+}$: Moore–Penrose pseudo-inverse
- $\hat\theta$: the solution gradient descent from zero converges to

Example: one equation, two unknowns, $\theta_1+2\theta_2=5$. Every point on that line fits; the minimum-norm one is $\Phi^{+}y=(1,2)$ with $\|\theta\|=\sqrt5$.

As $p$ grows past $n$, the set of interpolants grows and the smallest norm among them **shrinks**. A smaller norm means a smoother function, so variance falls. This is the mechanism behind the second descent. Deep networks trained with SGD show the same bias toward "simple" solutions, though for nonlinear networks it is understood only partially (it depends on the architecture, initialization scale, learning rate and batch size).

> Figure: Random Fourier-features regression on n = 20 noisy samples of sin(πx), fitted with the minimum-norm least-squares solution. Right: median test MSE over 20 feature draws as the number of features p goes from 1 to 200 (log axes). Drag the slider or click the chart to choose p, and watch the fit on the left: smooth at small p, wild at p = 20, smooth again at p = 200 while still passing through every point. Switch to λ = 0.1 (ridge) and the peak disappears.

## Worked example: the numbers behind the figure

Medians over 20 random feature draws. Train MSE is against the noisy labels, test MSE against the noise-free sin(πx) on [−1, 1]. Ridge column uses λ = 0.1.

| p | train MSE | test MSE (min-norm) | ‖θ‖ (min-norm) | test MSE (ridge) |
| --- | --- | --- | --- | --- |
| 5 | 0.502 | 0.479 | 0.8 | 0.477 |
| 10 | 0.212 | 0.259 | 5.0 | 0.292 |
| 16 | 0.011 | 0.064 | 30.2 | 0.126 |
| 20 = n | ≈ 0 | 1.329 | 533.4 | 0.046 |
| 25 | ≈ 0 | 0.412 | 14.3 | 0.050 |
| 40 | ≈ 0 | 0.072 | 4.2 | 0.044 |
| 200 | ≈ 0 | 0.032 | 3.1 | 0.029 |

1. The classical sweet spot is $p=16$: test MSE $0.064$.
2. At $p=n=20$ the fit interpolates for the first time; $\|\theta\|$ jumps about $18\times$ and test MSE is about $20\times$ worse than at $p=16$.
3. By $p=200$ the minimum-norm interpolant has $\|\theta\|=3.1$ and test MSE $0.032$, **half** the classical optimum, with zero training error.
4. Ridge at $\lambda=0.1$ removes the spike entirely (0.046 at $p=n$): the peak is a symptom of under-regularized interpolation, not of size itself.

## Double descent in deep networks

**Belkin et al. (2019)** named the phenomenon and showed it for random-feature models, random forests and small networks: the U-curve on the left, a peak at the interpolation threshold, a second descent on the right.

**Nakkiran et al. (2019)**, "Deep Double Descent", showed it in ResNets and transformers along three axes:

- **Model-wise:** fix the data and training budget, grow the width: test error peaks where the network can just barely fit the training set.
- **Epoch-wise:** fix a large model, train longer: test error falls, rises, then falls again. Early stopping at the first dip is not always best.
- **Sample-wise:** near the threshold, **more data can hurt**, because adding samples moves the threshold toward the current model size.

They tie all three to **effective model complexity**: the largest training set a given model and training procedure can fit to near-zero error. The peak sits where that number crosses $n$. Label noise makes the peak much taller; with clean labels it can be faint or invisible.

> **The identity never breaks:** bias² + variance + noise always holds. What breaks is the assumption that parameter count measures complexity. Measured directly in wide networks (Yang et al., 2020), bias keeps falling with width while variance rises and then falls again, so the total can keep improving.

## Practical consequences

- **Go big, then regularize:** A larger network with weight decay, data augmentation and early stopping usually beats a carefully "right-sized" one. Being just big enough to memorize is the worst place to be.
- **Tune regularization with size:** Weight decay and augmentation flatten the peak, as ridge does in the figure. If error rises as you grow the model, add regularization before shrinking it.
- **Keep a validation set:** Double descent is a tendency, not a guarantee. The curve depends on noise, data size and training procedure; only validation error tells you where your model sits.
- **Train long enough:** Epoch-wise double descent means a rising validation curve may still turn around. Patience-based early stopping with a generous window, or checkpoint selection, handles both cases.

```python
import torch

torch.manual_seed(0)
n, p = 20, 200
x = torch.linspace(-0.95, 0.95, n, dtype=torch.float64)
y = torch.sin(torch.pi * x) + 0.2 * torch.randn(n, dtype=torch.float64)
w, b = 20 * torch.randn(p, dtype=torch.float64), 2 * torch.pi * torch.rand(p, dtype=torch.float64)
Phi = (2 / p) ** 0.5 * torch.cos(x[:, None] * w + b)        # n x p, p > n

theta_pinv = torch.linalg.pinv(Phi) @ y                     # minimum-norm interpolant

theta = torch.zeros(p, dtype=torch.float64, requires_grad=True)
opt = torch.optim.SGD([theta], lr=1 / torch.linalg.matrix_norm(Phi, 2).item() ** 2)
for _ in range(20_000):
    opt.zero_grad()
    (0.5 * (Phi @ theta - y).pow(2).sum()).backward()
    opt.step()

print((theta - theta_pinv).norm() / theta_pinv.norm())    # ≈ 0: the same solution
```

Minimum-norm least squares = gradient descent from zero (verified with the numpy equivalent: relative difference ~1e-15).

> **Reading the peak as "too big":** If test error gets worse when you widen a network, you may be sitting on the interpolation peak, not past the sweet spot of a U. Try a much larger model with regularization, or more regularization at the current size, before shrinking it.

**Question:** In the figure (n = 20 training points, minimum-norm fit), where is the median test error largest?

- A. p = 1
- B. p = 10
- C. p = 20
- D. p = 200

**Answer:** C. p = 20 — At $p=n=20$ the model can fit every noisy point in exactly one way, which needs a huge weight norm (533 vs 3.1 at $p=200$). Test MSE peaks at 1.33 there.

**Question:** Past the interpolation threshold, training error is already zero. Why does test error keep falling as p grows?

- A. The label noise shrinks with more features
- B. The minimum-norm interpolant gets a smaller norm, so the fitted function is smoother and its variance drops
- C. Bias increases, which cancels the variance
- D. More features always reduce bias, and bias is the only term that matters past p = n

**Answer:** B. The minimum-norm interpolant gets a smaller norm, so the fitted function is smoother and its variance drops — More features means more interpolating solutions to choose from, so the smallest-norm one gets smaller (3.1 at $p=200$ vs 533 at $p=20$). Smoother fits vary less between training sets. Noise $\sigma^2$ is fixed, and variance, not bias, is what drives the peak and its decline.

**Key points**

- Error $=$ bias² $+$ variance $+$ $\sigma^2$ always; parameter count is just a poor measure of complexity.
- Test error peaks at the interpolation threshold ($p\approx n$, or "just able to fit the training set") and falls again beyond it.
- Gradient descent from small initialization picks low-norm, smooth interpolants: implicit regularization.
- Double descent appears model-wise, epoch-wise and sample-wise (Nakkiran et al.); label noise makes it worse.
- In practice: big and regularized beats right-sized, and validation error decides.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
