Definition
Double descent is the shape test error takes as model size grows past the point where the model can fit the training set exactly: error first follows the classical U, peaks at the interpolation threshold (parameters training points), then falls again as the model keeps growing.
The decomposition, briefly
Train the same architecture on many independent training sets and average. The expected squared error at an input splits into three parts.
- the true function; labels are
- the trained model's prediction (random through the training set and, for networks, the initialization and data order)
- average over those sources of randomness
- variance of the label noise : irreducible
The classical story, told in full in the ML article on the bias–variance trade-off, says more capacity lowers bias and raises variance, so test error is U-shaped and you should pick the size at the bottom of the U.
Deep learning contradicts the advice, not the identity. Modern networks have far more parameters than training examples, reach (near) zero training loss, and still generalize well. Zhang et al. (2017) showed the same networks can even memorize randomly relabelled images. Parameter count alone clearly does not set the variance.
Three regimes
| Regime | Capacity vs data | Training error | What dominates |
|---|---|---|---|
| Under-parameterized | > 0 | bias at small , variance as approaches | |
| Interpolation threshold | , just barely | variance: exactly one fit through the points, and it is wild | |
| Over-parameterized | 0 | neither: many interpolants exist, and training picks a smooth one |
At there is (generically) exactly one way to pass through every noisy point. The model has no slack: to hit each label it needs huge, finely balanced weights, and small changes in the data swing the fit wildly. That is a variance spike.
For there are infinitely many interpolating solutions. Which one you get depends on the training algorithm, and the usual ones prefer small, smooth solutions.
Implicit regularization: the minimum-norm solution
For linear least squares in features with , gradient descent started at never leaves the row space of (every gradient lies in it). It therefore converges to the interpolant with the smallest norm.
- feature matrix: training points, features
- Moore–Penrose pseudo-inverse
- the solution gradient descent from zero converges to
Example: one equation, two unknowns, . Every point on that line fits; the minimum-norm one is with .
As grows past , the set of interpolants grows and the smallest norm among them shrinks. A smaller norm means a smoother function, so variance falls. This is the mechanism behind the second descent. Deep networks trained with SGD show the same bias toward "simple" solutions, though for nonlinear networks it is understood only partially (it depends on the architecture, initialization scale, learning rate and batch size).
Worked example: the numbers behind the figure
| p | train MSE | test MSE (min-norm) | ‖θ‖ (min-norm) | test MSE (ridge) |
|---|---|---|---|---|
| 5 | 0.502 | 0.479 | 0.8 | 0.477 |
| 10 | 0.212 | 0.259 | 5.0 | 0.292 |
| 16 | 0.011 | 0.064 | 30.2 | 0.126 |
| 20 = n | ≈ 0 | 1.329 | 533.4 | 0.046 |
| 25 | ≈ 0 | 0.412 | 14.3 | 0.050 |
| 40 | ≈ 0 | 0.072 | 4.2 | 0.044 |
| 200 | ≈ 0 | 0.032 | 3.1 | 0.029 |
- 1The classical sweet spot is : test MSE .
- 2At the fit interpolates for the first time; jumps about and test MSE is about worse than at .
- 3By the minimum-norm interpolant has and test MSE , half the classical optimum, with zero training error.
- 4Ridge at removes the spike entirely (0.046 at ): the peak is a symptom of under-regularized interpolation, not of size itself.
Double descent in deep networks
Belkin et al. (2019) named the phenomenon and showed it for random-feature models, random forests and small networks: the U-curve on the left, a peak at the interpolation threshold, a second descent on the right.
Nakkiran et al. (2019), "Deep Double Descent", showed it in ResNets and transformers along three axes:
At a glance
- Model-wise
- fix the data and training budget, grow the width: test error peaks where the network can just barely fit the training set.
- Epoch-wise
- fix a large model, train longer: test error falls, rises, then falls again. Early stopping at the first dip is not always best.
- Sample-wise
- near the threshold, more data can hurt, because adding samples moves the threshold toward the current model size.
They tie all three to effective model complexity: the largest training set a given model and training procedure can fit to near-zero error. The peak sits where that number crosses . Label noise makes the peak much taller; with clean labels it can be faint or invisible.
Practical consequences
Where it's used
Go big, then regularize
A larger network with weight decay, data augmentation and early stopping usually beats a carefully "right-sized" one. Being just big enough to memorize is the worst place to be.
Tune regularization with size
Weight decay and augmentation flatten the peak, as ridge does in the figure. If error rises as you grow the model, add regularization before shrinking it.
Keep a validation set
Double descent is a tendency, not a guarantee. The curve depends on noise, data size and training procedure; only validation error tells you where your model sits.
Train long enough
Epoch-wise double descent means a rising validation curve may still turn around. Patience-based early stopping with a generous window, or checkpoint selection, handles both cases.
1import torch23torch.manual_seed(0)4n, p = 20, 2005x = torch.linspace(-0.95, 0.95, n, dtype=torch.float64)6y = torch.sin(torch.pi * x) + 0.2 * torch.randn(n, dtype=torch.float64)7w, b = 20 * torch.randn(p, dtype=torch.float64), 2 * torch.pi * torch.rand(p, dtype=torch.float64)8Phi = (2 / p) ** 0.5 * torch.cos(x[:, None] * w + b) # n x p, p > n910theta_pinv = torch.linalg.pinv(Phi) @ y # minimum-norm interpolant1112theta = torch.zeros(p, dtype=torch.float64, requires_grad=True)13opt = torch.optim.SGD([theta], lr=1 / torch.linalg.matrix_norm(Phi, 2).item() ** 2)14for _ in range(20_000):15 opt.zero_grad()16 (0.5 * (Phi @ theta - y).pow(2).sum()).backward()17 opt.step()1819print((theta - theta_pinv).norm() / theta_pinv.norm()) # ≈ 0: the same solutionCheck yourself
In the figure (n = 20 training points, minimum-norm fit), where is the median test error largest?
Check yourself
Past the interpolation threshold, training error is already zero. Why does test error keep falling as p grows?
Remember
- ✓Error bias² variance always; parameter count is just a poor measure of complexity.
- ✓Test error peaks at the interpolation threshold (, or "just able to fit the training set") and falls again beyond it.
- ✓Gradient descent from small initialization picks low-norm, smooth interpolants: implicit regularization.
- ✓Double descent appears model-wise, epoch-wise and sample-wise (Nakkiran et al.); label noise makes it worse.
- ✓In practice: big and regularized beats right-sized, and validation error decides.