Underfitting & overfitting

Underfitting means high training error; overfitting means low training error but a large gap to validation error, which early stopping and regularization close.

7 min read527 wordscore0 viewsupdated 2026-10-09

Definition

Underfitting and overfitting are the two ways a trained network can fail to generalize. An underfit model has high error on the training set itself: it has not captured the pattern. An overfit model has low training error but much higher validation error: it has fitted noise and quirks of the training sample that do not recur.

Two numbers diagnose it

You never see the true error on future data. You see the training loss and the validation loss. Training loss tells you whether the model can fit. The difference between the two tells you whether what it learned transfers.

gap=Lval−Ltrain\text{gap}=L_{\text{val}}-L_{\text{train}}
LtrainL_{\text{train}}
average loss on the examples the weights were fitted to
LvalL_{\text{val}}
average loss on held-out examples from the same distribution
gap\text{gap}
the generalization gap; near zero means what was learned transfers

High LtrainL_{\text{train}} (well above the noise floor, or above what a reasonable baseline reaches) means underfitting. Low LtrainL_{\text{train}} with a large gap means overfitting. Low train loss with a small gap is the goal. Both high, with the gap also large, happens too: then fix the fit first.

A network you can solve exactly

The figure fits a noisy 1-D curve with a one-hidden-layer network of KK Gaussian hidden units at fixed, evenly spaced centres. Only the output layer is learned, so training has a closed-form solution. KK is the capacity knob, and λ\lambda is weight decay (an L2 penalty).

f^(x)=b+∑k=1Kwk e−(x−ck)2/2h2,w^=arg⁡min⁡w,b 1n∑i=1n(f^(xi)−yi)2+λ∥w∥2\hat f(x)=b+\sum_{k=1}^{K}w_k\,e^{-(x-c_k)^2/2h^2},\qquad \hat w=\arg\min_{w,b}\ \frac1n\sum_{i=1}^{n}\big(\hat f(x_i)-y_i\big)^2+\lambda\lVert w\rVert^2
KK
number of hidden units (capacity)
ck, hc_k,\ h
fixed centres on [0, 1] and width = spacing between centres
wk, bw_k,\ b
learned output weights and bias (the bias is not penalised)
λ\lambda
weight decay strength
nn
number of training points (20, or 120 with “more data”)
train (20) · val · fit K = 6-2-101200.51dashed = true f · x →good fitMSE vs K: train · val0.010.11≥10noise floor σ² = 0.062516121824capacity (hidden units) →best valshaded bar = generalization gap (val − train) at chosen K · click a column to pick K
weight decay λ|
presets
train MSE 0.021 · val MSE 0.078 · gap 0.058
diagnosis good fit · best val at K = 6 (0.078)
24 widths, each refit exactly: K Gaussian hidden units, output layer solved by ridge least squares.
fig.Data: y = sin 2πx + 0.4 cos 5πx plus Gaussian noise with σ = 0.25 (noise floor σ² = 0.0625); the 200 validation points are the faint dots. Slide K from 1 to 24. At small K the curve cannot bend and both errors are high. At K = 6 validation error bottoms out. Beyond that, training error keeps falling while validation error climbs, and from K = 15 it explodes. Then press “overfit + λ” and “overfit + data” to see weight decay and more data close the gap at the same K.
Kλtrain MSEval MSEgapdiagnosis
200.3190.308−0.011underfit: training error 5× the noise floor
600.0210.0780.058good: lowest validation error of any K
1400.0100.6050.595overfit: train ↓, val ↑ 8×
140.010.0210.0710.051weight decay closes the gap
14 (120 pts)00.0700.062−0.009more data closes the gap
Worked example: numbers read off the figure (20 training points unless noted; MSE, noise floor 0.0625)

Note the third row: training error is the lowest of the five, and the model is the worst. Training loss alone can never tell you a network has overfit. Also note that the good model's training error (0.021) is below the noise floor. A small gap is normal and not a reason to stop.

Reading learning curves

In a real network, capacity grows during training as well as with size. Early on, train and val loss fall together. If training continues past the best point, train keeps falling while val turns upward: the gap opens epoch by epoch. Early stopping keeps the checkpoint at the validation minimum, which is why it counts as a regularizer.

symptomlikely causewhat to try
Train loss high and flat; val ≈ trainUnderfitting: too little capacity, too few steps, or a learning rate that is too low or too highBigger or deeper model, train longer, tune the learning rate, less regularization, better features or architecture
Train loss low; val much higher, rising late in trainingOverfitting: the network memorises the training sampleMore data, data augmentation, dropout, weight decay, early stopping, a smaller model
Train loss not decreasing at allA bug, not a capacity problemOverfit a single batch (below); check labels, loss and learning rate
Val lower than trainUsually dropout or augmentation active during training only, or an easy val splitCompare with train loss measured in eval mode; check the split for leakage
Diagnosis from learning curves

First, overfit a single batch

Before tuning anything, check that the model can memorise. Take one small batch and train on it repeatedly with regularization off. The loss should go to nearly zero within a few hundred steps. If it does not, the problem is a bug (wrong labels, a detached graph, a forgotten zero_grad, a bad learning rate), and no amount of data or regularization will fix it.

python
1import torch
2
3x, y = next(iter(train_loader)) # one batch, e.g. 32 examples
4model.train()
5opt = torch.optim.Adam(model.parameters(), lr=1e-3, weight_decay=0)
6for step in range(500):
7 opt.zero_grad()
8 loss = loss_fn(model(x), y)
9 loss.backward()
10 opt.step()
11print(loss.item()) # should be close to 0; if not, debug first
Sanity check: a working pipeline drives the loss on one batch close to zero

Check yourself

Training loss is 0.02 and still falling. Validation loss bottomed out at 0.31 at epoch 12 and is 0.45 at epoch 40. Which change is least likely to help?

Check yourself

In the worked example, why does K = 14 with λ = 0.01 generalize so much better than K = 14 with λ = 0?

Remember

  • ✓Underfit: training error is high. Overfit: training error is low but the gap to validation is large.
  • ✓Diagnose with both curves. Training loss alone can never reveal overfitting.
  • ✓Fix underfitting with capacity, training time and a tuned learning rate. Fix overfitting with data, augmentation, dropout, weight decay and early stopping.
  • ✓Fix underfitting first, because regularizers raise training error.
  • ✓Overfit one batch before anything else: if the network cannot memorise 32 examples, look for a bug.
← back to Deep Learning roadmap