Definition
Underfitting and overfitting are the two ways a trained network can fail to generalize. An underfit model has high error on the training set itself: it has not captured the pattern. An overfit model has low training error but much higher validation error: it has fitted noise and quirks of the training sample that do not recur.
Two numbers diagnose it
You never see the true error on future data. You see the training loss and the validation loss. Training loss tells you whether the model can fit. The difference between the two tells you whether what it learned transfers.
- average loss on the examples the weights were fitted to
- average loss on held-out examples from the same distribution
- the generalization gap; near zero means what was learned transfers
High (well above the noise floor, or above what a reasonable baseline reaches) means underfitting. Low with a large gap means overfitting. Low train loss with a small gap is the goal. Both high, with the gap also large, happens too: then fix the fit first.
A network you can solve exactly
The figure fits a noisy 1-D curve with a one-hidden-layer network of Gaussian hidden units at fixed, evenly spaced centres. Only the output layer is learned, so training has a closed-form solution. is the capacity knob, and is weight decay (an L2 penalty).
- number of hidden units (capacity)
- fixed centres on [0, 1] and width = spacing between centres
- learned output weights and bias (the bias is not penalised)
- weight decay strength
- number of training points (20, or 120 with “more data”)
| K | λ | train MSE | val MSE | gap | diagnosis |
|---|---|---|---|---|---|
| 2 | 0 | 0.319 | 0.308 | −0.011 | underfit: training error 5× the noise floor |
| 6 | 0 | 0.021 | 0.078 | 0.058 | good: lowest validation error of any K |
| 14 | 0 | 0.010 | 0.605 | 0.595 | overfit: train ↓, val ↑ 8× |
| 14 | 0.01 | 0.021 | 0.071 | 0.051 | weight decay closes the gap |
| 14 (120 pts) | 0 | 0.070 | 0.062 | −0.009 | more data closes the gap |
Note the third row: training error is the lowest of the five, and the model is the worst. Training loss alone can never tell you a network has overfit. Also note that the good model's training error (0.021) is below the noise floor. A small gap is normal and not a reason to stop.
Reading learning curves
In a real network, capacity grows during training as well as with size. Early on, train and val loss fall together. If training continues past the best point, train keeps falling while val turns upward: the gap opens epoch by epoch. Early stopping keeps the checkpoint at the validation minimum, which is why it counts as a regularizer.
| symptom | likely cause | what to try |
|---|---|---|
| Train loss high and flat; val ≈ train | Underfitting: too little capacity, too few steps, or a learning rate that is too low or too high | Bigger or deeper model, train longer, tune the learning rate, less regularization, better features or architecture |
| Train loss low; val much higher, rising late in training | Overfitting: the network memorises the training sample | More data, data augmentation, dropout, weight decay, early stopping, a smaller model |
| Train loss not decreasing at all | A bug, not a capacity problem | Overfit a single batch (below); check labels, loss and learning rate |
| Val lower than train | Usually dropout or augmentation active during training only, or an easy val split | Compare with train loss measured in eval mode; check the split for leakage |
First, overfit a single batch
Before tuning anything, check that the model can memorise. Take one small batch and train on it repeatedly with regularization off. The loss should go to nearly zero within a few hundred steps. If it does not, the problem is a bug (wrong labels, a detached graph, a forgotten zero_grad, a bad learning rate), and no amount of data or regularization will fix it.
1import torch23x, y = next(iter(train_loader)) # one batch, e.g. 32 examples4model.train()5opt = torch.optim.Adam(model.parameters(), lr=1e-3, weight_decay=0)6for step in range(500):7 opt.zero_grad()8 loss = loss_fn(model(x), y)9 loss.backward()10 opt.step()11print(loss.item()) # should be close to 0; if not, debug firstCheck yourself
Training loss is 0.02 and still falling. Validation loss bottomed out at 0.31 at epoch 12 and is 0.45 at epoch 40. Which change is least likely to help?
Check yourself
In the worked example, why does K = 14 with λ = 0.01 generalize so much better than K = 14 with λ = 0?
Remember
- ✓Underfit: training error is high. Overfit: training error is low but the gap to validation is large.
- ✓Diagnose with both curves. Training loss alone can never reveal overfitting.
- ✓Fix underfitting with capacity, training time and a tuned learning rate. Fix overfitting with data, augmentation, dropout, weight decay and early stopping.
- ✓Fix underfitting first, because regularizers raise training error.
- ✓Overfit one batch before anything else: if the network cannot memorise 32 examples, look for a bug.