# Underfitting & overfitting

> Underfitting means high training error; overfitting means low training error but a large gap to validation error, which early stopping and regularization close.

Source: https://www.mlfoundry.in/learn/underfitting-overfitting  
Section: Deep Learning › Deep Learning Fundamentals  
Level: core · 7 min read · updated 2026-10-09

**Underfitting and overfitting:** are the two ways a trained network can fail to generalize. An **underfit** model has high error on the training set itself: it has not captured the pattern. An **overfit** model has low training error but much higher validation error: it has fitted noise and quirks of the training sample that do not recur.

## Two numbers diagnose it

You never see the true error on future data. You see the training loss and the validation loss. Training loss tells you whether the model *can* fit. The difference between the two tells you whether what it learned *transfers*.

$$
\text{gap}=L_{\text{val}}-L_{\text{train}}
$$

- $L_{\text{train}}$: average loss on the examples the weights were fitted to
- $L_{\text{val}}$: average loss on held-out examples from the same distribution
- $\text{gap}$: the generalization gap; near zero means what was learned transfers

High $L_{\text{train}}$ (well above the noise floor, or above what a reasonable baseline reaches) means underfitting. Low $L_{\text{train}}$ with a large gap means overfitting. Low train loss with a small gap is the goal. Both high, with the gap also large, happens too: then fix the fit first.

## A network you can solve exactly

The figure fits a noisy 1-D curve with a one-hidden-layer network of $K$ Gaussian hidden units at fixed, evenly spaced centres. Only the output layer is learned, so training has a closed-form solution. $K$ is the capacity knob, and $\lambda$ is weight decay (an L2 penalty).

$$
\hat f(x)=b+\sum_{k=1}^{K}w_k\,e^{-(x-c_k)^2/2h^2},\qquad \hat w=\arg\min_{w,b}\ \frac1n\sum_{i=1}^{n}\big(\hat f(x_i)-y_i\big)^2+\lambda\lVert w\rVert^2
$$

- $K$: number of hidden units (capacity)
- $c_k,\ h$: fixed centres on [0, 1] and width = spacing between centres
- $w_k,\ b$: learned output weights and bias (the bias is not penalised)
- $\lambda$: weight decay strength
- $n$: number of training points (20, or 120 with “more data”)

> Figure: Data: y = sin 2πx + 0.4 cos 5πx plus Gaussian noise with σ = 0.25 (noise floor σ² = 0.0625); the 200 validation points are the faint dots. Slide K from 1 to 24. At small K the curve cannot bend and both errors are high. At K = 6 validation error bottoms out. Beyond that, training error keeps falling while validation error climbs, and from K = 15 it explodes. Then press “overfit + λ” and “overfit + data” to see weight decay and more data close the gap at the same K.

Worked example: numbers read off the figure (20 training points unless noted; MSE, noise floor 0.0625)

| K | λ | train MSE | val MSE | gap | diagnosis |
| --- | --- | --- | --- | --- | --- |
| 2 | 0 | 0.319 | 0.308 | −0.011 | underfit: training error 5× the noise floor |
| 6 | 0 | 0.021 | 0.078 | 0.058 | good: lowest validation error of any K |
| 14 | 0 | 0.010 | 0.605 | 0.595 | overfit: train ↓, val ↑ 8× |
| 14 | 0.01 | 0.021 | 0.071 | 0.051 | weight decay closes the gap |
| 14 (120 pts) | 0 | 0.070 | 0.062 | −0.009 | more data closes the gap |

Note the third row: training error is the *lowest* of the five, and the model is the worst. Training loss alone can never tell you a network has overfit. Also note that the good model's training error (0.021) is below the noise floor. A small gap is normal and not a reason to stop.

## Reading learning curves

In a real network, capacity grows during training as well as with size. Early on, train and val loss fall together. If training continues past the best point, train keeps falling while val turns upward: the gap opens epoch by epoch. **Early stopping** keeps the checkpoint at the validation minimum, which is why it counts as a regularizer.

Diagnosis from learning curves

| symptom | likely cause | what to try |
| --- | --- | --- |
| Train loss high and flat; val ≈ train | Underfitting: too little capacity, too few steps, or a learning rate that is too low or too high | Bigger or deeper model, train longer, tune the learning rate, less regularization, better features or architecture |
| Train loss low; val much higher, rising late in training | Overfitting: the network memorises the training sample | More data, data augmentation, dropout, weight decay, early stopping, a smaller model |
| Train loss not decreasing at all | A bug, not a capacity problem | Overfit a single batch (below); check labels, loss and learning rate |
| Val lower than train | Usually dropout or augmentation active during training only, or an easy val split | Compare with train loss measured in eval mode; check the split for leakage |

> **Big networks overfit less than you would expect:** Modern networks often have far more parameters than training examples, and can fit random labels perfectly. Yet with SGD, augmentation and weight decay they still generalize. Parameter count is a poor measure of effective capacity. Trust the measured gap, not the parameter count. The bias–variance article goes further into this, including double descent.

## First, overfit a single batch

Before tuning anything, check that the model can memorise. Take one small batch and train on it repeatedly with regularization off. The loss should go to nearly zero within a few hundred steps. If it does not, the problem is a bug (wrong labels, a detached graph, a forgotten `zero_grad`, a bad learning rate), and no amount of data or regularization will fix it.

```python
import torch

x, y = next(iter(train_loader))          # one batch, e.g. 32 examples
model.train()
opt = torch.optim.Adam(model.parameters(), lr=1e-3, weight_decay=0)
for step in range(500):
    opt.zero_grad()
    loss = loss_fn(model(x), y)
    loss.backward()
    opt.step()
print(loss.item())                       # should be close to 0; if not, debug first
```

Sanity check: a working pipeline drives the loss on one batch close to zero

> **Regularizing an underfit model:** Dropout, weight decay and smaller models all *raise* training error. Applied to a model whose training loss is already too high, they make things worse. Always fix underfitting first (train loss low), then close the gap.

**Question:** Training loss is 0.02 and still falling. Validation loss bottomed out at 0.31 at epoch 12 and is 0.45 at epoch 40. Which change is least likely to help?

- A. Restore the epoch-12 checkpoint (early stopping)
- B. Add data augmentation
- C. Double the width of every layer and keep training
- D. Increase weight decay

**Answer:** C. Double the width of every layer and keep training — This is overfitting: low and falling training loss with a widening gap. Early stopping, augmentation and weight decay all target the gap. More capacity, with training continued, typically lets the network memorise even more, so the gap grows.

**Question:** In the worked example, why does K = 14 with λ = 0.01 generalize so much better than K = 14 with λ = 0?

- A. It has fewer hidden units
- B. The penalty keeps the output weights small, so the curve cannot swing wildly between training points
- C. It is trained on more data
- D. It has a lower training error

**Answer:** B. The penalty keeps the output weights small, so the curve cannot swing wildly between training points — Same architecture, same 20 points. Without the penalty, the solution uses large, opposing weights to pass close to every noisy point, and it oscillates in between. Weight decay trades a slightly higher training error (0.010 → 0.021) for a much smaller gap (0.595 → 0.051).

**Key points**

- Underfit: training error is high. Overfit: training error is low but the gap to validation is large.
- Diagnose with both curves. Training loss alone can never reveal overfitting.
- Fix underfitting with capacity, training time and a tuned learning rate. Fix overfitting with data, augmentation, dropout, weight decay and early stopping.
- Fix underfitting first, because regularizers raise training error.
- Overfit one batch before anything else: if the network cannot memorise 32 examples, look for a bug.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
