# Early stopping

> Watch validation loss during training, keep the best checkpoint, stop once it has not improved for patience evaluations. For a quadratic loss, stopping gradient descent after τ steps acts like L2 with λ ≈ 1/(ητ).

Source: https://www.mlfoundry.in/learn/early-stopping  
Section: Deep Learning › Regularization & Normalization  
Level: core · 11 min read · updated 2026-10-11

**Early stopping:** ends training when a held-out **validation** metric stops improving, and returns the parameters from the evaluation where that metric was best. The number of training steps becomes a hyperparameter that is tuned automatically, on the fly, from validation data.

A large network trained long enough fits its training set almost perfectly, noise included. The training loss keeps falling, but the validation loss usually falls, bottoms out and then rises as the network starts to memorise. **Underfitting and overfitting** describes that U-shape. Early stopping keeps the checkpoint from the bottom of the U. It costs almost nothing, it needs no change to the model or the loss, and it is used in nearly every training run, often next to weight decay and dropout.

The idea is older than deep learning. Morgan and Bourlard (1990) reported that stopping backprop training at the minimum of a cross-validation error gave better speech-recognition networks than training to convergence. Prechelt's chapter "Early stopping — but when?" (1998, *Neural Networks: Tricks of the Trade*) compared stopping criteria systematically and recommended patience-style rules. Sjöberg and Ljung (1995) and Bishop (1995) showed that, for a quadratic error, stopping gradient descent early is approximately equivalent to L2 regularization. Goodfellow, Bengio and Courville (2016, §7.8) give the version derived below.

## The rule

$$
t^\star=\arg\min_{t\in\{0,\,E,\,2E,\dots\}} L_{\text{val}}(\theta_t),\qquad \text{stop at the first } t \text{ with } t-t^\star_{\le t}\ \ge\ P\cdot E,\qquad \text{return } \theta_{t^\star}
$$

- $E$: evaluation interval: usually one epoch, or a fixed number of steps for very large datasets
- $L_{\text{val}}$: the monitored quantity: validation loss, or a metric such as accuracy (then maximize)
- $t^\star_{\le t}$: the best evaluation seen up to time $t$; a new best must beat the old one by more than `min_delta`
- $P$: patience: how many evaluations without improvement to tolerate before stopping
- $\theta_{t^\star}$: the **restored** weights from the best evaluation, not the weights at the moment training stopped

Three things make it work in practice. **Patience** stops the rule from quitting on a single noisy evaluation. **Checkpointing** (saving a copy of the weights each time the metric improves) means that waiting $P$ evaluations past the optimum costs only compute: the extra steps are thrown away. **Restoring** the best checkpoint is what turns "stop" into a regularizer. Without it, you keep weights that are $P$ evaluations past the optimum.

**At a glance**

- **Hyperparameters:** patience $P$, evaluation interval, `min_delta`, monitored metric and direction
- **Cost:** one validation pass per evaluation plus a stored checkpoint; saves the steps after the stop
- **Regularizes:** the effective size of the weights reached from the initialization; for quadratics, like L2 with $\lambda\approx1/(\eta\tau)$
- **Needs:** a validation set the model never trains on, from the same distribution as the test data
- **Keras:** `EarlyStopping(monitor="val_loss", patience=…, restore_best_weights=True)`
- **PyTorch:** no built-in; a few lines in the loop (below), or Lightning's `EarlyStopping` plus `ModelCheckpoint`

> Figure: A regression with 24 Gaussian bump features fit to 20 noisy points (noise σ = 0.35) by full-batch gradient descent from w = 0, evaluated every 20 steps on 200 validation points. At the default η = 0.06 the validation MSE falls from 0.793 to its minimum 0.1436 at step 120 (train 0.0729), then climbs to 0.2225 by step 8000 while the training MSE heads to 0.0148. Patience 10 stops at step 320 and restores step 120. Move η: the best step moves inversely (220 at η = 0.03, 60 at η = 0.12), while η·t* stays near 7. The dashed curve is ridge regression with λ = 1/(ηt): its best validation MSE is 0.1446 at λ ≈ 0.056, almost the same model. Switch to SGD with batch 1 and η = 0.3: the noisy curve fools patience 1 into stopping at step 40 (restoring step 20, val 0.1756), while patience 3 reaches the real minimum at step 120 (0.1596).

## Why it regularizes: time is a complexity knob

Start gradient descent at small weights (here $w=0$). On a quadratic loss with Hessian $H=Q\Lambda Q^\top$, each eigen-direction of the weights relaxes toward the least-squares solution $w^*$ at its own rate. Directions with large curvature $h_i$ (smooth functions that many data points agree on) are fitted within a few steps. Directions with tiny curvature (wiggly functions supported by one or two points, which is where the label noise lives) take $\sim 1/(\eta h_i)$ steps. Stopping at step $\tau$ keeps the first group and leaves the second near zero.

$$
w_\tau=Q\big[I-(I-\eta\Lambda)^{\tau}\big]Q^\top w^*\qquad\text{vs.}\qquad w_\lambda=Q\big[\Lambda(\Lambda+\lambda I)^{-1}\big]Q^\top w^*
$$

- $w_\tau$: gradient descent with step $\eta$ after $\tau$ steps from $w_0=0$
- $w_\lambda$: the minimizer of the same loss plus $\tfrac\lambda2\lVert w\rVert^2$ (ridge / L2)
- $h_i$: an eigenvalue of $H$; both methods keep a fraction of $w^*$ along each eigenvector

Both are **spectral filters** on $w^*$: they keep directions with large $h_i$ and shrink those with small $h_i$. For $\eta h_i\ll1$ and $h_i\ll\lambda$, $(1-\eta h_i)^\tau\approx e^{-\eta h_i\tau}\approx1-\eta h_i\tau$ and $\lambda/(h_i+\lambda)\approx1-h_i/\lambda$. The two filters match when

$$
\tau\,\eta\;\approx\;\frac{1}{\lambda}
$$

- $\tau\eta$: the "training time" in units of learning rate: halving $\eta$ doubles the number of steps to the same model

So the number of training steps plays the role of an inverse weight-decay strength. Training longer means weaker regularization, and the effective capacity of the network grows during training. This is also why the best step in the figure scales like $1/\eta$. The correspondence is approximate: the two filters have different shapes (an exponential step versus a rational one), so the best $\lambda$ and the best $\tau$ agree only up to a constant factor. In the figure the ridge optimum sits at $\lambda\approx0.056$, which corresponds to $t=1/(\eta\lambda)\approx300$ rather than 120, yet the two validation errors (0.1446 and 0.1436) are nearly equal. For a nonlinear network the analysis applies locally, around the path that training takes.

> **Early stopping is cheap hyperparameter search:** Tuning weight decay takes one training run per value of $\lambda$. Early stopping evaluates a whole path of models, one per checkpoint, in a **single** run, and picks the best by validation loss. That is its main practical advantage over an explicit penalty, and the reason it is so often left on even when other regularizers are used.

## Worked example: the two filters side by side

Take $\eta=0.1$ and $\tau=100$ steps, so $\eta\tau=10$ and the matching ridge strength is $\lambda=1/(\eta\tau)=0.1$. Four directions with curvature from 1 down to 0.001:

Fraction of w* recovered along each eigen-direction (checked in Node)

| curvature hᵢ | GD after 100 steps: 1 − (1 − ηhᵢ)¹⁰⁰ | ridge λ = 0.1: hᵢ/(hᵢ + λ) | what the direction is |
| --- | --- | --- | --- |
| 1 | 1.0000 | 0.9091 | well determined by the data |
| 0.1 | 0.6340 | 0.5000 | at the cut-off hᵢ ≈ 1/(ητ) |
| 0.01 | 0.0952 | 0.0909 | weakly determined |
| 0.001 | 0.0100 | 0.0099 | mostly noise: left near 0 |

For the small-curvature directions, which are the ones that matter for overfitting, the two filters are almost identical. They differ near the cut-off and at the top: gradient descent eventually recovers the stiff directions fully, while ridge always shrinks them a little.

**Patience in action: validation loss per epoch, patience 3**

1. Epochs 1–4: $0.90,\,0.62,\,0.48,\,0.45$. Each is a new best; save a checkpoint each time.
2. Epoch 5: $0.46$. No improvement, wait counter 1. Epoch 6: $0.44$, a new best: save, reset the counter to 0.
3. Epochs 7–9: $0.45,\,0.47,\,0.46$. Counter 1, 2, 3: patience exhausted, stop after epoch 9 and restore epoch 6 (0.44).
4. With `min_delta = 0.02` a new best must beat the old one by more than 0.02. Epoch 6's 0.44 is not 0.02 below 0.45, so the counter reaches 3 at epoch 7 and epoch 4 is restored. A larger `min_delta` stops earlier.

```python
import copy, torch

def fit(model, opt, loss_fn, train_loader, val_loader, max_epochs=200, patience=10, min_delta=0.0):
    best, best_state, best_epoch, wait = float("inf"), None, -1, 0
    for epoch in range(max_epochs):
        model.train()
        for x, y in train_loader:
            opt.zero_grad(set_to_none=True)
            loss_fn(model(x), y).backward()
            opt.step()

        model.eval()                                   # dropout off, BatchNorm uses running stats
        with torch.no_grad():
            val = sum(loss_fn(model(x), y).item() * len(x) for x, y in val_loader)
            val /= len(val_loader.dataset)

        if val < best - min_delta:                     # improvement: checkpoint and reset the counter
            best, best_epoch, wait = val, epoch, 0
            best_state = copy.deepcopy(model.state_dict())   # or torch.save(...) to disk
        else:
            wait += 1
            if wait >= patience:
                break

    model.load_state_dict(best_state)                  # restore, do not keep the last weights
    return best_epoch, best

# Keras:     keras.callbacks.EarlyStopping(monitor="val_loss", patience=10, restore_best_weights=True)
# Lightning: EarlyStopping(monitor="val_loss", patience=10, mode="min") stops training;
#            ModelCheckpoint(monitor="val_loss", save_top_k=1) keeps the best weights to reload.
```

Early stopping with patience and best-weight restoration in a plain PyTorch loop

> **Stopping without restoring:** The weights at the moment the rule fires are $P$ evaluations past the optimum, and with a generous patience they can be clearly worse (in the figure, patience 10 stops at step 320 with val 0.163, versus 0.1436 at the restored step 120). Always reload the best checkpoint. Keras only does this with `restore_best_weights=True`; Lightning's `EarlyStopping` callback stops training but does not reload, so pair it with `ModelCheckpoint`.

> **The validation set is now part of training:** Early stopping uses the validation set to choose the parameters, so the best validation loss is an optimistically biased estimate of test error. Report a separate test set. And when the learning-rate schedule depends on the number of epochs (cosine annealing, one-cycle), stopping mid-schedule leaves the learning rate high; with such schedules it is often better to pick the training length once and keep the best checkpoint than to stop dynamically.

> **Double descent: the U is not guaranteed:** Nakkiran et al. (2020) found **epoch-wise double descent**: with label noise and a model just large enough to memorise, test error can fall, rise and then fall again, ending lower than the first minimum. A short patience stops at the first dip. If validation loss rises slowly rather than sharply, try a longer patience or simply train to a fixed budget and keep the best checkpoint (see **Bias–variance in deep networks**). Also note that validation **loss** can rise while validation **accuracy** still improves, because cross-entropy punishes the growing over-confidence on the few examples the model gets wrong. Monitor the metric you actually care about.

> **After stopping: retrain on everything?:** Goodfellow et al. (2016) describe two ways to use the data held out for validation once the best step count is known. Retrain from scratch on train + validation for the same number of **steps** (or epochs, which then cover more data), or continue training the stopped model on all the data until the training loss falls to the value it had at the early-stopping point. Neither is guaranteed to help; with plenty of data, most practitioners just keep the restored checkpoint.

**Strengths**

- Almost free: one validation pass per evaluation and a stored checkpoint, and it saves the wasted epochs
- Searches a whole regularization path in one run, instead of one run per weight-decay value
- Model- and loss-agnostic: nothing to change in the architecture or objective
- Has a clean theory for quadratics: $\tau\eta\approx1/\lambda$, the same spectral filter as L2

**Limitations**

- Consumes data: needs a validation set that is not trained on
- Noisy validation curves make the stopping point noisy; patience and min_delta are extra knobs
- Interacts with LR schedules: stopping mid-schedule leaves a high learning rate
- Can stop too early when the curve has a plateau or epoch-wise double descent

**Question:** A model trained with gradient descent at learning rate 0.01 reaches its best validation loss at step 4000. You rerun with learning rate 0.02 and everything else equal. Roughly where should you expect the best step now, if training is in the regime the quadratic analysis describes?

- A. Around step 2000
- B. Around step 4000
- C. Around step 8000
- D. There is no relation between the two

**Answer:** A. Around step 2000 — The effective regularization depends on the product $\eta\tau\approx1/\lambda$. Doubling $\eta$ reaches the same model in half the steps, so about 2000. The figure shows the same thing: best step 220, 120 and 60 at $\eta=0.03,\ 0.06,\ 0.12$.

**Question:** Your early-stopping callback has patience 20 and fires at epoch 95. The best validation loss was at epoch 75. Which weights should you deploy?

- A. Epoch 95: it has trained the longest
- B. Epoch 75: the restored best checkpoint
- C. The average of epochs 75 and 95
- D. Retrain to epoch 95 with a smaller learning rate

**Answer:** B. Epoch 75: the restored best checkpoint — The 20 epochs after the optimum only confirmed that the validation loss was not going to improve. Their weights are further along the overfitting path. Restoring the checkpoint from epoch 75 is what makes early stopping a regularizer.

**Key points**

- Evaluate on validation data every epoch (or every E steps), checkpoint on improvement, stop after `patience` evaluations without improvement, and restore the best checkpoint.
- Gradient descent from small weights fits high-curvature directions first; stopping at step $\tau$ filters the spectrum like L2 with $\lambda\approx1/(\eta\tau)$.
- The best step scales like $1/\eta$: training time is a capacity knob.
- Patience protects against noisy curves; min_delta ignores tiny improvements. Use a separate test set, because validation data now selects the model.
- Epoch-wise double descent and LR schedules can make a short patience stop too early; a fixed budget plus best-checkpoint selection is the safe alternative.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
