Early stopping

Watch validation loss during training, keep the best checkpoint, stop once it has not improved for patience evaluations. For a quadratic loss, stopping gradient descent after τ steps acts like L2 with λ ≈ 1/(ητ).

11 min read1159 wordscore0 viewsupdated 2026-10-11

Definition

Early stopping ends training when a held-out validation metric stops improving, and returns the parameters from the evaluation where that metric was best. The number of training steps becomes a hyperparameter that is tuned automatically, on the fly, from validation data.

A large network trained long enough fits its training set almost perfectly, noise included. The training loss keeps falling, but the validation loss usually falls, bottoms out and then rises as the network starts to memorise. Underfitting and overfitting describes that U-shape. Early stopping keeps the checkpoint from the bottom of the U. It costs almost nothing, it needs no change to the model or the loss, and it is used in nearly every training run, often next to weight decay and dropout.

The idea is older than deep learning. Morgan and Bourlard (1990) reported that stopping backprop training at the minimum of a cross-validation error gave better speech-recognition networks than training to convergence. Prechelt's chapter "Early stopping — but when?" (1998, Neural Networks: Tricks of the Trade) compared stopping criteria systematically and recommended patience-style rules. Sjöberg and Ljung (1995) and Bishop (1995) showed that, for a quadratic error, stopping gradient descent early is approximately equivalent to L2 regularization. Goodfellow, Bengio and Courville (2016, §7.8) give the version derived below.

The rule

t⋆=arg⁡min⁡t∈{0, E, 2E,… }Lval(θt),stop at the first t with t−t≤t⋆ ≥ P⋅E,return θt⋆t^\star=\arg\min_{t\in\{0,\,E,\,2E,\dots\}} L_{\text{val}}(\theta_t),\qquad \text{stop at the first } t \text{ with } t-t^\star_{\le t}\ \ge\ P\cdot E,\qquad \text{return } \theta_{t^\star}
EE
evaluation interval: usually one epoch, or a fixed number of steps for very large datasets
LvalL_{\text{val}}
the monitored quantity: validation loss, or a metric such as accuracy (then maximize)
t≤t⋆t^\star_{\le t}
the best evaluation seen up to time tt; a new best must beat the old one by more than min_delta
PP
patience: how many evaluations without improvement to tolerate before stopping
θt⋆\theta_{t^\star}
the restored weights from the best evaluation, not the weights at the moment training stopped

Three things make it work in practice. Patience stops the rule from quitting on a single noisy evaluation. Checkpointing (saving a copy of the weights each time the metric improves) means that waiting PP evaluations past the optimum costs only compute: the extra steps are thrown away. Restoring the best checkpoint is what turns "stop" into a regularizer. Without it, you keep weights that are PP evaluations past the optimum.

At a glance

Hyperparameters
patience PP, evaluation interval, min_delta, monitored metric and direction
Cost
one validation pass per evaluation plus a stored checkpoint; saves the steps after the stop
Regularizes
the effective size of the weights reached from the initialization; for quadratics, like L2 with λ≈1/(ητ)\lambda\approx1/(\eta\tau)
Needs
a validation set the model never trains on, from the same distribution as the test data
Keras
EarlyStopping(monitor="val_loss", patience=…, restore_best_weights=True)
PyTorch
no built-in; a few lines in the loop (below), or Lightning's EarlyStopping plus ModelCheckpoint
20 train points · 24 bump features−2−1012−101━ restored best ┄ truthMSE vs step (log–log), eval / 20 steps10.30.10.030.01noise floor σ² = 0.12252010010008000━ train ━ val ╌ ridge λ=1/(ηt) │ best ┆ stopstep 320 · train 0.0464 · val 0.1631 · best val 0.1436 @ 120 · stopped @ 320, restored
whole-run val minimum 0.1436 at step 120 (η·t* = 7.2) · final val 0.2225 · best ridge 0.1446 at λ = 0.056
fig.A regression with 24 Gaussian bump features fit to 20 noisy points (noise σ = 0.35) by full-batch gradient descent from w = 0, evaluated every 20 steps on 200 validation points. At the default η = 0.06 the validation MSE falls from 0.793 to its minimum 0.1436 at step 120 (train 0.0729), then climbs to 0.2225 by step 8000 while the training MSE heads to 0.0148. Patience 10 stops at step 320 and restores step 120. Move η: the best step moves inversely (220 at η = 0.03, 60 at η = 0.12), while η·t* stays near 7. The dashed curve is ridge regression with λ = 1/(ηt): its best validation MSE is 0.1446 at λ ≈ 0.056, almost the same model. Switch to SGD with batch 1 and η = 0.3: the noisy curve fools patience 1 into stopping at step 40 (restoring step 20, val 0.1756), while patience 3 reaches the real minimum at step 120 (0.1596).

Why it regularizes: time is a complexity knob

Start gradient descent at small weights (here w=0w=0). On a quadratic loss with Hessian H=QΛQ⊤H=Q\Lambda Q^\top, each eigen-direction of the weights relaxes toward the least-squares solution w∗w^* at its own rate. Directions with large curvature hih_i (smooth functions that many data points agree on) are fitted within a few steps. Directions with tiny curvature (wiggly functions supported by one or two points, which is where the label noise lives) take ∼1/(ηhi)\sim 1/(\eta h_i) steps. Stopping at step τ\tau keeps the first group and leaves the second near zero.

wτ=Q[I−(I−ηΛ)τ]Q⊤w∗vs.wλ=Q[Λ(Λ+λI)−1]Q⊤w∗w_\tau=Q\big[I-(I-\eta\Lambda)^{\tau}\big]Q^\top w^*\qquad\text{vs.}\qquad w_\lambda=Q\big[\Lambda(\Lambda+\lambda I)^{-1}\big]Q^\top w^*
wτw_\tau
gradient descent with step η\eta after τ\tau steps from w0=0w_0=0
wλw_\lambda
the minimizer of the same loss plus λ2∥w∥2\tfrac\lambda2\lVert w\rVert^2 (ridge / L2)
hih_i
an eigenvalue of HH; both methods keep a fraction of w∗w^* along each eigenvector

Both are spectral filters on w∗w^*: they keep directions with large hih_i and shrink those with small hih_i. For ηhi≪1\eta h_i\ll1 and hi≪λh_i\ll\lambda, (1−ηhi)τ≈e−ηhiτ≈1−ηhiτ(1-\eta h_i)^\tau\approx e^{-\eta h_i\tau}\approx1-\eta h_i\tau and λ/(hi+λ)≈1−hi/λ\lambda/(h_i+\lambda)\approx1-h_i/\lambda. The two filters match when

τ η  ≈  1λ\tau\,\eta\;\approx\;\frac{1}{\lambda}
τη\tau\eta
the "training time" in units of learning rate: halving η\eta doubles the number of steps to the same model

So the number of training steps plays the role of an inverse weight-decay strength. Training longer means weaker regularization, and the effective capacity of the network grows during training. This is also why the best step in the figure scales like 1/η1/\eta. The correspondence is approximate: the two filters have different shapes (an exponential step versus a rational one), so the best λ\lambda and the best τ\tau agree only up to a constant factor. In the figure the ridge optimum sits at λ≈0.056\lambda\approx0.056, which corresponds to t=1/(ηλ)≈300t=1/(\eta\lambda)\approx300 rather than 120, yet the two validation errors (0.1446 and 0.1436) are nearly equal. For a nonlinear network the analysis applies locally, around the path that training takes.

Worked example: the two filters side by side

Take η=0.1\eta=0.1 and τ=100\tau=100 steps, so ητ=10\eta\tau=10 and the matching ridge strength is λ=1/(ητ)=0.1\lambda=1/(\eta\tau)=0.1. Four directions with curvature from 1 down to 0.001:

curvature hᵢGD after 100 steps: 1 − (1 − ηhᵢ)¹⁰⁰ridge λ = 0.1: hᵢ/(hᵢ + λ)what the direction is
11.00000.9091well determined by the data
0.10.63400.5000at the cut-off hᵢ ≈ 1/(ητ)
0.010.09520.0909weakly determined
0.0010.01000.0099mostly noise: left near 0
Fraction of w* recovered along each eigen-direction (checked in Node)

For the small-curvature directions, which are the ones that matter for overfitting, the two filters are almost identical. They differ near the cut-off and at the top: gradient descent eventually recovers the stiff directions fully, while ridge always shrinks them a little.

Patience in action: validation loss per epoch, patience 3

  1. 1Epochs 1–4: 0.90, 0.62, 0.48, 0.450.90,\,0.62,\,0.48,\,0.45. Each is a new best; save a checkpoint each time.
  2. 2Epoch 5: 0.460.46. No improvement, wait counter 1. Epoch 6: 0.440.44, a new best: save, reset the counter to 0.
  3. 3Epochs 7–9: 0.45, 0.47, 0.460.45,\,0.47,\,0.46. Counter 1, 2, 3: patience exhausted, stop after epoch 9 and restore epoch 6 (0.44).
  4. 4With min_delta = 0.02 a new best must beat the old one by more than 0.02. Epoch 6's 0.44 is not 0.02 below 0.45, so the counter reaches 3 at epoch 7 and epoch 4 is restored. A larger min_delta stops earlier.
python
1import copy, torch
2
3def fit(model, opt, loss_fn, train_loader, val_loader, max_epochs=200, patience=10, min_delta=0.0):
4 best, best_state, best_epoch, wait = float("inf"), None, -1, 0
5 for epoch in range(max_epochs):
6 model.train()
7 for x, y in train_loader:
8 opt.zero_grad(set_to_none=True)
9 loss_fn(model(x), y).backward()
10 opt.step()
11
12 model.eval() # dropout off, BatchNorm uses running stats
13 with torch.no_grad():
14 val = sum(loss_fn(model(x), y).item() * len(x) for x, y in val_loader)
15 val /= len(val_loader.dataset)
16
17 if val < best - min_delta: # improvement: checkpoint and reset the counter
18 best, best_epoch, wait = val, epoch, 0
19 best_state = copy.deepcopy(model.state_dict()) # or torch.save(...) to disk
20 else:
21 wait += 1
22 if wait >= patience:
23 break
24
25 model.load_state_dict(best_state) # restore, do not keep the last weights
26 return best_epoch, best
27
28# Keras: keras.callbacks.EarlyStopping(monitor="val_loss", patience=10, restore_best_weights=True)
29# Lightning: EarlyStopping(monitor="val_loss", patience=10, mode="min") stops training;
30# ModelCheckpoint(monitor="val_loss", save_top_k=1) keeps the best weights to reload.
Early stopping with patience and best-weight restoration in a plain PyTorch loop

Use when

  • +Almost free: one validation pass per evaluation and a stored checkpoint, and it saves the wasted epochs
  • +Searches a whole regularization path in one run, instead of one run per weight-decay value
  • +Model- and loss-agnostic: nothing to change in the architecture or objective
  • +Has a clean theory for quadratics: τη≈1/λ\tau\eta\approx1/\lambda, the same spectral filter as L2

Avoid when

  • −Consumes data: needs a validation set that is not trained on
  • −Noisy validation curves make the stopping point noisy; patience and min_delta are extra knobs
  • −Interacts with LR schedules: stopping mid-schedule leaves a high learning rate
  • −Can stop too early when the curve has a plateau or epoch-wise double descent

Check yourself

A model trained with gradient descent at learning rate 0.01 reaches its best validation loss at step 4000. You rerun with learning rate 0.02 and everything else equal. Roughly where should you expect the best step now, if training is in the regime the quadratic analysis describes?

Check yourself

Your early-stopping callback has patience 20 and fires at epoch 95. The best validation loss was at epoch 75. Which weights should you deploy?

Remember

  • ✓Evaluate on validation data every epoch (or every E steps), checkpoint on improvement, stop after patience evaluations without improvement, and restore the best checkpoint.
  • ✓Gradient descent from small weights fits high-curvature directions first; stopping at step τ\tau filters the spectrum like L2 with λ≈1/(ητ)\lambda\approx1/(\eta\tau).
  • ✓The best step scales like 1/η1/\eta: training time is a capacity knob.
  • ✓Patience protects against noisy curves; min_delta ignores tiny improvements. Use a separate test set, because validation data now selects the model.
  • ✓Epoch-wise double descent and LR schedules can make a short patience stop too early; a fixed budget plus best-checkpoint selection is the safe alternative.
← back to Deep Learning roadmap