Definition
Early stopping ends training when a held-out validation metric stops improving, and returns the parameters from the evaluation where that metric was best. The number of training steps becomes a hyperparameter that is tuned automatically, on the fly, from validation data.
A large network trained long enough fits its training set almost perfectly, noise included. The training loss keeps falling, but the validation loss usually falls, bottoms out and then rises as the network starts to memorise. Underfitting and overfitting describes that U-shape. Early stopping keeps the checkpoint from the bottom of the U. It costs almost nothing, it needs no change to the model or the loss, and it is used in nearly every training run, often next to weight decay and dropout.
The idea is older than deep learning. Morgan and Bourlard (1990) reported that stopping backprop training at the minimum of a cross-validation error gave better speech-recognition networks than training to convergence. Prechelt's chapter "Early stopping — but when?" (1998, Neural Networks: Tricks of the Trade) compared stopping criteria systematically and recommended patience-style rules. Sjöberg and Ljung (1995) and Bishop (1995) showed that, for a quadratic error, stopping gradient descent early is approximately equivalent to L2 regularization. Goodfellow, Bengio and Courville (2016, §7.8) give the version derived below.
The rule
- evaluation interval: usually one epoch, or a fixed number of steps for very large datasets
- the monitored quantity: validation loss, or a metric such as accuracy (then maximize)
- the best evaluation seen up to time ; a new best must beat the old one by more than
min_delta - patience: how many evaluations without improvement to tolerate before stopping
- the restored weights from the best evaluation, not the weights at the moment training stopped
Three things make it work in practice. Patience stops the rule from quitting on a single noisy evaluation. Checkpointing (saving a copy of the weights each time the metric improves) means that waiting evaluations past the optimum costs only compute: the extra steps are thrown away. Restoring the best checkpoint is what turns "stop" into a regularizer. Without it, you keep weights that are evaluations past the optimum.
At a glance
- Hyperparameters
- patience , evaluation interval,
min_delta, monitored metric and direction - Cost
- one validation pass per evaluation plus a stored checkpoint; saves the steps after the stop
- Regularizes
- the effective size of the weights reached from the initialization; for quadratics, like L2 with
- Needs
- a validation set the model never trains on, from the same distribution as the test data
- Keras
EarlyStopping(monitor="val_loss", patience=…, restore_best_weights=True)- PyTorch
- no built-in; a few lines in the loop (below), or Lightning's
EarlyStoppingplusModelCheckpoint
Why it regularizes: time is a complexity knob
Start gradient descent at small weights (here ). On a quadratic loss with Hessian , each eigen-direction of the weights relaxes toward the least-squares solution at its own rate. Directions with large curvature (smooth functions that many data points agree on) are fitted within a few steps. Directions with tiny curvature (wiggly functions supported by one or two points, which is where the label noise lives) take steps. Stopping at step keeps the first group and leaves the second near zero.
- gradient descent with step after steps from
- the minimizer of the same loss plus (ridge / L2)
- an eigenvalue of ; both methods keep a fraction of along each eigenvector
Both are spectral filters on : they keep directions with large and shrink those with small . For and , and . The two filters match when
- the "training time" in units of learning rate: halving doubles the number of steps to the same model
So the number of training steps plays the role of an inverse weight-decay strength. Training longer means weaker regularization, and the effective capacity of the network grows during training. This is also why the best step in the figure scales like . The correspondence is approximate: the two filters have different shapes (an exponential step versus a rational one), so the best and the best agree only up to a constant factor. In the figure the ridge optimum sits at , which corresponds to rather than 120, yet the two validation errors (0.1446 and 0.1436) are nearly equal. For a nonlinear network the analysis applies locally, around the path that training takes.
Worked example: the two filters side by side
Take and steps, so and the matching ridge strength is . Four directions with curvature from 1 down to 0.001:
| curvature hᵢ | GD after 100 steps: 1 − (1 − ηhᵢ)¹⁰⁰ | ridge λ = 0.1: hᵢ/(hᵢ + λ) | what the direction is |
|---|---|---|---|
| 1 | 1.0000 | 0.9091 | well determined by the data |
| 0.1 | 0.6340 | 0.5000 | at the cut-off hᵢ ≈ 1/(ητ) |
| 0.01 | 0.0952 | 0.0909 | weakly determined |
| 0.001 | 0.0100 | 0.0099 | mostly noise: left near 0 |
For the small-curvature directions, which are the ones that matter for overfitting, the two filters are almost identical. They differ near the cut-off and at the top: gradient descent eventually recovers the stiff directions fully, while ridge always shrinks them a little.
Patience in action: validation loss per epoch, patience 3
- 1Epochs 1–4: . Each is a new best; save a checkpoint each time.
- 2Epoch 5: . No improvement, wait counter 1. Epoch 6: , a new best: save, reset the counter to 0.
- 3Epochs 7–9: . Counter 1, 2, 3: patience exhausted, stop after epoch 9 and restore epoch 6 (0.44).
- 4With
min_delta = 0.02a new best must beat the old one by more than 0.02. Epoch 6's 0.44 is not 0.02 below 0.45, so the counter reaches 3 at epoch 7 and epoch 4 is restored. A largermin_deltastops earlier.
1import copy, torch23def fit(model, opt, loss_fn, train_loader, val_loader, max_epochs=200, patience=10, min_delta=0.0):4 best, best_state, best_epoch, wait = float("inf"), None, -1, 05 for epoch in range(max_epochs):6 model.train()7 for x, y in train_loader:8 opt.zero_grad(set_to_none=True)9 loss_fn(model(x), y).backward()10 opt.step()1112 model.eval() # dropout off, BatchNorm uses running stats13 with torch.no_grad():14 val = sum(loss_fn(model(x), y).item() * len(x) for x, y in val_loader)15 val /= len(val_loader.dataset)1617 if val < best - min_delta: # improvement: checkpoint and reset the counter18 best, best_epoch, wait = val, epoch, 019 best_state = copy.deepcopy(model.state_dict()) # or torch.save(...) to disk20 else:21 wait += 122 if wait >= patience:23 break2425 model.load_state_dict(best_state) # restore, do not keep the last weights26 return best_epoch, best2728# Keras: keras.callbacks.EarlyStopping(monitor="val_loss", patience=10, restore_best_weights=True)29# Lightning: EarlyStopping(monitor="val_loss", patience=10, mode="min") stops training;30# ModelCheckpoint(monitor="val_loss", save_top_k=1) keeps the best weights to reload.Use when
- +Almost free: one validation pass per evaluation and a stored checkpoint, and it saves the wasted epochs
- +Searches a whole regularization path in one run, instead of one run per weight-decay value
- +Model- and loss-agnostic: nothing to change in the architecture or objective
- +Has a clean theory for quadratics: , the same spectral filter as L2
Avoid when
- −Consumes data: needs a validation set that is not trained on
- −Noisy validation curves make the stopping point noisy; patience and min_delta are extra knobs
- −Interacts with LR schedules: stopping mid-schedule leaves a high learning rate
- −Can stop too early when the curve has a plateau or epoch-wise double descent
Check yourself
A model trained with gradient descent at learning rate 0.01 reaches its best validation loss at step 4000. You rerun with learning rate 0.02 and everything else equal. Roughly where should you expect the best step now, if training is in the regime the quadratic analysis describes?
Check yourself
Your early-stopping callback has patience 20 and fires at epoch 95. The best validation loss was at epoch 75. Which weights should you deploy?
Remember
- ✓Evaluate on validation data every epoch (or every E steps), checkpoint on improvement, stop after
patienceevaluations without improvement, and restore the best checkpoint. - ✓Gradient descent from small weights fits high-curvature directions first; stopping at step filters the spectrum like L2 with .
- ✓The best step scales like : training time is a capacity knob.
- ✓Patience protects against noisy curves; min_delta ignores tiny improvements. Use a separate test set, because validation data now selects the model.
- ✓Epoch-wise double descent and LR schedules can make a short patience stop too early; a fixed budget plus best-checkpoint selection is the safe alternative.