Mini-batch gradient descent

Average the gradient over B random examples: noise variance falls as 1/B and the batch runs in parallel on a GPU. Up to a critical batch size, scaling the learning rate with B (Goyal et al. 2017) turns k× bigger batches into k× fewer steps; beyond it, extra batch buys little.

11 min read1052 wordscore0 viewsupdated 2026-10-11

Definition

Mini-batch gradient descent (mini-batch SGD) updates the parameters with the average gradient of a random subset Bt\mathcal B_t of BB examples: θt+1=θt−η 1B∑i∈Bt∇ℓi(θt)\theta_{t+1}=\theta_t-\eta\,\frac1B\sum_{i\in\mathcal B_t}\nabla\ell_i(\theta_t). B=1B=1 is SGD, B=NB=N is batch gradient descent; everything used in practice lies in between.

This is what "SGD" means in nearly every deep-learning paper and in torch.optim.SGD: the gradient of a mini-batch of, say, 32 to 4096 examples. It keeps SGD's key property, a cost per step that does not depend on NN, while fixing its two weaknesses: the gradient noise shrinks, and the BB examples are processed as one matrix operation that keeps a GPU busy.

The batch size is then a hyperparameter coupled to the learning rate. Goyal et al. (2017), "Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour", trained ResNet-50 with batches of 8192 images on 256 GPUs and matched the accuracy of the standard batch of 256, using two rules: scale the learning rate linearly with BB, and warm it up gradually over the first 5 epochs. McCandlish et al. (2018) and Shallue et al. (2019) then showed that this scaling works only up to a problem-dependent critical batch size.

The update and its noise

gt=1B∑i∈Bt∇ℓi(θt),E[gt]=∇f(θt),E∥gt−∇f∥2=tr⁡ΣB⋅N−BN−1g_t=\frac1B\sum_{i\in\mathcal B_t}\nabla\ell_i(\theta_t),\qquad \mathbb E[g_t]=\nabla f(\theta_t),\qquad \mathbb E\lVert g_t-\nabla f\rVert^2=\frac{\operatorname{tr}\Sigma}{B}\cdot\frac{N-B}{N-1}
BB
batch size: examples averaged per update
Σ\Sigma
covariance of a single example's gradient at θt\theta_t; tr⁡Σ\operatorname{tr}\Sigma is SGD's (B=1B=1) noise variance
N−BN−1\tfrac{N-B}{N-1}
finite-population correction for sampling without replacement (a shuffled epoch): exactly 0 at B=NB=N; drop it when N≫BN\gg B

So the noise standard deviation falls as 1/B1/\sqrt B: 16 times the batch gives a 4 times less noisy gradient, at 16 times the compute per step. That is a poor deal if compute is the bottleneck, which is why the learning rate must change with BB for a bigger batch to pay off.

Batch size at a glance

Steps per epoch
⌈N/B⌉\lceil N/B\rceil, or ⌊N/B⌋\lfloor N/B\rfloor with drop_last=True
Gradient noise
variance ∝1/B\propto 1/B (std ∝1/B\propto1/\sqrt B)
Noise scale
SGD's stochasticity is governed by roughly η/B\eta/B (Smith and Le, 2018); keep it fixed when changing BB
Linear scaling rule
η=ηbase⋅B/Bbase\eta=\eta_{\text{base}}\cdot B/B_{\text{base}}, with warmup; e.g. 0.1⋅B/2560.1\cdot B/256 for ResNet-50
Critical batch size
below it, k×k\times batch with k×ηk\times\eta needs k×k\times fewer steps; above it, the step count barely falls
Memory
activations grow linearly in BB; weights and optimizer state do not
Typical sizes
32–512 for vision on one GPU; millions of tokens per batch for large language models

The linear scaling rule

Why multiply η\eta by kk when the batch grows kk-fold? Compare kk SGD steps on small batches with one step on their union. If the gradient barely changes over those kk steps, the small-batch steps move η∑j<kgt+j≈kη gˉ\eta\sum_{j<k}g_{t+j}\approx k\eta\,\bar g, while the big batch moves η′gˉ\eta'\bar g. They agree when η′=kη\eta'=k\eta. Two caveats are built into the argument: it fails early in training when the gradient changes quickly (hence warmup), and it fails once kηk\eta approaches the curvature limit 2/L2/L of plain gradient descent (hence a critical batch size).

η(B)=η1 B  for B≲B∗,B∗≈ηmax⁡η1,Bnoise=tr⁡Σ∥∇f∥2\eta(B)=\eta_1\,B\ \ \text{for } B\lesssim B^*,\qquad B^*\approx\frac{\eta_{\max}}{\eta_1},\qquad \mathcal B_{\text{noise}}=\frac{\operatorname{tr}\Sigma}{\lVert\nabla f\rVert^2}
η1\eta_1
learning rate that works at batch size 1 (or per example)
ηmax⁡\eta_{\max}
the largest step the curvature allows (of order 1/L1/L)
Bnoise\mathcal B_{\text{noise}}
McCandlish et al.'s gradient noise scale: batches much smaller than it are noise-dominated (scaling works), much larger ones are curvature-limited (it does not)

For adaptive optimizers the rule is different. Adam divides by the gradient RMS, so its effective step does not grow with the gradient's signal-to-noise ratio in the same way; a square-root rule η∝B\eta\propto\sqrt B is the common heuristic, with theoretical support in Malladi et al. (2022).

N = 256, B = 16, η = 0.160, 16 steps/epochg_B − ∇f at θ*f − f* (log) vs epoch1e-41e-21epoch 0 … 12 · grey: B = 1steps to f − f* < 0.002 vs B (log-log)101001000116256— η ∝ B — fixed η ┆ B*
B =
|
epoch 12.0 · step 192 · 3,072 examples · f − f* = 1.5e-4
gradient noise at θ*: RMS 0.129 sampled, 0.134 exact = 0.551·√((N−B)/(B(N−1))) · to target: 137 steps, 2186 examples · B* ≈ 86
fig.Mini-batch SGD on a 256-example least-squares problem, 12 epochs, batches drawn like DataLoader(shuffle=True). The orange inset is 120 sampled values of g_B − ∇f at θ*, on a fixed scale, with the exact RMS circled: 0.551 at B = 1, 0.134 at B = 16, 0.060 at B = 64 and exactly 0 at B = N = 256. Lower right: steps needed to reach f − f* < 2.3·10⁻³ (5 seeds). With linear scaling η = min(0.01·B, 1/L) the steps halve with every doubling, from 2204 at B = 1 to 34 at B = 64, and the total examples stay near 2,200. Past B* = (1/L)/η₁ ≈ 86 the learning rate hits its cap: B = 256 needs 24 steps but 6,144 examples. With a fixed η every batch size needs about 2,200 steps (grey): bigger batches then only cost more. Per epoch, every linearly scaled batch size below B* traces nearly the same loss curve (compare the grey B = 1 curve).

Critical batch size: steps versus examples

Shallue et al. (2019) measured, across many models and datasets, the number of steps needed to reach a target validation error as a function of batch size, with the learning rate re-tuned for each BB. They found three regimes: perfect scaling, where doubling BB halves the steps; diminishing returns; and maximal data parallelism, where more batch does not reduce steps at all. Where the transition happens depends on the model, the dataset and the optimizer, and it tends to move to larger batches as training progresses and the gradient signal shrinks relative to its noise.

McCandlish et al. (2018) summarized the trade-off as (S/Smin⁡−1)(E/Emin⁡−1)=1(S/S_{\min}-1)(E/E_{\min}-1)=1 between steps SS and examples processed EE: you can trade more compute for fewer sequential steps, but only down to Smin⁡S_{\min}, and the critical batch ≈Bnoise\approx\mathcal B_{\text{noise}} is the knee of that curve. Below the knee, larger batches are almost free speed on parallel hardware; above it, they waste compute.

Worked example: what a batch size implies

BηRMS of g_B − ∇fsteps to targetexamples to target
10.0100.55122042204
40.0400.2745502200
160.1600.1341372186
640.6400.060342163
1280.857 (cap)0.035253149
2560.857 (cap)0246144
Gradient noise at θ* and steps to target in the figure (N = 256, η₁ = 0.01, linear scaling capped at 1/L = 0.857); computed by the figure's code
  1. 1Noise: 0.551(256−16)/(16⋅255)=0.551×0.243=0.1340.551\sqrt{(256-16)/(16\cdot255)}=0.551\times0.243=0.134. Without the finite-population factor it would be 0.551/4=0.1380.551/4=0.138; at B=NB=N the batch is the whole dataset and the noise is exactly zero.
  2. 2Up to B=64B=64 the rule works perfectly: ηB−1\eta B^{-1} stays at 0.010.01, so the noise floor and the per-epoch progress are unchanged, and the steps fall by exactly the factor the batch grew.
  3. 3At B=128B=128 the scaled rate would be 1.281.28, which exceeds the cap 1/L=0.8571/L=0.857 (and would approach the divergence limit 2/L=1.712/L=1.71). From there on the step count stays near 24: the problem is now curvature-limited, like batch gradient descent.
  4. 4Data loading: with N=1000N=1000 and B=64B=64, an epoch has ⌈1000/64⌉=16\lceil1000/64\rceil=16 steps, the last on only 40 examples. drop_last=True gives 15 steps of exactly 64 and skips 40 examples per epoch (different ones each epoch when shuffled).
  5. 5Goyal et al.'s rule η=0.1⋅B/256\eta=0.1\cdot B/256 gives η=0.4\eta=0.4 at B=1024B=1024 and η=3.2\eta=3.2 at B=8192B=8192; at 8192 they needed the 5-epoch linear warmup from 0.10.1 to avoid early divergence.

GPU efficiency

A mini-batch turns BB matrix-vector products into one matrix-matrix product, which GPUs execute far more efficiently: the weights are read from memory once per batch instead of once per example. Throughput (examples per second) therefore rises steeply with BB until the device is saturated, and then flattens; past that point a bigger batch only takes proportionally longer. NVIDIA recommends dimensions, batch size included, that are multiples of 8 for FP16 Tensor Cores (larger multiples can help on newer GPUs). Memory sets the upper limit: activations grow linearly in BB, and gradient accumulation lets you exceed it at the cost of more passes per step.

python
1import torch, torch.nn as nn
2from torch.utils.data import DataLoader, TensorDataset
3
4ds = TensorDataset(torch.randn(50_000, 32), torch.randint(0, 10, (50_000,)))
5B = 1024
6loader = DataLoader(ds, batch_size=B, shuffle=True, # new permutation every epoch
7 drop_last=True, # every step sees exactly B examples
8 num_workers=4, pin_memory=True) # overlap loading with GPU compute
9steps_per_epoch = len(loader) # 50_000 // 1024 = 48
10
11model = nn.Sequential(nn.Linear(32, 256), nn.ReLU(), nn.Linear(256, 10))
12base_lr, base_B = 0.1, 256
13lr = base_lr * B / base_B # linear scaling rule: 0.4
14opt = torch.optim.SGD(model.parameters(), lr=lr, momentum=0.9, weight_decay=1e-4)
15
16warmup = 5 * steps_per_epoch # 5 epochs, as in Goyal et al.
17sched = torch.optim.lr_scheduler.SequentialLR(opt, [
18 torch.optim.lr_scheduler.LinearLR(opt, start_factor=base_B / B, total_iters=warmup),
19 torch.optim.lr_scheduler.CosineAnnealingLR(opt, T_max=85 * steps_per_epoch),
20], milestones=[warmup])
21
22loss_fn = nn.CrossEntropyLoss() # mean over the batch: gradient is (1/B)·Σ
23for epoch in range(90):
24 for xb, yb in loader:
25 opt.zero_grad(set_to_none=True)
26 loss_fn(model(xb), yb).backward()
27 opt.step()
28 sched.step()
Mini-batch training with a shuffled DataLoader, the linear scaling rule and warmup

Use when

  • +Noise variance ∝1/B\propto1/B: smoother, more reliable steps than one-example SGD
  • +Matrix-matrix work saturates GPUs; far higher throughput than B = 1
  • +Below the critical batch size, linear LR scaling converts more parallel hardware into proportionally fewer steps
  • +Cost per step still independent of N; noise still regularizes

Avoid when

  • −BB and η\eta are coupled: changing one without the other gives misleading comparisons
  • −Above the critical batch size extra examples per step are mostly wasted compute
  • −Very large batches need warmup and careful tuning, and may generalize slightly worse
  • −Memory grows linearly in BB; the largest batch is limited by the GPU

Check yourself

You move from batch size 64 to 1024 on the same model with plain SGD. Following the linear scaling rule, what should the learning rate become if it was 0.05, and what happens to the gradient-noise standard deviation?

Check yourself

Doubling the batch size (with the learning rate doubled) no longer reduces the number of steps to reach your target loss. What does that indicate?

Remember

  • ✓Mini-batch GD averages BB example gradients: unbiased, variance tr⁡ΣBN−BN−1\frac{\operatorname{tr}\Sigma}{B}\frac{N-B}{N-1}, cost per step independent of NN.
  • ✓Steps per epoch: ⌈N/B⌉\lceil N/B\rceil; shuffle=True reshuffles each epoch, drop_last=True keeps every batch full.
  • ✓Linear scaling rule (Goyal et al., 2017): η∝B\eta\propto B with warmup keeps the noise scale η/B\eta/B fixed.
  • ✓It works up to a critical batch size ≈Bnoise\approx\mathcal B_{\text{noise}}; beyond it the curvature limit caps η\eta and steps stop falling.
  • ✓Bigger batches use GPUs better until throughput saturates; memory grows linearly with B.
← back to Deep Learning roadmap