Definition
Mini-batch gradient descent (mini-batch SGD) updates the parameters with the average gradient of a random subset of examples: . is SGD, is batch gradient descent; everything used in practice lies in between.
This is what "SGD" means in nearly every deep-learning paper and in torch.optim.SGD: the gradient of a mini-batch of, say, 32 to 4096 examples. It keeps SGD's key property, a cost per step that does not depend on , while fixing its two weaknesses: the gradient noise shrinks, and the examples are processed as one matrix operation that keeps a GPU busy.
The batch size is then a hyperparameter coupled to the learning rate. Goyal et al. (2017), "Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour", trained ResNet-50 with batches of 8192 images on 256 GPUs and matched the accuracy of the standard batch of 256, using two rules: scale the learning rate linearly with , and warm it up gradually over the first 5 epochs. McCandlish et al. (2018) and Shallue et al. (2019) then showed that this scaling works only up to a problem-dependent critical batch size.
The update and its noise
- batch size: examples averaged per update
- covariance of a single example's gradient at ; is SGD's () noise variance
- finite-population correction for sampling without replacement (a shuffled epoch): exactly 0 at ; drop it when
So the noise standard deviation falls as : 16 times the batch gives a 4 times less noisy gradient, at 16 times the compute per step. That is a poor deal if compute is the bottleneck, which is why the learning rate must change with for a bigger batch to pay off.
Batch size at a glance
- Steps per epoch
- , or with
drop_last=True - Gradient noise
- variance (std )
- Noise scale
- SGD's stochasticity is governed by roughly (Smith and Le, 2018); keep it fixed when changing
- Linear scaling rule
- , with warmup; e.g. for ResNet-50
- Critical batch size
- below it, batch with needs fewer steps; above it, the step count barely falls
- Memory
- activations grow linearly in ; weights and optimizer state do not
- Typical sizes
- 32–512 for vision on one GPU; millions of tokens per batch for large language models
The linear scaling rule
Why multiply by when the batch grows -fold? Compare SGD steps on small batches with one step on their union. If the gradient barely changes over those steps, the small-batch steps move , while the big batch moves . They agree when . Two caveats are built into the argument: it fails early in training when the gradient changes quickly (hence warmup), and it fails once approaches the curvature limit of plain gradient descent (hence a critical batch size).
- learning rate that works at batch size 1 (or per example)
- the largest step the curvature allows (of order )
- McCandlish et al.'s gradient noise scale: batches much smaller than it are noise-dominated (scaling works), much larger ones are curvature-limited (it does not)
For adaptive optimizers the rule is different. Adam divides by the gradient RMS, so its effective step does not grow with the gradient's signal-to-noise ratio in the same way; a square-root rule is the common heuristic, with theoretical support in Malladi et al. (2022).
Critical batch size: steps versus examples
Shallue et al. (2019) measured, across many models and datasets, the number of steps needed to reach a target validation error as a function of batch size, with the learning rate re-tuned for each . They found three regimes: perfect scaling, where doubling halves the steps; diminishing returns; and maximal data parallelism, where more batch does not reduce steps at all. Where the transition happens depends on the model, the dataset and the optimizer, and it tends to move to larger batches as training progresses and the gradient signal shrinks relative to its noise.
McCandlish et al. (2018) summarized the trade-off as between steps and examples processed : you can trade more compute for fewer sequential steps, but only down to , and the critical batch is the knee of that curve. Below the knee, larger batches are almost free speed on parallel hardware; above it, they waste compute.
Worked example: what a batch size implies
| B | η | RMS of g_B − ∇f | steps to target | examples to target |
|---|---|---|---|---|
| 1 | 0.010 | 0.551 | 2204 | 2204 |
| 4 | 0.040 | 0.274 | 550 | 2200 |
| 16 | 0.160 | 0.134 | 137 | 2186 |
| 64 | 0.640 | 0.060 | 34 | 2163 |
| 128 | 0.857 (cap) | 0.035 | 25 | 3149 |
| 256 | 0.857 (cap) | 0 | 24 | 6144 |
- 1Noise: . Without the finite-population factor it would be ; at the batch is the whole dataset and the noise is exactly zero.
- 2Up to the rule works perfectly: stays at , so the noise floor and the per-epoch progress are unchanged, and the steps fall by exactly the factor the batch grew.
- 3At the scaled rate would be , which exceeds the cap (and would approach the divergence limit ). From there on the step count stays near 24: the problem is now curvature-limited, like batch gradient descent.
- 4Data loading: with and , an epoch has steps, the last on only 40 examples.
drop_last=Truegives 15 steps of exactly 64 and skips 40 examples per epoch (different ones each epoch when shuffled). - 5Goyal et al.'s rule gives at and at ; at 8192 they needed the 5-epoch linear warmup from to avoid early divergence.
GPU efficiency
A mini-batch turns matrix-vector products into one matrix-matrix product, which GPUs execute far more efficiently: the weights are read from memory once per batch instead of once per example. Throughput (examples per second) therefore rises steeply with until the device is saturated, and then flattens; past that point a bigger batch only takes proportionally longer. NVIDIA recommends dimensions, batch size included, that are multiples of 8 for FP16 Tensor Cores (larger multiples can help on newer GPUs). Memory sets the upper limit: activations grow linearly in , and gradient accumulation lets you exceed it at the cost of more passes per step.
1import torch, torch.nn as nn2from torch.utils.data import DataLoader, TensorDataset34ds = TensorDataset(torch.randn(50_000, 32), torch.randint(0, 10, (50_000,)))5B = 10246loader = DataLoader(ds, batch_size=B, shuffle=True, # new permutation every epoch7 drop_last=True, # every step sees exactly B examples8 num_workers=4, pin_memory=True) # overlap loading with GPU compute9steps_per_epoch = len(loader) # 50_000 // 1024 = 481011model = nn.Sequential(nn.Linear(32, 256), nn.ReLU(), nn.Linear(256, 10))12base_lr, base_B = 0.1, 25613lr = base_lr * B / base_B # linear scaling rule: 0.414opt = torch.optim.SGD(model.parameters(), lr=lr, momentum=0.9, weight_decay=1e-4)1516warmup = 5 * steps_per_epoch # 5 epochs, as in Goyal et al.17sched = torch.optim.lr_scheduler.SequentialLR(opt, [18 torch.optim.lr_scheduler.LinearLR(opt, start_factor=base_B / B, total_iters=warmup),19 torch.optim.lr_scheduler.CosineAnnealingLR(opt, T_max=85 * steps_per_epoch),20], milestones=[warmup])2122loss_fn = nn.CrossEntropyLoss() # mean over the batch: gradient is (1/B)·Σ23for epoch in range(90):24 for xb, yb in loader:25 opt.zero_grad(set_to_none=True)26 loss_fn(model(xb), yb).backward()27 opt.step()28 sched.step()Use when
- +Noise variance : smoother, more reliable steps than one-example SGD
- +Matrix-matrix work saturates GPUs; far higher throughput than B = 1
- +Below the critical batch size, linear LR scaling converts more parallel hardware into proportionally fewer steps
- +Cost per step still independent of N; noise still regularizes
Avoid when
- − and are coupled: changing one without the other gives misleading comparisons
- −Above the critical batch size extra examples per step are mostly wasted compute
- −Very large batches need warmup and careful tuning, and may generalize slightly worse
- −Memory grows linearly in ; the largest batch is limited by the GPU
Check yourself
You move from batch size 64 to 1024 on the same model with plain SGD. Following the linear scaling rule, what should the learning rate become if it was 0.05, and what happens to the gradient-noise standard deviation?
Check yourself
Doubling the batch size (with the learning rate doubled) no longer reduces the number of steps to reach your target loss. What does that indicate?
Remember
- ✓Mini-batch GD averages example gradients: unbiased, variance , cost per step independent of .
- ✓Steps per epoch: ;
shuffle=Truereshuffles each epoch,drop_last=Truekeeps every batch full. - ✓Linear scaling rule (Goyal et al., 2017): with warmup keeps the noise scale fixed.
- ✓It works up to a critical batch size ; beyond it the curvature limit caps and steps stop falling.
- ✓Bigger batches use GPUs better until throughput saturates; memory grows linearly with B.