# Mini-batch gradient descent

> Average the gradient over B random examples: noise variance falls as 1/B and the batch runs in parallel on a GPU. Up to a critical batch size, scaling the learning rate with B (Goyal et al. 2017) turns k× bigger batches into k× fewer steps; beyond it, extra batch buys little.

Source: https://www.mlfoundry.in/learn/mini-batch-gradient-descent  
Section: Deep Learning › Optimization Algorithms  
Level: core · 11 min read · updated 2026-10-11

**Mini-batch gradient descent:** (mini-batch SGD) updates the parameters with the average gradient of a random subset $\mathcal B_t$ of $B$ examples: $\theta_{t+1}=\theta_t-\eta\,\frac1B\sum_{i\in\mathcal B_t}\nabla\ell_i(\theta_t)$. $B=1$ is SGD, $B=N$ is batch gradient descent; everything used in practice lies in between.

This is what "SGD" means in nearly every deep-learning paper and in `torch.optim.SGD`: the gradient of a mini-batch of, say, 32 to 4096 examples. It keeps SGD's key property, a cost per step that does not depend on $N$, while fixing its two weaknesses: the gradient noise shrinks, and the $B$ examples are processed as one matrix operation that keeps a GPU busy.

The batch size is then a hyperparameter coupled to the learning rate. Goyal et al. (2017), "Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour", trained ResNet-50 with batches of 8192 images on 256 GPUs and matched the accuracy of the standard batch of 256, using two rules: scale the learning rate linearly with $B$, and warm it up gradually over the first 5 epochs. McCandlish et al. (2018) and Shallue et al. (2019) then showed that this scaling works only up to a problem-dependent **critical batch size**.

## The update and its noise

$$
g_t=\frac1B\sum_{i\in\mathcal B_t}\nabla\ell_i(\theta_t),\qquad \mathbb E[g_t]=\nabla f(\theta_t),\qquad \mathbb E\lVert g_t-\nabla f\rVert^2=\frac{\operatorname{tr}\Sigma}{B}\cdot\frac{N-B}{N-1}
$$

- $B$: batch size: examples averaged per update
- $\Sigma$: covariance of a single example's gradient at $\theta_t$; $\operatorname{tr}\Sigma$ is SGD's ($B=1$) noise variance
- $\tfrac{N-B}{N-1}$: finite-population correction for sampling without replacement (a shuffled epoch): exactly 0 at $B=N$; drop it when $N\gg B$

So the noise **standard deviation** falls as $1/\sqrt B$: 16 times the batch gives a 4 times less noisy gradient, at 16 times the compute per step. That is a poor deal if compute is the bottleneck, which is why the learning rate must change with $B$ for a bigger batch to pay off.

**Batch size at a glance**

- **Steps per epoch:** $\lceil N/B\rceil$, or $\lfloor N/B\rfloor$ with `drop_last=True`
- **Gradient noise:** variance $\propto 1/B$ (std $\propto1/\sqrt B$)
- **Noise scale:** SGD's stochasticity is governed by roughly $\eta/B$ (Smith and Le, 2018); keep it fixed when changing $B$
- **Linear scaling rule:** $\eta=\eta_{\text{base}}\cdot B/B_{\text{base}}$, with warmup; e.g. $0.1\cdot B/256$ for ResNet-50
- **Critical batch size:** below it, $k\times$ batch with $k\times\eta$ needs $k\times$ fewer steps; above it, the step count barely falls
- **Memory:** activations grow linearly in $B$; weights and optimizer state do not
- **Typical sizes:** 32–512 for vision on one GPU; millions of tokens per batch for large language models

## The linear scaling rule

Why multiply $\eta$ by $k$ when the batch grows $k$-fold? Compare $k$ SGD steps on small batches with one step on their union. If the gradient barely changes over those $k$ steps, the small-batch steps move $\eta\sum_{j<k}g_{t+j}\approx k\eta\,\bar g$, while the big batch moves $\eta'\bar g$. They agree when $\eta'=k\eta$. Two caveats are built into the argument: it fails early in training when the gradient changes quickly (hence warmup), and it fails once $k\eta$ approaches the curvature limit $2/L$ of plain gradient descent (hence a critical batch size).

$$
\eta(B)=\eta_1\,B\ \ \text{for } B\lesssim B^*,\qquad B^*\approx\frac{\eta_{\max}}{\eta_1},\qquad \mathcal B_{\text{noise}}=\frac{\operatorname{tr}\Sigma}{\lVert\nabla f\rVert^2}
$$

- $\eta_1$: learning rate that works at batch size 1 (or per example)
- $\eta_{\max}$: the largest step the curvature allows (of order $1/L$)
- $\mathcal B_{\text{noise}}$: McCandlish et al.'s gradient noise scale: batches much smaller than it are noise-dominated (scaling works), much larger ones are curvature-limited (it does not)

For adaptive optimizers the rule is different. Adam divides by the gradient RMS, so its effective step does not grow with the gradient's signal-to-noise ratio in the same way; a square-root rule $\eta\propto\sqrt B$ is the common heuristic, with theoretical support in Malladi et al. (2022).

> Figure: Mini-batch SGD on a 256-example least-squares problem, 12 epochs, batches drawn like DataLoader(shuffle=True). The orange inset is 120 sampled values of g_B − ∇f at θ*, on a fixed scale, with the exact RMS circled: 0.551 at B = 1, 0.134 at B = 16, 0.060 at B = 64 and exactly 0 at B = N = 256. Lower right: steps needed to reach f − f* < 2.3·10⁻³ (5 seeds). With linear scaling η = min(0.01·B, 1/L) the steps halve with every doubling, from 2204 at B = 1 to 34 at B = 64, and the total examples stay near 2,200. Past B* = (1/L)/η₁ ≈ 86 the learning rate hits its cap: B = 256 needs 24 steps but 6,144 examples. With a fixed η every batch size needs about 2,200 steps (grey): bigger batches then only cost more. Per epoch, every linearly scaled batch size below B* traces nearly the same loss curve (compare the grey B = 1 curve).

## Critical batch size: steps versus examples

Shallue et al. (2019) measured, across many models and datasets, the number of steps needed to reach a target validation error as a function of batch size, with the learning rate re-tuned for each $B$. They found three regimes: **perfect scaling**, where doubling $B$ halves the steps; **diminishing returns**; and **maximal data parallelism**, where more batch does not reduce steps at all. Where the transition happens depends on the model, the dataset and the optimizer, and it tends to move to larger batches as training progresses and the gradient signal shrinks relative to its noise.

McCandlish et al. (2018) summarized the trade-off as $(S/S_{\min}-1)(E/E_{\min}-1)=1$ between steps $S$ and examples processed $E$: you can trade more compute for fewer sequential steps, but only down to $S_{\min}$, and the critical batch $\approx\mathcal B_{\text{noise}}$ is the knee of that curve. Below the knee, larger batches are almost free speed on parallel hardware; above it, they waste compute.

## Worked example: what a batch size implies

Gradient noise at θ* and steps to target in the figure (N = 256, η₁ = 0.01, linear scaling capped at 1/L = 0.857); computed by the figure's code

| B | η | RMS of g_B − ∇f | steps to target | examples to target |
| --- | --- | --- | --- | --- |
| 1 | 0.010 | 0.551 | 2204 | 2204 |
| 4 | 0.040 | 0.274 | 550 | 2200 |
| 16 | 0.160 | 0.134 | 137 | 2186 |
| 64 | 0.640 | 0.060 | 34 | 2163 |
| 128 | 0.857 (cap) | 0.035 | 25 | 3149 |
| 256 | 0.857 (cap) | 0 | 24 | 6144 |

1. Noise: $0.551\sqrt{(256-16)/(16\cdot255)}=0.551\times0.243=0.134$. Without the finite-population factor it would be $0.551/4=0.138$; at $B=N$ the batch is the whole dataset and the noise is exactly zero.
2. Up to $B=64$ the rule works perfectly: $\eta B^{-1}$ stays at $0.01$, so the noise floor and the per-epoch progress are unchanged, and the steps fall by exactly the factor the batch grew.
3. At $B=128$ the scaled rate would be $1.28$, which exceeds the cap $1/L=0.857$ (and would approach the divergence limit $2/L=1.71$). From there on the step count stays near 24: the problem is now curvature-limited, like batch gradient descent.
4. Data loading: with $N=1000$ and $B=64$, an epoch has $\lceil1000/64\rceil=16$ steps, the last on only 40 examples. `drop_last=True` gives 15 steps of exactly 64 and skips 40 examples per epoch (different ones each epoch when shuffled).
5. Goyal et al.'s rule $\eta=0.1\cdot B/256$ gives $\eta=0.4$ at $B=1024$ and $\eta=3.2$ at $B=8192$; at 8192 they needed the 5-epoch linear warmup from $0.1$ to avoid early divergence.

## GPU efficiency

A mini-batch turns $B$ matrix-vector products into one matrix-matrix product, which GPUs execute far more efficiently: the weights are read from memory once per batch instead of once per example. Throughput (examples per second) therefore rises steeply with $B$ until the device is saturated, and then flattens; past that point a bigger batch only takes proportionally longer. NVIDIA recommends dimensions, batch size included, that are multiples of 8 for FP16 Tensor Cores (larger multiples can help on newer GPUs). Memory sets the upper limit: activations grow linearly in $B$, and gradient accumulation lets you exceed it at the cost of more passes per step.

```python
import torch, torch.nn as nn
from torch.utils.data import DataLoader, TensorDataset

ds = TensorDataset(torch.randn(50_000, 32), torch.randint(0, 10, (50_000,)))
B = 1024
loader = DataLoader(ds, batch_size=B, shuffle=True,   # new permutation every epoch
                    drop_last=True,                    # every step sees exactly B examples
                    num_workers=4, pin_memory=True)    # overlap loading with GPU compute
steps_per_epoch = len(loader)                          # 50_000 // 1024 = 48

model = nn.Sequential(nn.Linear(32, 256), nn.ReLU(), nn.Linear(256, 10))
base_lr, base_B = 0.1, 256
lr = base_lr * B / base_B                              # linear scaling rule: 0.4
opt = torch.optim.SGD(model.parameters(), lr=lr, momentum=0.9, weight_decay=1e-4)

warmup = 5 * steps_per_epoch                           # 5 epochs, as in Goyal et al.
sched = torch.optim.lr_scheduler.SequentialLR(opt, [
    torch.optim.lr_scheduler.LinearLR(opt, start_factor=base_B / B, total_iters=warmup),
    torch.optim.lr_scheduler.CosineAnnealingLR(opt, T_max=85 * steps_per_epoch),
], milestones=[warmup])

loss_fn = nn.CrossEntropyLoss()                        # mean over the batch: gradient is (1/B)·Σ
for epoch in range(90):
    for xb, yb in loader:
        opt.zero_grad(set_to_none=True)
        loss_fn(model(xb), yb).backward()
        opt.step()
        sched.step()
```

Mini-batch training with a shuffled DataLoader, the linear scaling rule and warmup

> **Changing B without changing η:** If you raise the batch size and keep the learning rate, each epoch makes $k\times$ fewer steps of the same length: training per epoch gets slower, not faster, and it looks as if large batches "generalize worse". Rescale $\eta$ (linear for SGD, roughly square-root for Adam) and add warmup before comparing batch sizes. Compare runs by steps or by examples, not by epochs alone.

> **BatchNorm and tiny last batches:** With `drop_last=False` the last batch of each epoch can be very small (a single example if $N\bmod B=1$), and BatchNorm in training mode then normalizes with a mean and variance estimated from almost nothing; with one example per channel it raises an error. Use `drop_last=True` for training; keep `False` for evaluation so no example is skipped.

> **The noise scale is η/B:** In the figure every linearly scaled batch size below $B^*$ follows nearly the same loss curve per epoch, because $\eta/B$ (the noise temperature) and $\eta\times$steps per epoch (distance travelled) are both unchanged. Smith et al. (2018) used the same equivalence in reverse: instead of decaying the learning rate, increase the batch size, and get the same accuracy with fewer parameter updates.

**Strengths**

- Noise variance $\propto1/B$: smoother, more reliable steps than one-example SGD
- Matrix-matrix work saturates GPUs; far higher throughput than B = 1
- Below the critical batch size, linear LR scaling converts more parallel hardware into proportionally fewer steps
- Cost per step still independent of N; noise still regularizes

**Limitations**

- $B$ and $\eta$ are coupled: changing one without the other gives misleading comparisons
- Above the critical batch size extra examples per step are mostly wasted compute
- Very large batches need warmup and careful tuning, and may generalize slightly worse
- Memory grows linearly in $B$; the largest batch is limited by the GPU

**Question:** You move from batch size 64 to 1024 on the same model with plain SGD. Following the linear scaling rule, what should the learning rate become if it was 0.05, and what happens to the gradient-noise standard deviation?

- A. 0.8; the noise std falls by 4×
- B. 0.2; the noise std falls by 16×
- C. 0.05; the noise std falls by 4×
- D. 0.8; the noise std falls by 16×

**Answer:** A. 0.8; the noise std falls by 4× — The batch grows $16\times$, so $\eta=0.05\cdot16=0.8$ (with warmup). The noise variance falls by 16, so its standard deviation falls by $\sqrt{16}=4$. The noise scale $\eta/B$ is unchanged, which is the point of the rule.

**Question:** Doubling the batch size (with the learning rate doubled) no longer reduces the number of steps to reach your target loss. What does that indicate?

- A. You are above the critical batch size: progress is limited by curvature, not gradient noise
- B. The DataLoader is not shuffling
- C. The learning rate is too small
- D. The model has too few parameters

**Answer:** A. You are above the critical batch size: progress is limited by curvature, not gradient noise — Below the critical batch size, steps are noise-limited and a less noisy gradient with a proportionally larger step helps. Above it, the gradient is already accurate and the step is capped by the curvature ($\eta\lesssim1/L$), as for batch gradient descent, so more examples per step buy almost nothing.

**Key points**

- Mini-batch GD averages $B$ example gradients: unbiased, variance $\frac{\operatorname{tr}\Sigma}{B}\frac{N-B}{N-1}$, cost per step independent of $N$.
- Steps per epoch: $\lceil N/B\rceil$; `shuffle=True` reshuffles each epoch, `drop_last=True` keeps every batch full.
- Linear scaling rule (Goyal et al., 2017): $\eta\propto B$ with warmup keeps the noise scale $\eta/B$ fixed.
- It works up to a critical batch size $\approx\mathcal B_{\text{noise}}$; beyond it the curvature limit caps $\eta$ and steps stop falling.
- Bigger batches use GPUs better until throughput saturates; memory grows linearly with B.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
