# Gradient accumulation

> Run K small micro-batches, let backward() add their gradients into .grad, and step once: the update equals one batch K times larger at the memory cost of one micro-batch. Divide each loss by K, zero the gradients once per step, and mind BatchNorm.

Source: https://www.mlfoundry.in/learn/gradient-accumulation  
Section: Deep Learning › Optimization Algorithms  
Level: core · 10 min read · updated 2026-10-11

**Gradient accumulation:** splits one optimizer step over $K$ forward/backward passes. Each micro-batch of $b$ examples adds its gradient into the same buffer, and `optimizer.step()` runs once after the $K$-th, on an **effective batch** of $B=b\times K$ examples.

Memory on an accelerator is dominated by activations saved for the backward pass, and they grow linearly with the batch. When the batch a recipe needs (thousands of sequences for a language model, 256 images for a ResNet schedule) does not fit, accumulation trades time for memory: $K$ passes of $b$ examples cost the same compute as one pass of $B$, but only one micro-batch's activations are alive at a time.

The trick works because the gradient of a sum is the sum of the gradients, and PyTorch's `backward()` **adds** into `.grad` instead of overwriting it. It has been standard practice for years; Ott et al. (2018) used it in fairseq ("delayed updates") to train translation models with the batch of many more GPUs than they had, and Hugging Face's Trainer and Accelerate expose it as a single setting.

## Why the gradients are identical

$$
\nabla_\theta\,\frac1B\sum_{i=1}^{B}\ell_i\;=\;\sum_{k=1}^{K}\frac1K\,\underbrace{\nabla_\theta\Big(\frac1b\sum_{i\in M_k}\ell_i\Big)}_{\text{backward of micro-batch }k}\qquad B=bK
$$

- $\ell_i$: the loss of example i
- $M_k$: micro-batch $k$, with $b$ examples; the $M_k$ partition the big batch
- $\tfrac1b\sum$: what a loss with `reduction="mean"` returns for one micro-batch
- $\tfrac1K$: the factor you must apply yourself: `(loss / K).backward()`

A mean over $B$ is the average of $K$ means over $b$, provided every micro-batch has the same size. So if each micro-batch's mean loss is divided by $K$ before `backward()`, the buffer ends up holding exactly the big-batch gradient, and the optimizer cannot tell the difference. Only floating-point summation order changes, at the level of $10^{-15}$ in float64 or $10^{-7}$ in float32.

**At a glance**

- **Effective batch:** $b\times K$ per device, times the number of data-parallel devices
- **Memory:** activations of one micro-batch; weights, gradients and optimizer state are unchanged
- **Compute:** the same FLOPs as the big batch, but $K$ sequential passes, so no faster; small micro-batches may use the GPU less efficiently
- **Exact when:** the loss is a mean over equally sized micro-batches and no layer couples examples (BatchNorm does)
- **Counts as:** one optimizer step: schedulers, clipping, logging and `zero_grad` happen once per $K$ micro-batches

> Figure: Linear regression on 64 points, effective batch 16, learning rate 0.1. Left: each micro-batch's own mean gradient (orange dots) and what each backward() adds to .grad, head to tail; the dashed arrow is the gradient of one big batch of the same 16 examples. With loss / K, the chain ends exactly on the dashed arrow for every K from 1 to 16 (largest gap 2·10⁻¹⁵), and the loss curve is the big-batch curve, ending at 0.2458 (optimum 0.2456). Forget / K and .grad is K times too long: it acts like lr·K. At K = 4 that is lr 0.4 and happens to train faster; at K = 8 (lr 0.8, above the stability limit of 0.73) the loss climbs to 345 after 40 steps, and at K = 16 it diverges within 11 steps. Zero the gradients inside the loop and only the last micro-batch survives, divided by K: at K = 4 the loss after 10 steps is 4.38 instead of 0.30.

The orange dots are worth a look too. Each micro-batch gradient is a noisy estimate of the big-batch gradient, and the big-batch gradient is their average. Accumulation does not change that average; it only computes it in pieces.

## Worked example: four examples, K = 2

Model $\hat y=wx$ at $w=0$, squared error, so $\partial\ell_i/\partial w=2(wx_i-y_i)x_i=-2x_iy_i$. The big batch is $(x,y)=(1,2),(2,3),(3,5),(-1,-1)$, split into micro-batches $A=\{(1,2),(2,3)\}$ and $B=\{(3,5),(-1,-1)\}$.

What .grad holds when optimizer.step() runs

| loop | micro A adds | micro B adds | .grad at step() | equals |
| --- | --- | --- | --- | --- |
| one batch of 4 | – | – | $-2\cdot\frac{2+6+15+1}{4}=-12$ | reference |
| accumulate, loss / 2 | $-8/2=-4$ | $-16/2=-8$ | $-12$ | reference ✓ |
| accumulate, forgot / 2 | $-8$ | $-16$ | $-24$ | $2\times$: lr doubled |
| zero_grad before each micro | $-4$, then erased | $-8$ | $-8$ | half of B only |

The micro-batch means are $-2\cdot\frac{2+6}{2}=-8$ and $-2\cdot\frac{15+1}{2}=-16$; their average is the reference $-12$.

## Where accumulation stops being exact

> **Uneven micro-batches and token-level losses:** "Average of means equals the overall mean" needs equal sizes. In language modelling the loss is a mean over **tokens**, and micro-batches hold different numbers of real (non-padding) tokens. Example: micro-batch A has 3 tokens with loss 2 each, B has 1 token with loss 6. The true mean over the 4 tokens is $12/4=3$, but $(\text{mean}_A+\text{mean}_B)/2=(2+6)/2=4$: B's single token gets three times the weight it should. The fix is to sum token losses (`reduction="sum"`) and divide by the total token count of the whole accumulation window. Several popular fine-tuning libraries shipped this bug until 2024. The same applies to a final, smaller micro-batch at the end of an epoch.

> **BatchNorm sees the micro-batch:** Batch normalization computes its mean and variance over the examples in **one forward pass**, so with accumulation it normalizes over $b$ examples, not $B$. For unit-variance features the standard deviation of a batch mean is $1/\sqrt b$: $0.5$ at $b=4$ versus $0.25$ at $B=16$. The statistics are noisier, the running averages differ, and very small micro-batches (2 to 8) hurt accuracy. Options: GroupNorm or LayerNorm (no batch dependence), SyncBatchNorm across devices, or a larger micro-batch. Transformers use LayerNorm or RMSNorm, so they are unaffected.

Everything else that runs per forward pass is unaffected in expectation: dropout draws independent masks per example anyway, and data augmentation is per example.

## Writing the loop

**One optimizer step with accumulation**

1. `optimizer.zero_grad()` **once**, before the first micro-batch of the step (or right after the previous `step()`).
2. For each of the $K$ micro-batches: forward, `loss = loss / K`, `loss.backward()`. Gradients add up in `.grad`.
3. After the $K$-th backward only: unscale (fp16 AMP), clip the gradient norm, `optimizer.step()`, `scheduler.step()`, then `zero_grad()`.
4. Count steps in optimizer steps. An epoch of $n$ micro-batches has $\lceil n/K\rceil$ of them; size warmup and decay schedules with that number, not $n$.

```python
import contextlib, torch
from torch.nn.parallel import DistributedDataParallel as DDP

K = 8                                    # accumulation steps
model = DDP(model)                       # effective batch = micro_bs * K * world_size
optimizer = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.1)
steps_per_epoch = -(-len(loader) // K)   # ceil: optimizer steps, not micro-batches
scheduler = torch.optim.lr_scheduler.OneCycleLR(optimizer, max_lr=3e-4,
                                                total_steps=epochs * steps_per_epoch)

optimizer.zero_grad(set_to_none=True)
for i, (x, y) in enumerate(loader):
    boundary = (i + 1) % K == 0 or (i + 1) == len(loader)
    # no_sync(): skip the gradient all-reduce on all but the last micro-batch
    sync = contextlib.nullcontext() if boundary else model.no_sync()
    with sync:
        with torch.autocast(device_type="cuda", dtype=torch.bfloat16):
            loss = loss_fn(model(x), y)
        (loss / K).backward()            # .grad += grad(loss) / K
    if boundary:
        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)   # once per step
        optimizer.step()
        scheduler.step()                 # once per OPTIMIZER step
        optimizer.zero_grad(set_to_none=True)
# A short final group (len(loader) not divisible by K) is still divided by K,
# so that step is smaller; drop it or divide by its real size instead.
```

Accumulation with DDP, bf16 autocast, clipping and a per-step scheduler

> **DDP and no_sync:** DistributedDataParallel all-reduces gradients during every `backward()`. With accumulation that is $K-1$ wasted rounds of communication per step. Inside `with model.no_sync():` the reduction is skipped and gradients accumulate locally; the final micro-batch, outside the context, triggers one all-reduce of the accumulated sum. The result is the same, with $1/K$ of the traffic. FSDP has the same context manager, though skipping the reduce-scatter there costs memory, because each rank must then keep full unsharded gradients.

> **With fp16 and GradScaler:** Call `scaler.scale(loss / K).backward()` on every micro-batch, and `scaler.unscale_`, clipping, `scaler.step(optimizer)` and `scaler.update()` only at the boundary. The scale must stay constant within one accumulation window, otherwise the micro-batches in `.grad` are scaled differently. With bf16, as in the code above, no scaler is needed.

> **Accumulation is not a free lunch on batch size:** It reproduces a large batch exactly; it does not make a large batch a good idea. If you raise the effective batch from 256 to 2048 you usually need a larger learning rate (the linear scaling rule with warmup, Goyal et al., 2017) and you take 8 times fewer optimizer steps per epoch. Past a problem-dependent "critical batch size", more examples per step stop buying faster progress.

**Strengths**

- Train with the batch size a recipe needs on hardware that cannot hold it
- Mathematically the same update as the big batch for mean losses without BatchNorm
- Two extra lines in the loop; frameworks expose it as one setting
- With no_sync, communication per optimizer step drops by a factor of K

**Limitations**

- No speed-up: K sequential passes, and tiny micro-batches can under-use the GPU
- BatchNorm statistics come from the micro-batch, not the effective batch
- Silent bugs: missing / K (acts like lr × K), zero_grad inside the loop, schedulers stepped per micro-batch
- Token-averaged losses need normalization by the total token count of the window

**Question:** You accumulate over K = 4 micro-batches but call loss.backward() without dividing by K. Using SGD, what is the effect?

- A. None: the optimizer averages automatically
- B. The update is 4 times larger, exactly as if the learning rate were 4 times higher
- C. The gradient becomes noisier but has the same size
- D. Only the last micro-batch is used

**Answer:** B. The update is 4 times larger, exactly as if the learning rate were 4 times higher — `.grad` holds the sum of four micro-batch means, which is $4\times$ the big-batch mean. With SGD the update is $\eta\cdot4g$, so training behaves exactly as with learning rate $4\eta$ (in the figure, K = 4 with lr 0.1 reproduces lr 0.4 to within $10^{-16}$). With Adam the factor mostly cancels in $m/\sqrt v$, so the bug hides there; only the relative size of $\epsilon$ (and a coupled L2 term) changes.

**Question:** An epoch has 1000 micro-batches and you accumulate over K = 8. How many times should a per-step learning-rate scheduler (e.g. cosine decay) be stepped per epoch?

- A. 1000
- B. 125
- C. 8
- D. Once per epoch

**Answer:** B. 125 — Schedulers should advance once per **optimizer** step: $1000/8=125$ times. Stepping per micro-batch would run the schedule 8 times too fast, finishing warmup and decay after 1/8 of training.

**Key points**

- Effective batch $=$ micro-batch $\times K$ (× data-parallel devices); memory is that of one micro-batch.
- Divide each micro-batch loss by $K$: then `.grad` equals the big-batch gradient exactly. Forgetting it multiplies the learning rate by $K$.
- `zero_grad` once per optimizer step, not per micro-batch; step the optimizer, clip, and step the scheduler only at the boundary.
- Exactness needs equal-size micro-batches and per-example layers: normalize token losses by the window's total tokens, and expect BatchNorm to see only the micro-batch.
- Under DDP wrap all but the last micro-batch in `model.no_sync()` to avoid $K-1$ redundant all-reduces.
- It reproduces a large batch; it does not make training faster or a large batch better.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
