Gradient accumulation

Run K small micro-batches, let backward() add their gradients into .grad, and step once: the update equals one batch K times larger at the memory cost of one micro-batch. Divide each loss by K, zero the gradients once per step, and mind BatchNorm.

10 min read896 wordscore0 viewsupdated 2026-10-11

Definition

Gradient accumulation splits one optimizer step over KK forward/backward passes. Each micro-batch of bb examples adds its gradient into the same buffer, and optimizer.step() runs once after the KK-th, on an effective batch of B=b×KB=b\times K examples.

Memory on an accelerator is dominated by activations saved for the backward pass, and they grow linearly with the batch. When the batch a recipe needs (thousands of sequences for a language model, 256 images for a ResNet schedule) does not fit, accumulation trades time for memory: KK passes of bb examples cost the same compute as one pass of BB, but only one micro-batch's activations are alive at a time.

The trick works because the gradient of a sum is the sum of the gradients, and PyTorch's backward() adds into .grad instead of overwriting it. It has been standard practice for years; Ott et al. (2018) used it in fairseq ("delayed updates") to train translation models with the batch of many more GPUs than they had, and Hugging Face's Trainer and Accelerate expose it as a single setting.

Why the gradients are identical

∇θ 1B∑i=1Bℓi  =  ∑k=1K1K ∇θ(1b∑i∈Mkℓi)⏟backward of micro-batch kB=bK\nabla_\theta\,\frac1B\sum_{i=1}^{B}\ell_i\;=\;\sum_{k=1}^{K}\frac1K\,\underbrace{\nabla_\theta\Big(\frac1b\sum_{i\in M_k}\ell_i\Big)}_{\text{backward of micro-batch }k}\qquad B=bK
ℓi\ell_i
the loss of example i
MkM_k
micro-batch kk, with bb examples; the MkM_k partition the big batch
1b∑\tfrac1b\sum
what a loss with reduction="mean" returns for one micro-batch
1K\tfrac1K
the factor you must apply yourself: (loss / K).backward()

A mean over BB is the average of KK means over bb, provided every micro-batch has the same size. So if each micro-batch's mean loss is divided by KK before backward(), the buffer ends up holding exactly the big-batch gradient, and the optimizer cannot tell the difference. Only floating-point summation order changes, at the level of 10−1510^{-15} in float64 or 10−710^{-7} in float32.

At a glance

Effective batch
b×Kb\times K per device, times the number of data-parallel devices
Memory
activations of one micro-batch; weights, gradients and optimizer state are unchanged
Compute
the same FLOPs as the big batch, but KK sequential passes, so no faster; small micro-batches may use the GPU less efficiently
Exact when
the loss is a mean over equally sized micro-batches and no layer couples examples (BatchNorm does)
Counts as
one optimizer step: schedulers, clipping, logging and zero_grad happen once per KK micro-batches
.grad in (∂L/∂w, ∂L/∂c), optimizer step 1∂L/∂w∂L/∂chalf-width 19.76full-data loss per optimizer step (log)0.3110100optimum 0.246010203040optimizer steps (1 done)b = 4 per micro-batch × K = 4 → effective batch 16; micro-batch 4/4.grad = (−10.36, 7.30) · big batch = (−10.36, 7.30) · gap 2e−15largest gap over the run: 2e−15 (rounding only: identical updates)● micro-batch mean gradients ━ added by backward() ━ .grad at step() ┄ big batch ━/┄ loss
|
accumulation steps K
fig.Linear regression on 64 points, effective batch 16, learning rate 0.1. Left: each micro-batch's own mean gradient (orange dots) and what each backward() adds to .grad, head to tail; the dashed arrow is the gradient of one big batch of the same 16 examples. With loss / K, the chain ends exactly on the dashed arrow for every K from 1 to 16 (largest gap 2·10⁻¹⁵), and the loss curve is the big-batch curve, ending at 0.2458 (optimum 0.2456). Forget / K and .grad is K times too long: it acts like lr·K. At K = 4 that is lr 0.4 and happens to train faster; at K = 8 (lr 0.8, above the stability limit of 0.73) the loss climbs to 345 after 40 steps, and at K = 16 it diverges within 11 steps. Zero the gradients inside the loop and only the last micro-batch survives, divided by K: at K = 4 the loss after 10 steps is 4.38 instead of 0.30.

The orange dots are worth a look too. Each micro-batch gradient is a noisy estimate of the big-batch gradient, and the big-batch gradient is their average. Accumulation does not change that average; it only computes it in pieces.

Worked example: four examples, K = 2

Model y^=wx\hat y=wx at w=0w=0, squared error, so ∂ℓi/∂w=2(wxi−yi)xi=−2xiyi\partial\ell_i/\partial w=2(wx_i-y_i)x_i=-2x_iy_i. The big batch is (x,y)=(1,2),(2,3),(3,5),(−1,−1)(x,y)=(1,2),(2,3),(3,5),(-1,-1), split into micro-batches A={(1,2),(2,3)}A=\{(1,2),(2,3)\} and B={(3,5),(−1,−1)}B=\{(3,5),(-1,-1)\}.

loopmicro A addsmicro B adds.grad at step()equals
one batch of 4––−2⋅2+6+15+14=−12-2\cdot\frac{2+6+15+1}{4}=-12reference
accumulate, loss / 2−8/2=−4-8/2=-4−16/2=−8-16/2=-8−12-12reference ✓
accumulate, forgot / 2−8-8−16-16−24-242×2\times: lr doubled
zero_grad before each micro−4-4, then erased−8-8−8-8half of B only
What .grad holds when optimizer.step() runs

The micro-batch means are −2⋅2+62=−8-2\cdot\frac{2+6}{2}=-8 and −2⋅15+12=−16-2\cdot\frac{15+1}{2}=-16; their average is the reference −12-12.

Where accumulation stops being exact

Everything else that runs per forward pass is unaffected in expectation: dropout draws independent masks per example anyway, and data augmentation is per example.

Writing the loop

One optimizer step with accumulation

  1. 1optimizer.zero_grad() once, before the first micro-batch of the step (or right after the previous step()).
  2. 2For each of the KK micro-batches: forward, loss = loss / K, loss.backward(). Gradients add up in .grad.
  3. 3After the KK-th backward only: unscale (fp16 AMP), clip the gradient norm, optimizer.step(), scheduler.step(), then zero_grad().
  4. 4Count steps in optimizer steps. An epoch of nn micro-batches has ⌈n/K⌉\lceil n/K\rceil of them; size warmup and decay schedules with that number, not nn.
python
1import contextlib, torch
2from torch.nn.parallel import DistributedDataParallel as DDP
3
4K = 8 # accumulation steps
5model = DDP(model) # effective batch = micro_bs * K * world_size
6optimizer = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.1)
7steps_per_epoch = -(-len(loader) // K) # ceil: optimizer steps, not micro-batches
8scheduler = torch.optim.lr_scheduler.OneCycleLR(optimizer, max_lr=3e-4,
9 total_steps=epochs * steps_per_epoch)
10
11optimizer.zero_grad(set_to_none=True)
12for i, (x, y) in enumerate(loader):
13 boundary = (i + 1) % K == 0 or (i + 1) == len(loader)
14 # no_sync(): skip the gradient all-reduce on all but the last micro-batch
15 sync = contextlib.nullcontext() if boundary else model.no_sync()
16 with sync:
17 with torch.autocast(device_type="cuda", dtype=torch.bfloat16):
18 loss = loss_fn(model(x), y)
19 (loss / K).backward() # .grad += grad(loss) / K
20 if boundary:
21 torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0) # once per step
22 optimizer.step()
23 scheduler.step() # once per OPTIMIZER step
24 optimizer.zero_grad(set_to_none=True)
25# A short final group (len(loader) not divisible by K) is still divided by K,
26# so that step is smaller; drop it or divide by its real size instead.
Accumulation with DDP, bf16 autocast, clipping and a per-step scheduler

Use when

  • +Train with the batch size a recipe needs on hardware that cannot hold it
  • +Mathematically the same update as the big batch for mean losses without BatchNorm
  • +Two extra lines in the loop; frameworks expose it as one setting
  • +With no_sync, communication per optimizer step drops by a factor of K

Avoid when

  • −No speed-up: K sequential passes, and tiny micro-batches can under-use the GPU
  • −BatchNorm statistics come from the micro-batch, not the effective batch
  • −Silent bugs: missing / K (acts like lr × K), zero_grad inside the loop, schedulers stepped per micro-batch
  • −Token-averaged losses need normalization by the total token count of the window

Check yourself

You accumulate over K = 4 micro-batches but call loss.backward() without dividing by K. Using SGD, what is the effect?

Check yourself

An epoch has 1000 micro-batches and you accumulate over K = 8. How many times should a per-step learning-rate scheduler (e.g. cosine decay) be stepped per epoch?

Remember

  • ✓Effective batch == micro-batch ×K\times K (× data-parallel devices); memory is that of one micro-batch.
  • ✓Divide each micro-batch loss by KK: then .grad equals the big-batch gradient exactly. Forgetting it multiplies the learning rate by KK.
  • ✓zero_grad once per optimizer step, not per micro-batch; step the optimizer, clip, and step the scheduler only at the boundary.
  • ✓Exactness needs equal-size micro-batches and per-example layers: normalize token losses by the window's total tokens, and expect BatchNorm to see only the micro-batch.
  • ✓Under DDP wrap all but the last micro-batch in model.no_sync() to avoid K−1K-1 redundant all-reduces.
  • ✓It reproduces a large batch; it does not make training faster or a large batch better.
← back to Deep Learning roadmap