Definition
Epoch, mini-batch, iteration A mini-batch is a group of training examples whose average gradient produces one parameter update. That update is one iteration (also called a step). An epoch is one full pass in which every one of the training examples has been used once, which takes iterations.
Why not use all the data for every update?
The training loss is an average over examples, and so is its gradient. Computing the exact gradient means a forward and backward pass over the whole dataset (for ImageNet, 1.28 million images) just to take one step.
A random subset of examples gives an unbiased estimate of that gradient at a fraction of the cost. The estimate is noisy, but you get updates for the price of one exact one, and early in training many rough steps beat one precise step.
| Variant | Batch size | Updates per epoch | Gradient |
|---|---|---|---|
| Batch (full-batch) gradient descent | 1 | exact, deterministic | |
| Stochastic gradient descent (SGD) | noisiest estimate, no parallelism | ||
| Mini-batch gradient descent | (typically 32–1024) | noisy, and vectorizes well on a GPU |
In practice everyone uses mini-batches and still calls it "SGD": torch.optim.SGD simply updates with whatever batch you hand it.
The math
- iteration counter (one per update, never reset between epochs)
- the mini-batch used at iteration ; except possibly the last batch of an epoch
- learning rate
- per-example loss
- mini-batch gradient, an unbiased estimate of the full gradient
- number of training examples
- batch size
- number of epochs
How noisy is ? If individual gradients have variance around the true gradient, then averaging of them drawn without replacement gives
- variance of a single example's gradient (summed over parameters)
- finite-population correction; it is 0 at , where the gradient is exact
So quadrupling the batch only halves the noise while costing four times the compute per step. That square root is why huge batches give diminishing returns per example processed.
Worked example: N = 50,000 (CIFAR-10's training set)
| B | Iterations / epoch | Last batch | With drop_last | Iterations in 100 epochs | Relative noise |
|---|---|---|---|---|---|
| 1 | 50,000 | 1 | 50,000 | 5,000,000 | 1 |
| 32 | 1,563 | 16 | 1,562 | 156,300 | 0.177 |
| 128 | 391 | 80 | 390 | 39,100 | 0.088 |
| 512 | 98 | 336 | 97 | 9,800 | 0.044 |
| 50,000 | 1 | 50,000 | 1 | 100 | 0 |
The B = 128 row
- 1, so 390 full batches cover examples.
- 2The remaining examples form a 391st, smaller batch: iterations per epoch.
- 3With
drop_last=Truethose 80 are skipped each epoch (a different 80 each time, because of shuffling), leaving 390 iterations. - 4A 100-epoch run is iterations. Learning-rate schedules and logging are usually counted in these iterations, not in epochs.
Shuffling and the last batch
Reshuffle every epoch. If the order is fixed, every epoch sees identical batches in the same sequence, and any structure in that order (for example, data sorted by class) biases the updates. A fresh permutation each epoch keeps close to an independent random sample.
The partial batch still averages its loss, so the update has the usual scale, just more noise. It is harmless for most models. Drop it when every batch must have the same shape (some compiled or distributed setups) or when a layer misbehaves on tiny batches: BatchNorm in training mode raises an error on a batch of one sample.
Choosing B: memory versus time
Use when
- +Larger B: better GPU utilization, fewer iterations per epoch, smoother gradient estimate, so faster wall-clock time per epoch.
- +Activation memory grows roughly linearly in , so the largest batch that fits is bounded by GPU memory. Weights and optimizer state do not depend on .
Avoid when
- −Per-example returns shrink: noise falls only as , so past some size, doubling barely reduces the number of updates needed.
- −Very large batches usually need a retuned learning rate (a common heuristic scales in proportion to , with warm-up) and can generalize slightly worse without it.
If the batch you want does not fit, use gradient accumulation: run smaller micro-batches, add up their gradients, and call optimizer.step() once. That is one iteration with an effective batch of the micro-batch size.
Where the counters live in PyTorch
1import torch2from torch.utils.data import DataLoader, TensorDataset34X, y = torch.randn(50_000, 20), torch.randint(0, 10, (50_000,))5loader = DataLoader(TensorDataset(X, y), batch_size=128, shuffle=True, drop_last=False)6print(len(loader)) # 391 = ceil(50000 / 128)78model = torch.nn.Linear(20, 10)9opt = torch.optim.SGD(model.parameters(), lr=0.1)10loss_fn = torch.nn.CrossEntropyLoss()1112step = 0 # global iteration counter13for epoch in range(10): # one epoch = one full pass over the loader14 for xb, yb in loader: # new permutation each time this loop starts15 loss = loss_fn(model(xb), yb) # mean loss over this mini-batch16 opt.zero_grad()17 loss.backward()18 opt.step() # one iteration = one parameter update19 step += 120 print(epoch, step) # step == (epoch + 1) * 391Check yourself
, , drop_last=False, 5 epochs. How many parameter updates happen?
Check yourself
You raise the batch size from 32 to 512 (N is much larger than both). By roughly what factor does the standard deviation of the mini-batch gradient shrink?
Remember
- ✓Iteration = one mini-batch, one update. Epoch = every training example used once.
- ✓Iterations per epoch (or with
drop_last); for 50,000 examples at that is 391. - ✓Batch GD is , SGD is ; mini-batch sits in between and is what everyone actually uses.
- ✓Gradient noise falls as ; activation memory grows linearly in .
- ✓Reshuffle every epoch, and compare runs with different B by iterations or examples seen, not by epochs.