# Epoch, batch, mini-batch & iteration

> An iteration is one update from one mini-batch; an epoch is one full pass over the data, so iterations per epoch = ⌈N / batch size⌉.

Source: https://www.mlfoundry.in/learn/epoch-batch-mini-batch-iteration  
Section: Deep Learning › Deep Learning Fundamentals  
Level: core · 7 min read · updated 2026-10-09

**Epoch, mini-batch, iteration:** A **mini-batch** is a group of $B$ training examples whose average gradient produces one parameter update. That update is one **iteration** (also called a step). An **epoch** is one full pass in which every one of the $N$ training examples has been used once, which takes $\lceil N/B\rceil$ iterations.

## Why not use all the data for every update?

The training loss is an average over $N$ examples, and so is its gradient. Computing the exact gradient means a forward and backward pass over the whole dataset (for ImageNet, 1.28 million images) just to take **one** step.

A random subset of $B$ examples gives an unbiased estimate of that gradient at a fraction of the cost. The estimate is noisy, but you get $N/B$ updates for the price of one exact one, and early in training many rough steps beat one precise step.

Three names for the same loop, differing only in B

| Variant | Batch size | Updates per epoch | Gradient |
| --- | --- | --- | --- |
| Batch (full-batch) gradient descent | $B=N$ | 1 | exact, deterministic |
| Stochastic gradient descent (SGD) | $B=1$ | $N$ | noisiest estimate, no parallelism |
| Mini-batch gradient descent | $1<B<N$ (typically 32–1024) | $\lceil N/B\rceil$ | noisy, and vectorizes well on a GPU |

In practice everyone uses mini-batches and still calls it "SGD": `torch.optim.SGD` simply updates with whatever batch you hand it.

## The math

$$
\theta_{t+1}=\theta_t-\eta\,\hat g_t,\qquad \hat g_t=\frac{1}{|\mathcal B_t|}\sum_{i\in\mathcal B_t}\nabla_\theta\,\ell(x_i,y_i;\theta_t)
$$

- $t$: iteration counter (one per update, never reset between epochs)
- $\mathcal B_t$: the mini-batch used at iteration $t$; $|\mathcal B_t|=B$ except possibly the last batch of an epoch
- $\eta$: learning rate
- $\ell$: per-example loss
- $\hat g_t$: mini-batch gradient, an unbiased estimate of the full gradient

$$
\text{iterations per epoch}=\left\lceil \frac{N}{B}\right\rceil\;\;\text{or}\;\;\left\lfloor \frac{N}{B}\right\rfloor\text{ with drop\_last},\qquad \text{total iterations}=E\cdot\text{iterations per epoch}
$$

- $N$: number of training examples
- $B$: batch size
- $E$: number of epochs

How noisy is $\hat g_t$? If individual gradients have variance $\sigma^2$ around the true gradient, then averaging $B$ of them drawn without replacement gives

$$
\operatorname{Var}[\hat g]=\frac{\sigma^2}{B}\cdot\frac{N-B}{N-1}\quad\Rightarrow\quad \operatorname{std}[\hat g]\approx\frac{\sigma}{\sqrt B}\;\;(B\ll N)
$$

- $\sigma^2$: variance of a single example's gradient (summed over parameters)
- $\frac{N-B}{N-1}$: finite-population correction; it is 0 at $B=N$, where the gradient is exact

So quadrupling the batch only **halves** the noise while costing four times the compute per step. That square root is why huge batches give diminishing returns per example processed.

> Figure: Mini-batch SGD on a 60-point regression. At B = 16 an epoch is 4 iterations and the last batch has 12 points. Press play: one batch lights up per update, and the colours reshuffle at each epoch boundary. Then compare B = 1 (60 noisy updates per epoch, loss about 0.18 after one epoch) with B = 60 (one smooth update per epoch, loss still 0.78). Finally set B = 7 and turn on drop_last, which drops 4 points every epoch.

## Worked example: N = 50,000 (CIFAR-10's training set)

Relative noise = std of the mini-batch gradient divided by the B = 1 value, sqrt((N−B)/((N−1)B))

| B | Iterations / epoch | Last batch | With drop_last | Iterations in 100 epochs | Relative noise |
| --- | --- | --- | --- | --- | --- |
| 1 | 50,000 | 1 | 50,000 | 5,000,000 | 1 |
| 32 | 1,563 | 16 | 1,562 | 156,300 | 0.177 |
| 128 | 391 | 80 | 390 | 39,100 | 0.088 |
| 512 | 98 | 336 | 97 | 9,800 | 0.044 |
| 50,000 | 1 | 50,000 | 1 | 100 | 0 |

**The B = 128 row**

1. $50{,}000/128=390.625$, so 390 full batches cover $390\cdot128=49{,}920$ examples.
2. The remaining $50{,}000-49{,}920=80$ examples form a 391st, smaller batch: $\lceil 390.625\rceil=391$ iterations per epoch.
3. With `drop_last=True` those 80 are skipped each epoch (a different 80 each time, because of shuffling), leaving 390 iterations.
4. A 100-epoch run is $100\cdot391=39{,}100$ iterations. Learning-rate schedules and logging are usually counted in these iterations, not in epochs.

## Shuffling and the last batch

**Reshuffle every epoch.** If the order is fixed, every epoch sees identical batches in the same sequence, and any structure in that order (for example, data sorted by class) biases the updates. A fresh permutation each epoch keeps $\hat g_t$ close to an independent random sample.

**The partial batch** still averages its loss, so the update has the usual scale, just more noise. It is harmless for most models. Drop it when every batch must have the same shape (some compiled or distributed setups) or when a layer misbehaves on tiny batches: BatchNorm in training mode raises an error on a batch of one sample.

## Choosing B: memory versus time

**Strengths**

- **Larger B**: better GPU utilization, fewer iterations per epoch, smoother gradient estimate, so faster wall-clock time per epoch.
- Activation memory grows roughly linearly in $B$, so the largest batch that fits is bounded by GPU memory. Weights and optimizer state do not depend on $B$.

**Limitations**

- Per-example returns shrink: noise falls only as $1/\sqrt B$, so past some size, doubling $B$ barely reduces the number of updates needed.
- Very large batches usually need a retuned learning rate (a common heuristic scales $\eta$ in proportion to $B$, with warm-up) and can generalize slightly worse without it.

If the batch you want does not fit, use **gradient accumulation**: run $k$ smaller micro-batches, add up their gradients, and call `optimizer.step()` once. That is one iteration with an effective batch of $k\times$ the micro-batch size.

## Where the counters live in PyTorch

```python
import torch
from torch.utils.data import DataLoader, TensorDataset

X, y = torch.randn(50_000, 20), torch.randint(0, 10, (50_000,))
loader = DataLoader(TensorDataset(X, y), batch_size=128, shuffle=True, drop_last=False)
print(len(loader))                         # 391 = ceil(50000 / 128)

model = torch.nn.Linear(20, 10)
opt = torch.optim.SGD(model.parameters(), lr=0.1)
loss_fn = torch.nn.CrossEntropyLoss()

step = 0                                   # global iteration counter
for epoch in range(10):                    # one epoch = one full pass over the loader
    for xb, yb in loader:                  # new permutation each time this loop starts
        loss = loss_fn(model(xb), yb)      # mean loss over this mini-batch
        opt.zero_grad()
        loss.backward()
        opt.step()                         # one iteration = one parameter update
        step += 1
    print(epoch, step)                     # step == (epoch + 1) * 391
```

len(loader) is the number of iterations per epoch; the outer loop counts epochs, the inner loop counts iterations.

> **Epoch counts are not comparable across batch sizes:** "Trained for 10 epochs" means 3,910 updates at $B=128$ but only 980 at $B=512$. When you change $B$, compare runs by iterations or by examples seen, and remember that a schedule written in steps (warm-up, decay) changes meaning too. In multi-GPU training with `DistributedSampler`, call `sampler.set_epoch(epoch)` each epoch, or every epoch reuses the same shuffle.

> **Noise is not only a cost:** The gradient noise from small batches lets SGD jitter out of sharp, narrow minima. That is one reason small and medium batches often generalize at least as well as huge ones trained with the same learning rate.

**Question:** $N=10{,}000$, $B=64$, `drop_last=False`, 5 epochs. How many parameter updates happen?

- A. 780
- B. 785
- C. 157
- D. 640

**Answer:** B. 785 — $10{,}000/64=156.25$, so each epoch has $\lceil156.25\rceil=157$ iterations (156 full batches plus one of 16). Five epochs give $5\cdot157=785$. With `drop_last=True` it would be $5\cdot156=780$.

**Question:** You raise the batch size from 32 to 512 (N is much larger than both). By roughly what factor does the standard deviation of the mini-batch gradient shrink?

- A. 16
- B. 4
- C. 2
- D. It does not change

**Answer:** B. 4 — The std scales as $1/\sqrt B$. The batch grew 16 times, so the noise shrinks by $\sqrt{16}=4$, while each step costs about 16 times as much compute.

**Key points**

- Iteration = one mini-batch, one update. Epoch = every training example used once.
- Iterations per epoch $=\lceil N/B\rceil$ (or $\lfloor N/B\rfloor$ with `drop_last`); for 50,000 examples at $B=128$ that is 391.
- Batch GD is $B=N$, SGD is $B=1$; mini-batch sits in between and is what everyone actually uses.
- Gradient noise falls as $1/\sqrt B$; activation memory grows linearly in $B$.
- Reshuffle every epoch, and compare runs with different B by iterations or examples seen, not by epochs.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
