# Batch normalization

> Normalize each channel with the mean and variance of the current mini-batch, then rescale with a learned γ and β. It allows larger learning rates and faster training, but it couples the examples in a batch, breaks down at small batch sizes and behaves differently in train and eval mode.

Source: https://www.mlfoundry.in/learn/batch-normalization  
Section: Deep Learning › Regularization & Normalization  
Level: core · 15 min read · updated 2026-10-11

**Batch normalization:** (BatchNorm, BN) is a layer that standardizes each channel of its input using statistics of the **current mini-batch**: subtract the channel's batch mean, divide by its batch standard deviation, then apply a learned per-channel scale $\gamma$ and shift $\beta$. At inference it uses running averages of those statistics instead, so it becomes a fixed affine map.

Sergey Ioffe and Christian Szegedy introduced it in "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift" (ICML 2015). Applied to an Inception network for ImageNet, BN matched the original model's accuracy in 14 times fewer training steps, tolerated much larger learning rates (their BN-x30 variant raised the initial rate 30-fold), let them drop dropout and cut L2 weight decay by a factor of 5, and an ensemble reached 4.9% top-5 validation error. Within a year it was in almost every convolutional network, including the **Residual neural network (ResNet)**.

BN sits in this module because it does two jobs. Its main effect is on **optimization**: it keeps every layer's pre-activations at a controlled scale, so you can train deeper networks with higher learning rates. Its side effect is **regularization**: each example is normalized with statistics that depend on which other examples happen to share its batch, and that noise acts a little like dropout.

## The forward pass

For a convolutional activation of shape $(N, C, H, W)$, BN treats each channel separately and pools over the batch **and** the spatial positions: one mean and one variance per channel, each computed from $m=N\cdot H\cdot W$ numbers. For a fully connected layer the input is $(N, C)$ and $m=N$. Pooling over space is what makes the normalization convolutional: every position of a feature map is normalized the same way, so the layer stays translation-equivariant.

$$
\mu_c=\frac1m\sum_{n,h,w}x_{nchw},\qquad \sigma_c^2=\frac1m\sum_{n,h,w}\big(x_{nchw}-\mu_c\big)^2,\qquad \hat x_{nchw}=\frac{x_{nchw}-\mu_c}{\sqrt{\sigma_c^2+\epsilon}},\qquad y_{nchw}=\gamma_c\,\hat x_{nchw}+\beta_c
$$

- $m$: number of values pooled per channel: $N\cdot H\cdot W$ for `BatchNorm2d`, $N$ for `BatchNorm1d` on $(N, C)$
- $\mu_c,\ \sigma_c^2$: batch mean and **biased** batch variance (divide by $m$) of channel $c$
- $\epsilon$: small constant for numerical safety, `eps=1e-5` in PyTorch
- $\gamma_c,\ \beta_c$: learned scale and shift, 2 parameters per channel, initialised to 1 and 0

Why $\gamma$ and $\beta$? Forcing every channel to zero mean and unit variance would restrict what the layer can represent (a sigmoid fed unit-variance inputs stays mostly in its linear region). With $\gamma_c=\sqrt{\sigma_c^2+\epsilon}$ and $\beta_c=\mu_c$ the layer can undo the normalization exactly, so BN never removes capacity; it only changes the parametrisation so that the scale and offset of each channel are controlled by two dedicated parameters instead of by all the weights upstream.

BN is usually placed between the linear or convolutional layer and the nonlinearity, as in the original paper: $\text{ReLU}(\text{BN}(Wx))$. The layer's own bias is then redundant, since BN subtracts the mean and $\beta$ takes over the offset, so it is created with `bias=False`.

**At a glance**

- **Statistics over:** $(N, H, W)$, separately for each channel $c$
- **Learnable:** $\gamma, \beta \in \mathbb R^C$: $2C$ parameters
- **Buffers:** `running_mean`, `running_var` (length $C$) and `num_batches_tracked`
- **Train mode:** normalize with batch statistics, update the running averages
- **Eval mode:** normalize with running statistics; a fixed per-channel affine map
- **Depends on batch size:** yes: poor below roughly 8 examples per device for images
- **Typical home:** CNNs (ResNet, VGG-BN, EfficientNet); rarely in transformers or RNNs

## Running statistics: what eval mode uses

At test time you want a deterministic output that does not depend on the other images in the request. So during training BN keeps an exponential moving average of the batch statistics, and `model.eval()` switches to it. PyTorch's convention is:

$$
\hat\mu\leftarrow(1-\alpha)\,\hat\mu+\alpha\,\mu_c,\qquad \hat\sigma^2\leftarrow(1-\alpha)\,\hat\sigma^2+\alpha\,\frac{m}{m-1}\,\sigma_c^2
$$

- $\alpha$: `momentum`, default $0.1$: the weight of the **new** batch. Keras and TensorFlow use the opposite convention, `momentum=0.99` meaning $0.99$ on the old value
- $\hat\mu,\ \hat\sigma^2$: `running_mean` and `running_var`, initialised to 0 and 1
- $\tfrac{m}{m-1}$: Bessel's correction: the running variance uses the **unbiased** batch variance, while the normalization itself uses the biased one

With $\alpha=0.1$ the average remembers about the last $1/\alpha=10$ batches, and after $k$ batches it has moved a fraction $1-0.9^k$ of the way from its initial value: 65% after 10, 99.5% after 50. Setting `momentum=None` replaces the EMA by a plain cumulative average over all batches seen.

> Figure: Mini-batches of shape (8, 4, 3, 3) streamed through one BN layer. The outlined column is everything channel c0 pools: N·H·W = 72 values. Dots are each batch's statistics, the line is the running average. After 80 batches at momentum 0.1, c0 has running_mean 2.92 and running_var 3.99 against true values 3 and 4. 'train x̂' shows each channel with mean 0 and sd 1 exactly; 'eval x̂' uses the running statistics, so c0 of the last batch comes out with mean 0.24 and sd 1.38. The bottom readout compares example n0 in the two modes: the RMS gap is 0.38 at N = 8, 0.27 at N = 16, 0.84 at N = 2 and 1.65 at N = 1. Within one image the values are correlated (70% of the variance is shared), so small batches also underestimate the variance: at N = 2, c3's running_var settles at 6.04 against a true 9. Momentum 0.5 tracks faster (running_mean 3.03 after 10 batches vs 1.84 at 0.1) but jitters three times as much. 1×1 maps with N = 1 is the case PyTorch refuses.

## Worked example: one channel, four values

Take a `BatchNorm1d(1)` in training mode with $\gamma=2$, $\beta=0.5$ and a batch $x=(1,2,4,9)$. Then $m=4$, $\mu=4$, the biased variance is $\sigma^2=(9+4+0+25)/4=9.5$, and $\sigma=\sqrt{9.5+10^{-5}}=3.0822$.

Forward pass, and the backward pass for an upstream gradient that touches only the first output: g = ∂L/∂y = (1, 0, 0, 0). Checked in Node, including a finite-difference check of ∂L/∂x.

| x | x − μ | x̂ | y = 2x̂ + 0.5 | ∂L/∂x |
| --- | --- | --- | --- | --- |
| 1 | −3 | −0.9733 | −1.4467 | 0.3330 |
| 2 | −2 | −0.6489 | −0.7978 | −0.2647 |
| 4 | 0 | 0.0000 | 0.5000 | −0.1622 |
| 9 | 5 | 1.6222 | 3.7444 | 0.0939 |

**What the numbers say**

1. The outputs $\hat x$ have mean 0 and biased variance 1 (up to $\epsilon$), whatever the input scale was.
2. Running stats after this one step: $\hat\mu=0.9\cdot0+0.1\cdot4=0.4$ and $\hat\sigma^2=0.9\cdot1+0.1\cdot\tfrac{38}{3}=2.1667$ (unbiased variance $12.667$, not $9.5$).
3. Eval mode right now would map $x=9$ to $2(9-0.4)/\sqrt{2.1667}+0.5=12.19$ instead of the training-mode $3.74$: the running averages need many batches before eval mode matches training.
4. The gradient reaches **all four** inputs although only $y_1$ was touched, because every input moved $\mu$ and $\sigma$. Treating $\mu,\sigma$ as constants would give $\gamma/\sigma=0.6489$ for $x_1$ and 0 elsewhere, which is wrong.
5. $\sum_i\partial L/\partial x_i=0$ and $\sum_i \hat x_i\,\partial L/\partial x_i=0$: shifting or scaling the whole batch cannot change BN's output, so the gradient has no component in those directions. Also $\partial L/\partial\gamma=\sum g_i\hat x_i=-0.9733$ and $\partial L/\partial\beta=\sum g_i=1$.

## The backward pass

Writing $g_i=\partial L/\partial y_i$ for the values of one channel and differentiating through $\mu$ and $\sigma^2$ as well as through $x_i$ directly gives a compact formula:

$$
\frac{\partial L}{\partial x_i}=\frac{\gamma}{m\,\sigma}\Big(m\,g_i-\sum_{j}g_j-\hat x_i\sum_{j}g_j\hat x_j\Big),\qquad \frac{\partial L}{\partial\gamma}=\sum_i g_i\hat x_i,\qquad \frac{\partial L}{\partial\beta}=\sum_i g_i
$$

- $\sigma$: $\sqrt{\sigma_c^2+\epsilon}$, the batch standard deviation used in the forward pass
- $\sum_j g_j$: removes the mean of the upstream gradient: the term that flows through $\mu$
- $\hat x_i\sum_j g_j\hat x_j$: removes the component along $\hat x$: the term that flows through $\sigma^2$

The bracket is the upstream gradient with its mean and its projection on $\hat x$ removed, then scaled by $\gamma/\sigma$. Two consequences matter in practice. First, the gradient of one example depends on every other example in the batch, so BN is not a per-example layer: per-example gradients (for differential privacy, say) and **Gradient accumulation** over micro-batches do not see the same function as a single large batch. Second, because $\text{BN}(a\,Wx)=\text{BN}(Wx)$ for any $a>0$, the loss is invariant to the scale of the weights feeding a BN layer, and the gradient with respect to those weights is orthogonal to them and shrinks like $1/\lVert W\rVert$.

## Why does it help? Two explanations

Ioffe and Szegedy's motivation was **internal covariate shift**: as earlier layers train, the distribution of each layer's inputs keeps changing, so later layers chase a moving target. Fixing the first two moments of every layer's inputs was supposed to stabilise that.

Santurkar, Tsipras, Ilyas and Madry tested this in "How Does Batch Normalization Help Optimization?" (NeurIPS 2018). They added random, time-varying noise with non-zero mean and non-unit variance **after** each BN layer, deliberately creating covariate shift; the network still trained almost as well as standard BN and far better than no BN. They also found that BN did not reliably reduce covariate shift by their measures. Their explanation is that BN makes the **loss landscape smoother**: the loss and its gradient change more slowly as the weights move (better Lipschitz constants and β-smoothness), so gradients are more predictive and larger steps are safe.

> **What is agreed, what is not:** Everyone agrees on the effects: BN permits higher learning rates, reduces sensitivity to initialization (see **Weight initialization**), and speeds up training of deep CNNs. The "reduces internal covariate shift" story is now widely considered at best incomplete. Smoothness, the scale invariance of the weights, and the noise from batch statistics each explain part of the benefit.

## When batch statistics break down

Every mechanism above relies on $\mu_c$ and $\sigma_c^2$ being decent estimates. With few examples per batch they are not, and three things go wrong. The noise in each example's normalized value grows, which is the RMS train/eval gap in the figure. The running variance is biased low when values within one image are correlated. And the network learns to rely on statistics that eval mode will not reproduce. Wu and He's Group Normalization paper (ECCV 2018) measured the effect: on ImageNet ResNet-50, BN's error rises steeply as the per-GPU batch shrinks, and at 2 images per GPU Group Normalization's error was 10.6% lower than BN's.

The extreme cases are degenerate. With one value per channel ($N=1$ for `BatchNorm1d`, or $N=1$ and $1\times1$ maps), the variance is 0 and every output equals $\beta$; PyTorch refuses this in training mode with "Expected more than 1 value per channel when training". With $N=1$ and larger maps, BN normalizes an image only against itself, which is **Instance normalization** in disguise, and eval mode then sees statistics it was never trained with.

- **SyncBatchNorm:** Under data parallelism each GPU normally computes statistics over its own shard. `nn.SyncBatchNorm` all-reduces the sums across processes so the statistics cover the global batch; it costs one extra communication per BN layer per step. Used for detection and segmentation, where each GPU holds only 1 or 2 images.
- **Group or layer normalization:** **Group normalization**, **Layer normalization** and **Instance normalization** compute statistics per example, so they are independent of batch size and identical in train and eval. They are the standard replacements when batches are small or variable.
- **Batch Renormalization:** Ioffe (2017) corrects the batch statistics toward the running ones with clipped factors $r,d$ that are treated as constants in the backward pass, which reduces the train/eval mismatch for small or non-i.i.d. batches.
- **Frozen BN for fine-tuning:** When fine-tuning a pretrained CNN with tiny batches, keep the BN layers in eval mode (and often freeze $\gamma,\beta$) so the pretrained running statistics are used throughout.

## Inference: fold BN into the convolution

In eval mode BN is $y=\gamma(z-\hat\mu)/\sqrt{\hat\sigma^2+\epsilon}+\beta$ applied to a convolution output $z=w*x+b$. Both steps are affine, so they collapse into one convolution with per-output-channel rescaled weights. Deployment toolchains do this automatically; the BN layer then costs nothing at inference.

$$
w'_c=\frac{\gamma_c}{\sqrt{\hat\sigma_c^2+\epsilon}}\,w_c,\qquad b'_c=\frac{\gamma_c\,(b_c-\hat\mu_c)}{\sqrt{\hat\sigma_c^2+\epsilon}}+\beta_c
$$

- $w_c,\ b_c$: the filter and bias of output channel $c$ ($b_c=0$ for a bias-free conv)
- $w'_c,\ b'_c$: the fused filter and bias: one conv that equals conv followed by eval-mode BN

Example: a 1×1 conv with $w=0.5$, $b=0.2$, followed by BN with $\hat\mu=4$, $\hat\sigma^2=9.5$, $\gamma=2$, $\beta=0.5$, fuses to $w'=0.3244$, $b'=-1.9658$. For input 3 both the two-layer and the fused version output $-0.9924$.

```python
import copy, torch, torch.nn as nn

torch.manual_seed(0)
x = torch.randn(8, 4, 3, 3) * 2 + 3            # (N, C, H, W)
bn = nn.BatchNorm2d(4)                          # eps=1e-5, momentum=0.1, affine=True

# --- training mode: batch statistics over (N, H, W) ---
y = bn(x)
mu  = x.mean(dim=(0, 2, 3))
var = x.var(dim=(0, 2, 3), correction=0)        # biased: used to normalize
xhat = (x - mu[None, :, None, None]) / torch.sqrt(var[None, :, None, None] + bn.eps)
print(torch.allclose(y, xhat * bn.weight[None, :, None, None] + bn.bias[None, :, None, None], atol=1e-5))

# running stats: start at 0 and 1, momentum weighs the NEW batch, variance is unbiased
m = x.shape[0] * x.shape[2] * x.shape[3]
print(torch.allclose(bn.running_mean, 0.1 * mu))
print(torch.allclose(bn.running_var, 0.9 * 1 + 0.1 * var * m / (m - 1)))

# --- eval mode: running statistics, outputs no longer depend on the batch ---
bn.eval()
print(torch.allclose(bn(x[:1]), bn(x)[:1]))     # True only in eval mode

# BatchNorm1d with one value per channel cannot train
try:
    nn.BatchNorm1d(4).train()(torch.randn(1, 4))
except ValueError as e:
    print(e)                                     # Expected more than 1 value per channel when training ...

# --- typical block: no conv bias, BN before the nonlinearity ---
block = nn.Sequential(nn.Conv2d(3, 16, 3, padding=1, bias=False), nn.BatchNorm2d(16), nn.ReLU())

# multi-GPU: share statistics across processes (use with DistributedDataParallel)
sync_block = nn.SyncBatchNorm.convert_sync_batchnorm(copy.deepcopy(block))

# --- inference: fold eval-mode BN into the conv ---
from torch.nn.utils.fusion import fuse_conv_bn_eval
block.eval()
fused = fuse_conv_bn_eval(block[0], block[1])
img = torch.randn(2, 3, 32, 32)
print(torch.allclose(fused(img), block[1](block[0](img)), atol=1e-5))
```

BatchNorm2d by hand, the running-stat convention, eval mode, SyncBatchNorm and conv-BN fusion

> **Forgetting model.eval():** In training mode BN normalizes each test batch with its own statistics: predictions change with batch composition, a batch of one gives garbage, and the running statistics are silently overwritten by test data. Always call `model.eval()` for validation and inference, and `model.train()` before resuming training. If eval accuracy is much worse than train accuracy on the **same** data, suspect the running statistics: too few training steps, a momentum that is too large, or batches whose statistics differ from the test distribution.

> **BN with dropout and weight decay:** **Dropout** placed before a BN layer changes the variance of its inputs between training (units dropped and rescaled) and eval (all units kept), so the running variance collected in training is wrong at test time. Li et al. (CVPR 2019) called this the variance shift; the usual fix is to put dropout only after the last BN layer, or not use it at all in BN networks. **Weight decay** on weights that feed a BN layer does not regularize the function, since BN cancels their scale. It shrinks $\lVert W\rVert$ and thereby raises the effective learning rate, $\eta/\lVert W\rVert^2$ (van Laarhoven, 2017). Decay on $\gamma$ and $\beta$ is usually switched off (see **AdamW**).

> **Small tricks that matter:** Initialise the last BN $\gamma$ of each residual branch to 0 (`zero_init_residual=True` in torchvision's ResNet) so every block starts as the identity, which helps large-batch training. Shuffle the data every epoch so batch statistics stay representative. When fine-tuning, BN statistics computed on the new data distribution often matter more than the learned weights.

**Strengths**

- Allows much higher learning rates and trains deep CNNs several times faster
- Makes training far less sensitive to initialization and weight scale
- Free at inference: folds into the preceding convolution
- Mild regularization from batch-statistic noise; often lets you drop dropout

**Limitations**

- Output of one example depends on the rest of its batch; not a per-example function in training
- Degrades at small per-device batch sizes (detection, segmentation, video, large models)
- Two modes: train/eval mismatch, running statistics that can go stale or be overwritten
- Awkward for RNNs and variable-length sequences; needs cross-device sync under data parallelism

**Question:** A conv layer outputs a tensor of shape (16, 64, 28, 28) into nn.BatchNorm2d(64). Over how many values is each mean computed, and how many mean values are there?

- A. 16·28·28 = 12,544 values each; 64 means
- B. 64·28·28 = 50,176 values each; 16 means
- C. 16 values each; 64·28·28 means
- D. 64 values each; 16·28·28 means

**Answer:** A. 16·28·28 = 12,544 values each; 64 means — BatchNorm2d keeps one mean and one variance per channel and pools over the batch and both spatial axes: $m=N\cdot H\cdot W=16\cdot28\cdot28=12{,}544$. A per-example scheme over the 64 channels at each position would be closer to **Layer normalization** applied per pixel.

**Question:** Your BN model reaches 92% accuracy when evaluated in train mode on the validation set, but 61% after model.eval(). Which is the most likely cause?

- A. The running mean and variance do not match the statistics the network was trained with
- B. γ and β are frozen in eval mode
- C. Eval mode disables the gradient of BN
- D. BN uses the unbiased variance only in eval mode

**Answer:** A. The running mean and variance do not match the statistics the network was trained with — Eval mode differs only in which statistics it uses. A big drop means the running averages are off: too few updates, a large momentum, tiny or correlated training batches (the figure's small-N case), or test data that differs in distribution. Recomputing the running statistics on a few hundred training batches (with momentum=None) is a quick check.

**Key points**

- BN standardizes each channel with batch statistics over $(N,H,W)$, then applies a learned $\gamma_c,\beta_c$; it can represent the identity, so no capacity is lost.
- Running stats: $\hat\mu\leftarrow0.9\hat\mu+0.1\mu_B$ in PyTorch, with the **unbiased** variance; eval mode uses them and becomes a fixed affine map that folds into the preceding conv.
- Backward: $\partial L/\partial x=\tfrac{\gamma}{m\sigma}\big(mg-\sum g-\hat x\sum g\hat x\big)$; gradients couple all examples in the batch.
- It works mostly by smoothing optimization (Santurkar et al., 2018) rather than by removing internal covariate shift.
- Small per-device batches break it: use SyncBatchNorm, or Group, Layer or Instance normalization.
- Call model.eval() for inference, avoid dropout before BN, and remember weight decay on BN-fed weights acts as a learning-rate change.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
