# Group normalization

> Split the channels into G groups and normalize each group of each sample over its channels and positions. It never looks at other samples, so it works the same at batch size 2 as at 64; G = 1 gives layer norm and G = C gives instance norm.

Source: https://www.mlfoundry.in/learn/group-normalization  
Section: Deep Learning › Regularization & Normalization  
Level: core · 11 min read · updated 2026-10-11

**Group normalization:** (GN) divides the $C$ channels of a layer into $G$ groups of $C/G$ consecutive channels. For each sample and each group it computes one mean and one variance over all channels of the group and all spatial positions, normalizes with them, and then applies a learned per-channel scale $\gamma_c$ and shift $\beta_c$.

**Batch normalization** estimates its statistics from the batch. That works well when each GPU holds 32 or more images, and breaks down when it holds 1 or 2. Object detection, instance segmentation and video models are in that regime because their inputs are large. Their per-batch means and variances are then noisy, the network trains on noisy normalization, and at test time it is evaluated with running averages it never saw during training. GN removes the batch from the computation entirely.

**Wu and He (2018)**, *Group Normalization* (ECCV), took the idea from classic hand-designed features: SIFT and HOG normalize **groups** of histogram bins rather than single bins or the whole descriptor. Channels of a conv layer are similarly related (filters for the same orientation at different frequencies, say), so normalizing them together is a reasonable middle ground. On ResNet-50 for ImageNet with 32 images per GPU they reported 24.1% top-1 error for GN against 23.6% for BN, 25.3% for **Layer normalization** and 28.4% for **Instance normalization**. At 2 images per GPU, GN's error was 10.6 points lower than BN's, because GN's accuracy does not depend on the batch size at all.

## Formula

$$
\mathcal S_{n,g}=\Big\{(c,h,w)\;:\;\Big\lfloor\tfrac{c}{C/G}\Big\rfloor=g\Big\},\qquad \mu_{ng}=\frac{1}{m}\sum_{\mathcal S_{n,g}}x_{nchw},\qquad \sigma^2_{ng}=\frac{1}{m}\sum_{\mathcal S_{n,g}}\big(x_{nchw}-\mu_{ng}\big)^2
$$

- $G$: number of groups; $C$ must be divisible by $G$. Wu & He's default is $G=32$
- $\mathcal S_{n,g}$: the values that share statistics: sample $n$, the $C/G$ channels of group $g$, all positions
- $m$: $|\mathcal S_{n,g}|=(C/G)\cdot H\cdot W$

$$
y_{nchw}=\gamma_c\,\frac{x_{nchw}-\mu_{n,g(c)}}{\sqrt{\sigma^2_{n,g(c)}+\varepsilon}}+\beta_c
$$

- $g(c)$: the group that channel $c$ belongs to
- $\gamma_c,\ \beta_c$: learned **per channel**, not per group: $2C$ parameters in total
- $\varepsilon$: $10^{-5}$ by default in PyTorch

The gradient is the standard normalization backward pass taken over each set $\mathcal S_{n,g}$: with $\hat x=(x-\mu)/\hat\sigma$ and $g_i=\gamma_c\,\partial L/\partial y_i$, $\;\partial L/\partial x_i=\hat\sigma^{-1}\big(g_i-\overline{g}-\hat x_i\,\overline{g\hat x}\big)$, where the bars average over the $m$ members of the set. Within a group the gradient sums to zero and is orthogonal to $\hat x$, so nothing upstream can change a group's mean or overall scale through this layer.

## One family, four members

All four normalizers apply the same formula. They differ only in which activations share a mean and variance. Lay the activations of a conv layer out as an $N\times C$ grid of cells, each cell holding the $H\times W$ values of one (sample, channel) pair:

Which cells share statistics, and how many values go into each mean (C channels, H×W positions, batch N)

| normalizer | statistics over | values per statistic | depends on batch? |
| --- | --- | --- | --- |
| Batch norm | $(N,H,W)$ for each $c$ | $N\cdot H\cdot W$ | yes |
| Layer norm | $(C,H,W)$ for each $n$ | $C\cdot H\cdot W$ | no |
| Instance norm | $(H,W)$ for each $(n,c)$ | $H\cdot W$ | no |
| Group norm | $(C/G,H,W)$ for each $(n,g)$ | $(C/G)\cdot H\cdot W$ | no |

With $G=1$ the single group is all channels, which is layer norm over $(C,H,W)$. With $G=C$ each group is one channel, which is instance norm. GN interpolates: layer norm assumes all channels deserve one scale, instance norm assumes none are related, and GN assumes channels come in related bundles of $C/G$.

**At a glance**

- **Statistics over:** $C/G$ channels $\times\,H\times W$ positions, per sample
- **Batch dependence:** none: identical in training and eval, no running statistics
- **Parameters:** $2C$ ($\gamma_c$, $\beta_c$), same as batch norm
- **Default G:** 32 (or a fixed $C/G$ such as 16 channels per group)
- **Special cases:** $G=1$: layer norm over $(C,H,W)$; $G=C$: instance norm
- **Used in:** detection and segmentation (Mask R-CNN with GN), diffusion U-Nets, BiT (with weight standardization)
- **Paper:** Wu & He (2018), ECCV

> Figure: Left: 4 samples × 16 channels, each cell one 3×3 feature map. Click a cell to see which cells share its statistics: a column for batch norm, a row for layer norm, the cell alone for instance norm, a row segment of 16/G channels for group norm. With G = 4 one statistic pools 4 cells, 36 values. Right: the batch-normalized value of one fixed pixel of sample 0, over 200 random batches drawn from 64 samples. Its spread across batches is 0.538 at batch size 2, 0.416 at 4, 0.303 at 8, 0.186 at 16 and 0.113 at 32, and its mean distance from the eval-mode value 0.561 is 0.501 at batch size 2. At batch size 1 the spread is 0 because BN then equals instance norm, but the distance from eval mode is still 0.502. Group norm gives one fixed value for every batch.

The right-hand panel is the whole argument for GN in one picture. Under BN, the same input produces a different output depending on which other images happen to share its batch. That noise shrinks only like $1/\sqrt{N}$, and the gap between training-mode and eval-mode outputs is just as large. GN's output depends on the sample alone, so there is nothing to average and no train/test discrepancy.

## Worked example: G = 1, 2 and 4

One sample, $C=4$ channels, two positions each: $c_0=(1,3)$, $c_1=(5,7)$, $c_2=(2,2)$, $c_3=(4,8)$. Take $\gamma=1$, $\beta=0$, $\varepsilon=10^{-5}$. Values checked in Node:

The same 8 activations under three group counts

| G | groups | μ, σ per group | output (c₀ \| c₁ \| c₂ \| c₃) |
| --- | --- | --- | --- |
| 1 (layer norm) | {c₀, c₁, c₂, c₃} | 4, 2.345 | (−1.279, −0.426) \| (0.426, 1.279) \| (−0.853, −0.853) \| (0, 1.706) |
| 2 | {c₀, c₁}, {c₂, c₃} | 4, 2.236 and 4, 2.449 | (−1.342, −0.447) \| (0.447, 1.342) \| (−0.816, −0.816) \| (0, 1.633) |
| 4 (instance norm) | each channel alone | (2,1), (6,1), (2,0), (6,2) | (−1, 1) \| (−1, 1) \| (0, 0) \| (−1, 1) |

Two things show up. First, with more groups each statistic is computed from fewer values: 8, 4 and 2 here. Second, at $G=C$ the constant channel $c_2$ has $\sigma=0$ and is mapped to exactly 0, and every two-value channel becomes $(-1,1)$ regardless of its content. Grouping keeps relative information between channels: in the $G=2$ row, $c_1$ still sits above $c_0$.

```python
import torch, torch.nn as nn
import torchvision

x = torch.randn(2, 64, 28, 28)                # batch of 2: fine for GN

gn = nn.GroupNorm(num_groups=32, num_channels=64)   # eps=1e-5, affine=True
y = gn(x)

# By hand: reshape to (N, G, C/G * H * W) and normalize the last axis
N, C, H, W = x.shape
G = 32
xg = x.view(N, G, -1)
mu, var = xg.mean(-1, keepdim=True), xg.var(-1, keepdim=True, unbiased=False)
y_ref = ((xg - mu) / torch.sqrt(var + 1e-5)).view(N, C, H, W)
print(torch.allclose(y, y_ref, atol=1e-5))         # True (gamma=1, beta=0 at init)

# Same output for each sample whether it is alone or in a batch: no batch dependence
print(torch.allclose(gn(x[:1]), y[:1], atol=1e-6))  # True

# Special cases
ln_like = nn.GroupNorm(1, 64)       # layer norm over (C, H, W), but gamma/beta per channel
in_like = nn.GroupNorm(64, 64)      # instance norm with affine=True

# Replace every BatchNorm2d of a ResNet-50 with GroupNorm(32, C)
model = torchvision.models.resnet50(norm_layer=lambda c: nn.GroupNorm(32, c))

# nn.GroupNorm(32, 48) raises: num_channels must be divisible by num_groups
```

nn.GroupNorm, its special cases, and swapping it into a torchvision ResNet

> **GroupNorm(1, C) is not nn.LayerNorm((C, H, W)):** The statistics are the same, the affine parameters are not. `nn.GroupNorm(1, C)` learns one $\gamma,\beta$ per channel ($2C$ numbers). `nn.LayerNorm([C, H, W])` learns one per element ($2CHW$ numbers) and fixes the input resolution. For conv nets the per-channel version is usually what you want.

> **Inference cost:** At inference, BN uses fixed running statistics, so it is an affine map that can be folded into the preceding convolution's weights and costs nothing. GN's statistics depend on the input, so it must be computed on every forward pass. GN is the right choice when batches are small; for large-batch image classification BN is usually as accurate and cheaper to deploy.

> **Choosing G:** Wu and He found $G=32$ best on ResNet-50, with every group count they tried between 2 and 64 within about a point of it and ahead of both extremes ($G=1$ and $G=C$). Two practical rules: $C$ must be divisible by $G$, and narrow early layers should keep a reasonable number of channels per group, so some code uses a fixed $C/G$ (e.g. 16) instead of a fixed $G$.

> **Where you meet it:** Mask R-CNN and other detectors trained with 1 to 2 images per GPU; the U-Nets of diffusion models (DDPM by Ho et al., 2020, and Stable Diffusion's U-Net use `GroupNorm(32, C)`); Big Transfer (BiT, Kolesnikov et al., 2020), which pairs GN with **weight standardization** (Qiao et al., 2019) in place of BN for large-scale pre-training and transfer; and reinforcement learning or federated settings, where batch statistics are unreliable or must not be shared.

**Strengths**

- Accuracy independent of batch size: same behaviour at 1, 2 or 64 samples per device
- No running statistics: training and evaluation compute exactly the same function
- No cross-device synchronization needed, unlike SyncBatchNorm
- Keeps relative scale between channels inside a group, unlike instance norm

**Limitations**

- Slightly worse than BN on large-batch image classification (24.1% vs 23.6% top-1 error on ResNet-50)
- Cannot be folded into the convolution at inference: a real per-input cost
- Adds a hyperparameter $G$ and the divisibility constraint $C\bmod G=0$
- Loses BN's mild regularizing noise, which large-batch classifiers sometimes benefit from

**Question:** A layer has C = 64 channels and 14×14 feature maps. With nn.GroupNorm(32, 64), how many values go into each mean, and how many means are computed for a batch of 8?

- A. 392 values per mean, 256 means
- B. 6272 values per mean, 64 means
- C. 196 values per mean, 512 means
- D. 12544 values per mean, 8 means

**Answer:** A. 392 values per mean, 256 means — Each group holds $64/32=2$ channels of $14\cdot14=196$ positions, so $m=392$. There is one mean per sample per group: $8\cdot32=256$. Batch norm would instead compute 64 means of $8\cdot196=1568$ values each.

**Question:** Why does replacing batch norm with group norm remove the train/test discrepancy?

- A. GN's statistics come from the sample itself, so evaluation uses the same computation as training
- B. GN tracks more accurate running averages
- C. GN uses unbiased variance
- D. GN has no learnable parameters

**Answer:** A. GN's statistics come from the sample itself, so evaluation uses the same computation as training — BN normalizes with batch statistics while training and with running averages at test time, two different functions. GN never uses other samples and keeps no running averages, so `model.train()` and `model.eval()` produce identical outputs. In the figure, BN's train/eval gap at batch size 2 is 0.501 on average; GN's is 0.

**Key points**

- GN normalizes each sample over groups of $C/G$ channels and all positions, then applies per-channel $\gamma_c,\beta_c$.
- $G=1$ is layer norm over $(C,H,W)$, $G=C$ is instance norm; $G=32$ is the usual default.
- It is independent of batch size and identical in training and eval: the fix for small-batch detection, segmentation and video.
- BN's per-sample output is noisy in small batches (spread 0.54 at batch size 2 vs 0.11 at 32 in the figure); GN's is not.
- Cost: it cannot be folded into the conv at inference, and $C$ must be divisible by $G$.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
