Definition
Layer normalization (LayerNorm, LN) standardizes the activations of one example across its features: subtract the mean of that example's values, divide by their standard deviation, then apply a learned per-feature gain and bias . Nothing is pooled across the batch, so there are no running statistics and no difference between train and eval mode.
Jimmy Lei Ba, Jamie Ryan Kiros and Geoffrey Hinton proposed it in "Layer Normalization" (arXiv, 2016) as a fix for the parts of Batch normalization that do not fit recurrent networks. BN needs a mini-batch to estimate statistics, separate statistics for every time step of an RNN, and a test-time sequence no longer than the training ones. LN replaces all of that with statistics computed from the summed inputs of a layer for a single example at a single time step. They showed faster convergence on several recurrent tasks, including skip-thought vectors, the attentive reader and DRAW.
Its biggest success came a year later. The Transformer (Vaswani et al., 2017) wrapped every attention and feed-forward sub-layer as , and nearly every transformer since normalizes each token with LN or with its cheaper relative RMS normalization (RMSNorm).
Formula: same maths as BN, different axis
- the features of one example (one token, one time step); is the product of
normalized_shape - mean and biased variance over those features: one pair per example, recomputed on every forward pass
- learned gain and bias per feature, initialised to 1 and 0
eps=1e-5in PyTorch'snn.LayerNorm
Compare with BN on a matrix of shape : BN computes one mean per column (a feature, across the examples), LN one mean per row (an example, across its features). Both then apply a per-feature affine map. For a transformer activation of shape , nn.LayerNorm(d) normalizes each of the token vectors independently. For images, LN over pools everything one image contains; Group normalization with one group does the same pooling with per-channel instead of per-element affine parameters, and Instance normalization pools only per channel.
At a glance
- Statistics over
- the last
len(normalized_shape)dimensions of one example - Learnable
- of shape
normalized_shape: parameters - Buffers
- none: no running mean or variance
- Train vs eval
- identical
- Batch-size dependence
- none; works with batch size 1 and variable-length sequences
- Cost
- two reductions over the feature axis per example, on every forward pass, including inference
- Typical home
- transformers, RNNs/LSTMs, ConvNeXt-style CNNs, MLP-Mixers
What it is invariant to
Because each example is standardized by its own statistics, LN's output does not change if you multiply that example's summed inputs by a positive constant or add a constant to all of them. That is the key property for RNNs: a hidden state that grows or drifts over time is rescaled at every step. Ba et al. compared the invariances of the two methods; the table restates their comparison.
| Transformation | Batch norm | Layer norm |
|---|---|---|
| Scale the whole weight matrix | invariant | invariant |
| Add the same vector to every weight vector (re-centre the matrix) | not invariant | invariant |
| Scale one unit's weight vector | invariant | not invariant |
| Scale the whole dataset | invariant | invariant |
| Shift the whole dataset | invariant | not invariant |
| Scale a single training example | not invariant | invariant |
The last row is the practical difference. A batch of one is fine for LN, and an example's output never depends on which other examples share its batch, which is why LN gives the same result in training, in evaluation, and in token-by-token generation with a KV cache.
Worked example: one token with D = 4
x = (2, −1, 0, 3), γ = 1, β = 0 (checked in Node)
- 1 and , so .
- 2: mean 0, biased variance 1.
- 3Feed instead: , , and is again . Scale and offset of the token are gone.
- 4Backward for : . It sums to 0 and is orthogonal to , the same structure as BN's gradient, but along the feature axis instead of the batch axis.
- 5RMS normalization (RMSNorm) skips the centring and divides by , giving : same direction of scaling, no mean removal.
Layer norm in RNNs
In a recurrent layer the summed input at step is . LN normalizes it over the hidden units before the nonlinearity, with gains and biases shared over time:
- number of hidden units
- statistics of one sequence at one step: nothing depends on other sequences or on the sequence length
If the recurrent weights would make explode or vanish over many steps, dividing by cancels that scale at every step, which stabilises the hidden-state dynamics. For LSTMs it is usually applied separately to the input-to-hidden and hidden-to-hidden contributions and to the cell state.
Transformers: Post-LN vs Pre-LN
- a sub-layer: multi-head attention or the feed-forward block
- the residual stream entering block
- the extra final LayerNorm that Pre-LN models need before the output head
The original Transformer and BERT are Post-LN: the normalization sits on the residual path, after the addition. GPT-2 moved it to the input of each sub-block and added a final LN, which is Pre-LN, now the default for large models. Xiong et al. ("On Layer Normalization in the Transformer Architecture", ICML 2020) explained why the choice matters for optimization. At initialization, the expected gradient of the parameters near the output of a Post-LN transformer is large and does not shrink with depth, so a large learning rate right away is unstable and learning-rate warm-up is essential. In Pre-LN the identity path is never normalized, the gradients are well behaved and shrink with depth roughly like , and they trained Pre-LN models without warm-up and faster.
The figure's second tab reproduces those trends in a toy residual MLP: Post-LN's top-block gradient stays near 9 to 12 for any depth, while Pre-LN's falls from 7.14 at 2 blocks to 3.28 at 24. It also shows the price of Pre-LN: the residual stream is never renormalized, so its norm keeps growing with depth (to 18.6 at 24 blocks), and each later block adds a relatively smaller update to it. Some studies (e.g. Liu et al., 2020) report that Post-LN reaches slightly better final quality when it can be trained stably, which motivated hybrids and careful initializations of the residual branches.
1import torch, torch.nn as nn, torch.nn.functional as F23torch.manual_seed(0)4x = torch.randn(2, 5, 16) # (batch B, tokens T, features d)5ln = nn.LayerNorm(16) # normalized_shape=16, eps=1e-5,6 # elementwise_affine=True, bias=True7y = ln(x)8mu = x.mean(dim=-1, keepdim=True)9var = x.var(dim=-1, keepdim=True, correction=0) # biased, per token10manual = (x - mu) / torch.sqrt(var + ln.eps) * ln.weight + ln.bias11print(torch.allclose(y, manual, atol=1e-6))12print(torch.allclose(y, F.layer_norm(x, (16,), ln.weight, ln.bias, ln.eps)))1314# no batch statistics: one example alone gives the same output, in train or eval mode15print(torch.allclose(ln(x[:1]), y[:1])) # True16print([n for n, _ in ln.named_buffers()]) # [] (no running stats)1718# invariant to scaling and shifting a single example19print(torch.allclose(ln(10 * x + 5), y, atol=1e-4)) # equal up to the eps term2021# normalized_shape covers the LAST dims: over (C, H, W) of an image22img = torch.randn(4, 8, 6, 6)23ln_chw = nn.LayerNorm([8, 6, 6], elementwise_affine=False) # ties the layer to 6x6 inputs24print(torch.allclose(ln_chw(img), F.group_norm(img, 1, eps=1e-5), atol=1e-5))2526class PreLNBlock(nn.Module):27 def __init__(self, d, heads):28 super().__init__()29 self.ln1, self.ln2 = nn.LayerNorm(d), nn.LayerNorm(d)30 self.attn = nn.MultiheadAttention(d, heads, batch_first=True)31 self.mlp = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d))32 def forward(self, x):33 h = self.ln1(x)34 x = x + self.attn(h, h, h, need_weights=False)[0] # LN inside the residual branch35 return x + self.mlp(self.ln2(x))3637# PyTorch's own layer: norm_first=True is Pre-LN, the default False is Post-LN38layer = nn.TransformerEncoderLayer(d_model=16, nhead=4, batch_first=True, norm_first=True)Use when
- +Independent of batch size; works with batch 1, variable-length sequences and autoregressive decoding
- +Same computation in training and inference: no running statistics to go stale
- +Per-example, so no cross-device synchronization under data parallelism
- +Invariant to the scale and offset of each example's summed inputs, which stabilises RNNs and deep transformers
Avoid when
- −Statistics must be computed at inference too, and cannot be folded into the preceding layer the way BN can
- −Mixes all features of an example: in CNNs it usually underperforms BN at normal batch sizes
- −Two reductions per token; RMSNorm saves the mean computation at little cost in quality
- −Placement matters: Post-LN needs warm-up, Pre-LN lets the residual stream grow with depth
Check yourself
A language model's activations have shape (B, T, d) = (8, 512, 768) and pass through nn.LayerNorm(768). How many means are computed, and over how many values each?
Check yourself
Why do Post-LN transformers typically need learning-rate warm-up while Pre-LN transformers can often train without it?
Remember
- ✓LN standardizes each example over its own features, , with one per example or token.
- ✓No batch statistics and no running buffers: identical in training and inference, and independent of batch size.
- ✓Invariant to scaling and shifting a single example's summed inputs, which suits RNNs (statistics per time step) and transformers (per token).
- ✓
nn.LayerNorm(normalized_shape, eps=1e-5, elementwise_affine=True)normalizes the trailing dimensions; for usenn.LayerNorm(d). - ✓Post-LN puts LN after the residual addition and needs warm-up; Pre-LN puts it inside the branch plus a final LN and trains more stably (Xiong et al., 2020).
- ✓BN is better for CNNs at normal batch sizes; LN, Group or RMS normalization take over when batches are small or sequences vary.