Layer normalization

Normalize each example over its own features (each token, in a transformer), then apply a learned per-feature gain and bias. No batch statistics, so it is identical in training and inference at any batch size. It is the normalization of RNNs and transformers.

13 min read1144 wordscore0 viewsupdated 2026-10-11

Definition

Layer normalization (LayerNorm, LN) standardizes the activations of one example across its features: subtract the mean of that example's DD values, divide by their standard deviation, then apply a learned per-feature gain γ\gamma and bias β\beta. Nothing is pooled across the batch, so there are no running statistics and no difference between train and eval mode.

Jimmy Lei Ba, Jamie Ryan Kiros and Geoffrey Hinton proposed it in "Layer Normalization" (arXiv, 2016) as a fix for the parts of Batch normalization that do not fit recurrent networks. BN needs a mini-batch to estimate statistics, separate statistics for every time step of an RNN, and a test-time sequence no longer than the training ones. LN replaces all of that with statistics computed from the summed inputs of a layer for a single example at a single time step. They showed faster convergence on several recurrent tasks, including skip-thought vectors, the attentive reader and DRAW.

Its biggest success came a year later. The Transformer (Vaswani et al., 2017) wrapped every attention and feed-forward sub-layer as LayerNorm(x+Sublayer(x))\mathrm{LayerNorm}(x+\mathrm{Sublayer}(x)), and nearly every transformer since normalizes each token with LN or with its cheaper relative RMS normalization (RMSNorm).

Formula: same maths as BN, different axis

μ=1D∑i=1Dxi,σ2=1D∑i=1D(xi−μ)2,yi=γi xi−μσ2+ϵ+βi\mu=\frac1D\sum_{i=1}^{D}x_i,\qquad \sigma^2=\frac1D\sum_{i=1}^{D}(x_i-\mu)^2,\qquad y_i=\gamma_i\,\frac{x_i-\mu}{\sqrt{\sigma^2+\epsilon}}+\beta_i
x∈RDx\in\mathbb R^D
the features of one example (one token, one time step); DD is the product of normalized_shape
μ, σ2\mu,\ \sigma^2
mean and biased variance over those DD features: one pair per example, recomputed on every forward pass
γi, βi\gamma_i,\ \beta_i
learned gain and bias per feature, initialised to 1 and 0
ϵ\epsilon
eps=1e-5 in PyTorch's nn.LayerNorm

Compare with BN on a matrix of shape (N,D)(N, D): BN computes one mean per column (a feature, across the NN examples), LN one mean per row (an example, across its DD features). Both then apply a per-feature affine map. For a transformer activation of shape (B,T,d)(B, T, d), nn.LayerNorm(d) normalizes each of the B⋅TB\cdot T token vectors independently. For images, LN over (C,H,W)(C, H, W) pools everything one image contains; Group normalization with one group does the same pooling with per-channel instead of per-element affine parameters, and Instance normalization pools only (H,W)(H, W) per channel.

At a glance

Statistics over
the last len(normalized_shape) dimensions of one example
Learnable
γ,β\gamma,\beta of shape normalized_shape: 2D2D parameters
Buffers
none: no running mean or variance
Train vs eval
identical
Batch-size dependence
none; works with batch size 1 and variable-length sequences
Cost
two reductions over the feature axis per example, on every forward pass, including inference
Typical home
transformers, RNNs/LSTMs, ConvNeXt-style CNNs, MLP-Mixers
x has shape (N, D) = (4, 8); example n1 is replaced by a·x + binput xrow μ, σ differ a lotLayerNorm(x)pool one row: D = 8 valuesBatchNorm(x), trainpool one column: N = 4 valuesn0−0.39, 1.24n13.88, 3.29n2−1.57, 0.40n31.37, 1.25μ, σexample n0 after ■ LayerNorm and ■ BatchNorm (faint = at a = 1, b = 0, N = 4)d0d1d2d3d4d5d6d7LayerNorm: n0 moved by max 0 (other examples never enter its statistics)LayerNorm: n1 itself moved by max 0 for a = 1.00, b = 0.00 (only ε breaks exact invariance)BatchNorm: n0 moved by max 0 · its output depends on its batch-mates
|batch size
fig.Tab 1: a batch of 4 examples × 8 features whose rows have very different scales (row means −0.39, 3.88, −1.57, 1.37; standard deviations 1.24, 3.29, 0.40, 1.25). LayerNorm pools one row, BatchNorm one column. Rescale or shift example n1: with a = 5 LayerNorm's output for n0 moves by exactly 0 and n1's own output by 6.4e−7 (only ε breaks the invariance), while BatchNorm's output for n0 moves by up to 0.511; a shift b = 8 moves it by 0.541. Shrink the batch to 2 and every BatchNorm output becomes ±1; at 1 they are all 0. LayerNorm does not change. Tab 2: a toy 16-wide residual stack of ReLU MLP blocks. At 12 blocks the top block's weight gradient is 3.94 with Pre-LN and 10.36 with Post-LN (2.63×). Post-LN's gradients grow toward the output (top/bottom ratio 1.51), Pre-LN's shrink (0.40). The ratio widens with depth: 1.25× at 2 blocks, 3.63× at 24. The Pre-LN residual stream grows to a norm of 14.1, while Post-LN resets it to √16 = 4 after every block.

What it is invariant to

Because each example is standardized by its own statistics, LN's output does not change if you multiply that example's summed inputs by a positive constant or add a constant to all of them. That is the key property for RNNs: a hidden state that grows or drifts over time is rescaled at every step. Ba et al. compared the invariances of the two methods; the table restates their comparison.

TransformationBatch normLayer norm
Scale the whole weight matrixinvariantinvariant
Add the same vector to every weight vector (re-centre the matrix)not invariantinvariant
Scale one unit's weight vectorinvariantnot invariant
Scale the whole datasetinvariantinvariant
Shift the whole datasetinvariantnot invariant
Scale a single training examplenot invariantinvariant
Invariance of the normalized output under transformations of the weights and data (after Ba, Kiros and Hinton, 2016, Table 1)

The last row is the practical difference. A batch of one is fine for LN, and an example's output never depends on which other examples share its batch, which is why LN gives the same result in training, in evaluation, and in token-by-token generation with a KV cache.

Worked example: one token with D = 4

x = (2, −1, 0, 3), γ = 1, β = 0 (checked in Node)

  1. 1μ=(2−1+0+3)/4=1\mu=(2-1+0+3)/4=1 and σ2=(1+4+1+4)/4=2.5\sigma^2=(1+4+1+4)/4=2.5, so σ=1.5811\sigma=1.5811.
  2. 2x^=(x−μ)/σ=(0.6325, −1.2649, −0.6325, 1.2649)\hat x=(x-\mu)/\sigma=(0.6325,\,-1.2649,\,-0.6325,\,1.2649): mean 0, biased variance 1.
  3. 3Feed 10x+5=(25,−5,5,35)10x+5=(25,-5,5,35) instead: μ=15\mu=15, σ2=250\sigma^2=250, and x^\hat x is again (0.6325,−1.2649,−0.6325,1.2649)(0.6325,-1.2649,-0.6325,1.2649). Scale and offset of the token are gone.
  4. 4Backward for ∂L/∂y=(1,0,0,0)\partial L/\partial y=(1,0,0,0): ∂L/∂x=1σ(g−gˉ−x^ gx^‾)=(0.4111, −0.0316, −0.0949, −0.2846)\partial L/\partial x=\tfrac1\sigma\big(g-\bar g-\hat x\,\overline{g\hat x}\big)=(0.4111,\,-0.0316,\,-0.0949,\,-0.2846). It sums to 0 and is orthogonal to x^\hat x, the same structure as BN's gradient, but along the feature axis instead of the batch axis.
  5. 5RMS normalization (RMSNorm) skips the centring and divides by 1D∑xi2=1.8708\sqrt{\tfrac1D\sum x_i^2}=1.8708, giving (1.0690,−0.5345,0,1.6036)(1.0690,-0.5345,0,1.6036): same direction of scaling, no mean removal.

Layer norm in RNNs

In a recurrent layer the summed input at step tt is at=Whhht−1+Wxhxta_t=W_{hh}h_{t-1}+W_{xh}x_t. LN normalizes it over the hidden units before the nonlinearity, with gains and biases shared over time:

ht=f(γσt⊙(at−μt)+β),μt=1H∑i=1Hat,i,σt=1H∑i=1H(at,i−μt)2h_t=f\Big(\frac{\gamma}{\sigma_t}\odot(a_t-\mu_t)+\beta\Big),\qquad \mu_t=\frac1H\sum_{i=1}^H a_{t,i},\quad \sigma_t=\sqrt{\frac1H\sum_{i=1}^H(a_{t,i}-\mu_t)^2}
HH
number of hidden units
μt, σt\mu_t,\ \sigma_t
statistics of one sequence at one step: nothing depends on other sequences or on the sequence length

If the recurrent weights would make ata_t explode or vanish over many steps, dividing by σt\sigma_t cancels that scale at every step, which stabilises the hidden-state dynamics. For LSTMs it is usually applied separately to the input-to-hidden and hidden-to-hidden contributions and to the cell state.

Transformers: Post-LN vs Pre-LN

Post-LN: xl+1=LN(xl+Fl(xl))Pre-LN: xl+1=xl+Fl(LN(xl)),  output=LN(xL)\text{Post-LN: } x_{l+1}=\mathrm{LN}\big(x_l+F_l(x_l)\big)\qquad\qquad \text{Pre-LN: } x_{l+1}=x_l+F_l\big(\mathrm{LN}(x_l)\big),\ \ \text{output}=\mathrm{LN}(x_L)
FlF_l
a sub-layer: multi-head attention or the feed-forward block
xlx_l
the residual stream entering block ll
LN(xL)\mathrm{LN}(x_L)
the extra final LayerNorm that Pre-LN models need before the output head

The original Transformer and BERT are Post-LN: the normalization sits on the residual path, after the addition. GPT-2 moved it to the input of each sub-block and added a final LN, which is Pre-LN, now the default for large models. Xiong et al. ("On Layer Normalization in the Transformer Architecture", ICML 2020) explained why the choice matters for optimization. At initialization, the expected gradient of the parameters near the output of a Post-LN transformer is large and does not shrink with depth, so a large learning rate right away is unstable and learning-rate warm-up is essential. In Pre-LN the identity path is never normalized, the gradients are well behaved and shrink with depth roughly like 1/L1/\sqrt L, and they trained Pre-LN models without warm-up and faster.

The figure's second tab reproduces those trends in a toy residual MLP: Post-LN's top-block gradient stays near 9 to 12 for any depth, while Pre-LN's falls from 7.14 at 2 blocks to 3.28 at 24. It also shows the price of Pre-LN: the residual stream is never renormalized, so its norm keeps growing with depth (to 18.6 at 24 blocks), and each later block adds a relatively smaller update to it. Some studies (e.g. Liu et al., 2020) report that Post-LN reaches slightly better final quality when it can be trained stably, which motivated hybrids and careful initializations of the residual branches.

python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3torch.manual_seed(0)
4x = torch.randn(2, 5, 16) # (batch B, tokens T, features d)
5ln = nn.LayerNorm(16) # normalized_shape=16, eps=1e-5,
6 # elementwise_affine=True, bias=True
7y = ln(x)
8mu = x.mean(dim=-1, keepdim=True)
9var = x.var(dim=-1, keepdim=True, correction=0) # biased, per token
10manual = (x - mu) / torch.sqrt(var + ln.eps) * ln.weight + ln.bias
11print(torch.allclose(y, manual, atol=1e-6))
12print(torch.allclose(y, F.layer_norm(x, (16,), ln.weight, ln.bias, ln.eps)))
13
14# no batch statistics: one example alone gives the same output, in train or eval mode
15print(torch.allclose(ln(x[:1]), y[:1])) # True
16print([n for n, _ in ln.named_buffers()]) # [] (no running stats)
17
18# invariant to scaling and shifting a single example
19print(torch.allclose(ln(10 * x + 5), y, atol=1e-4)) # equal up to the eps term
20
21# normalized_shape covers the LAST dims: over (C, H, W) of an image
22img = torch.randn(4, 8, 6, 6)
23ln_chw = nn.LayerNorm([8, 6, 6], elementwise_affine=False) # ties the layer to 6x6 inputs
24print(torch.allclose(ln_chw(img), F.group_norm(img, 1, eps=1e-5), atol=1e-5))
25
26class PreLNBlock(nn.Module):
27 def __init__(self, d, heads):
28 super().__init__()
29 self.ln1, self.ln2 = nn.LayerNorm(d), nn.LayerNorm(d)
30 self.attn = nn.MultiheadAttention(d, heads, batch_first=True)
31 self.mlp = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d))
32 def forward(self, x):
33 h = self.ln1(x)
34 x = x + self.attn(h, h, h, need_weights=False)[0] # LN inside the residual branch
35 return x + self.mlp(self.ln2(x))
36
37# PyTorch's own layer: norm_first=True is Pre-LN, the default False is Post-LN
38layer = nn.TransformerEncoderLayer(d_model=16, nhead=4, batch_first=True, norm_first=True)
nn.LayerNorm by hand, batch independence, and a Pre-LN transformer block

Use when

  • +Independent of batch size; works with batch 1, variable-length sequences and autoregressive decoding
  • +Same computation in training and inference: no running statistics to go stale
  • +Per-example, so no cross-device synchronization under data parallelism
  • +Invariant to the scale and offset of each example's summed inputs, which stabilises RNNs and deep transformers

Avoid when

  • −Statistics must be computed at inference too, and cannot be folded into the preceding layer the way BN can
  • −Mixes all features of an example: in CNNs it usually underperforms BN at normal batch sizes
  • −Two reductions per token; RMSNorm saves the mean computation at little cost in quality
  • −Placement matters: Post-LN needs warm-up, Pre-LN lets the residual stream grow with depth

Check yourself

A language model's activations have shape (B, T, d) = (8, 512, 768) and pass through nn.LayerNorm(768). How many means are computed, and over how many values each?

Check yourself

Why do Post-LN transformers typically need learning-rate warm-up while Pre-LN transformers can often train without it?

Remember

  • ✓LN standardizes each example over its own DD features, y=γ⊙(x−μ)/σ2+ϵ+βy=\gamma\odot(x-\mu)/\sqrt{\sigma^2+\epsilon}+\beta, with one μ,σ\mu,\sigma per example or token.
  • ✓No batch statistics and no running buffers: identical in training and inference, and independent of batch size.
  • ✓Invariant to scaling and shifting a single example's summed inputs, which suits RNNs (statistics per time step) and transformers (per token).
  • ✓nn.LayerNorm(normalized_shape, eps=1e-5, elementwise_affine=True) normalizes the trailing dimensions; for (B,T,d)(B,T,d) use nn.LayerNorm(d).
  • ✓Post-LN puts LN after the residual addition and needs warm-up; Pre-LN puts it inside the branch plus a final LN and trains more stably (Xiong et al., 2020).
  • ✓BN is better for CNNs at normal batch sizes; LN, Group or RMS normalization take over when batches are small or sequences vary.
← back to Deep Learning roadmap