# Layer normalization

> Normalize each example over its own features (each token, in a transformer), then apply a learned per-feature gain and bias. No batch statistics, so it is identical in training and inference at any batch size. It is the normalization of RNNs and transformers.

Source: https://www.mlfoundry.in/learn/layer-normalization  
Section: Deep Learning › Regularization & Normalization  
Level: core · 13 min read · updated 2026-10-11

**Layer normalization:** (LayerNorm, LN) standardizes the activations of **one example** across its features: subtract the mean of that example's $D$ values, divide by their standard deviation, then apply a learned per-feature gain $\gamma$ and bias $\beta$. Nothing is pooled across the batch, so there are no running statistics and no difference between train and eval mode.

Jimmy Lei Ba, Jamie Ryan Kiros and Geoffrey Hinton proposed it in "Layer Normalization" (arXiv, 2016) as a fix for the parts of **Batch normalization** that do not fit recurrent networks. BN needs a mini-batch to estimate statistics, separate statistics for every time step of an RNN, and a test-time sequence no longer than the training ones. LN replaces all of that with statistics computed from the summed inputs of a layer for a single example at a single time step. They showed faster convergence on several recurrent tasks, including skip-thought vectors, the attentive reader and DRAW.

Its biggest success came a year later. The Transformer (Vaswani et al., 2017) wrapped every attention and feed-forward sub-layer as $\mathrm{LayerNorm}(x+\mathrm{Sublayer}(x))$, and nearly every transformer since normalizes each token with LN or with its cheaper relative **RMS normalization (RMSNorm)**.

## Formula: same maths as BN, different axis

$$
\mu=\frac1D\sum_{i=1}^{D}x_i,\qquad \sigma^2=\frac1D\sum_{i=1}^{D}(x_i-\mu)^2,\qquad y_i=\gamma_i\,\frac{x_i-\mu}{\sqrt{\sigma^2+\epsilon}}+\beta_i
$$

- $x\in\mathbb R^D$: the features of **one** example (one token, one time step); $D$ is the product of `normalized_shape`
- $\mu,\ \sigma^2$: mean and biased variance over those $D$ features: one pair per example, recomputed on every forward pass
- $\gamma_i,\ \beta_i$: learned gain and bias **per feature**, initialised to 1 and 0
- $\epsilon$: `eps=1e-5` in PyTorch's `nn.LayerNorm`

Compare with BN on a matrix of shape $(N, D)$: BN computes one mean per **column** (a feature, across the $N$ examples), LN one mean per **row** (an example, across its $D$ features). Both then apply a per-feature affine map. For a transformer activation of shape $(B, T, d)$, `nn.LayerNorm(d)` normalizes each of the $B\cdot T$ token vectors independently. For images, LN over $(C, H, W)$ pools everything one image contains; **Group normalization** with one group does the same pooling with per-channel instead of per-element affine parameters, and **Instance normalization** pools only $(H, W)$ per channel.

**At a glance**

- **Statistics over:** the last `len(normalized_shape)` dimensions of one example
- **Learnable:** $\gamma,\beta$ of shape `normalized_shape`: $2D$ parameters
- **Buffers:** none: no running mean or variance
- **Train vs eval:** identical
- **Batch-size dependence:** none; works with batch size 1 and variable-length sequences
- **Cost:** two reductions over the feature axis per example, on every forward pass, including inference
- **Typical home:** transformers, RNNs/LSTMs, ConvNeXt-style CNNs, MLP-Mixers

> Figure: Tab 1: a batch of 4 examples × 8 features whose rows have very different scales (row means −0.39, 3.88, −1.57, 1.37; standard deviations 1.24, 3.29, 0.40, 1.25). LayerNorm pools one row, BatchNorm one column. Rescale or shift example n1: with a = 5 LayerNorm's output for n0 moves by exactly 0 and n1's own output by 6.4e−7 (only ε breaks the invariance), while BatchNorm's output for n0 moves by up to 0.511; a shift b = 8 moves it by 0.541. Shrink the batch to 2 and every BatchNorm output becomes ±1; at 1 they are all 0. LayerNorm does not change. Tab 2: a toy 16-wide residual stack of ReLU MLP blocks. At 12 blocks the top block's weight gradient is 3.94 with Pre-LN and 10.36 with Post-LN (2.63×). Post-LN's gradients grow toward the output (top/bottom ratio 1.51), Pre-LN's shrink (0.40). The ratio widens with depth: 1.25× at 2 blocks, 3.63× at 24. The Pre-LN residual stream grows to a norm of 14.1, while Post-LN resets it to √16 = 4 after every block.

## What it is invariant to

Because each example is standardized by its own statistics, LN's output does not change if you multiply that example's summed inputs by a positive constant or add a constant to all of them. That is the key property for RNNs: a hidden state that grows or drifts over time is rescaled at every step. Ba et al. compared the invariances of the two methods; the table restates their comparison.

Invariance of the normalized output under transformations of the weights and data (after Ba, Kiros and Hinton, 2016, Table 1)

| Transformation | Batch norm | Layer norm |
| --- | --- | --- |
| Scale the whole weight matrix | invariant | invariant |
| Add the same vector to every weight vector (re-centre the matrix) | not invariant | invariant |
| Scale one unit's weight vector | invariant | not invariant |
| Scale the whole dataset | invariant | invariant |
| Shift the whole dataset | invariant | not invariant |
| Scale a single training example | not invariant | invariant |

The last row is the practical difference. A batch of one is fine for LN, and an example's output never depends on which other examples share its batch, which is why LN gives the same result in training, in evaluation, and in token-by-token generation with a KV cache.

## Worked example: one token with D = 4

**x = (2, −1, 0, 3), γ = 1, β = 0 (checked in Node)**

1. $\mu=(2-1+0+3)/4=1$ and $\sigma^2=(1+4+1+4)/4=2.5$, so $\sigma=1.5811$.
2. $\hat x=(x-\mu)/\sigma=(0.6325,\,-1.2649,\,-0.6325,\,1.2649)$: mean 0, biased variance 1.
3. Feed $10x+5=(25,-5,5,35)$ instead: $\mu=15$, $\sigma^2=250$, and $\hat x$ is again $(0.6325,-1.2649,-0.6325,1.2649)$. Scale and offset of the token are gone.
4. Backward for $\partial L/\partial y=(1,0,0,0)$: $\partial L/\partial x=\tfrac1\sigma\big(g-\bar g-\hat x\,\overline{g\hat x}\big)=(0.4111,\,-0.0316,\,-0.0949,\,-0.2846)$. It sums to 0 and is orthogonal to $\hat x$, the same structure as BN's gradient, but along the feature axis instead of the batch axis.
5. **RMS normalization (RMSNorm)** skips the centring and divides by $\sqrt{\tfrac1D\sum x_i^2}=1.8708$, giving $(1.0690,-0.5345,0,1.6036)$: same direction of scaling, no mean removal.

## Layer norm in RNNs

In a recurrent layer the summed input at step $t$ is $a_t=W_{hh}h_{t-1}+W_{xh}x_t$. LN normalizes it over the hidden units before the nonlinearity, with gains and biases shared over time:

$$
h_t=f\Big(\frac{\gamma}{\sigma_t}\odot(a_t-\mu_t)+\beta\Big),\qquad \mu_t=\frac1H\sum_{i=1}^H a_{t,i},\quad \sigma_t=\sqrt{\frac1H\sum_{i=1}^H(a_{t,i}-\mu_t)^2}
$$

- $H$: number of hidden units
- $\mu_t,\ \sigma_t$: statistics of one sequence at one step: nothing depends on other sequences or on the sequence length

If the recurrent weights would make $a_t$ explode or vanish over many steps, dividing by $\sigma_t$ cancels that scale at every step, which stabilises the hidden-state dynamics. For LSTMs it is usually applied separately to the input-to-hidden and hidden-to-hidden contributions and to the cell state.

## Transformers: Post-LN vs Pre-LN

$$
\text{Post-LN: } x_{l+1}=\mathrm{LN}\big(x_l+F_l(x_l)\big)\qquad\qquad \text{Pre-LN: } x_{l+1}=x_l+F_l\big(\mathrm{LN}(x_l)\big),\ \ \text{output}=\mathrm{LN}(x_L)
$$

- $F_l$: a sub-layer: multi-head attention or the feed-forward block
- $x_l$: the residual stream entering block $l$
- $\mathrm{LN}(x_L)$: the extra final LayerNorm that Pre-LN models need before the output head

The original Transformer and BERT are **Post-LN**: the normalization sits on the residual path, after the addition. GPT-2 moved it to the input of each sub-block and added a final LN, which is **Pre-LN**, now the default for large models. Xiong et al. ("On Layer Normalization in the Transformer Architecture", ICML 2020) explained why the choice matters for optimization. At initialization, the expected gradient of the parameters near the output of a Post-LN transformer is large and does not shrink with depth, so a large learning rate right away is unstable and **learning-rate warm-up** is essential. In Pre-LN the identity path is never normalized, the gradients are well behaved and shrink with depth roughly like $1/\sqrt L$, and they trained Pre-LN models without warm-up and faster.

The figure's second tab reproduces those trends in a toy residual MLP: Post-LN's top-block gradient stays near 9 to 12 for any depth, while Pre-LN's falls from 7.14 at 2 blocks to 3.28 at 24. It also shows the price of Pre-LN: the residual stream is never renormalized, so its norm keeps growing with depth (to 18.6 at 24 blocks), and each later block adds a relatively smaller update to it. Some studies (e.g. Liu et al., 2020) report that Post-LN reaches slightly better final quality when it can be trained stably, which motivated hybrids and careful initializations of the residual branches.

> **See also:** **Warmup scheduling** covers the learning-rate side of Post-LN instability, and **AdamW** the usual optimizer for transformers. Many recent language models (LLaMA, for example) keep the Pre-LN layout but swap LayerNorm for **RMS normalization (RMSNorm)**, which drops the mean subtraction and $\beta$.

```python
import torch, torch.nn as nn, torch.nn.functional as F

torch.manual_seed(0)
x = torch.randn(2, 5, 16)                       # (batch B, tokens T, features d)
ln = nn.LayerNorm(16)                           # normalized_shape=16, eps=1e-5,
                                                # elementwise_affine=True, bias=True
y = ln(x)
mu  = x.mean(dim=-1, keepdim=True)
var = x.var(dim=-1, keepdim=True, correction=0)  # biased, per token
manual = (x - mu) / torch.sqrt(var + ln.eps) * ln.weight + ln.bias
print(torch.allclose(y, manual, atol=1e-6))
print(torch.allclose(y, F.layer_norm(x, (16,), ln.weight, ln.bias, ln.eps)))

# no batch statistics: one example alone gives the same output, in train or eval mode
print(torch.allclose(ln(x[:1]), y[:1]))         # True
print([n for n, _ in ln.named_buffers()])       # []  (no running stats)

# invariant to scaling and shifting a single example
print(torch.allclose(ln(10 * x + 5), y, atol=1e-4))   # equal up to the eps term

# normalized_shape covers the LAST dims: over (C, H, W) of an image
img = torch.randn(4, 8, 6, 6)
ln_chw = nn.LayerNorm([8, 6, 6], elementwise_affine=False)   # ties the layer to 6x6 inputs
print(torch.allclose(ln_chw(img), F.group_norm(img, 1, eps=1e-5), atol=1e-5))

class PreLNBlock(nn.Module):
    def __init__(self, d, heads):
        super().__init__()
        self.ln1, self.ln2 = nn.LayerNorm(d), nn.LayerNorm(d)
        self.attn = nn.MultiheadAttention(d, heads, batch_first=True)
        self.mlp = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d))
    def forward(self, x):
        h = self.ln1(x)
        x = x + self.attn(h, h, h, need_weights=False)[0]   # LN inside the residual branch
        return x + self.mlp(self.ln2(x))

# PyTorch's own layer: norm_first=True is Pre-LN, the default False is Post-LN
layer = nn.TransformerEncoderLayer(d_model=16, nhead=4, batch_first=True, norm_first=True)
```

nn.LayerNorm by hand, batch independence, and a Pre-LN transformer block

> **normalized_shape is the trailing dimensions:** `nn.LayerNorm(d)` normalizes over the **last** dimension only. Passing a channels-first image $(N, C, H, W)$ to `nn.LayerNorm(C)` fails (or, if $W$ happens to equal $C$, silently normalizes over the width), and `nn.LayerNorm([C, H, W])` pools the whole image and learns one $\gamma,\beta$ per element, which ties the layer to one input resolution. ConvNeXt-style "LayerNorm over channels" permutes to channels-last first; `nn.GroupNorm(1, C)` pools over $(C,H,W)$ with per-channel affine parameters instead.

> **A feature axis of size 1:** LN over a single feature has variance 0, so the normalized value is always 0 and the output is just $\beta$. The same happens for any input whose features are all equal. Check that the axis you normalize over is really the feature axis and not, say, a length-1 time axis.

> **Regularizer or not?:** Unlike BN, LN injects no noise: the output of an example is a deterministic function of that example. Its benefit is purely about optimization and signal propagation, keeping each token's representation at a fixed scale so that attention logits, residual sums and gradients stay in a trainable range. Regularization in transformers comes from elsewhere (**Dropout**, **Weight decay**, **Label smoothing**). Like BN gains, LN's $\gamma$ and $\beta$ are normally excluded from weight decay.

**Strengths**

- Independent of batch size; works with batch 1, variable-length sequences and autoregressive decoding
- Same computation in training and inference: no running statistics to go stale
- Per-example, so no cross-device synchronization under data parallelism
- Invariant to the scale and offset of each example's summed inputs, which stabilises RNNs and deep transformers

**Limitations**

- Statistics must be computed at inference too, and cannot be folded into the preceding layer the way BN can
- Mixes all features of an example: in CNNs it usually underperforms BN at normal batch sizes
- Two reductions per token; RMSNorm saves the mean computation at little cost in quality
- Placement matters: Post-LN needs warm-up, Pre-LN lets the residual stream grow with depth

**Question:** A language model's activations have shape (B, T, d) = (8, 512, 768) and pass through nn.LayerNorm(768). How many means are computed, and over how many values each?

- A. 8·512 = 4,096 means over 768 values each
- B. 768 means over 8·512 values each
- C. 8 means over 512·768 values each
- D. One mean over all 8·512·768 values

**Answer:** A. 8·512 = 4,096 means over 768 values each — LN pools the trailing `normalized_shape` dimensions of each position: every token vector of 768 features gets its own mean and variance, so there are $B\cdot T=4{,}096$ of them. 768 means over the batch-and-time axis would be BN-style statistics.

**Question:** Why do Post-LN transformers typically need learning-rate warm-up while Pre-LN transformers can often train without it?

- A. At initialization Post-LN has large gradients for the parameters near the output, so early large steps are unstable
- B. Post-LN has running statistics that need time to converge
- C. Pre-LN has no learnable parameters in its normalization
- D. Post-LN uses batch statistics, Pre-LN uses per-token statistics

**Answer:** A. At initialization Post-LN has large gradients for the parameters near the output, so early large steps are unstable — Xiong et al. (2020) showed that in Post-LN the expected gradient of the last layers' parameters is large and independent of depth at initialization, while in Pre-LN it shrinks with depth because the unnormalized residual path carries the signal. Both variants use the same per-token LayerNorm with no running statistics.

**Key points**

- LN standardizes each example over its own $D$ features, $y=\gamma\odot(x-\mu)/\sqrt{\sigma^2+\epsilon}+\beta$, with one $\mu,\sigma$ per example or token.
- No batch statistics and no running buffers: identical in training and inference, and independent of batch size.
- Invariant to scaling and shifting a single example's summed inputs, which suits RNNs (statistics per time step) and transformers (per token).
- `nn.LayerNorm(normalized_shape, eps=1e-5, elementwise_affine=True)` normalizes the trailing dimensions; for $(B,T,d)$ use `nn.LayerNorm(d)`.
- Post-LN puts LN after the residual addition and needs warm-up; Pre-LN puts it inside the branch plus a final LN and trains more stably (Xiong et al., 2020).
- BN is better for CNNs at normal batch sizes; LN, Group or RMS normalization take over when batches are small or sequences vary.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
