RMS normalization (RMSNorm)

Layer norm without the mean: divide each feature vector by its root mean square and multiply by a learned gain. It keeps invariance to rescaling, drops invariance to shifts, needs one reduction instead of two, and is the normalizer of T5, Gopher, LLaMA and most recent LLMs.

11 min read956 wordscore0 viewsupdated 2026-10-11

Definition

RMSNorm (root mean square layer normalization) rescales a feature vector x∈Rdx\in\mathbb R^d to unit root mean square, xˉ=x/RMS(x)\bar x=x/\mathrm{RMS}(x) with RMS(x)=1d∑ixi2\mathrm{RMS}(x)=\sqrt{\tfrac1d\sum_i x_i^2}, then multiplies by a learned per-feature gain gg. Unlike Layer normalization, it does not subtract the mean and has no bias.

Layer normalization does two things to each token's vector: it re-centres it (subtract the mean) and re-scales it (divide by the standard deviation). Zhang and Sennrich (2019), Root Mean Square Layer Normalization (NeurIPS), asked which of the two matters. Their hypothesis was that the benefit comes from re-scaling invariance: the output does not change when the input or the incoming weights are multiplied by a constant. Re-centring, they argued, is dispensable. Dropping it removes the mean, the bias and one reduction over the features. Across their RNN and Transformer experiments, RMSNorm matched layer norm's quality while cutting running time by 7% to 64%, depending on model and implementation.

Large language models adopted it quickly. T5 (Raffel et al., 2020) used a layer norm that "only rescales" and has no additive bias, which is RMSNorm. Gopher (Rae et al., 2021) and LLaMA (Touvron et al., 2023) use RMSNorm in a pre-norm layout, and most open LLMs since then follow LLaMA.

Formula and gradient

RMS(x)=1d∑i=1dxi2+ε,yi=xiRMS(x) givs. LayerNorm: yi=xi−μσ2+ε γi+βi\mathrm{RMS}(x)=\sqrt{\frac1d\sum_{i=1}^d x_i^2+\varepsilon},\qquad y_i=\frac{x_i}{\mathrm{RMS}(x)}\,g_i\qquad\text{vs. LayerNorm: } y_i=\frac{x_i-\mu}{\sqrt{\sigma^2+\varepsilon}}\,\gamma_i+\beta_i
x∈Rdx\in\mathbb R^d
one token's feature vector (normalization is over the last dimension)
gg
learned gain, one per feature, initialized to 1 (weight in PyTorch)
ε\varepsilon
added inside the square root, as in PyTorch, T5 and LLaMA; LLaMA uses 10−610^{-6}
μ, σ2, β\mu,\ \sigma^2,\ \beta
layer norm's mean, variance and bias, all absent from RMSNorm

With xˉ=x/RMS(x)\bar x=x/\mathrm{RMS}(x) and ε=0\varepsilon=0, the Jacobian is ∂xˉ/∂x=1RMS(I−1dxˉxˉ⊤)\partial\bar x/\partial x=\frac{1}{\mathrm{RMS}}\big(I-\frac{1}{d}\bar x\bar x^\top\big). So for an upstream gradient u=g⊙∂L/∂yu=g\odot\partial L/\partial y:

∂L∂x=1RMS(x)(u−xˉ xˉ⊤ud),∂L∂gi=∑tokens∂L∂yi xˉi\frac{\partial L}{\partial x}=\frac{1}{\mathrm{RMS}(x)}\Big(u-\bar x\,\frac{\bar x^\top u}{d}\Big),\qquad \frac{\partial L}{\partial g_i}=\sum_{\text{tokens}}\frac{\partial L}{\partial y_i}\,\bar x_i
xˉ xˉ⊤ud\bar x\,\tfrac{\bar x^\top u}{d}
the component of uu along xx, removed because rescaling xx cannot change the output
1/RMS1/\mathrm{RMS}
larger inputs get proportionally smaller gradients

Two consequences follow. The input gradient is orthogonal to xx: x⊤∂L/∂x=0x^\top\partial L/\partial x=0. And it scales like 1/RMS(x)1/\mathrm{RMS}(x): if the weights WW feeding the norm grow by a factor α\alpha, the output is unchanged and the gradient on WW shrinks by 1/α1/\alpha. Zhang and Sennrich call this an implicit learning-rate adaptation. Layer norm has the same two properties and, in addition, a gradient that sums to zero (shift invariance). RMSNorm gives that up.

At a glance

Statistics over
the dd features of one token: a single sum of squares
Invariant to
x↦αxx\mapsto\alpha x (α>0\alpha>0) and to scaling the incoming weights
Not invariant to
x↦x+t1x\mapsto x+t\mathbf 1: a shared offset changes the output
Parameters
dd (gain only), versus 2d2d for layer norm
Relation to LN
LN(x)=RMSNorm(x−μ1)\mathrm{LN}(x)=\mathrm{RMSNorm}(x-\mu\mathbf 1) with the same ε\varepsilon (and β=0\beta=0)
Used in
T5, Gopher, Chinchilla, LLaMA and its descendants (Mistral, Qwen, Gemma …), as pre-norm and as QK-norm
Paper
Zhang & Sennrich (2019), NeurIPS

What is kept and what is lost

Geometrically, RMSNorm maps every nonzero xx radially onto the sphere of radius d\sqrt d (then gg stretches the axes). Layer norm first projects xx onto the hyperplane ∑ixi=0\sum_i x_i=0 and then onto the same sphere, so its outputs live on a (d−2)(d-2)-dimensional sphere instead of a (d−1)(d-1)-dimensional one. In other words, layer norm is RMSNorm applied to the centred vector: removing the mean is the only difference.

d = 2: drag xRMSNorm → circle r = √d · LN → 2 pointsx (2.000, 0.800)LN(x) (1.000, −1.000)RMS(x) (1.313, 0.525)‖RMS(x)‖ = 1.414 ≈ √2d = 8: x = s·v + t, outputs vs those of v (outlines)input: mean 0.450 std 1.078 RMS 1.168LayerNorm Δ vs v: 0.000 (2 reductions: μ, σ²)RMSNorm Δ vs v: 0.000 (1 reduction: mean x²)scale only: both outputs unchangedRMS of RMSNorm output vs input RMS, ε = 1.0e−600.511e−51e−41e−31e−21e−11e01e1━ x/√(ms + ε) ━ x/(RMS + ε) · now 1.000┄ LN line x₁ + x₂ = 0
ε
fig.Left: in 2-D, RMSNorm sends a point radially to the circle of radius √2, while layer norm first removes the mean (moves along (1,1) to the line x₁ + x₂ = 0) and can only land on (1, −1) or (−1, 1). The default x = (2, 0.8) maps to (1.313, 0.525) and (1.000, −1.000). Middle: an 8-feature vector v with mean 0.45, std 1.078 and RMS 1.168 is transformed to s·v + t. Any scale leaves both outputs unchanged (Δ ≤ 0.001 down to s = 0.1). A shift t = 1 leaves layer norm unchanged but moves the RMSNorm output by up to 0.886, t = 2 by 1.278. Right: the RMS of the RMSNorm output against the input RMS. With ε = 1e−6 inside the root, an input of RMS 1.17e−3 (s = 0.001) comes out at 0.760, not 1; with ε added outside the root it comes out at 0.999. With bf16's machine epsilon 7.8e−3 inside the root, already s = 0.1 gives 0.798.

Is losing shift invariance a problem? In a pre-norm Transformer the input to each norm is the residual stream, and the mean over features is just one more direction in it. The next linear layer can learn to use or ignore it. Zhang and Sennrich's experiments, and every LLaMA-style model since, suggest the network does not need the norm to remove it.

Worked example

Take x=(1,2,3,4)x=(1,2,3,4), g=1g=\mathbf 1, ε=0\varepsilon=0. The mean of squares is (1+4+9+16)/4=7.5(1+4+9+16)/4=7.5, so RMS=7.5=2.7386\mathrm{RMS}=\sqrt{7.5}=2.7386. For layer norm, μ=2.5\mu=2.5 and σ=1.25=1.1180\sigma=\sqrt{1.25}=1.1180. Checked in Node:

inputRMSRMSNorm outputlayer norm output
(1, 2, 3, 4)2.739(0.365, 0.730, 1.095, 1.461)(−1.342, −0.447, 0.447, 1.342)
(2, 4, 6, 8)5.477(0.365, 0.730, 1.095, 1.461)(−1.342, −0.447, 0.447, 1.342)
(11, 12, 13, 14)12.550(0.877, 0.956, 1.036, 1.116)(−1.342, −0.447, 0.447, 1.342)
RMSNorm and layer norm of x, 2x and x + 10

Backward for x = (1, 2, 3, 4), upstream gradient ∂L/∂y = (1, 0, 0, 0)

  1. 1x⊤u=1x^\top u=1, so the component along xx is x x⊤ud RMS2=x⋅130=(0.0333, 0.0667, 0.1, 0.1333)x\,\frac{x^\top u}{d\,\mathrm{RMS}^2}=x\cdot\frac{1}{30}=(0.0333,\,0.0667,\,0.1,\,0.1333).
  2. 2uu minus that: (0.9667, −0.0667, −0.1, −0.1333)(0.9667,\,-0.0667,\,-0.1,\,-0.1333).
  3. 3Divide by RMS=2.7386\mathrm{RMS}=2.7386: ∂L/∂x=(0.3530, −0.0243, −0.0365, −0.0487)\partial L/\partial x=(0.3530,\,-0.0243,\,-0.0365,\,-0.0487), matching finite differences.
  4. 4Check: x⊤∂L/∂x=0.3530−0.0487−0.1095−0.1947≈0x^\top\partial L/\partial x=0.3530-0.0487-0.1095-0.1947\approx0. The entries do not sum to zero (0.2435): RMSNorm lets the gradient move the mean.
python
1import torch, torch.nn as nn
2
3d = 4096
4x = torch.randn(2, 16, d) # (batch, tokens, features)
5
6norm = nn.RMSNorm(d, eps=1e-6) # weight = g, initialized to ones; no bias
7y = norm(x) # eps=None (default) means torch.finfo(x.dtype).eps
8
9class LlamaRMSNorm(nn.Module):
10 """As in the LLaMA reference code: eps inside the root, statistics in float32."""
11 def __init__(self, dim, eps=1e-6):
12 super().__init__()
13 self.eps = eps
14 self.weight = nn.Parameter(torch.ones(dim))
15 def forward(self, x):
16 xf = x.float()
17 xf = xf * torch.rsqrt(xf.pow(2).mean(-1, keepdim=True) + self.eps)
18 return xf.type_as(x) * self.weight
19
20print(torch.allclose(y, LlamaRMSNorm(d)(x), atol=1e-5)) # True
21
22# Scale invariance holds, shift invariance does not
23print(torch.allclose(norm(3 * x), y, atol=1e-5)) # True
24print(torch.allclose(norm(x + 1.0), y, atol=1e-2)) # False
25
26# LayerNorm = RMSNorm of the centred vector
27ln = nn.LayerNorm(d, eps=1e-6, elementwise_affine=False)
28xc = x - x.mean(-1, keepdim=True)
29print(torch.allclose(ln(x), nn.RMSNorm(d, eps=1e-6)(xc), atol=1e-5)) # True
30
31# Pre-norm block (LLaMA): h = x + Attn(RMSNorm(x)); out = h + MLP(RMSNorm(h))
nn.RMSNorm (PyTorch ≥ 2.4), the LLaMA-style module, and the relation to LayerNorm

Where ε goes, and QK-norm

PyTorch, T5 and LLaMA add ε\varepsilon to the mean square, inside the root. It then matters once RMS2\mathrm{RMS}^2 is comparable to ε\varepsilon, i.e. for RMS≲ε=10−3\mathrm{RMS}\lesssim\sqrt\varepsilon=10^{-3} when ε=10−6\varepsilon=10^{-6}. Adding it to the RMS itself, x/(RMS+ε)x/(\mathrm{RMS}+\varepsilon), only matters for RMS≲ε\mathrm{RMS}\lesssim\varepsilon. The two conventions are not interchangeable when porting weights, and the figure shows how far apart they are for small inputs. Also note that nn.RMSNorm's default eps=None means torch.finfo(x.dtype).eps: 1.2×10−71.2\times10^{-7} in float32 but 9.8×10−49.8\times10^{-4} in float16 and 7.8×10−37.8\times10^{-3} in bfloat16. Set it explicitly.

RMSNorm is also applied inside attention. QK-norm normalizes each head's queries and keys before the dot product, which bounds the attention logits q⊤k/dhq^\top k/\sqrt{d_h} and prevents the logit growth that destabilizes large-scale training. Henry et al. (2020) proposed normalizing queries and keys; Dehghani et al. (2023) used layer norm on them to train the 22-billion-parameter ViT; Wortsman et al. (2023) showed it removes a whole class of instabilities at high learning rates. OLMo 2, Qwen3 and Gemma 3 use RMSNorm as their QK-norm.

Use when

  • +One reduction instead of two, no mean subtraction, no bias: simpler and somewhat faster
  • +Keeps the property that matters for optimization: invariance to the scale of inputs and incoming weights
  • +Independent of batch and sequence length, like layer norm: one token at a time
  • +The default of LLaMA-style LLMs, so kernels, configs and checkpoints all expect it

Avoid when

  • −Not invariant to a shared offset: a drifting mean in the residual stream passes through
  • −Sensitive to ε\varepsilon placement and value for small inputs; defaults differ by dtype
  • −Gain-parametrization differences (gg vs 1+w1+w) make weight porting error-prone
  • −Speed advantage over a fused layer norm is often small

Check yourself

A token vector x is replaced by x + 5·1 (5 added to every feature). Which output changes?

Check yourself

With ε = 1e−6 added inside the square root, below roughly which input RMS does RMSNorm stop producing unit-RMS outputs?

Remember

  • ✓y=g⊙x/1d∑xi2+εy=g\odot x/\sqrt{\tfrac1d\sum x_i^2+\varepsilon}: layer norm without the mean subtraction and without β\beta.
  • ✓Keeps scale invariance (x↦αxx\mapsto\alpha x) and gradients orthogonal to xx; loses shift invariance.
  • ✓LN(x)=RMSNorm(x−μ)\mathrm{LN}(x)=\mathrm{RMSNorm}(x-\mu): the centring is the only difference.
  • ✓Zhang & Sennrich (2019); used in T5, Gopher, LLaMA and most later LLMs, as pre-norm and as QK-norm.
  • ✓Practicalities: nn.RMSNorm (PyTorch ≥ 2.4), set eps explicitly, compute in float32, check the gain convention.
← back to Deep Learning roadmap