# RMS normalization (RMSNorm)

> Layer norm without the mean: divide each feature vector by its root mean square and multiply by a learned gain. It keeps invariance to rescaling, drops invariance to shifts, needs one reduction instead of two, and is the normalizer of T5, Gopher, LLaMA and most recent LLMs.

Source: https://www.mlfoundry.in/learn/rms-normalization-rmsnorm  
Section: Deep Learning › Regularization & Normalization  
Level: core · 11 min read · updated 2026-10-11

**RMSNorm:** (root mean square layer normalization) rescales a feature vector $x\in\mathbb R^d$ to unit root mean square, $\bar x=x/\mathrm{RMS}(x)$ with $\mathrm{RMS}(x)=\sqrt{\tfrac1d\sum_i x_i^2}$, then multiplies by a learned per-feature gain $g$. Unlike **Layer normalization**, it does not subtract the mean and has no bias.

Layer normalization does two things to each token's vector: it re-centres it (subtract the mean) and re-scales it (divide by the standard deviation). **Zhang and Sennrich (2019)**, *Root Mean Square Layer Normalization* (NeurIPS), asked which of the two matters. Their hypothesis was that the benefit comes from **re-scaling invariance**: the output does not change when the input or the incoming weights are multiplied by a constant. Re-centring, they argued, is dispensable. Dropping it removes the mean, the bias and one reduction over the features. Across their RNN and Transformer experiments, RMSNorm matched layer norm's quality while cutting running time by 7% to 64%, depending on model and implementation.

Large language models adopted it quickly. T5 (Raffel et al., 2020) used a layer norm that "only rescales" and has no additive bias, which is RMSNorm. Gopher (Rae et al., 2021) and LLaMA (Touvron et al., 2023) use RMSNorm in a pre-norm layout, and most open LLMs since then follow LLaMA.

## Formula and gradient

$$
\mathrm{RMS}(x)=\sqrt{\frac1d\sum_{i=1}^d x_i^2+\varepsilon},\qquad y_i=\frac{x_i}{\mathrm{RMS}(x)}\,g_i\qquad\text{vs. LayerNorm: } y_i=\frac{x_i-\mu}{\sqrt{\sigma^2+\varepsilon}}\,\gamma_i+\beta_i
$$

- $x\in\mathbb R^d$: one token's feature vector (normalization is over the last dimension)
- $g$: learned gain, one per feature, initialized to 1 (`weight` in PyTorch)
- $\varepsilon$: added **inside** the square root, as in PyTorch, T5 and LLaMA; LLaMA uses $10^{-6}$
- $\mu,\ \sigma^2,\ \beta$: layer norm's mean, variance and bias, all absent from RMSNorm

With $\bar x=x/\mathrm{RMS}(x)$ and $\varepsilon=0$, the Jacobian is $\partial\bar x/\partial x=\frac{1}{\mathrm{RMS}}\big(I-\frac{1}{d}\bar x\bar x^\top\big)$. So for an upstream gradient $u=g\odot\partial L/\partial y$:

$$
\frac{\partial L}{\partial x}=\frac{1}{\mathrm{RMS}(x)}\Big(u-\bar x\,\frac{\bar x^\top u}{d}\Big),\qquad \frac{\partial L}{\partial g_i}=\sum_{\text{tokens}}\frac{\partial L}{\partial y_i}\,\bar x_i
$$

- $\bar x\,\tfrac{\bar x^\top u}{d}$: the component of $u$ along $x$, removed because rescaling $x$ cannot change the output
- $1/\mathrm{RMS}$: larger inputs get proportionally smaller gradients

Two consequences follow. The input gradient is orthogonal to $x$: $x^\top\partial L/\partial x=0$. And it scales like $1/\mathrm{RMS}(x)$: if the weights $W$ feeding the norm grow by a factor $\alpha$, the output is unchanged and the gradient on $W$ shrinks by $1/\alpha$. Zhang and Sennrich call this an implicit learning-rate adaptation. Layer norm has the same two properties and, in addition, a gradient that sums to zero (shift invariance). RMSNorm gives that up.

**At a glance**

- **Statistics over:** the $d$ features of one token: a single sum of squares
- **Invariant to:** $x\mapsto\alpha x$ ($\alpha>0$) and to scaling the incoming weights
- **Not invariant to:** $x\mapsto x+t\mathbf 1$: a shared offset changes the output
- **Parameters:** $d$ (gain only), versus $2d$ for layer norm
- **Relation to LN:** $\mathrm{LN}(x)=\mathrm{RMSNorm}(x-\mu\mathbf 1)$ with the same $\varepsilon$ (and $\beta=0$)
- **Used in:** T5, Gopher, Chinchilla, LLaMA and its descendants (Mistral, Qwen, Gemma …), as pre-norm and as QK-norm
- **Paper:** Zhang & Sennrich (2019), NeurIPS

## What is kept and what is lost

Geometrically, RMSNorm maps every nonzero $x$ radially onto the sphere of radius $\sqrt d$ (then $g$ stretches the axes). Layer norm first projects $x$ onto the hyperplane $\sum_i x_i=0$ and then onto the same sphere, so its outputs live on a $(d-2)$-dimensional sphere instead of a $(d-1)$-dimensional one. In other words, layer norm is RMSNorm applied to the centred vector: removing the mean is the only difference.

> Figure: Left: in 2-D, RMSNorm sends a point radially to the circle of radius √2, while layer norm first removes the mean (moves along (1,1) to the line x₁ + x₂ = 0) and can only land on (1, −1) or (−1, 1). The default x = (2, 0.8) maps to (1.313, 0.525) and (1.000, −1.000). Middle: an 8-feature vector v with mean 0.45, std 1.078 and RMS 1.168 is transformed to s·v + t. Any scale leaves both outputs unchanged (Δ ≤ 0.001 down to s = 0.1). A shift t = 1 leaves layer norm unchanged but moves the RMSNorm output by up to 0.886, t = 2 by 1.278. Right: the RMS of the RMSNorm output against the input RMS. With ε = 1e−6 inside the root, an input of RMS 1.17e−3 (s = 0.001) comes out at 0.760, not 1; with ε added outside the root it comes out at 0.999. With bf16's machine epsilon 7.8e−3 inside the root, already s = 0.1 gives 0.798.

Is losing shift invariance a problem? In a pre-norm Transformer the input to each norm is the residual stream, and the mean over features is just one more direction in it. The next linear layer can learn to use or ignore it. Zhang and Sennrich's experiments, and every LLaMA-style model since, suggest the network does not need the norm to remove it.

## Worked example

Take $x=(1,2,3,4)$, $g=\mathbf 1$, $\varepsilon=0$. The mean of squares is $(1+4+9+16)/4=7.5$, so $\mathrm{RMS}=\sqrt{7.5}=2.7386$. For layer norm, $\mu=2.5$ and $\sigma=\sqrt{1.25}=1.1180$. Checked in Node:

RMSNorm and layer norm of x, 2x and x + 10

| input | RMS | RMSNorm output | layer norm output |
| --- | --- | --- | --- |
| (1, 2, 3, 4) | 2.739 | (0.365, 0.730, 1.095, 1.461) | (−1.342, −0.447, 0.447, 1.342) |
| (2, 4, 6, 8) | 5.477 | (0.365, 0.730, 1.095, 1.461) | (−1.342, −0.447, 0.447, 1.342) |
| (11, 12, 13, 14) | 12.550 | (0.877, 0.956, 1.036, 1.116) | (−1.342, −0.447, 0.447, 1.342) |

**Backward for x = (1, 2, 3, 4), upstream gradient ∂L/∂y = (1, 0, 0, 0)**

1. $x^\top u=1$, so the component along $x$ is $x\,\frac{x^\top u}{d\,\mathrm{RMS}^2}=x\cdot\frac{1}{30}=(0.0333,\,0.0667,\,0.1,\,0.1333)$.
2. $u$ minus that: $(0.9667,\,-0.0667,\,-0.1,\,-0.1333)$.
3. Divide by $\mathrm{RMS}=2.7386$: $\partial L/\partial x=(0.3530,\,-0.0243,\,-0.0365,\,-0.0487)$, matching finite differences.
4. Check: $x^\top\partial L/\partial x=0.3530-0.0487-0.1095-0.1947\approx0$. The entries do **not** sum to zero (0.2435): RMSNorm lets the gradient move the mean.

```python
import torch, torch.nn as nn

d = 4096
x = torch.randn(2, 16, d)                      # (batch, tokens, features)

norm = nn.RMSNorm(d, eps=1e-6)                 # weight = g, initialized to ones; no bias
y = norm(x)                                    # eps=None (default) means torch.finfo(x.dtype).eps

class LlamaRMSNorm(nn.Module):
    """As in the LLaMA reference code: eps inside the root, statistics in float32."""
    def __init__(self, dim, eps=1e-6):
        super().__init__()
        self.eps = eps
        self.weight = nn.Parameter(torch.ones(dim))
    def forward(self, x):
        xf = x.float()
        xf = xf * torch.rsqrt(xf.pow(2).mean(-1, keepdim=True) + self.eps)
        return xf.type_as(x) * self.weight

print(torch.allclose(y, LlamaRMSNorm(d)(x), atol=1e-5))                    # True

# Scale invariance holds, shift invariance does not
print(torch.allclose(norm(3 * x), y, atol=1e-5))                           # True
print(torch.allclose(norm(x + 1.0), y, atol=1e-2))                         # False

# LayerNorm = RMSNorm of the centred vector
ln = nn.LayerNorm(d, eps=1e-6, elementwise_affine=False)
xc = x - x.mean(-1, keepdim=True)
print(torch.allclose(ln(x), nn.RMSNorm(d, eps=1e-6)(xc), atol=1e-5))     # True

# Pre-norm block (LLaMA): h = x + Attn(RMSNorm(x)); out = h + MLP(RMSNorm(h))
```

nn.RMSNorm (PyTorch ≥ 2.4), the LLaMA-style module, and the relation to LayerNorm

## Where ε goes, and QK-norm

PyTorch, T5 and LLaMA add $\varepsilon$ to the mean square, inside the root. It then matters once $\mathrm{RMS}^2$ is comparable to $\varepsilon$, i.e. for $\mathrm{RMS}\lesssim\sqrt\varepsilon=10^{-3}$ when $\varepsilon=10^{-6}$. Adding it to the RMS itself, $x/(\mathrm{RMS}+\varepsilon)$, only matters for $\mathrm{RMS}\lesssim\varepsilon$. The two conventions are not interchangeable when porting weights, and the figure shows how far apart they are for small inputs. Also note that `nn.RMSNorm`'s default `eps=None` means `torch.finfo(x.dtype).eps`: $1.2\times10^{-7}$ in float32 but $9.8\times10^{-4}$ in float16 and $7.8\times10^{-3}$ in bfloat16. Set it explicitly.

RMSNorm is also applied inside attention. **QK-norm** normalizes each head's queries and keys before the dot product, which bounds the attention logits $q^\top k/\sqrt{d_h}$ and prevents the logit growth that destabilizes large-scale training. Henry et al. (2020) proposed normalizing queries and keys; Dehghani et al. (2023) used layer norm on them to train the 22-billion-parameter ViT; Wortsman et al. (2023) showed it removes a whole class of instabilities at high learning rates. OLMo 2, Qwen3 and Gemma 3 use RMSNorm as their QK-norm.

> **Compute the statistic in float32:** In float16 the squares overflow easily: a single entry of 256 squares to 65536, above the float16 maximum of 65504, and in bfloat16 the mean of squares loses precision. Reference implementations cast to float32 for the normalization and cast back afterwards, as in the code above. Fused kernels do the same internally.

> **Gain conventions differ:** Most models store $g$ directly, initialized to 1. Some, such as Gemma, store $w$ and multiply by $(1+w)$ with $w$ initialized to 0. Loading one convention into the other silently shifts every gain by 1. Also, weight decay on $g$ pulls it toward 0 (or toward 1 in the $1+w$ form); most LLM recipes exclude norm gains from weight decay (see **AdamW**).

> **Why the saving is real but modest:** Layer norm needs the mean before it can compute the variance (or a one-pass formula that is less stable), then a subtraction; RMSNorm needs one sum of squares. The normalization is memory-bound, so the gain comes from fewer passes over the data rather than fewer FLOPs, and a well-fused layer norm narrows the gap. The 7% to 64% of Zhang & Sennrich were measured on their own implementations; the main reason LLMs use RMSNorm today is that it is simpler and works as well.

> **Partial RMSNorm:** Zhang and Sennrich also proposed pRMSNorm, which estimates the RMS from only the first $p\%$ of the features (e.g. 6.25%). It worked in their experiments, but the extra saving is small next to the matrix multiplies, and it is rarely used.

**Strengths**

- One reduction instead of two, no mean subtraction, no bias: simpler and somewhat faster
- Keeps the property that matters for optimization: invariance to the scale of inputs and incoming weights
- Independent of batch and sequence length, like layer norm: one token at a time
- The default of LLaMA-style LLMs, so kernels, configs and checkpoints all expect it

**Limitations**

- Not invariant to a shared offset: a drifting mean in the residual stream passes through
- Sensitive to $\varepsilon$ placement and value for small inputs; defaults differ by dtype
- Gain-parametrization differences ($g$ vs $1+w$) make weight porting error-prone
- Speed advantage over a fused layer norm is often small

**Question:** A token vector x is replaced by x + 5·1 (5 added to every feature). Which output changes?

- A. Only the RMSNorm output
- B. Only the layer norm output
- C. Both
- D. Neither

**Answer:** A. Only the RMSNorm output — Layer norm subtracts the mean, so a shared offset cancels exactly. RMSNorm divides by $\sqrt{\tfrac1d\sum x_i^2}$, which grows with the offset, and keeps the offset in the numerator: in the worked example, $x+10$ turns $(0.365,\dots,1.461)$ into $(0.877,\dots,1.116)$.

**Question:** With ε = 1e−6 added inside the square root, below roughly which input RMS does RMSNorm stop producing unit-RMS outputs?

- A. About 1e−3
- B. About 1e−6
- C. About 1e−12
- D. Never: ε only prevents division by zero

**Answer:** A. About 1e−3 — The output RMS is $r/\sqrt{r^2+\varepsilon}$, which departs from 1 once $r^2\approx\varepsilon$, i.e. $r\approx\sqrt{10^{-6}}=10^{-3}$. The figure gives 0.760 at $r=1.17\times10^{-3}$. With $\varepsilon$ outside the root the threshold would be $r\approx10^{-6}$.

**Key points**

- $y=g\odot x/\sqrt{\tfrac1d\sum x_i^2+\varepsilon}$: layer norm without the mean subtraction and without $\beta$.
- Keeps scale invariance ($x\mapsto\alpha x$) and gradients orthogonal to $x$; loses shift invariance.
- $\mathrm{LN}(x)=\mathrm{RMSNorm}(x-\mu)$: the centring is the only difference.
- Zhang & Sennrich (2019); used in T5, Gopher, LLaMA and most later LLMs, as pre-norm and as QK-norm.
- Practicalities: `nn.RMSNorm` (PyTorch ≥ 2.4), set `eps` explicitly, compute in float32, check the gain convention.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
