# Rotary positional embeddings (RoPE)

> RoPE (Su et al., 2021) rotates each 2-D pair of a query and a key by its position times a fixed frequency θᵢ = 10000^(−2i/d). The rotations cancel in the dot product, so the attention score depends on content and the offset m − n only.

Source: https://www.mlfoundry.in/learn/rotary-positional-embeddings-rope  
Section: Deep Learning › Attention and Transformer Architectures  
Level: advanced · 15 min read · updated 2026-10-12

**Rotary positional embedding (RoPE):** encodes position by **rotating** queries and keys instead of adding a vector to the input. The head vector of size $d$ is split into $d/2$ two-dimensional pairs. At position $m$, pair $i$ of the query (and of the key) is rotated by the angle $m\theta_i$, with $\theta_i=10000^{-2i/d}$. Because $R_m^\top R_n=R_{n-m}$, the score $\langle R_mq,\,R_nk\rangle=q^\top R_{n-m}k$ depends only on the two vectors and the offset. Values are not rotated, and there are no parameters.

Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen and Yunfeng Liu introduced RoPE in "RoFormer: Enhanced Transformer with Rotary Position Embedding" (arXiv 2021; Neurocomputing 2024), after Su first described it on his blog. They asked for a function $f(x,m)$ such that $\langle f(q,m),\,f(k,n)\rangle=g(q,k,m-n)$ for some $g$. Additive schemes (see **Positional encoding**) only give that for one of the four terms of the expanded score. Rotation gives it exactly. EleutherAI's GPT-J and GPT-NeoX adopted it in 2021–22, followed by PaLM and LLaMA, and it has since become the default for decoder-only language models.

## The rotation

$$
f(x,m)=R^d_{\Theta,m}\,x,\qquad R^d_{\Theta,m}=\begin{pmatrix}\cos m\theta_0&-\sin m\theta_0&&&\\ \sin m\theta_0&\cos m\theta_0&&&\\ &&\ddots&&\\ &&&\cos m\theta_{d/2-1}&-\sin m\theta_{d/2-1}\\ &&&\sin m\theta_{d/2-1}&\cos m\theta_{d/2-1}\end{pmatrix},\qquad \theta_i=10000^{-2i/d}
$$

- $x$: a query $q_m=W_Qx_m$ or key $k_n=W_Kx_n$ of one head, after projection
- $R^d_{\Theta,m}$: block-diagonal and orthogonal: it changes no norm, only angles. Never materialised; see the elementwise form below
- $\theta_i$: frequency of pair $i$: pair 0 turns 1 rad per position (wavelength $2\pi$), the last pair $10000^{-(d-2)/d}$ rad (wavelength about $2\pi\cdot10^4$). The same geometric ladder as sinusoidal encodings
- $10000$: the **base**; later models raise it (Code Llama $10^6$, Llama 3 $5\cdot10^5$) to slow the low frequencies down for long contexts

$$
\langle R_mq,\;R_nk\rangle=q^\top R_m^\top R_n\,k=q^\top R_{n-m}\,k=\mathrm{Re}\sum_{i=0}^{d/2-1}q_{[i]}\,\overline{k_{[i]}}\;e^{\,\mathrm i(m-n)\theta_i}
$$

- $q_{[i]}$: pair $i$ read as a complex number, $q_{2i}+\mathrm i\,q_{2i+1}$; rotation by $m\theta_i$ is multiplication by $e^{\mathrm im\theta_i}$
- $R_m^\top R_n=R_{n-m}$: rotations by angles compose by adding angles, and the transpose rotates backwards
- $(m-n)$: the only place position enters. Shifting both positions by any $t$ leaves the score unchanged; swapping $m$ and $n$ changes it (direction is encoded)

In code the matrix product becomes two elementwise products: $R_mx=x\odot\cos(m\Theta)+\mathrm{rot}(x)\odot\sin(m\Theta)$, where $\mathrm{rot}$ maps each pair $(x_a,x_b)$ to $(-x_b,x_a)$. The cost is $O(Td)$ per head, negligible next to attention. The rotation is applied in every layer to $Q$ and $K$ just before the scores, so nothing positional is ever written into the residual stream. With a KV cache, keys are rotated once at their absolute position and stored; the relative property guarantees that the score with any later query is still a function of the offset.

## Long-term decay

Su et al. also argue that the score tends to shrink as $|m-n|$ grows. Write $h_i=q_{[i]}\overline{k_{[i]}}$ and $S_j=\sum_{i<j}e^{\mathrm i(m-n)\theta_i}$. Summation by parts (an Abel transformation) gives $\big|\sum_ih_ie^{\mathrm i(m-n)\theta_i}\big|\le\big(\max_i|h_{i+1}-h_i|\big)\sum_j|S_j|$. The factor $\frac{1}{d/2}\sum_j|S_j|$ decreases with the distance on average, because the phases $e^{\mathrm i(m-n)\theta_i}$ spread around the circle and cancel. It is a bound, not a law: the curve oscillates, and a trained model can still attend sharply to a distant token whose content matches. The figure plots this factor next to the actual score for $q=k=\mathbf 1$.

> Figure: One head of size d = 64 (32 pairs) with fixed random q and k. The dial shows pair i rotated by its position: at the defaults m = 9, n = 4, pair 0 (θ₀ = 1 rad per position) is 5 rad apart, and that pair contributes −0.755 to ⟨Rₘq, Rₙk⟩. The full logit ⟨Rₘq, Rₙk⟩/√d is −0.185. Moving both positions by 100 gives −0.185 again, while swapping m and n gives 0.550. Lower left: the score for q = k = 1, normalised to (1/32)Σcos((m − n)θᵢ), and Su et al.'s decay factor, both 1 at distance 0, on a log axis. The score is 0.966 at distance 1, 0.658 at 10, 0.559 at 100 and 0.020 at 2000. Lower right: how each extension method rescales θᵢ to run at s·L with L = 2048. Unmodified at s = 4, the 11 slowest pairs (21–31) never complete a turn within 2048 tokens and would see angles they were never trained on (red). Position interpolation divides every θᵢ by s: no pair is extrapolated, but pair 0 now steps only 0.25 rad per token. NTK-aware scaling raises the base to 10000·s^(64/62), which leaves pair 0 at 1 rad and divides only the slowest pair by 4; pairs 21–30 still extrapolate slightly. YaRN keeps pairs 0–8 (more than 32 turns in L) exactly, fully interpolates pairs 21–31 (less than one turn), ramps between, and scales the logits by (0.1 ln s + 1)² = 1.297.

## Worked example: one pair

Take a single pair with $\theta=0.5$, query $q=(1,2)$ and key $k=(3,1)$. Without rotation $q^\top k=5$. As complex numbers, $q\bar k=(1+2\mathrm i)(3-\mathrm i)=5+5\mathrm i$, so the RoPE score is $\mathrm{Re}\big[(5+5\mathrm i)e^{\mathrm i(m-n)\theta}\big]=5\cos((m-n)\theta)-5\sin((m-n)\theta)$.

Rotated vectors and scores (computed in Node)

| m | n | R_m q | R_n k | ⟨R_m q, R_n k⟩ |
| --- | --- | --- | --- | --- |
| 3 | 1 | (−1.9243, 1.1390) | (2.1533, 2.3159) | −1.5058 |
| 7 | 5 | (−0.2349, −2.2237) | (−3.0019, 0.9943) | −1.5058 |
| 1 | 3 | (−0.0813, 2.2346) | (−0.7853, 3.0632) | 6.9089 |
| 3 | 3 | (−1.9243, 1.1390) | (−0.7853, 3.0632) | 5.0000 |

**Reading the table**

1. $m=3$: rotate $q$ by $1.5$ rad: $(1\cos1.5-2\sin1.5,\ 1\sin1.5+2\cos1.5)=(-1.9243,\ 1.1390)$. $n=1$: rotate $k$ by $0.5$ rad to $(2.1533,\ 2.3159)$.
2. Their dot product is $-1.5058=5\cos1-5\sin1=2.7015-4.2074$, which depends only on $(m-n)\theta=1$.
3. Shifting both positions by 4 changes both vectors completely but leaves the score at $-1.5058$.
4. Swapping to $m=1$, $n=3$ gives $5\cos(-1)-5\sin(-1)=6.9089$: the sign of the offset matters. At $m=n$ the rotations cancel and the score is the raw $q^\top k=5$.

## Extending the context

A model trained on length $L$ has only seen angles $m\theta_i$ with $m<L$. For high-frequency pairs this is no problem: they have turned many times, and every angle on the circle has been seen. Low-frequency pairs with $L\theta_i<2\pi$ have only covered part of the circle, so at $m>L$ they produce angles never seen in training. Perplexity of a plain RoPE model rises sharply just beyond $L$. Three fixes are widely used. All of them change $\theta_i$ or $m$, then usually fine-tune briefly.

$$
\text{PI: }\ \theta'_i=\frac{\theta_i}{s}\qquad\quad \text{NTK-aware: }\ \theta'_i=\big(b\,s^{d/(d-2)}\big)^{-2i/d}\qquad\quad \text{YaRN: }\ \theta'_i=\Big(1-\gamma(r_i)\Big)\frac{\theta_i}{s}+\gamma(r_i)\,\theta_i,\quad r_i=\frac{L\,\theta_i}{2\pi}
$$

- $s$: extension factor $L'/L$ (e.g. 2048 → 8192 gives $s=4$)
- $\text{PI}$: position interpolation (Chen, Wong, Chen and Tian, 2023): equivalent to using position $m/s$. Every angle stays inside the trained range, but neighbouring tokens become $s$ times closer in every pair
- $b\,s^{d/(d-2)}$: NTK-aware scaling (posted by "bloc97", 2023): a larger base. Pair 0 is unchanged and the slowest pair is divided by exactly $s$, so high frequencies keep their resolution
- $\gamma(r)$: YaRN's ramp: 0 if a pair makes fewer than $\alpha=1$ turns within $L$ (interpolate), 1 if more than $\beta=32$ (keep), linear in between. YaRN also multiplies the logits by $(0.1\ln s+1)^2$

Chen et al. extended LLaMA models from 2048 to 32768 tokens with position interpolation and about 1000 fine-tuning steps; direct fine-tuning without interpolation adapted far more slowly. NTK-aware scaling worked reasonably even without fine-tuning, which made it popular in open-source inference tools. YaRN (Peng, Quesnelle, Fan and Shippole, ICLR 2024) combined the "by parts" frequency split with the temperature and released LLaMA 2 models with 64k and 128k context. Raising the base before or during long-context training, as Code Llama and Llama 3 do, is the same idea applied from the start.

**Settings in practice**

- **RoFormer, LLaMA 1/2:** base 10000, rotation on every query/key dimension; LLaMA trained at 2048 tokens, Llama 2 at 4096
- **GPT-J:** rotates only 64 of each head's 256 dimensions; the rest carry no position
- **Code Llama:** base raised to $10^6$ for 16k-token training sequences
- **Llama 3:** base $5\cdot10^5$
- **Position interpolation:** LLaMA 7B–65B extended to 32768 tokens with about 1000 fine-tuning steps
- **YaRN:** $\alpha=1$, $\beta=32$ for LLaMA; logits scaled by $(0.1\ln s+1)^2$

```python
import torch

def rope_cache(seq_len, head_dim, base=10000.0, pi_scale=1.0):
    theta = base ** (-torch.arange(0, head_dim, 2).float() / head_dim)  # (head_dim/2,)
    pos = torch.arange(seq_len).float() / pi_scale                      # PI: positions m / s
    ang = torch.outer(pos, theta)                                         # (T, head_dim/2)
    ang = torch.cat([ang, ang], dim=-1)                                   # (T, head_dim)
    return ang.cos(), ang.sin()

def rotate_half(x):                       # pairs are (x[j], x[j + head_dim/2])
    x1, x2 = x.chunk(2, dim=-1)
    return torch.cat([-x2, x1], dim=-1)

def apply_rope(x, cos, sin):              # x: (..., T, head_dim)
    return x * cos + rotate_half(x) * sin

B, H, T, Dh = 2, 8, 16, 64
q, k = torch.randn(B, H, T, Dh), torch.randn(B, H, T, Dh)
cos, sin = rope_cache(T, Dh)              # (16, 64) each, broadcast over B and H
qr, kr = apply_rope(q, cos, sin), apply_rope(k, cos, sin)
scores = qr @ kr.transpose(-1, -2) / Dh ** 0.5
print(scores.shape)                       # torch.Size([2, 8, 16, 16])

# the score depends only on m - n
def score(qv, kv, m, n, **kw):
    c, s = rope_cache(max(m, n) + 1, Dh, **kw)
    return (apply_rope(qv, c[m], s[m]) * apply_rope(kv, c[n], s[n])).sum()

qv, kv = torch.randn(Dh), torch.randn(Dh)
print(torch.allclose(score(qv, kv, 3, 1), score(qv, kv, 103, 101), atol=1e-3))  # True
print(torch.allclose(score(qv, kv, 3, 1), score(qv, kv, 1, 3), atol=1e-3))      # False (almost surely)

# context extension by s = 4
s = 4
cos_pi, sin_pi = rope_cache(4 * 2048, Dh, pi_scale=s)                        # position interpolation
cos_ntk, sin_ntk = rope_cache(4 * 2048, Dh, base=10000.0 * s ** (Dh / (Dh - 2)))  # NTK-aware
print(cos_pi.shape)                       # torch.Size([8192, 64])
```

RoPE in the rotate-half layout used by GPT-NeoX and Hugging Face LLaMA, with position interpolation and NTK-aware base

> **Interleaved versus rotate-half:** The paper pairs dimensions $(2i,2i+1)$. GPT-NeoX and the Hugging Face LLaMA code pair $(j,\,j+d/2)$ and use `rotate_half`. Both are valid RoPE, but with different pairings. Converting Meta's original LLaMA checkpoints to the Hugging Face layout therefore permutes the rows of $W_Q$ and $W_K$. Mixing the two layouts without that permutation gives a model that loads without error and produces nonsense.

> **Why rotate instead of add:** An added encoding makes the score a sum of content–content, content–position, position–content and position–position terms, and only the last is relative. Rotation is multiplicative, and $R_m^\top R_n=R_{n-m}$ makes the **whole** score relative while leaving norms untouched. It also commutes with the KV cache and costs no parameters. Su et al. further note that it works with kernelised **Linear attention**, where an additive bias such as **ALiBi** has no place to go.

> **Which dimensions carry long-range position:** With base 10000 and $d=128$, only the slowest pairs have wavelengths longer than a few thousand tokens. These pairs are the ones that break first beyond the training length, and the ones every extension method rescales most. Partial RoPE (GPT-J's 64 of 256) and larger bases are both ways of controlling how much of each head is spent on position.

**Strengths**

- The score is an exact function of content and $m-n$; shift-invariant, direction-aware
- No parameters, applied to queries and keys only, nothing added to the residual stream
- Compatible with KV caching, FlashAttention-style kernels and linear attention
- Context can be extended later by rescaling frequencies (PI, NTK-aware, YaRN) with little fine-tuning

**Limitations**

- Extrapolates poorly beyond the training length without rescaling: slow pairs meet unseen angles
- Position interpolation blurs nearby positions; methods trade resolution against range
- Two incompatible layouts (interleaved, rotate-half) in circulation
- Long-term decay is only an average bound, not a guaranteed preference for nearby tokens

**Question:** With RoPE, a query at position 10 and a key at position 4 give logit 0.8. Which pair of positions gives the same logit for the same q and k vectors?

- A. Query 106, key 100
- B. Query 4, key 10
- C. Query 10, key 4 with the base doubled
- D. Query 20, key 8

**Answer:** A. Query 106, key 100 — The score is $q^\top R_{n-m}k$, which depends only on the offset. Query 106 and key 100 have the same offset of 6. Swapping the positions reverses the offset, which changes the score. Doubling the positions doubles the offset to 12, and changing the base changes every $\theta_i$.

**Question:** A model with head size 64 was trained with RoPE (base 10000) on 2048 tokens. You want 8192 tokens. What does NTK-aware scaling do that position interpolation does not?

- A. It keeps the highest-frequency pairs almost unchanged and slows only the low-frequency pairs, the slowest by the full factor 4
- B. It divides every frequency by 4
- C. It removes RoPE from the slowest pairs
- D. It adds a learned bias for positions beyond 2048

**Answer:** A. It keeps the highest-frequency pairs almost unchanged and slows only the low-frequency pairs, the slowest by the full factor 4 — NTK-aware scaling raises the base to $10000\cdot4^{64/62}\approx41{,}829$. Then $\theta'_i/\theta_i=4^{-2i/62}$, which is 1 for pair 0 and $1/4$ for pair 31. Position interpolation divides every $\theta_i$ by 4, so neighbouring tokens also become 4 times harder to tell apart in the fastest pairs.

**Key points**

- RoPE: rotate pair $i$ of $q$ and $k$ at position $m$ by $m\theta_i$, $\theta_i=10000^{-2i/d}$; values are not rotated.
- $\langle R_mq,R_nk\rangle=q^\top R_{n-m}k=\mathrm{Re}\sum_iq_{[i]}\overline{k_{[i]}}e^{\mathrm i(m-n)\theta_i}$: the score depends only on content and $m-n$.
- Implemented as $x\odot\cos+\mathrm{rot}(x)\odot\sin$ in every layer; no parameters, cache-friendly.
- Long-term decay: Su et al. bound the score by $\max_i|h_{i+1}-h_i|\sum_j|S_j|$, which shrinks with distance on average.
- Context extension: PI uses $m/s$; NTK-aware uses base $b\,s^{d/(d-2)}$; YaRN interpolates only slow pairs and scales logits by $(0.1\ln s+1)^2$.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
