Definition
Rotary positional embedding (RoPE) encodes position by rotating queries and keys instead of adding a vector to the input. The head vector of size is split into two-dimensional pairs. At position , pair of the query (and of the key) is rotated by the angle , with . Because , the score depends only on the two vectors and the offset. Values are not rotated, and there are no parameters.
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen and Yunfeng Liu introduced RoPE in "RoFormer: Enhanced Transformer with Rotary Position Embedding" (arXiv 2021; Neurocomputing 2024), after Su first described it on his blog. They asked for a function such that for some . Additive schemes (see Positional encoding) only give that for one of the four terms of the expanded score. Rotation gives it exactly. EleutherAI's GPT-J and GPT-NeoX adopted it in 2021–22, followed by PaLM and LLaMA, and it has since become the default for decoder-only language models.
The rotation
- a query or key of one head, after projection
- block-diagonal and orthogonal: it changes no norm, only angles. Never materialised; see the elementwise form below
- frequency of pair : pair 0 turns 1 rad per position (wavelength ), the last pair rad (wavelength about ). The same geometric ladder as sinusoidal encodings
- the base; later models raise it (Code Llama , Llama 3 ) to slow the low frequencies down for long contexts
- pair read as a complex number, ; rotation by is multiplication by
- rotations by angles compose by adding angles, and the transpose rotates backwards
- the only place position enters. Shifting both positions by any leaves the score unchanged; swapping and changes it (direction is encoded)
In code the matrix product becomes two elementwise products: , where maps each pair to . The cost is per head, negligible next to attention. The rotation is applied in every layer to and just before the scores, so nothing positional is ever written into the residual stream. With a KV cache, keys are rotated once at their absolute position and stored; the relative property guarantees that the score with any later query is still a function of the offset.
Long-term decay
Su et al. also argue that the score tends to shrink as grows. Write and . Summation by parts (an Abel transformation) gives . The factor decreases with the distance on average, because the phases spread around the circle and cancel. It is a bound, not a law: the curve oscillates, and a trained model can still attend sharply to a distant token whose content matches. The figure plots this factor next to the actual score for .
Worked example: one pair
Take a single pair with , query and key . Without rotation . As complex numbers, , so the RoPE score is .
| m | n | R_m q | R_n k | ⟨R_m q, R_n k⟩ |
|---|---|---|---|---|
| 3 | 1 | (−1.9243, 1.1390) | (2.1533, 2.3159) | −1.5058 |
| 7 | 5 | (−0.2349, −2.2237) | (−3.0019, 0.9943) | −1.5058 |
| 1 | 3 | (−0.0813, 2.2346) | (−0.7853, 3.0632) | 6.9089 |
| 3 | 3 | (−1.9243, 1.1390) | (−0.7853, 3.0632) | 5.0000 |
Reading the table
- 1: rotate by rad: . : rotate by rad to .
- 2Their dot product is , which depends only on .
- 3Shifting both positions by 4 changes both vectors completely but leaves the score at .
- 4Swapping to , gives : the sign of the offset matters. At the rotations cancel and the score is the raw .
Extending the context
A model trained on length has only seen angles with . For high-frequency pairs this is no problem: they have turned many times, and every angle on the circle has been seen. Low-frequency pairs with have only covered part of the circle, so at they produce angles never seen in training. Perplexity of a plain RoPE model rises sharply just beyond . Three fixes are widely used. All of them change or , then usually fine-tune briefly.
- extension factor (e.g. 2048 → 8192 gives )
- position interpolation (Chen, Wong, Chen and Tian, 2023): equivalent to using position . Every angle stays inside the trained range, but neighbouring tokens become times closer in every pair
- NTK-aware scaling (posted by "bloc97", 2023): a larger base. Pair 0 is unchanged and the slowest pair is divided by exactly , so high frequencies keep their resolution
- YaRN's ramp: 0 if a pair makes fewer than turns within (interpolate), 1 if more than (keep), linear in between. YaRN also multiplies the logits by
Chen et al. extended LLaMA models from 2048 to 32768 tokens with position interpolation and about 1000 fine-tuning steps; direct fine-tuning without interpolation adapted far more slowly. NTK-aware scaling worked reasonably even without fine-tuning, which made it popular in open-source inference tools. YaRN (Peng, Quesnelle, Fan and Shippole, ICLR 2024) combined the "by parts" frequency split with the temperature and released LLaMA 2 models with 64k and 128k context. Raising the base before or during long-context training, as Code Llama and Llama 3 do, is the same idea applied from the start.
Settings in practice
- RoFormer, LLaMA 1/2
- base 10000, rotation on every query/key dimension; LLaMA trained at 2048 tokens, Llama 2 at 4096
- GPT-J
- rotates only 64 of each head's 256 dimensions; the rest carry no position
- Code Llama
- base raised to for 16k-token training sequences
- Llama 3
- base
- Position interpolation
- LLaMA 7B–65B extended to 32768 tokens with about 1000 fine-tuning steps
- YaRN
- , for LLaMA; logits scaled by
1import torch23def rope_cache(seq_len, head_dim, base=10000.0, pi_scale=1.0):4 theta = base ** (-torch.arange(0, head_dim, 2).float() / head_dim) # (head_dim/2,)5 pos = torch.arange(seq_len).float() / pi_scale # PI: positions m / s6 ang = torch.outer(pos, theta) # (T, head_dim/2)7 ang = torch.cat([ang, ang], dim=-1) # (T, head_dim)8 return ang.cos(), ang.sin()910def rotate_half(x): # pairs are (x[j], x[j + head_dim/2])11 x1, x2 = x.chunk(2, dim=-1)12 return torch.cat([-x2, x1], dim=-1)1314def apply_rope(x, cos, sin): # x: (..., T, head_dim)15 return x * cos + rotate_half(x) * sin1617B, H, T, Dh = 2, 8, 16, 6418q, k = torch.randn(B, H, T, Dh), torch.randn(B, H, T, Dh)19cos, sin = rope_cache(T, Dh) # (16, 64) each, broadcast over B and H20qr, kr = apply_rope(q, cos, sin), apply_rope(k, cos, sin)21scores = qr @ kr.transpose(-1, -2) / Dh ** 0.522print(scores.shape) # torch.Size([2, 8, 16, 16])2324# the score depends only on m - n25def score(qv, kv, m, n, **kw):26 c, s = rope_cache(max(m, n) + 1, Dh, **kw)27 return (apply_rope(qv, c[m], s[m]) * apply_rope(kv, c[n], s[n])).sum()2829qv, kv = torch.randn(Dh), torch.randn(Dh)30print(torch.allclose(score(qv, kv, 3, 1), score(qv, kv, 103, 101), atol=1e-3)) # True31print(torch.allclose(score(qv, kv, 3, 1), score(qv, kv, 1, 3), atol=1e-3)) # False (almost surely)3233# context extension by s = 434s = 435cos_pi, sin_pi = rope_cache(4 * 2048, Dh, pi_scale=s) # position interpolation36cos_ntk, sin_ntk = rope_cache(4 * 2048, Dh, base=10000.0 * s ** (Dh / (Dh - 2))) # NTK-aware37print(cos_pi.shape) # torch.Size([8192, 64])Use when
- +The score is an exact function of content and ; shift-invariant, direction-aware
- +No parameters, applied to queries and keys only, nothing added to the residual stream
- +Compatible with KV caching, FlashAttention-style kernels and linear attention
- +Context can be extended later by rescaling frequencies (PI, NTK-aware, YaRN) with little fine-tuning
Avoid when
- −Extrapolates poorly beyond the training length without rescaling: slow pairs meet unseen angles
- −Position interpolation blurs nearby positions; methods trade resolution against range
- −Two incompatible layouts (interleaved, rotate-half) in circulation
- −Long-term decay is only an average bound, not a guaranteed preference for nearby tokens
Check yourself
With RoPE, a query at position 10 and a key at position 4 give logit 0.8. Which pair of positions gives the same logit for the same q and k vectors?
Check yourself
A model with head size 64 was trained with RoPE (base 10000) on 2048 tokens. You want 8192 tokens. What does NTK-aware scaling do that position interpolation does not?
Remember
- ✓RoPE: rotate pair of and at position by , ; values are not rotated.
- ✓: the score depends only on content and .
- ✓Implemented as in every layer; no parameters, cache-friendly.
- ✓Long-term decay: Su et al. bound the score by , which shrinks with distance on average.
- ✓Context extension: PI uses ; NTK-aware uses base ; YaRN interpolates only slow pairs and scales logits by .