Rotary positional embeddings (RoPE)

RoPE (Su et al., 2021) rotates each 2-D pair of a query and a key by its position times a fixed frequency θᵢ = 10000^(−2i/d). The rotations cancel in the dot product, so the attention score depends on content and the offset m − n only.

15 min read877 wordsadvanced0 viewsupdated 2026-10-12

Definition

Rotary positional embedding (RoPE) encodes position by rotating queries and keys instead of adding a vector to the input. The head vector of size dd is split into d/2d/2 two-dimensional pairs. At position mm, pair ii of the query (and of the key) is rotated by the angle mθim\theta_i, with θi=10000−2i/d\theta_i=10000^{-2i/d}. Because Rm⊤Rn=Rn−mR_m^\top R_n=R_{n-m}, the score ⟨Rmq, Rnk⟩=q⊤Rn−mk\langle R_mq,\,R_nk\rangle=q^\top R_{n-m}k depends only on the two vectors and the offset. Values are not rotated, and there are no parameters.

Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen and Yunfeng Liu introduced RoPE in "RoFormer: Enhanced Transformer with Rotary Position Embedding" (arXiv 2021; Neurocomputing 2024), after Su first described it on his blog. They asked for a function f(x,m)f(x,m) such that ⟨f(q,m), f(k,n)⟩=g(q,k,m−n)\langle f(q,m),\,f(k,n)\rangle=g(q,k,m-n) for some gg. Additive schemes (see Positional encoding) only give that for one of the four terms of the expanded score. Rotation gives it exactly. EleutherAI's GPT-J and GPT-NeoX adopted it in 2021–22, followed by PaLM and LLaMA, and it has since become the default for decoder-only language models.

The rotation

f(x,m)=RΘ,md x,RΘ,md=(cos⁡mθ0−sin⁡mθ0sin⁡mθ0cos⁡mθ0⋱cos⁡mθd/2−1−sin⁡mθd/2−1sin⁡mθd/2−1cos⁡mθd/2−1),θi=10000−2i/df(x,m)=R^d_{\Theta,m}\,x,\qquad R^d_{\Theta,m}=\begin{pmatrix}\cos m\theta_0&-\sin m\theta_0&&&\\ \sin m\theta_0&\cos m\theta_0&&&\\ &&\ddots&&\\ &&&\cos m\theta_{d/2-1}&-\sin m\theta_{d/2-1}\\ &&&\sin m\theta_{d/2-1}&\cos m\theta_{d/2-1}\end{pmatrix},\qquad \theta_i=10000^{-2i/d}
xx
a query qm=WQxmq_m=W_Qx_m or key kn=WKxnk_n=W_Kx_n of one head, after projection
RΘ,mdR^d_{\Theta,m}
block-diagonal and orthogonal: it changes no norm, only angles. Never materialised; see the elementwise form below
θi\theta_i
frequency of pair ii: pair 0 turns 1 rad per position (wavelength 2π2\pi), the last pair 10000−(d−2)/d10000^{-(d-2)/d} rad (wavelength about 2π⋅1042\pi\cdot10^4). The same geometric ladder as sinusoidal encodings
1000010000
the base; later models raise it (Code Llama 10610^6, Llama 3 5⋅1055\cdot10^5) to slow the low frequencies down for long contexts
⟨Rmq,  Rnk⟩=q⊤Rm⊤Rn k=q⊤Rn−m k=Re∑i=0d/2−1q[i] k[i]‾  e i(m−n)θi\langle R_mq,\;R_nk\rangle=q^\top R_m^\top R_n\,k=q^\top R_{n-m}\,k=\mathrm{Re}\sum_{i=0}^{d/2-1}q_{[i]}\,\overline{k_{[i]}}\;e^{\,\mathrm i(m-n)\theta_i}
q[i]q_{[i]}
pair ii read as a complex number, q2i+i q2i+1q_{2i}+\mathrm i\,q_{2i+1}; rotation by mθim\theta_i is multiplication by eimθie^{\mathrm im\theta_i}
Rm⊤Rn=Rn−mR_m^\top R_n=R_{n-m}
rotations by angles compose by adding angles, and the transpose rotates backwards
(m−n)(m-n)
the only place position enters. Shifting both positions by any tt leaves the score unchanged; swapping mm and nn changes it (direction is encoded)

In code the matrix product becomes two elementwise products: Rmx=x⊙cos⁡(mΘ)+rot(x)⊙sin⁡(mΘ)R_mx=x\odot\cos(m\Theta)+\mathrm{rot}(x)\odot\sin(m\Theta), where rot\mathrm{rot} maps each pair (xa,xb)(x_a,x_b) to (−xb,xa)(-x_b,x_a). The cost is O(Td)O(Td) per head, negligible next to attention. The rotation is applied in every layer to QQ and KK just before the scores, so nothing positional is ever written into the residual stream. With a KV cache, keys are rotated once at their absolute position and stored; the relative property guarantees that the score with any later query is still a function of the offset.

Long-term decay

Su et al. also argue that the score tends to shrink as ∣m−n∣|m-n| grows. Write hi=q[i]k[i]‾h_i=q_{[i]}\overline{k_{[i]}} and Sj=∑i<jei(m−n)θiS_j=\sum_{i<j}e^{\mathrm i(m-n)\theta_i}. Summation by parts (an Abel transformation) gives ∣∑ihiei(m−n)θi∣≤(max⁡i∣hi+1−hi∣)∑j∣Sj∣\big|\sum_ih_ie^{\mathrm i(m-n)\theta_i}\big|\le\big(\max_i|h_{i+1}-h_i|\big)\sum_j|S_j|. The factor 1d/2∑j∣Sj∣\frac{1}{d/2}\sum_j|S_j| decreases with the distance on average, because the phases ei(m−n)θie^{\mathrm i(m-n)\theta_i} spread around the circle and cancel. It is a bound, not a law: the curve oscillates, and a trained model can still attend sharply to a distant token whose content matches. The figure plots this factor next to the actual score for q=k=1q=k=\mathbf 1.

q at position m: pair i turns by mθᵢ, θᵢ = 10000^(−2i/64); the logit depends on m − npair 0: Rₘq · Rₙk · (m−n)θᵢθᵢ 1.00 rad/positionwavelength 2π/θᵢ 6.28 positionsm, n, m − n 9, 4, 5(m − n)θᵢ 5.00 radpair 0 term of ⟨Rₘq, Rₙk⟩ −0.755logit ⟨Rₘq, Rₙk⟩/√d −0.185same, m+100 and n+100 −0.185swapped: m ↔ n 0.550d = 64 (32 pairs), q, k fixed random vectors; shifting bothpositions leaves the logit unchanged, swapping them does not— score for q = k = 1 · — Su et al. decay term (1 at 0)10.501162564096LsLrelative distance m − n (log scale)θ′ᵢ/θᵢ per pair · ■ extrapolated pairsi = 031high freq. ← → low freq.L = 2048, s = 4 · extrapolated pairs 11 · pair-0 step 1.00 rad
extension
fig.One head of size d = 64 (32 pairs) with fixed random q and k. The dial shows pair i rotated by its position: at the defaults m = 9, n = 4, pair 0 (θ₀ = 1 rad per position) is 5 rad apart, and that pair contributes −0.755 to ⟨Rₘq, Rₙk⟩. The full logit ⟨Rₘq, Rₙk⟩/√d is −0.185. Moving both positions by 100 gives −0.185 again, while swapping m and n gives 0.550. Lower left: the score for q = k = 1, normalised to (1/32)Σcos((m − n)θᵢ), and Su et al.'s decay factor, both 1 at distance 0, on a log axis. The score is 0.966 at distance 1, 0.658 at 10, 0.559 at 100 and 0.020 at 2000. Lower right: how each extension method rescales θᵢ to run at s·L with L = 2048. Unmodified at s = 4, the 11 slowest pairs (21–31) never complete a turn within 2048 tokens and would see angles they were never trained on (red). Position interpolation divides every θᵢ by s: no pair is extrapolated, but pair 0 now steps only 0.25 rad per token. NTK-aware scaling raises the base to 10000·s^(64/62), which leaves pair 0 at 1 rad and divides only the slowest pair by 4; pairs 21–30 still extrapolate slightly. YaRN keeps pairs 0–8 (more than 32 turns in L) exactly, fully interpolates pairs 21–31 (less than one turn), ramps between, and scales the logits by (0.1 ln s + 1)² = 1.297.

Worked example: one pair

Take a single pair with θ=0.5\theta=0.5, query q=(1,2)q=(1,2) and key k=(3,1)k=(3,1). Without rotation q⊤k=5q^\top k=5. As complex numbers, qkˉ=(1+2i)(3−i)=5+5iq\bar k=(1+2\mathrm i)(3-\mathrm i)=5+5\mathrm i, so the RoPE score is Re[(5+5i)ei(m−n)θ]=5cos⁡((m−n)θ)−5sin⁡((m−n)θ)\mathrm{Re}\big[(5+5\mathrm i)e^{\mathrm i(m-n)\theta}\big]=5\cos((m-n)\theta)-5\sin((m-n)\theta).

mnR_m qR_n k⟨R_m q, R_n k⟩
31(−1.9243, 1.1390)(2.1533, 2.3159)−1.5058
75(−0.2349, −2.2237)(−3.0019, 0.9943)−1.5058
13(−0.0813, 2.2346)(−0.7853, 3.0632)6.9089
33(−1.9243, 1.1390)(−0.7853, 3.0632)5.0000
Rotated vectors and scores (computed in Node)

Reading the table

  1. 1m=3m=3: rotate qq by 1.51.5 rad: (1cos⁡1.5−2sin⁡1.5, 1sin⁡1.5+2cos⁡1.5)=(−1.9243, 1.1390)(1\cos1.5-2\sin1.5,\ 1\sin1.5+2\cos1.5)=(-1.9243,\ 1.1390). n=1n=1: rotate kk by 0.50.5 rad to (2.1533, 2.3159)(2.1533,\ 2.3159).
  2. 2Their dot product is −1.5058=5cos⁡1−5sin⁡1=2.7015−4.2074-1.5058=5\cos1-5\sin1=2.7015-4.2074, which depends only on (m−n)θ=1(m-n)\theta=1.
  3. 3Shifting both positions by 4 changes both vectors completely but leaves the score at −1.5058-1.5058.
  4. 4Swapping to m=1m=1, n=3n=3 gives 5cos⁡(−1)−5sin⁡(−1)=6.90895\cos(-1)-5\sin(-1)=6.9089: the sign of the offset matters. At m=nm=n the rotations cancel and the score is the raw q⊤k=5q^\top k=5.

Extending the context

A model trained on length LL has only seen angles mθim\theta_i with m<Lm<L. For high-frequency pairs this is no problem: they have turned many times, and every angle on the circle has been seen. Low-frequency pairs with Lθi<2πL\theta_i<2\pi have only covered part of the circle, so at m>Lm>L they produce angles never seen in training. Perplexity of a plain RoPE model rises sharply just beyond LL. Three fixes are widely used. All of them change θi\theta_i or mm, then usually fine-tune briefly.

PI:  θi′=θisNTK-aware:  θi′=(b sd/(d−2))−2i/dYaRN:  θi′=(1−γ(ri))θis+γ(ri) θi,ri=L θi2π\text{PI: }\ \theta'_i=\frac{\theta_i}{s}\qquad\quad \text{NTK-aware: }\ \theta'_i=\big(b\,s^{d/(d-2)}\big)^{-2i/d}\qquad\quad \text{YaRN: }\ \theta'_i=\Big(1-\gamma(r_i)\Big)\frac{\theta_i}{s}+\gamma(r_i)\,\theta_i,\quad r_i=\frac{L\,\theta_i}{2\pi}
ss
extension factor L′/LL'/L (e.g. 2048 → 8192 gives s=4s=4)
PI\text{PI}
position interpolation (Chen, Wong, Chen and Tian, 2023): equivalent to using position m/sm/s. Every angle stays inside the trained range, but neighbouring tokens become ss times closer in every pair
b sd/(d−2)b\,s^{d/(d-2)}
NTK-aware scaling (posted by "bloc97", 2023): a larger base. Pair 0 is unchanged and the slowest pair is divided by exactly ss, so high frequencies keep their resolution
γ(r)\gamma(r)
YaRN's ramp: 0 if a pair makes fewer than α=1\alpha=1 turns within LL (interpolate), 1 if more than β=32\beta=32 (keep), linear in between. YaRN also multiplies the logits by (0.1ln⁡s+1)2(0.1\ln s+1)^2

Chen et al. extended LLaMA models from 2048 to 32768 tokens with position interpolation and about 1000 fine-tuning steps; direct fine-tuning without interpolation adapted far more slowly. NTK-aware scaling worked reasonably even without fine-tuning, which made it popular in open-source inference tools. YaRN (Peng, Quesnelle, Fan and Shippole, ICLR 2024) combined the "by parts" frequency split with the temperature and released LLaMA 2 models with 64k and 128k context. Raising the base before or during long-context training, as Code Llama and Llama 3 do, is the same idea applied from the start.

Settings in practice

RoFormer, LLaMA 1/2
base 10000, rotation on every query/key dimension; LLaMA trained at 2048 tokens, Llama 2 at 4096
GPT-J
rotates only 64 of each head's 256 dimensions; the rest carry no position
Code Llama
base raised to 10610^6 for 16k-token training sequences
Llama 3
base 5⋅1055\cdot10^5
Position interpolation
LLaMA 7B–65B extended to 32768 tokens with about 1000 fine-tuning steps
YaRN
α=1\alpha=1, β=32\beta=32 for LLaMA; logits scaled by (0.1ln⁡s+1)2(0.1\ln s+1)^2
python
1import torch
2
3def rope_cache(seq_len, head_dim, base=10000.0, pi_scale=1.0):
4 theta = base ** (-torch.arange(0, head_dim, 2).float() / head_dim) # (head_dim/2,)
5 pos = torch.arange(seq_len).float() / pi_scale # PI: positions m / s
6 ang = torch.outer(pos, theta) # (T, head_dim/2)
7 ang = torch.cat([ang, ang], dim=-1) # (T, head_dim)
8 return ang.cos(), ang.sin()
9
10def rotate_half(x): # pairs are (x[j], x[j + head_dim/2])
11 x1, x2 = x.chunk(2, dim=-1)
12 return torch.cat([-x2, x1], dim=-1)
13
14def apply_rope(x, cos, sin): # x: (..., T, head_dim)
15 return x * cos + rotate_half(x) * sin
16
17B, H, T, Dh = 2, 8, 16, 64
18q, k = torch.randn(B, H, T, Dh), torch.randn(B, H, T, Dh)
19cos, sin = rope_cache(T, Dh) # (16, 64) each, broadcast over B and H
20qr, kr = apply_rope(q, cos, sin), apply_rope(k, cos, sin)
21scores = qr @ kr.transpose(-1, -2) / Dh ** 0.5
22print(scores.shape) # torch.Size([2, 8, 16, 16])
23
24# the score depends only on m - n
25def score(qv, kv, m, n, **kw):
26 c, s = rope_cache(max(m, n) + 1, Dh, **kw)
27 return (apply_rope(qv, c[m], s[m]) * apply_rope(kv, c[n], s[n])).sum()
28
29qv, kv = torch.randn(Dh), torch.randn(Dh)
30print(torch.allclose(score(qv, kv, 3, 1), score(qv, kv, 103, 101), atol=1e-3)) # True
31print(torch.allclose(score(qv, kv, 3, 1), score(qv, kv, 1, 3), atol=1e-3)) # False (almost surely)
32
33# context extension by s = 4
34s = 4
35cos_pi, sin_pi = rope_cache(4 * 2048, Dh, pi_scale=s) # position interpolation
36cos_ntk, sin_ntk = rope_cache(4 * 2048, Dh, base=10000.0 * s ** (Dh / (Dh - 2))) # NTK-aware
37print(cos_pi.shape) # torch.Size([8192, 64])
RoPE in the rotate-half layout used by GPT-NeoX and Hugging Face LLaMA, with position interpolation and NTK-aware base

Use when

  • +The score is an exact function of content and m−nm-n; shift-invariant, direction-aware
  • +No parameters, applied to queries and keys only, nothing added to the residual stream
  • +Compatible with KV caching, FlashAttention-style kernels and linear attention
  • +Context can be extended later by rescaling frequencies (PI, NTK-aware, YaRN) with little fine-tuning

Avoid when

  • −Extrapolates poorly beyond the training length without rescaling: slow pairs meet unseen angles
  • −Position interpolation blurs nearby positions; methods trade resolution against range
  • −Two incompatible layouts (interleaved, rotate-half) in circulation
  • −Long-term decay is only an average bound, not a guaranteed preference for nearby tokens

Check yourself

With RoPE, a query at position 10 and a key at position 4 give logit 0.8. Which pair of positions gives the same logit for the same q and k vectors?

Check yourself

A model with head size 64 was trained with RoPE (base 10000) on 2048 tokens. You want 8192 tokens. What does NTK-aware scaling do that position interpolation does not?

Remember

  • ✓RoPE: rotate pair ii of qq and kk at position mm by mθim\theta_i, θi=10000−2i/d\theta_i=10000^{-2i/d}; values are not rotated.
  • ✓⟨Rmq,Rnk⟩=q⊤Rn−mk=Re∑iq[i]k[i]‾ei(m−n)θi\langle R_mq,R_nk\rangle=q^\top R_{n-m}k=\mathrm{Re}\sum_iq_{[i]}\overline{k_{[i]}}e^{\mathrm i(m-n)\theta_i}: the score depends only on content and m−nm-n.
  • ✓Implemented as x⊙cos⁡+rot(x)⊙sin⁡x\odot\cos+\mathrm{rot}(x)\odot\sin in every layer; no parameters, cache-friendly.
  • ✓Long-term decay: Su et al. bound the score by max⁡i∣hi+1−hi∣∑j∣Sj∣\max_i|h_{i+1}-h_i|\sum_j|S_j|, which shrinks with distance on average.
  • ✓Context extension: PI uses m/sm/s; NTK-aware uses base b sd/(d−2)b\,s^{d/(d-2)}; YaRN interpolates only slow pairs and scales logits by (0.1ln⁡s+1)2(0.1\ln s+1)^2.
← back to Deep Learning roadmap