Positional encoding

Self-attention ignores token order, so position must be injected. Absolute schemes add a vector per position (fixed sinusoids or a learned table); relative schemes make each attention score depend on the offset j − i (Shaw et al., T5 buckets, RoPE, ALiBi).

14 min read1082 wordscore0 viewsupdated 2026-10-12

Definition

Positional encoding is any mechanism that tells a Transformer where each token sits in the sequence. Absolute encodings give position tt a vector pt∈Rdp_t\in\mathbb R^{d} that is added to the token embedding: either fixed sinusoids, pt,2i=sin⁡(t ωi)p_{t,2i}=\sin(t\,\omega_i) and pt,2i+1=cos⁡(t ωi)p_{t,2i+1}=\cos(t\,\omega_i) with ωi=10000−2i/d\omega_i=10000^{-2i/d} (Vaswani et al., 2017), or a learned table with one row per position. Relative encodings leave the token vectors alone and change the attention score between query ii and key jj as a function of the offset j−ij-i.

Self-attention is a weighted average whose weights come from dot products (see Self-attention). Nothing in it refers to an index. Permute the input tokens and an encoder layer returns the same output vectors, permuted the same way: the layer is permutation-equivariant. For language this is fatal, because "dog bites man" and "man bites dog" contain the same bag of tokens. Recurrent and convolutional networks get order for free from their structure; a Transformer has to be told. (A causal mask breaks the symmetry a little, because token tt sees exactly t+1t+1 tokens, and decoder-only models without any encoding can still infer some position from it, as Haviv et al. showed in 2022. Every widely used model still adds an explicit scheme.)

Vaswani et al. introduced sinusoidal encodings in "Attention Is All You Need" (NeurIPS 2017). ConvS2S (Gehring et al., 2017) had used learned position embeddings, and Vaswani et al. tried those too: on English→German development BLEU the two were almost identical (25.7 learned vs 25.8 sinusoidal). They kept the sinusoids because they might extrapolate to sequences longer than those seen in training. BERT and GPT-2 went the learned route. Shaw, Uszkoreit and Vaswani (NAACL 2018) then moved position into the attention score itself, which started the line of relative methods that ends, for now, in Rotary positional embeddings (RoPE) and ALiBi.

Sinusoidal encoding

pt,2i=sin⁡(t ωi),pt,2i+1=cos⁡(t ωi),ωi=10000−2i/d,i=0,…,d2−1,xt=d  E[tokent]+ptp_{t,2i}=\sin(t\,\omega_i),\qquad p_{t,2i+1}=\cos(t\,\omega_i),\qquad \omega_i=10000^{-2i/d},\quad i=0,\dots,\tfrac d2-1,\qquad x_t=\sqrt d\;E[\text{token}_t]+p_t
tt
position index, 0, 1, 2, …
dd
model width dmodeld_{\text{model}} (512 in the base Transformer); pair ii occupies dimensions 2i2i and 2i+12i+1
ωi\omega_i
angular frequency of pair ii. Wavelengths 2π/ωi2\pi/\omega_i form a geometric sequence from 2π2\pi (pair 0) to 2π⋅10000(d−2)/d2\pi\cdot10000^{(d-2)/d}, about 60,60060{,}600 positions for d=512d=512
EE
token embedding matrix; the paper multiplies it by d\sqrt d before adding ptp_t, so the encoding does not swamp the token signal

Read each pair as a clock hand. Pair 0 turns one radian per position and repeats every 6.28 tokens. Pair d/2−1d/2-1 turns so slowly that over a few thousand positions it never completes a revolution. Together the hands work like the digits of a number in a mixed radix: fast pairs separate neighbours, slow pairs separate distant positions. Every value lies in [−1,1][-1,1], and ∥pt∥2=d/2\lVert p_t\rVert^2=d/2 for every tt.

(pt+k,2ipt+k,2i+1)=(cos⁡kωisin⁡kωi−sin⁡kωicos⁡kωi)(pt,2ipt,2i+1),pt⊤pt+k=∑icos⁡(k ωi)\begin{pmatrix}p_{t+k,2i}\\ p_{t+k,2i+1}\end{pmatrix}=\begin{pmatrix}\cos k\omega_i & \sin k\omega_i\\ -\sin k\omega_i & \cos k\omega_i\end{pmatrix}\begin{pmatrix}p_{t,2i}\\ p_{t,2i+1}\end{pmatrix},\qquad p_t^\top p_{t+k}=\sum_{i}\cos(k\,\omega_i)
MkM_k
the block-diagonal matrix of these 2×22\times2 rotations; pt+k=Mkptp_{t+k}=M_kp_t with MkM_k independent of tt. This is the "attend by relative positions via a linear function" argument of the paper
pt⊤pt+kp_t^\top p_{t+k}
depends only on the offset kk, is symmetric in ±k\pm k, and equals d/2d/2 at k=0k=0

The caveat is that the model never sees ptp_t alone. It sees xt=d E[⋅]+ptx_t=\sqrt d\,E[\cdot]+p_t, and the attention score (WQxi)⊤(WKxj)(W_Qx_i)^\top(W_Kx_j) expands into content–content, content–position, position–content and position–position terms. Only the last is a clean function of j−ij-i, and only if WQ⊤WKW_Q^\top W_K preserves the rotation structure, which nothing enforces. In practice sinusoidal models do not extrapolate well: Press et al. (2022) found perplexity rising sharply within a few dozen tokens past the training length.

PE(pos, 2i) = sin(pos·ωᵢ) · PE(pos, 2i+1) = cos(pos·ωᵢ) · ωᵢ = base^(−2i/d)encoding matrix: column = position, row = dimension · ■ +1 ■ −1031i=3pos 047pp+kPE(p)·PE(q) / (d/2) = mean cos((p − q)ωᵢ), for every q10−1q = 0 … 47 · ○ most similar other positiond · base 32 · 10000pair i 3 (dims 6, 7)ωᵢ 0.1778λᵢ = 2π/ωᵢ 35.3 positionsp · k 10 · 5sim(p, p+k) 0.736sim(0, k) 0.736|MₖPE(p) − PE(p+k)| 8e-16most similar q 9 (0.957)pair i as a point (sin, cos):p → p+k: turn kωᵢ = 0.889 radMₖ is the same for every pwavelengths 2π … 2π·base^((d−2)/d) = 35.3k · sim(p, p+k) = sim(0, k): only the offset matters
d_modelbase
fig.The 48 × 32 sinusoidal matrix for d = 32 and base 10000: columns are positions, rows are dimensions, blue is +1 and red −1. The top rows flip sign every few positions; the bottom rows barely change. Below it, the curve is PE(p)·PE(q)/(d/2) for p = 10 against every q. It peaks at 1 for q = p, and the nearest neighbour (q = 9) scores 0.957. The default offset k = 5 gives sim(10, 15) = 0.736, exactly sim(0, 5). Drag p and the curve slides without changing shape: the similarity depends only on p − q. The dial shows pair i = 3 (ω₃ = 0.1778, wavelength 35.3 positions) as a point on the unit circle. Moving from p to p + k turns it by kω₃ = 0.889 rad whatever p is, and applying that fixed rotation Mₖ to PE(p) reproduces PE(p + k) to 8·10⁻¹⁶. Switch the base to 100 and the slow pairs speed up: sim(10, 30) falls from 0.654 to 0.183 and the curve dips to −0.009. With d = 16 and base 100 it reaches −0.247.

Learned absolute embeddings

The simplest alternative is a trainable matrix P∈RLmax⁡×dP\in\mathbb R^{L_{\max}\times d}, with row tt added to token tt. BERT uses Lmax⁡=512L_{\max}=512, GPT-2 1024, and Vision Transformers (ViT) learn one embedding per patch slot. The table can learn whatever the task needs, and in the 2017 comparison it cost nothing in quality. Its hard limit is Lmax⁡L_{\max}: position 1024 in GPT-2 has no row at all. Rows for rarely seen positions are also poorly trained, because only long training sequences ever reach them. ViT works around the limit at a new resolution by interpolating the 2-D grid of embeddings, which is a form of the same idea that Rotary positional embeddings (RoPE) later use to extend context.

Relative positions in the score

eij=(xiWQ)(xjWK+aijK)⊤dz,zi=∑jαij (xjWV+aijV),aijK=wclip(j−i, k)K,  aijV=wclip(j−i, k)Ve_{ij}=\frac{(x_iW^Q)(x_jW^K+a^K_{ij})^\top}{\sqrt{d_z}},\qquad z_i=\sum_j\alpha_{ij}\,(x_jW^V+a^V_{ij}),\qquad a^K_{ij}=w^K_{\mathrm{clip}(j-i,\,k)},\ \ a^V_{ij}=w^V_{\mathrm{clip}(j-i,\,k)}
wrK, wrVw^K_r,\ w^V_r
learned vectors for each clipped offset r∈{−k,…,k}r\in\{-k,\dots,k\}, so 2k+12k+1 of each, shared across heads
clip(r,k)\mathrm{clip}(r,k)
max⁡(−k,min⁡(k,r))\max(-k,\min(k,r)): all offsets beyond ±k\pm k share one vector, which lets the model run on any length
αij\alpha_{ij}
softmax of eije_{ij} over jj, as in Scaled dot-product attention

Shaw et al. replaced absolute encodings in the Transformer with these offset vectors. On WMT'14 they reported gains of 1.3 BLEU on English→German and 0.3 BLEU on English→French over absolute positions, and adding sinusoids back on top did not help further. Transformer-XL (Dai et al., 2019) needed relative positions for a different reason: it reuses cached states from the previous segment, and absolute indices would collide. It uses a sinusoidal function of j−ij-i plus two global bias vectors.

T5 (Raffel et al., JMLR 2020) reduced the idea to its minimum. Each head adds a single learned scalar bh(bucket(j−i))b_h(\text{bucket}(j-i)) to its logit, and the 32 buckets are shared by all layers. Small offsets get their own bucket; larger ones are grouped logarithmically up to 128, beyond which all offsets share the last bucket. In the encoder, 16 buckets serve keys before the query and 16 serve keys after it; in the decoder all 32 cover the past. ALiBi goes one step further and fixes the scalar to −mh (i−j)-m_h\,(i-j) with no parameters at all.

key − queryencoder (bidirectional) bucketdecoder (causal) bucket
000
−333
+3190 (future, masked)
−108 (shared by −8 … −11)10
−2010 (−16 … −22)17
−5013 (−46 … −63)24
−20015 (all ≤ −91)31
T5 relative-position buckets (32 buckets, max distance 128), computed with the reference bucketing function

Worked example: d = 4

With d=4d=4 there are two pairs, ω0=100000=1\omega_0=10000^{0}=1 and ω1=10000−2/4=0.01\omega_1=10000^{-2/4}=0.01. So pt=(sin⁡t, cos⁡t, sin⁡0.01t, cos⁡0.01t)p_t=(\sin t,\ \cos t,\ \sin 0.01t,\ \cos 0.01t).

tsin tcos tsin 0.01tcos 0.01t
00.00001.00000.00001.0000
10.84150.54030.01001.0000
20.9093−0.41610.02000.9998
30.1411−0.99000.03000.9996
Sinusoidal encodings for positions 0–3 (computed in Node)

Shift by a rotation, and the offset-only dot product

  1. 1M2M_2 rotates pair 0 by 2ω0=22\omega_0=2 rad and pair 1 by 2ω1=0.022\omega_1=0.02 rad. Applied to p1p_1: sin⁡1cos⁡2+cos⁡1sin⁡2=0.1411\sin1\cos2+\cos1\sin2=0.1411 and cos⁡1cos⁡2−sin⁡1sin⁡2=−0.9900\cos1\cos2-\sin1\sin2=-0.9900, then (0.0300, 0.9996)(0.0300,\ 0.9996) for pair 1. That is p3p_3 exactly, and the same M2M_2 maps every ptp_t to pt+2p_{t+2}.
  2. 2p1⊤p3=0.8415⋅0.1411+0.5403⋅(−0.9900)+0.0100⋅0.0300+1.0000⋅0.9996=0.5837p_1^\top p_3=0.8415\cdot0.1411+0.5403\cdot(-0.9900)+0.0100\cdot0.0300+1.0000\cdot0.9996=0.5837.
  3. 3p0⊤p2=cos⁡2+cos⁡0.02=−0.4161+0.9998=0.5837p_0^\top p_2=\cos2+\cos0.02=-0.4161+0.9998=0.5837. Same offset, same value.
  4. 4Offset 1 in both directions: p1⊤p2=p1⊤p0=cos⁡1+cos⁡0.01=1.5403p_1^\top p_2=p_1^\top p_0=\cos1+\cos0.01=1.5403. The plain dot product cannot tell "one before" from "one after"; the score only becomes direction-aware through the learned WQ,WKW_Q,W_K.

Choices in well-known models

Transformer (2017)
sinusoidal, base 10000, added after scaling embeddings by d\sqrt{d}; learned embeddings gave 25.7 vs 25.8 BLEU (dev, En→De)
BERT, GPT-2
learned absolute tables of 512 and 1024 positions
Shaw et al. (2018)
learned offset vectors on keys and values, clipped at distance kk; +1.3 BLEU En→De, +0.3 En→Fr over absolute
T5 (2020)
one scalar bias per head per bucket, 32 buckets up to distance 128, shared across layers
LLaMA, PaLM, GPT-NeoX
Rotary positional embeddings (RoPE): rotate queries and keys, nothing added to the residual stream
BLOOM, MPT
ALiBi: a fixed linear distance penalty per head
python
1import math, torch, torch.nn as nn
2
3def sinusoidal_pe(n_pos, d_model, base=10000.0):
4 pos = torch.arange(n_pos, dtype=torch.float32)[:, None] # (n_pos, 1)
5 two_i = torch.arange(0, d_model, 2, dtype=torch.float32) # (d_model/2,)
6 omega = base ** (-two_i / d_model) # (d_model/2,)
7 pe = torch.zeros(n_pos, d_model)
8 pe[:, 0::2] = torch.sin(pos * omega) # even dims: sin
9 pe[:, 1::2] = torch.cos(pos * omega) # odd dims: cos
10 return pe # (n_pos, d_model)
11
12class TokenAndPosition(nn.Module):
13 def __init__(self, vocab, d_model, max_len=2048, learned=False):
14 super().__init__()
15 self.tok, self.d, self.learned = nn.Embedding(vocab, d_model), d_model, learned
16 if learned:
17 self.pos = nn.Embedding(max_len, d_model) # BERT / GPT-2 style
18 else:
19 self.register_buffer("pe", sinusoidal_pe(max_len, d_model), persistent=False)
20
21 def forward(self, ids): # ids: (B, T)
22 T = ids.shape[1]
23 x = self.tok(ids) * math.sqrt(self.d) # (B, T, d)
24 p = self.pos(torch.arange(T, device=ids.device)) if self.learned else self.pe[:T]
25 return x + p # (B, T, d)
26
27def t5_bucket(rel, bidirectional=True, num_buckets=32, max_distance=128):
28 """rel = key position - query position (int64 tensor)."""
29 out = torch.zeros_like(rel)
30 if bidirectional:
31 num_buckets //= 2
32 out += (rel > 0).long() * num_buckets # keys after the query
33 n = rel.abs()
34 else:
35 n = (-rel).clamp(min=0) # only the past counts
36 exact = num_buckets // 2
37 large = exact + (torch.log(n.float() / exact) / math.log(max_distance / exact)
38 * (num_buckets - exact)).long()
39 large = large.clamp(max=num_buckets - 1)
40 return out + torch.where(n < exact, n, large)
41
42pe = sinusoidal_pe(50, 4)
43print(pe[1] @ pe[3], pe[0] @ pe[2]) # tensor(0.5837) tensor(0.5837)
44emb = TokenAndPosition(vocab=1000, d_model=64)
45print(emb(torch.randint(0, 1000, (2, 10))).shape) # torch.Size([2, 10, 64])
46
47rel = torch.tensor([-200, -50, -20, -10, -3, 0, 3, 10, 50])
48print(t5_bucket(rel)) # tensor([15, 13, 10, 8, 3, 0, 19, 24, 29])
49print(t5_bucket(rel, False)) # tensor([31, 24, 17, 10, 3, 0, 0, 0, 0])
Sinusoidal and learned absolute encodings, and T5's relative-position buckets

Use when

  • +Sinusoidal: no parameters, any length can be computed, pt+k=Mkptp_{t+k}=M_kp_t
  • +Learned absolute: simplest possible, matches sinusoids in quality at trained lengths
  • +Relative (Shaw, T5): invariant to shifting the whole sequence, natural for cached segments, any length
  • +T5 buckets: a few hundred parameters in total, coarse far away where precision matters least

Avoid when

  • −Absolute schemes extrapolate poorly past the training length; a learned table cannot go past it at all
  • −Added encodings share the residual stream with content, and the score's cross terms are not purely relative
  • −Shaw et al. add aijKa^K_{ij}, aijVa^V_{ij} inside attention, which costs memory proportional to T2T^2 unless computed cleverly
  • −Bias-style relative schemes need a modified attention kernel, which can be slower than plain attention

Check yourself

You remove all positional information from a Transformer encoder and feed it a permuted sentence. What happens to the outputs?

Check yourself

With sinusoidal encodings and d = 4 (ω = 1 and 0.01), which pair of positions has the same encoding dot product as positions 0 and 2?

Remember

  • ✓Sinusoidal: pt,2i=sin⁡(t ωi)p_{t,2i}=\sin(t\,\omega_i), pt,2i+1=cos⁡(t ωi)p_{t,2i+1}=\cos(t\,\omega_i), ωi=10000−2i/d\omega_i=10000^{-2i/d}; wavelengths from 2π2\pi to about 2π⋅1042\pi\cdot10^4.
  • ✓Self-attention is permutation-equivariant, so order must be injected: absolute (add ptp_t) or relative (score depends on j−ij-i).
  • ✓pt+k=Mkptp_{t+k}=M_kp_t with MkM_k a fixed block rotation, and pt⊤pt+k=∑icos⁡(kωi)p_t^\top p_{t+k}=\sum_i\cos(k\omega_i) depends only on kk.
  • ✓Learned tables (BERT 512, GPT-2 1024) match sinusoids at trained lengths but cannot go beyond them.
  • ✓Relative schemes: Shaw et al. learn clipped offset vectors on keys and values; T5 adds one scalar per head per log-spaced bucket (32 buckets, up to 128).
← back to Deep Learning roadmap