Definition
Positional encoding is any mechanism that tells a Transformer where each token sits in the sequence. Absolute encodings give position a vector that is added to the token embedding: either fixed sinusoids, and with (Vaswani et al., 2017), or a learned table with one row per position. Relative encodings leave the token vectors alone and change the attention score between query and key as a function of the offset .
Self-attention is a weighted average whose weights come from dot products (see Self-attention). Nothing in it refers to an index. Permute the input tokens and an encoder layer returns the same output vectors, permuted the same way: the layer is permutation-equivariant. For language this is fatal, because "dog bites man" and "man bites dog" contain the same bag of tokens. Recurrent and convolutional networks get order for free from their structure; a Transformer has to be told. (A causal mask breaks the symmetry a little, because token sees exactly tokens, and decoder-only models without any encoding can still infer some position from it, as Haviv et al. showed in 2022. Every widely used model still adds an explicit scheme.)
Vaswani et al. introduced sinusoidal encodings in "Attention Is All You Need" (NeurIPS 2017). ConvS2S (Gehring et al., 2017) had used learned position embeddings, and Vaswani et al. tried those too: on English→German development BLEU the two were almost identical (25.7 learned vs 25.8 sinusoidal). They kept the sinusoids because they might extrapolate to sequences longer than those seen in training. BERT and GPT-2 went the learned route. Shaw, Uszkoreit and Vaswani (NAACL 2018) then moved position into the attention score itself, which started the line of relative methods that ends, for now, in Rotary positional embeddings (RoPE) and ALiBi.
Sinusoidal encoding
- position index, 0, 1, 2, …
- model width (512 in the base Transformer); pair occupies dimensions and
- angular frequency of pair . Wavelengths form a geometric sequence from (pair 0) to , about positions for
- token embedding matrix; the paper multiplies it by before adding , so the encoding does not swamp the token signal
Read each pair as a clock hand. Pair 0 turns one radian per position and repeats every 6.28 tokens. Pair turns so slowly that over a few thousand positions it never completes a revolution. Together the hands work like the digits of a number in a mixed radix: fast pairs separate neighbours, slow pairs separate distant positions. Every value lies in , and for every .
- the block-diagonal matrix of these rotations; with independent of . This is the "attend by relative positions via a linear function" argument of the paper
- depends only on the offset , is symmetric in , and equals at
The caveat is that the model never sees alone. It sees , and the attention score expands into content–content, content–position, position–content and position–position terms. Only the last is a clean function of , and only if preserves the rotation structure, which nothing enforces. In practice sinusoidal models do not extrapolate well: Press et al. (2022) found perplexity rising sharply within a few dozen tokens past the training length.
Learned absolute embeddings
The simplest alternative is a trainable matrix , with row added to token . BERT uses , GPT-2 1024, and Vision Transformers (ViT) learn one embedding per patch slot. The table can learn whatever the task needs, and in the 2017 comparison it cost nothing in quality. Its hard limit is : position 1024 in GPT-2 has no row at all. Rows for rarely seen positions are also poorly trained, because only long training sequences ever reach them. ViT works around the limit at a new resolution by interpolating the 2-D grid of embeddings, which is a form of the same idea that Rotary positional embeddings (RoPE) later use to extend context.
Relative positions in the score
- learned vectors for each clipped offset , so of each, shared across heads
- : all offsets beyond share one vector, which lets the model run on any length
- softmax of over , as in Scaled dot-product attention
Shaw et al. replaced absolute encodings in the Transformer with these offset vectors. On WMT'14 they reported gains of 1.3 BLEU on English→German and 0.3 BLEU on English→French over absolute positions, and adding sinusoids back on top did not help further. Transformer-XL (Dai et al., 2019) needed relative positions for a different reason: it reuses cached states from the previous segment, and absolute indices would collide. It uses a sinusoidal function of plus two global bias vectors.
T5 (Raffel et al., JMLR 2020) reduced the idea to its minimum. Each head adds a single learned scalar to its logit, and the 32 buckets are shared by all layers. Small offsets get their own bucket; larger ones are grouped logarithmically up to 128, beyond which all offsets share the last bucket. In the encoder, 16 buckets serve keys before the query and 16 serve keys after it; in the decoder all 32 cover the past. ALiBi goes one step further and fixes the scalar to with no parameters at all.
| key − query | encoder (bidirectional) bucket | decoder (causal) bucket |
|---|---|---|
| 0 | 0 | 0 |
| −3 | 3 | 3 |
| +3 | 19 | 0 (future, masked) |
| −10 | 8 (shared by −8 … −11) | 10 |
| −20 | 10 (−16 … −22) | 17 |
| −50 | 13 (−46 … −63) | 24 |
| −200 | 15 (all ≤ −91) | 31 |
Worked example: d = 4
With there are two pairs, and . So .
| t | sin t | cos t | sin 0.01t | cos 0.01t |
|---|---|---|---|---|
| 0 | 0.0000 | 1.0000 | 0.0000 | 1.0000 |
| 1 | 0.8415 | 0.5403 | 0.0100 | 1.0000 |
| 2 | 0.9093 | −0.4161 | 0.0200 | 0.9998 |
| 3 | 0.1411 | −0.9900 | 0.0300 | 0.9996 |
Shift by a rotation, and the offset-only dot product
- 1 rotates pair 0 by rad and pair 1 by rad. Applied to : and , then for pair 1. That is exactly, and the same maps every to .
- 2.
- 3. Same offset, same value.
- 4Offset 1 in both directions: . The plain dot product cannot tell "one before" from "one after"; the score only becomes direction-aware through the learned .
Choices in well-known models
- Transformer (2017)
- sinusoidal, base 10000, added after scaling embeddings by ; learned embeddings gave 25.7 vs 25.8 BLEU (dev, En→De)
- BERT, GPT-2
- learned absolute tables of 512 and 1024 positions
- Shaw et al. (2018)
- learned offset vectors on keys and values, clipped at distance ; +1.3 BLEU En→De, +0.3 En→Fr over absolute
- T5 (2020)
- one scalar bias per head per bucket, 32 buckets up to distance 128, shared across layers
- LLaMA, PaLM, GPT-NeoX
- Rotary positional embeddings (RoPE): rotate queries and keys, nothing added to the residual stream
- BLOOM, MPT
- ALiBi: a fixed linear distance penalty per head
1import math, torch, torch.nn as nn23def sinusoidal_pe(n_pos, d_model, base=10000.0):4 pos = torch.arange(n_pos, dtype=torch.float32)[:, None] # (n_pos, 1)5 two_i = torch.arange(0, d_model, 2, dtype=torch.float32) # (d_model/2,)6 omega = base ** (-two_i / d_model) # (d_model/2,)7 pe = torch.zeros(n_pos, d_model)8 pe[:, 0::2] = torch.sin(pos * omega) # even dims: sin9 pe[:, 1::2] = torch.cos(pos * omega) # odd dims: cos10 return pe # (n_pos, d_model)1112class TokenAndPosition(nn.Module):13 def __init__(self, vocab, d_model, max_len=2048, learned=False):14 super().__init__()15 self.tok, self.d, self.learned = nn.Embedding(vocab, d_model), d_model, learned16 if learned:17 self.pos = nn.Embedding(max_len, d_model) # BERT / GPT-2 style18 else:19 self.register_buffer("pe", sinusoidal_pe(max_len, d_model), persistent=False)2021 def forward(self, ids): # ids: (B, T)22 T = ids.shape[1]23 x = self.tok(ids) * math.sqrt(self.d) # (B, T, d)24 p = self.pos(torch.arange(T, device=ids.device)) if self.learned else self.pe[:T]25 return x + p # (B, T, d)2627def t5_bucket(rel, bidirectional=True, num_buckets=32, max_distance=128):28 """rel = key position - query position (int64 tensor)."""29 out = torch.zeros_like(rel)30 if bidirectional:31 num_buckets //= 232 out += (rel > 0).long() * num_buckets # keys after the query33 n = rel.abs()34 else:35 n = (-rel).clamp(min=0) # only the past counts36 exact = num_buckets // 237 large = exact + (torch.log(n.float() / exact) / math.log(max_distance / exact)38 * (num_buckets - exact)).long()39 large = large.clamp(max=num_buckets - 1)40 return out + torch.where(n < exact, n, large)4142pe = sinusoidal_pe(50, 4)43print(pe[1] @ pe[3], pe[0] @ pe[2]) # tensor(0.5837) tensor(0.5837)44emb = TokenAndPosition(vocab=1000, d_model=64)45print(emb(torch.randint(0, 1000, (2, 10))).shape) # torch.Size([2, 10, 64])4647rel = torch.tensor([-200, -50, -20, -10, -3, 0, 3, 10, 50])48print(t5_bucket(rel)) # tensor([15, 13, 10, 8, 3, 0, 19, 24, 29])49print(t5_bucket(rel, False)) # tensor([31, 24, 17, 10, 3, 0, 0, 0, 0])Use when
- +Sinusoidal: no parameters, any length can be computed,
- +Learned absolute: simplest possible, matches sinusoids in quality at trained lengths
- +Relative (Shaw, T5): invariant to shifting the whole sequence, natural for cached segments, any length
- +T5 buckets: a few hundred parameters in total, coarse far away where precision matters least
Avoid when
- −Absolute schemes extrapolate poorly past the training length; a learned table cannot go past it at all
- −Added encodings share the residual stream with content, and the score's cross terms are not purely relative
- −Shaw et al. add , inside attention, which costs memory proportional to unless computed cleverly
- −Bias-style relative schemes need a modified attention kernel, which can be slower than plain attention
Check yourself
You remove all positional information from a Transformer encoder and feed it a permuted sentence. What happens to the outputs?
Check yourself
With sinusoidal encodings and d = 4 (ω = 1 and 0.01), which pair of positions has the same encoding dot product as positions 0 and 2?
Remember
- ✓Sinusoidal: , , ; wavelengths from to about .
- ✓Self-attention is permutation-equivariant, so order must be injected: absolute (add ) or relative (score depends on ).
- ✓ with a fixed block rotation, and depends only on .
- ✓Learned tables (BERT 512, GPT-2 1024) match sinusoids at trained lengths but cannot go beyond them.
- ✓Relative schemes: Shaw et al. learn clipped offset vectors on keys and values; T5 adds one scalar per head per log-spaced bucket (32 buckets, up to 128).