# Positional encoding

> Self-attention ignores token order, so position must be injected. Absolute schemes add a vector per position (fixed sinusoids or a learned table); relative schemes make each attention score depend on the offset j − i (Shaw et al., T5 buckets, RoPE, ALiBi).

Source: https://www.mlfoundry.in/learn/positional-encoding  
Section: Deep Learning › Attention and Transformer Architectures  
Level: core · 14 min read · updated 2026-10-12

**Positional encoding:** is any mechanism that tells a Transformer where each token sits in the sequence. **Absolute** encodings give position $t$ a vector $p_t\in\mathbb R^{d}$ that is added to the token embedding: either fixed sinusoids, $p_{t,2i}=\sin(t\,\omega_i)$ and $p_{t,2i+1}=\cos(t\,\omega_i)$ with $\omega_i=10000^{-2i/d}$ (Vaswani et al., 2017), or a learned table with one row per position. **Relative** encodings leave the token vectors alone and change the attention score between query $i$ and key $j$ as a function of the offset $j-i$.

Self-attention is a weighted average whose weights come from dot products (see **Self-attention**). Nothing in it refers to an index. Permute the input tokens and an encoder layer returns the same output vectors, permuted the same way: the layer is **permutation-equivariant**. For language this is fatal, because "dog bites man" and "man bites dog" contain the same bag of tokens. Recurrent and convolutional networks get order for free from their structure; a Transformer has to be told. (A causal mask breaks the symmetry a little, because token $t$ sees exactly $t+1$ tokens, and decoder-only models without any encoding can still infer some position from it, as Haviv et al. showed in 2022. Every widely used model still adds an explicit scheme.)

Vaswani et al. introduced sinusoidal encodings in "Attention Is All You Need" (NeurIPS 2017). ConvS2S (Gehring et al., 2017) had used learned position embeddings, and Vaswani et al. tried those too: on English→German development BLEU the two were almost identical (25.7 learned vs 25.8 sinusoidal). They kept the sinusoids because they might extrapolate to sequences longer than those seen in training. BERT and GPT-2 went the learned route. Shaw, Uszkoreit and Vaswani (NAACL 2018) then moved position into the attention score itself, which started the line of **relative** methods that ends, for now, in **Rotary positional embeddings (RoPE)** and **ALiBi**.

## Sinusoidal encoding

$$
p_{t,2i}=\sin(t\,\omega_i),\qquad p_{t,2i+1}=\cos(t\,\omega_i),\qquad \omega_i=10000^{-2i/d},\quad i=0,\dots,\tfrac d2-1,\qquad x_t=\sqrt d\;E[\text{token}_t]+p_t
$$

- $t$: position index, 0, 1, 2, …
- $d$: model width $d_{\text{model}}$ (512 in the base Transformer); pair $i$ occupies dimensions $2i$ and $2i+1$
- $\omega_i$: angular frequency of pair $i$. Wavelengths $2\pi/\omega_i$ form a geometric sequence from $2\pi$ (pair 0) to $2\pi\cdot10000^{(d-2)/d}$, about $60{,}600$ positions for $d=512$
- $E$: token embedding matrix; the paper multiplies it by $\sqrt d$ before adding $p_t$, so the encoding does not swamp the token signal

Read each pair as a clock hand. Pair 0 turns one radian per position and repeats every 6.28 tokens. Pair $d/2-1$ turns so slowly that over a few thousand positions it never completes a revolution. Together the hands work like the digits of a number in a mixed radix: fast pairs separate neighbours, slow pairs separate distant positions. Every value lies in $[-1,1]$, and $\lVert p_t\rVert^2=d/2$ for every $t$.

$$
\begin{pmatrix}p_{t+k,2i}\\ p_{t+k,2i+1}\end{pmatrix}=\begin{pmatrix}\cos k\omega_i & \sin k\omega_i\\ -\sin k\omega_i & \cos k\omega_i\end{pmatrix}\begin{pmatrix}p_{t,2i}\\ p_{t,2i+1}\end{pmatrix},\qquad p_t^\top p_{t+k}=\sum_{i}\cos(k\,\omega_i)
$$

- $M_k$: the block-diagonal matrix of these $2\times2$ rotations; $p_{t+k}=M_kp_t$ with $M_k$ independent of $t$. This is the "attend by relative positions via a linear function" argument of the paper
- $p_t^\top p_{t+k}$: depends only on the offset $k$, is symmetric in $\pm k$, and equals $d/2$ at $k=0$

The caveat is that the model never sees $p_t$ alone. It sees $x_t=\sqrt d\,E[\cdot]+p_t$, and the attention score $(W_Qx_i)^\top(W_Kx_j)$ expands into content–content, content–position, position–content and position–position terms. Only the last is a clean function of $j-i$, and only if $W_Q^\top W_K$ preserves the rotation structure, which nothing enforces. In practice sinusoidal models do **not** extrapolate well: Press et al. (2022) found perplexity rising sharply within a few dozen tokens past the training length.

> Figure: The 48 × 32 sinusoidal matrix for d = 32 and base 10000: columns are positions, rows are dimensions, blue is +1 and red −1. The top rows flip sign every few positions; the bottom rows barely change. Below it, the curve is PE(p)·PE(q)/(d/2) for p = 10 against every q. It peaks at 1 for q = p, and the nearest neighbour (q = 9) scores 0.957. The default offset k = 5 gives sim(10, 15) = 0.736, exactly sim(0, 5). Drag p and the curve slides without changing shape: the similarity depends only on p − q. The dial shows pair i = 3 (ω₃ = 0.1778, wavelength 35.3 positions) as a point on the unit circle. Moving from p to p + k turns it by kω₃ = 0.889 rad whatever p is, and applying that fixed rotation Mₖ to PE(p) reproduces PE(p + k) to 8·10⁻¹⁶. Switch the base to 100 and the slow pairs speed up: sim(10, 30) falls from 0.654 to 0.183 and the curve dips to −0.009. With d = 16 and base 100 it reaches −0.247.

## Learned absolute embeddings

The simplest alternative is a trainable matrix $P\in\mathbb R^{L_{\max}\times d}$, with row $t$ added to token $t$. BERT uses $L_{\max}=512$, GPT-2 1024, and **Vision Transformers (ViT)** learn one embedding per patch slot. The table can learn whatever the task needs, and in the 2017 comparison it cost nothing in quality. Its hard limit is $L_{\max}$: position 1024 in GPT-2 has no row at all. Rows for rarely seen positions are also poorly trained, because only long training sequences ever reach them. ViT works around the limit at a new resolution by interpolating the 2-D grid of embeddings, which is a form of the same idea that **Rotary positional embeddings (RoPE)** later use to extend context.

## Relative positions in the score

$$
e_{ij}=\frac{(x_iW^Q)(x_jW^K+a^K_{ij})^\top}{\sqrt{d_z}},\qquad z_i=\sum_j\alpha_{ij}\,(x_jW^V+a^V_{ij}),\qquad a^K_{ij}=w^K_{\mathrm{clip}(j-i,\,k)},\ \ a^V_{ij}=w^V_{\mathrm{clip}(j-i,\,k)}
$$

- $w^K_r,\ w^V_r$: learned vectors for each clipped offset $r\in\{-k,\dots,k\}$, so $2k+1$ of each, shared across heads
- $\mathrm{clip}(r,k)$: $\max(-k,\min(k,r))$: all offsets beyond $\pm k$ share one vector, which lets the model run on any length
- $\alpha_{ij}$: softmax of $e_{ij}$ over $j$, as in **Scaled dot-product attention**

Shaw et al. replaced absolute encodings in the Transformer with these offset vectors. On WMT'14 they reported gains of 1.3 BLEU on English→German and 0.3 BLEU on English→French over absolute positions, and adding sinusoids back on top did not help further. **Transformer-XL** (Dai et al., 2019) needed relative positions for a different reason: it reuses cached states from the previous segment, and absolute indices would collide. It uses a sinusoidal function of $j-i$ plus two global bias vectors.

T5 (Raffel et al., JMLR 2020) reduced the idea to its minimum. Each head adds a single learned **scalar** $b_h(\text{bucket}(j-i))$ to its logit, and the 32 buckets are shared by all layers. Small offsets get their own bucket; larger ones are grouped logarithmically up to 128, beyond which all offsets share the last bucket. In the encoder, 16 buckets serve keys before the query and 16 serve keys after it; in the decoder all 32 cover the past. **ALiBi** goes one step further and fixes the scalar to $-m_h\,(i-j)$ with no parameters at all.

T5 relative-position buckets (32 buckets, max distance 128), computed with the reference bucketing function

| key − query | encoder (bidirectional) bucket | decoder (causal) bucket |
| --- | --- | --- |
| 0 | 0 | 0 |
| −3 | 3 | 3 |
| +3 | 19 | 0 (future, masked) |
| −10 | 8 (shared by −8 … −11) | 10 |
| −20 | 10 (−16 … −22) | 17 |
| −50 | 13 (−46 … −63) | 24 |
| −200 | 15 (all ≤ −91) | 31 |

## Worked example: d = 4

With $d=4$ there are two pairs, $\omega_0=10000^{0}=1$ and $\omega_1=10000^{-2/4}=0.01$. So $p_t=(\sin t,\ \cos t,\ \sin 0.01t,\ \cos 0.01t)$.

Sinusoidal encodings for positions 0–3 (computed in Node)

| t | sin t | cos t | sin 0.01t | cos 0.01t |
| --- | --- | --- | --- | --- |
| 0 | 0.0000 | 1.0000 | 0.0000 | 1.0000 |
| 1 | 0.8415 | 0.5403 | 0.0100 | 1.0000 |
| 2 | 0.9093 | −0.4161 | 0.0200 | 0.9998 |
| 3 | 0.1411 | −0.9900 | 0.0300 | 0.9996 |

**Shift by a rotation, and the offset-only dot product**

1. $M_2$ rotates pair 0 by $2\omega_0=2$ rad and pair 1 by $2\omega_1=0.02$ rad. Applied to $p_1$: $\sin1\cos2+\cos1\sin2=0.1411$ and $\cos1\cos2-\sin1\sin2=-0.9900$, then $(0.0300,\ 0.9996)$ for pair 1. That is $p_3$ exactly, and the same $M_2$ maps every $p_t$ to $p_{t+2}$.
2. $p_1^\top p_3=0.8415\cdot0.1411+0.5403\cdot(-0.9900)+0.0100\cdot0.0300+1.0000\cdot0.9996=0.5837$.
3. $p_0^\top p_2=\cos2+\cos0.02=-0.4161+0.9998=0.5837$. Same offset, same value.
4. Offset 1 in both directions: $p_1^\top p_2=p_1^\top p_0=\cos1+\cos0.01=1.5403$. The plain dot product cannot tell "one before" from "one after"; the score only becomes direction-aware through the learned $W_Q,W_K$.

**Choices in well-known models**

- **Transformer (2017):** sinusoidal, base 10000, added after scaling embeddings by $\sqrt{d}$; learned embeddings gave 25.7 vs 25.8 BLEU (dev, En→De)
- **BERT, GPT-2:** learned absolute tables of 512 and 1024 positions
- **Shaw et al. (2018):** learned offset vectors on keys and values, clipped at distance $k$; +1.3 BLEU En→De, +0.3 En→Fr over absolute
- **T5 (2020):** one scalar bias per head per bucket, 32 buckets up to distance 128, shared across layers
- **LLaMA, PaLM, GPT-NeoX:** **Rotary positional embeddings (RoPE)**: rotate queries and keys, nothing added to the residual stream
- **BLOOM, MPT:** **ALiBi**: a fixed linear distance penalty per head

```python
import math, torch, torch.nn as nn

def sinusoidal_pe(n_pos, d_model, base=10000.0):
    pos = torch.arange(n_pos, dtype=torch.float32)[:, None]        # (n_pos, 1)
    two_i = torch.arange(0, d_model, 2, dtype=torch.float32)        # (d_model/2,)
    omega = base ** (-two_i / d_model)                              # (d_model/2,)
    pe = torch.zeros(n_pos, d_model)
    pe[:, 0::2] = torch.sin(pos * omega)                            # even dims: sin
    pe[:, 1::2] = torch.cos(pos * omega)                            # odd dims:  cos
    return pe                                                       # (n_pos, d_model)

class TokenAndPosition(nn.Module):
    def __init__(self, vocab, d_model, max_len=2048, learned=False):
        super().__init__()
        self.tok, self.d, self.learned = nn.Embedding(vocab, d_model), d_model, learned
        if learned:
            self.pos = nn.Embedding(max_len, d_model)               # BERT / GPT-2 style
        else:
            self.register_buffer("pe", sinusoidal_pe(max_len, d_model), persistent=False)

    def forward(self, ids):                                         # ids: (B, T)
        T = ids.shape[1]
        x = self.tok(ids) * math.sqrt(self.d)                       # (B, T, d)
        p = self.pos(torch.arange(T, device=ids.device)) if self.learned else self.pe[:T]
        return x + p                                                # (B, T, d)

def t5_bucket(rel, bidirectional=True, num_buckets=32, max_distance=128):
    """rel = key position - query position (int64 tensor)."""
    out = torch.zeros_like(rel)
    if bidirectional:
        num_buckets //= 2
        out += (rel > 0).long() * num_buckets                       # keys after the query
        n = rel.abs()
    else:
        n = (-rel).clamp(min=0)                                     # only the past counts
    exact = num_buckets // 2
    large = exact + (torch.log(n.float() / exact) / math.log(max_distance / exact)
                     * (num_buckets - exact)).long()
    large = large.clamp(max=num_buckets - 1)
    return out + torch.where(n < exact, n, large)

pe = sinusoidal_pe(50, 4)
print(pe[1] @ pe[3], pe[0] @ pe[2])        # tensor(0.5837) tensor(0.5837)
emb = TokenAndPosition(vocab=1000, d_model=64)
print(emb(torch.randint(0, 1000, (2, 10))).shape)   # torch.Size([2, 10, 64])

rel = torch.tensor([-200, -50, -20, -10, -3, 0, 3, 10, 50])
print(t5_bucket(rel))          # tensor([15, 13, 10,  8,  3,  0, 19, 24, 29])
print(t5_bucket(rel, False))   # tensor([31, 24, 17, 10,  3,  0,  0,  0,  0])
```

Sinusoidal and learned absolute encodings, and T5's relative-position buckets

> **Interleaved and concatenated layouts are not interchangeable:** The paper interleaves sin and cos (even and odd dimensions). Many codebases instead concatenate all sines and then all cosines. The two are the same set of features in a different order, so training from scratch with either is fine. Loading weights trained with one layout into code that uses the other silently scrambles every position. The same trap exists for RoPE's "interleaved" versus "rotate-half" layouts.

> **From adding a rotation to applying one:** Sinusoids make $p_{t+k}$ a rotation of $p_t$, but adding $p_t$ to the token vector mixes position into content, and the cross terms of the attention score spoil the relative structure. **Rotary positional embeddings (RoPE)** keep the rotation and drop the addition: they rotate the query and key themselves by their positions, so the score is an exact function of content and $j-i$.

> **Absolute does not mean useless for relative patterns:** Learned absolute tables in trained models often look smooth and partly sinusoidal, and attention heads learn offset patterns such as "previous token" from them. The practical differences are elsewhere: whether the scheme has a maximum length, whether it works on lengths not seen in training, and whether it costs anything inside attention.

**Strengths**

- Sinusoidal: no parameters, any length can be computed, $p_{t+k}=M_kp_t$
- Learned absolute: simplest possible, matches sinusoids in quality at trained lengths
- Relative (Shaw, T5): invariant to shifting the whole sequence, natural for cached segments, any length
- T5 buckets: a few hundred parameters in total, coarse far away where precision matters least

**Limitations**

- Absolute schemes extrapolate poorly past the training length; a learned table cannot go past it at all
- Added encodings share the residual stream with content, and the score's cross terms are not purely relative
- Shaw et al. add $a^K_{ij}$, $a^V_{ij}$ inside attention, which costs memory proportional to $T^2$ unless computed cleverly
- Bias-style relative schemes need a modified attention kernel, which can be slower than plain attention

**Question:** You remove all positional information from a Transformer encoder and feed it a permuted sentence. What happens to the outputs?

- A. They are the original outputs, permuted the same way
- B. They are unchanged, in the original order
- C. They change unpredictably because attention is non-linear
- D. The model raises an error because positions are missing

**Answer:** A. They are the original outputs, permuted the same way — Each output is a function of its own token and of the multiset of all tokens. Permuting the inputs permutes the rows of $Q$, $K$, $V$, so the attention matrix is permuted on both axes and the outputs come back permuted: the layer is permutation-equivariant. Pooled outputs would not change at all, which is why order must be injected.

**Question:** With sinusoidal encodings and d = 4 (ω = 1 and 0.01), which pair of positions has the same encoding dot product as positions 0 and 2?

- A. 1 and 3
- B. 0 and 3
- C. 2 and 2
- D. 1 and 2

**Answer:** A. 1 and 3 — $p_t^\top p_{t+k}=\sum_i\cos(k\omega_i)$ depends only on the offset $k$. Positions 1 and 3 have offset 2, like 0 and 2, so both give $\cos2+\cos0.02=0.5837$. Offset 1 gives 1.5403, offset 0 gives 2.

**Key points**

- Sinusoidal: $p_{t,2i}=\sin(t\,\omega_i)$, $p_{t,2i+1}=\cos(t\,\omega_i)$, $\omega_i=10000^{-2i/d}$; wavelengths from $2\pi$ to about $2\pi\cdot10^4$.
- Self-attention is permutation-equivariant, so order must be injected: absolute (add $p_t$) or relative (score depends on $j-i$).
- $p_{t+k}=M_kp_t$ with $M_k$ a fixed block rotation, and $p_t^\top p_{t+k}=\sum_i\cos(k\omega_i)$ depends only on $k$.
- Learned tables (BERT 512, GPT-2 1024) match sinusoids at trained lengths but cannot go beyond them.
- Relative schemes: Shaw et al. learn clipped offset vectors on keys and values; T5 adds one scalar per head per log-spaced bucket (32 buckets, up to 128).

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
