ALiBi

ALiBi (Press, Smith and Lewis, 2022) uses no position embeddings. Each head h adds −mₕ·(i − j) to the logit of query i and key j, with fixed slopes in a geometric sequence (1/2, 1/4, …, 1/256 for 8 heads). This lets a model trained on short sequences run on longer ones.

12 min read882 wordscore0 viewsupdated 2026-10-12

Definition

ALiBi (Attention with Linear Biases) is a position method for causal Transformers that adds nothing to the token embeddings. Instead, every head hh subtracts a penalty proportional to the distance from each attention logit: softmax(qi⊤kj/d−mh (i−j))\mathrm{softmax}\big(q_i^\top k_j/\sqrt{d}-m_h\,(i-j)\big) for j≤ij\le i. The slopes mhm_h are fixed, not learned. For nn heads they form the geometric sequence mh=2−8h/nm_h=2^{-8h/n}, h=1,…,nh=1,\dots,n. Steep heads look only a few tokens back; shallow heads see far.

Ofir Press, Noah A. Smith and Mike Lewis asked a narrow question in "Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation" (ICLR 2022): can a language model trained on sequences of length LL be evaluated on longer ones? With sinusoidal encodings (see Positional encoding) the answer was essentially no: perplexity rose sharply once inputs ran a few dozen tokens past LL. Rotary positional embeddings (RoPE) extrapolated a little further, and T5's relative bias further still, but it trained more slowly. ALiBi was designed to extrapolate by construction and to cost nothing. Its headline result: a 1.3B-parameter model trained on L=1024L=1024 and evaluated on 2048 tokens matched the perplexity of a sinusoidal model trained on 2048, while training 11% faster and using 11% less memory.

The bias

softmax(qi⊤K⊤d+mh⋅[−(i−1), …, −2, −1, 0]),mh=2−8h/n,h=1,…,n\mathrm{softmax}\Big(\frac{q_i^\top K^\top}{\sqrt d}+m_h\cdot\big[-(i-1),\ \dots,\ -2,\ -1,\ 0\big]\Big),\qquad m_h=2^{-8h/n},\quad h=1,\dots,n
qi⊤K⊤q_i^\top K^\top
the content scores of query ii against keys 1,…,i1,\dots,i (causal), as in Scaled dot-product attention
−(i−j)-(i-j)
the distance to key jj, negated: the current token gets 0, the previous one −1-1, and so on
mhm_h
head-specific slope, a fixed constant. For 8 heads: 12,14,…,1256\tfrac12,\tfrac14,\dots,\tfrac1{256}. For 16 heads: 2−0.5,2−1,…,2−82^{-0.5},2^{-1},\dots,2^{-8}
nn
number of heads. If nn is not a power of 2, take the 2⌊log⁡2n⌋2^{\lfloor\log_2n\rfloor} slopes and fill up with every other slope of the sequence for twice that many heads

After the softmax the bias becomes a factor: key jj's weight is multiplied by e−mh(i−j)e^{-m_h(i-j)} before normalising. A head with m=12m=\tfrac12 halves a key's weight every ln⁡2/m=1.4\ln2/m=1.4 positions, so it is close to a "last few tokens" head. A head with m=1256m=\tfrac1{256} halves it only every 177 positions. Content can overrule the penalty, but only by a logit margin larger than m Δm\,\Delta. The set of slopes therefore gives the model a range of soft windows, from very local to nearly global, chosen without any training.

The bias is added in every layer, and the input embeddings carry no position. Because the softmax is invariant to adding a constant to a row, −m(i−j)-m(i-j) can be replaced by +m j+m\,j for every row. That is one (n×1×T)(n\times1\times T) tensor that broadcasts over queries and can be folded into the causal mask. The extra compute is negligible, and fused kernels accept the slopes directly (for example the alibi_slopes argument of the flash-attn package).

Why it extrapolates

With learned or sinusoidal embeddings, positions beyond LL are new inputs: either rows that do not exist or vectors the network never saw. With ALiBi a longer input only adds keys at larger distances, and their bias continues the same straight line. For most heads those keys already had negligible weight at distance LL. Nothing the model computes falls outside its training distribution, it just has more almost-ignored history. The flip side is that ALiBi does not make a model use far context it was not trained on. Its perplexity at 2L2L or more stays flat instead of exploding, mostly because it behaves like a soft sliding window. Retrieving a specific fact from far back still requires a head with a small enough slope and a large enough content score.

logitₕ(i, j) = qᵢ·kⱼ/√d − mₕ·(i − j), j ≤ i · no position embeddings · L_train = 16head 4 weights, query i ↓ key j →┅ distance 16: farther keys (below-left) were never seen in traininghslope mln2/mmass at Δ ≥ 16needle 10.50001.40.0000.000 20.25002.80.0180.002 30.12505.50.1290.052 40.062511.10.3110.167 50.031322.20.4490.254 60.015644.40.5240.299 70.0078188.70.5620.320 80.00391177.40.5810.330mass: last query, no content · click a headquery 39, head 4: weight on each key · ■ needle (+3.0 content) at distance 30 · — bias e^(−mΔ)key 0key 39 (Δ = 0) · bars scaled to max 0.167head 4: m = 0.0625 · bias halves weight every 11.1 · L = 40 = 2.50× L_train · needle 0.167
heads n
fig.Eight heads with ALiBi slopes 1/2 … 1/256, trained length L_train = 16, evaluated on L = 40 (2.5×). Content logits are fixed noise (std 0.5) plus a 'needle' key 30 tokens back with +3 extra score. The heat map shows head 4 (m = 0.0625, weight halves every 11.1 tokens). Every query's weights fade along a band near the diagonal, and the needle column stands out where the penalty allows it. The red dashed diagonal marks distance 16: keys below it are farther away than anything seen in training. The table gives, for the last query with content ignored, the attention mass beyond distance 16. It is 0.000 for head 1 and 0.018 for head 2, but 0.311 for head 4 and 0.581 for head 8. It also gives the weight on the needle: 0.000, 0.002, 0.052, 0.167 … 0.330 from steep to shallow heads. Only the shallow heads can pick up a fact 30 tokens away. Raise L to 64 and head 1's far mass is still 0.0003, which is why a steep head behaves identically at any length. Switch to 12 heads to see the non-power-of-2 rule: the 8 slopes 1/2 … 1/256, then 0.7071, 0.3536, 0.1768, 0.0884 from the 16-head sequence.

Worked example: one query, two heads

Query at position 4 (0-based) attends to keys 0–4 with content scores q4⊤kj/d=(0.2, 1.0, −0.3, 0.5, 0.1)q_4^\top k_j/\sqrt d=(0.2,\ 1.0,\ -0.3,\ 0.5,\ 0.1). Compare head 1 of 8 (m=12m=\tfrac12) with head 8 (m=1256m=\tfrac1{256}).

key jdistance 4 − jcontentbias, m = 1/2weight, m = 1/2weight, m = 1/256weight, no bias
040.2−2.00.05250.16310.1643
131.0−1.50.19260.36450.3656
22−0.3−1.00.08650.09970.0996
310.5−0.50.31750.22280.2218
400.10.00.35090.14990.1487
ALiBi logits and weights for query 4 (computed in Node)

What the slopes do

  1. 1Head 1 adds −12(4−j)-\tfrac12(4-j): logits (−1.8, −0.5, −1.3, 0.0, 0.1)(-1.8,\ -0.5,\ -1.3,\ 0.0,\ 0.1). Softmax puts 0.668 on the two most recent keys, and the best content match (key 1, score 1.0) drops from 0.366 to 0.193.
  2. 2Head 8 adds at most −4/256=−0.0156-4/256=-0.0156. Its weights are within 0.002 of plain content attention: key 1 keeps 0.365.
  3. 3Neither head has a parameter for position 4 or for distance 4. Run the same query at position 4000 with the same content, and head 8 still sees a tiny linear tilt while head 1 still looks at the last few tokens.
  4. 4With 8 heads in a layer, the model gets both behaviours at once, plus six intermediate windows (ln⁡2/m\ln2/m from 1.4 to 177 tokens).

From the paper and later use

Slopes
mh=2−8h/nm_h=2^{-8h/n}: 8 heads 12…1256\tfrac12\dots\tfrac1{256}; 16 heads 2−0.5…2−82^{-0.5}\dots2^{-8}; not learned (trainable slopes did not extrapolate well in the paper)
Placement
every layer, added to q⊤k/dq^\top k/\sqrt d before the softmax; no position embeddings at the input
Headline result
1.3B model trained at L=1024L=1024, evaluated at 2048: same perplexity as a sinusoidal model trained at 2048, 11% faster training, 11% less memory
Benchmarks
WikiText-103 and the Toronto BookCorpus; a larger setup on the RoBERTa/CC-100 training corpus for the 1.3B model
Adopted by
BLOOM (BigScience, 176B) and the MPT models from MosaicML
python
1import math, torch
2
3def alibi_slopes(n_heads):
4 def pow2(n): # n a power of 2
5 start = 2 ** (-2 ** -(math.log2(n) - 3)) # = 2^(-8/n)
6 return [start * start ** i for i in range(n)]
7 if math.log2(n_heads).is_integer():
8 return pow2(n_heads)
9 c = 2 ** math.floor(math.log2(n_heads))
10 return pow2(c) + alibi_slopes(2 * c)[0::2][: n_heads - c]
11
12def alibi_bias(n_heads, T):
13 m = torch.tensor(alibi_slopes(n_heads)) # (H,)
14 i = torch.arange(T)[:, None]
15 j = torch.arange(T)[None, :]
16 bias = -m[:, None, None] * (i - j).clamp(min=0) # (H, T, T)
17 causal = torch.ones(T, T, dtype=torch.bool).tril()
18 return bias.masked_fill(~causal, float("-inf")) # (H, T, T)
19
20def alibi_attention(q, k, v): # (B, H, T, Dh) each
21 B, H, T, Dh = q.shape
22 logits = q @ k.transpose(-1, -2) / math.sqrt(Dh) + alibi_bias(H, T) # (B, H, T, T)
23 return logits.softmax(-1) @ v # (B, H, T, Dh)
24
25print([round(s, 5) for s in alibi_slopes(8)])
26# [0.5, 0.25, 0.125, 0.0625, 0.03125, 0.01562, 0.00781, 0.00391]
27print([round(s, 4) for s in alibi_slopes(12)][8:]) # [0.7071, 0.3536, 0.1768, 0.0884]
28
29q, k, v = (torch.randn(2, 8, 2048, 64) for _ in range(3)) # any length: no tables to outgrow
30print(alibi_attention(q, k, v).shape) # torch.Size([2, 8, 2048, 64])
ALiBi slopes (as in the authors' code), the bias tensor, and causal attention with it

Use when

  • +No parameters and no position embeddings; any sequence length works
  • +Trains at short L and evaluates at longer L without perplexity blow-up; cheaper training for a given evaluation length
  • +Implementation is one bias tensor (or m jm\,j folded into the mask); supported by fused attention kernels
  • +Gives heads a built-in range of receptive fields

Avoid when

  • −Strong recency prior: far tokens are penalised linearly and can be retrieved only through large content scores
  • −Flat perplexity beyond L does not mean the extra context is used
  • −Designed for causal attention; bidirectional use needs changes
  • −Most recent large models chose RoPE with frequency rescaling for long context instead

Check yourself

A model has 8 ALiBi heads. What is the slope of head 3, and after how many tokens does its bias halve a key's weight?

Check yourself

Why can an ALiBi model trained on 1024 tokens run on 2048 tokens without a perplexity explosion, unlike a model with learned absolute position embeddings?

Remember

  • ✓ALiBi logit: qi⊤kj/d−mh(i−j)q_i^\top k_j/\sqrt d-m_h(i-j) for j≤ij\le i; no position embeddings anywhere.
  • ✓Slopes mh=2−8h/nm_h=2^{-8h/n}: 12,14,…,1256\tfrac12,\tfrac14,\dots,\tfrac1{256} for 8 heads; a key's weight is scaled by e−mh(i−j)e^{-m_h(i-j)}, halving every ln⁡2/mh\ln2/m_h tokens.
  • ✓Press, Smith and Lewis (ICLR 2022): 1.3B model trained at 1024 matched a sinusoidal model trained at 2048 when evaluated at 2048, 11% faster and with 11% less memory.
  • ✓Implementation: one (H,1,T)(H,1,T) bias m jm\,j folded into the causal mask (softmax is row-shift invariant).
  • ✓Extrapolation means no breakdown, not use of distant context: steep heads stay local at any length.
← back to Deep Learning roadmap