Definition
ALiBi (Attention with Linear Biases) is a position method for causal Transformers that adds nothing to the token embeddings. Instead, every head subtracts a penalty proportional to the distance from each attention logit: for . The slopes are fixed, not learned. For heads they form the geometric sequence , . Steep heads look only a few tokens back; shallow heads see far.
Ofir Press, Noah A. Smith and Mike Lewis asked a narrow question in "Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation" (ICLR 2022): can a language model trained on sequences of length be evaluated on longer ones? With sinusoidal encodings (see Positional encoding) the answer was essentially no: perplexity rose sharply once inputs ran a few dozen tokens past . Rotary positional embeddings (RoPE) extrapolated a little further, and T5's relative bias further still, but it trained more slowly. ALiBi was designed to extrapolate by construction and to cost nothing. Its headline result: a 1.3B-parameter model trained on and evaluated on 2048 tokens matched the perplexity of a sinusoidal model trained on 2048, while training 11% faster and using 11% less memory.
The bias
- the content scores of query against keys (causal), as in Scaled dot-product attention
- the distance to key , negated: the current token gets 0, the previous one , and so on
- head-specific slope, a fixed constant. For 8 heads: . For 16 heads:
- number of heads. If is not a power of 2, take the slopes and fill up with every other slope of the sequence for twice that many heads
After the softmax the bias becomes a factor: key 's weight is multiplied by before normalising. A head with halves a key's weight every positions, so it is close to a "last few tokens" head. A head with halves it only every 177 positions. Content can overrule the penalty, but only by a logit margin larger than . The set of slopes therefore gives the model a range of soft windows, from very local to nearly global, chosen without any training.
The bias is added in every layer, and the input embeddings carry no position. Because the softmax is invariant to adding a constant to a row, can be replaced by for every row. That is one tensor that broadcasts over queries and can be folded into the causal mask. The extra compute is negligible, and fused kernels accept the slopes directly (for example the alibi_slopes argument of the flash-attn package).
Why it extrapolates
With learned or sinusoidal embeddings, positions beyond are new inputs: either rows that do not exist or vectors the network never saw. With ALiBi a longer input only adds keys at larger distances, and their bias continues the same straight line. For most heads those keys already had negligible weight at distance . Nothing the model computes falls outside its training distribution, it just has more almost-ignored history. The flip side is that ALiBi does not make a model use far context it was not trained on. Its perplexity at or more stays flat instead of exploding, mostly because it behaves like a soft sliding window. Retrieving a specific fact from far back still requires a head with a small enough slope and a large enough content score.
Worked example: one query, two heads
Query at position 4 (0-based) attends to keys 0–4 with content scores . Compare head 1 of 8 () with head 8 ().
| key j | distance 4 − j | content | bias, m = 1/2 | weight, m = 1/2 | weight, m = 1/256 | weight, no bias |
|---|---|---|---|---|---|---|
| 0 | 4 | 0.2 | −2.0 | 0.0525 | 0.1631 | 0.1643 |
| 1 | 3 | 1.0 | −1.5 | 0.1926 | 0.3645 | 0.3656 |
| 2 | 2 | −0.3 | −1.0 | 0.0865 | 0.0997 | 0.0996 |
| 3 | 1 | 0.5 | −0.5 | 0.3175 | 0.2228 | 0.2218 |
| 4 | 0 | 0.1 | 0.0 | 0.3509 | 0.1499 | 0.1487 |
What the slopes do
- 1Head 1 adds : logits . Softmax puts 0.668 on the two most recent keys, and the best content match (key 1, score 1.0) drops from 0.366 to 0.193.
- 2Head 8 adds at most . Its weights are within 0.002 of plain content attention: key 1 keeps 0.365.
- 3Neither head has a parameter for position 4 or for distance 4. Run the same query at position 4000 with the same content, and head 8 still sees a tiny linear tilt while head 1 still looks at the last few tokens.
- 4With 8 heads in a layer, the model gets both behaviours at once, plus six intermediate windows ( from 1.4 to 177 tokens).
From the paper and later use
- Slopes
- : 8 heads ; 16 heads ; not learned (trainable slopes did not extrapolate well in the paper)
- Placement
- every layer, added to before the softmax; no position embeddings at the input
- Headline result
- 1.3B model trained at , evaluated at 2048: same perplexity as a sinusoidal model trained at 2048, 11% faster training, 11% less memory
- Benchmarks
- WikiText-103 and the Toronto BookCorpus; a larger setup on the RoBERTa/CC-100 training corpus for the 1.3B model
- Adopted by
- BLOOM (BigScience, 176B) and the MPT models from MosaicML
1import math, torch23def alibi_slopes(n_heads):4 def pow2(n): # n a power of 25 start = 2 ** (-2 ** -(math.log2(n) - 3)) # = 2^(-8/n)6 return [start * start ** i for i in range(n)]7 if math.log2(n_heads).is_integer():8 return pow2(n_heads)9 c = 2 ** math.floor(math.log2(n_heads))10 return pow2(c) + alibi_slopes(2 * c)[0::2][: n_heads - c]1112def alibi_bias(n_heads, T):13 m = torch.tensor(alibi_slopes(n_heads)) # (H,)14 i = torch.arange(T)[:, None]15 j = torch.arange(T)[None, :]16 bias = -m[:, None, None] * (i - j).clamp(min=0) # (H, T, T)17 causal = torch.ones(T, T, dtype=torch.bool).tril()18 return bias.masked_fill(~causal, float("-inf")) # (H, T, T)1920def alibi_attention(q, k, v): # (B, H, T, Dh) each21 B, H, T, Dh = q.shape22 logits = q @ k.transpose(-1, -2) / math.sqrt(Dh) + alibi_bias(H, T) # (B, H, T, T)23 return logits.softmax(-1) @ v # (B, H, T, Dh)2425print([round(s, 5) for s in alibi_slopes(8)])26# [0.5, 0.25, 0.125, 0.0625, 0.03125, 0.01562, 0.00781, 0.00391]27print([round(s, 4) for s in alibi_slopes(12)][8:]) # [0.7071, 0.3536, 0.1768, 0.0884]2829q, k, v = (torch.randn(2, 8, 2048, 64) for _ in range(3)) # any length: no tables to outgrow30print(alibi_attention(q, k, v).shape) # torch.Size([2, 8, 2048, 64])Use when
- +No parameters and no position embeddings; any sequence length works
- +Trains at short L and evaluates at longer L without perplexity blow-up; cheaper training for a given evaluation length
- +Implementation is one bias tensor (or folded into the mask); supported by fused attention kernels
- +Gives heads a built-in range of receptive fields
Avoid when
- −Strong recency prior: far tokens are penalised linearly and can be retrieved only through large content scores
- −Flat perplexity beyond L does not mean the extra context is used
- −Designed for causal attention; bidirectional use needs changes
- −Most recent large models chose RoPE with frequency rescaling for long context instead
Check yourself
A model has 8 ALiBi heads. What is the slope of head 3, and after how many tokens does its bias halve a key's weight?
Check yourself
Why can an ALiBi model trained on 1024 tokens run on 2048 tokens without a perplexity explosion, unlike a model with learned absolute position embeddings?
Remember
- ✓ALiBi logit: for ; no position embeddings anywhere.
- ✓Slopes : for 8 heads; a key's weight is scaled by , halving every tokens.
- ✓Press, Smith and Lewis (ICLR 2022): 1.3B model trained at 1024 matched a sinusoidal model trained at 2048 when evaluated at 2048, 11% faster and with 11% less memory.
- ✓Implementation: one bias folded into the causal mask (softmax is row-shift invariant).
- ✓Extrapolation means no breakdown, not use of distant context: steep heads stay local at any length.