# ALiBi

> ALiBi (Press, Smith and Lewis, 2022) uses no position embeddings. Each head h adds −mₕ·(i − j) to the logit of query i and key j, with fixed slopes in a geometric sequence (1/2, 1/4, …, 1/256 for 8 heads). This lets a model trained on short sequences run on longer ones.

Source: https://www.mlfoundry.in/learn/alibi  
Section: Deep Learning › Attention and Transformer Architectures  
Level: core · 12 min read · updated 2026-10-12

**ALiBi (Attention with Linear Biases):** is a position method for causal Transformers that adds nothing to the token embeddings. Instead, every head $h$ subtracts a penalty proportional to the distance from each attention logit: $\mathrm{softmax}\big(q_i^\top k_j/\sqrt{d}-m_h\,(i-j)\big)$ for $j\le i$. The slopes $m_h$ are fixed, not learned. For $n$ heads they form the geometric sequence $m_h=2^{-8h/n}$, $h=1,\dots,n$. Steep heads look only a few tokens back; shallow heads see far.

Ofir Press, Noah A. Smith and Mike Lewis asked a narrow question in "Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation" (ICLR 2022): can a language model trained on sequences of length $L$ be evaluated on longer ones? With sinusoidal encodings (see **Positional encoding**) the answer was essentially no: perplexity rose sharply once inputs ran a few dozen tokens past $L$. **Rotary positional embeddings (RoPE)** extrapolated a little further, and T5's relative bias further still, but it trained more slowly. ALiBi was designed to extrapolate by construction and to cost nothing. Its headline result: a 1.3B-parameter model trained on $L=1024$ and evaluated on 2048 tokens matched the perplexity of a sinusoidal model trained on 2048, while training 11% faster and using 11% less memory.

## The bias

$$
\mathrm{softmax}\Big(\frac{q_i^\top K^\top}{\sqrt d}+m_h\cdot\big[-(i-1),\ \dots,\ -2,\ -1,\ 0\big]\Big),\qquad m_h=2^{-8h/n},\quad h=1,\dots,n
$$

- $q_i^\top K^\top$: the content scores of query $i$ against keys $1,\dots,i$ (causal), as in **Scaled dot-product attention**
- $-(i-j)$: the distance to key $j$, negated: the current token gets 0, the previous one $-1$, and so on
- $m_h$: head-specific slope, a fixed constant. For 8 heads: $\tfrac12,\tfrac14,\dots,\tfrac1{256}$. For 16 heads: $2^{-0.5},2^{-1},\dots,2^{-8}$
- $n$: number of heads. If $n$ is not a power of 2, take the $2^{\lfloor\log_2n\rfloor}$ slopes and fill up with every other slope of the sequence for twice that many heads

After the softmax the bias becomes a factor: key $j$'s weight is multiplied by $e^{-m_h(i-j)}$ before normalising. A head with $m=\tfrac12$ halves a key's weight every $\ln2/m=1.4$ positions, so it is close to a "last few tokens" head. A head with $m=\tfrac1{256}$ halves it only every 177 positions. Content can overrule the penalty, but only by a logit margin larger than $m\,\Delta$. The set of slopes therefore gives the model a range of soft windows, from very local to nearly global, chosen without any training.

The bias is added in every layer, and the input embeddings carry no position. Because the softmax is invariant to adding a constant to a row, $-m(i-j)$ can be replaced by $+m\,j$ for every row. That is one $(n\times1\times T)$ tensor that broadcasts over queries and can be folded into the causal mask. The extra compute is negligible, and fused kernels accept the slopes directly (for example the `alibi_slopes` argument of the `flash-attn` package).

## Why it extrapolates

With learned or sinusoidal embeddings, positions beyond $L$ are new inputs: either rows that do not exist or vectors the network never saw. With ALiBi a longer input only adds keys at larger distances, and their bias continues the same straight line. For most heads those keys already had negligible weight at distance $L$. Nothing the model computes falls outside its training distribution, it just has more almost-ignored history. The flip side is that ALiBi does not make a model **use** far context it was not trained on. Its perplexity at $2L$ or more stays flat instead of exploding, mostly because it behaves like a soft sliding window. Retrieving a specific fact from far back still requires a head with a small enough slope and a large enough content score.

> Figure: Eight heads with ALiBi slopes 1/2 … 1/256, trained length L_train = 16, evaluated on L = 40 (2.5×). Content logits are fixed noise (std 0.5) plus a 'needle' key 30 tokens back with +3 extra score. The heat map shows head 4 (m = 0.0625, weight halves every 11.1 tokens). Every query's weights fade along a band near the diagonal, and the needle column stands out where the penalty allows it. The red dashed diagonal marks distance 16: keys below it are farther away than anything seen in training. The table gives, for the last query with content ignored, the attention mass beyond distance 16. It is 0.000 for head 1 and 0.018 for head 2, but 0.311 for head 4 and 0.581 for head 8. It also gives the weight on the needle: 0.000, 0.002, 0.052, 0.167 … 0.330 from steep to shallow heads. Only the shallow heads can pick up a fact 30 tokens away. Raise L to 64 and head 1's far mass is still 0.0003, which is why a steep head behaves identically at any length. Switch to 12 heads to see the non-power-of-2 rule: the 8 slopes 1/2 … 1/256, then 0.7071, 0.3536, 0.1768, 0.0884 from the 16-head sequence.

## Worked example: one query, two heads

Query at position 4 (0-based) attends to keys 0–4 with content scores $q_4^\top k_j/\sqrt d=(0.2,\ 1.0,\ -0.3,\ 0.5,\ 0.1)$. Compare head 1 of 8 ($m=\tfrac12$) with head 8 ($m=\tfrac1{256}$).

ALiBi logits and weights for query 4 (computed in Node)

| key j | distance 4 − j | content | bias, m = 1/2 | weight, m = 1/2 | weight, m = 1/256 | weight, no bias |
| --- | --- | --- | --- | --- | --- | --- |
| 0 | 4 | 0.2 | −2.0 | 0.0525 | 0.1631 | 0.1643 |
| 1 | 3 | 1.0 | −1.5 | 0.1926 | 0.3645 | 0.3656 |
| 2 | 2 | −0.3 | −1.0 | 0.0865 | 0.0997 | 0.0996 |
| 3 | 1 | 0.5 | −0.5 | 0.3175 | 0.2228 | 0.2218 |
| 4 | 0 | 0.1 | 0.0 | 0.3509 | 0.1499 | 0.1487 |

**What the slopes do**

1. Head 1 adds $-\tfrac12(4-j)$: logits $(-1.8,\ -0.5,\ -1.3,\ 0.0,\ 0.1)$. Softmax puts 0.668 on the two most recent keys, and the best content match (key 1, score 1.0) drops from 0.366 to 0.193.
2. Head 8 adds at most $-4/256=-0.0156$. Its weights are within 0.002 of plain content attention: key 1 keeps 0.365.
3. Neither head has a parameter for position 4 or for distance 4. Run the same query at position 4000 with the same content, and head 8 still sees a tiny linear tilt while head 1 still looks at the last few tokens.
4. With 8 heads in a layer, the model gets both behaviours at once, plus six intermediate windows ($\ln2/m$ from 1.4 to 177 tokens).

**From the paper and later use**

- **Slopes:** $m_h=2^{-8h/n}$: 8 heads $\tfrac12\dots\tfrac1{256}$; 16 heads $2^{-0.5}\dots2^{-8}$; not learned (trainable slopes did not extrapolate well in the paper)
- **Placement:** every layer, added to $q^\top k/\sqrt d$ before the softmax; no position embeddings at the input
- **Headline result:** 1.3B model trained at $L=1024$, evaluated at 2048: same perplexity as a sinusoidal model trained at 2048, 11% faster training, 11% less memory
- **Benchmarks:** WikiText-103 and the Toronto BookCorpus; a larger setup on the RoBERTa/CC-100 training corpus for the 1.3B model
- **Adopted by:** BLOOM (BigScience, 176B) and the MPT models from MosaicML

```python
import math, torch

def alibi_slopes(n_heads):
    def pow2(n):                                        # n a power of 2
        start = 2 ** (-2 ** -(math.log2(n) - 3))        # = 2^(-8/n)
        return [start * start ** i for i in range(n)]
    if math.log2(n_heads).is_integer():
        return pow2(n_heads)
    c = 2 ** math.floor(math.log2(n_heads))
    return pow2(c) + alibi_slopes(2 * c)[0::2][: n_heads - c]

def alibi_bias(n_heads, T):
    m = torch.tensor(alibi_slopes(n_heads))             # (H,)
    i = torch.arange(T)[:, None]
    j = torch.arange(T)[None, :]
    bias = -m[:, None, None] * (i - j).clamp(min=0)     # (H, T, T)
    causal = torch.ones(T, T, dtype=torch.bool).tril()
    return bias.masked_fill(~causal, float("-inf"))     # (H, T, T)

def alibi_attention(q, k, v):                           # (B, H, T, Dh) each
    B, H, T, Dh = q.shape
    logits = q @ k.transpose(-1, -2) / math.sqrt(Dh) + alibi_bias(H, T)   # (B, H, T, T)
    return logits.softmax(-1) @ v                       # (B, H, T, Dh)

print([round(s, 5) for s in alibi_slopes(8)])
# [0.5, 0.25, 0.125, 0.0625, 0.03125, 0.01562, 0.00781, 0.00391]
print([round(s, 4) for s in alibi_slopes(12)][8:])     # [0.7071, 0.3536, 0.1768, 0.0884]

q, k, v = (torch.randn(2, 8, 2048, 64) for _ in range(3))   # any length: no tables to outgrow
print(alibi_attention(q, k, v).shape)                       # torch.Size([2, 8, 2048, 64])
```

ALiBi slopes (as in the authors' code), the bias tensor, and causal attention with it

> **Extrapolating is not the same as using long context:** A flat perplexity curve beyond $L$ shows that the model does not break, not that it reads the extra tokens. Most heads give keys at distance $\gg L$ almost no weight, and the shallowest head's window is fixed by its slope. If a task needs information from far back (retrieval over a long document), measure it directly instead of inferring it from perplexity.

> **A prior, not a feature:** Learned and rotary encodings give the network a coordinate and let it learn what to do with it. ALiBi instead hard-codes a **recency prior** with a spread of strengths, and lets content scores decide when to override it. For language modelling, where nearby tokens matter most, that prior is cheap and hard to beat; for tasks where order matters but distance does not, it can be the wrong one.

> **Encoders and bidirectional attention:** The paper treats causal language models, where $j\le i$ and the distance is one-sided. Bidirectional adaptations either apply the symmetric penalty $-m|i-j|$, which cannot tell left from right, or use different slopes for the two directions.

**Strengths**

- No parameters and no position embeddings; any sequence length works
- Trains at short L and evaluates at longer L without perplexity blow-up; cheaper training for a given evaluation length
- Implementation is one bias tensor (or $m\,j$ folded into the mask); supported by fused attention kernels
- Gives heads a built-in range of receptive fields

**Limitations**

- Strong recency prior: far tokens are penalised linearly and can be retrieved only through large content scores
- Flat perplexity beyond L does not mean the extra context is used
- Designed for causal attention; bidirectional use needs changes
- Most recent large models chose RoPE with frequency rescaling for long context instead

**Question:** A model has 8 ALiBi heads. What is the slope of head 3, and after how many tokens does its bias halve a key's weight?

- A. 1/8, about 5.5 tokens
- B. 3/8, about 1.8 tokens
- C. 1/3, about 2.1 tokens
- D. 1/64, about 44 tokens

**Answer:** A. 1/8, about 5.5 tokens — $m_h=2^{-8h/n}=2^{-3}=\tfrac18$ for $h=3$, $n=8$. The weight factor is $e^{-m\Delta}$, which halves when $\Delta=\ln2/m=0.693\cdot8\approx5.5$ tokens.

**Question:** Why can an ALiBi model trained on 1024 tokens run on 2048 tokens without a perplexity explosion, unlike a model with learned absolute position embeddings?

- A. Longer inputs only add keys at larger distances, whose bias continues the same linear penalty the model was trained with; no new position vectors appear
- B. ALiBi learns a position embedding for every length up to 2048
- C. ALiBi removes the causal mask, so length does not matter
- D. The slopes are retrained at test time for the new length

**Answer:** A. Longer inputs only add keys at larger distances, whose bias continues the same linear penalty the model was trained with; no new position vectors appear — ALiBi has no table to outgrow and no unseen input vectors. The penalty $-m(i-j)$ for distances above 1024 is the same straight line, and for most heads those keys already had negligible weight. The model sees more of the same, mostly ignored, history.

**Key points**

- ALiBi logit: $q_i^\top k_j/\sqrt d-m_h(i-j)$ for $j\le i$; no position embeddings anywhere.
- Slopes $m_h=2^{-8h/n}$: $\tfrac12,\tfrac14,\dots,\tfrac1{256}$ for 8 heads; a key's weight is scaled by $e^{-m_h(i-j)}$, halving every $\ln2/m_h$ tokens.
- Press, Smith and Lewis (ICLR 2022): 1.3B model trained at 1024 matched a sinusoidal model trained at 2048 when evaluated at 2048, 11% faster and with 11% less memory.
- Implementation: one $(H,1,T)$ bias $m\,j$ folded into the causal mask (softmax is row-shift invariant).
- Extrapolation means no breakdown, not use of distant context: steep heads stay local at any length.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
