# Causal attention

> Causal attention adds −∞ to every score where a key comes after its query, so position i attends only to positions j ≤ i. One parallel pass then trains all next-token predictions at once, and at inference past keys and values never change, so they can be cached.

Source: https://www.mlfoundry.in/learn/causal-attention  
Section: Deep Learning › Attention and Transformer Architectures  
Level: core · 12 min read · updated 2026-10-12

**Causal attention:** (also **masked** or **autoregressive self-attention**) is self-attention in which query position $i$ may attend only to key positions $j\le i$. Before the softmax, a mask adds $-\infty$ to every score with $j>i$, so the weights on future tokens are exactly 0. The weights form a lower-triangular matrix. The output at position $i$ then depends only on the prefix $x_1,\dots,x_i$. This is the property an autoregressive model $p(x)=\prod_t p(x_t\mid x_{<t})$ needs.

An RNN language model is causal by construction, because its state at step $t$ has seen only $x_{\le t}$. It is also sequential: step $t$ cannot start before step $t-1$ ends. Vaswani et al., "Attention Is All You Need" (NeurIPS 2017), removed the recurrence. The decoder of their Transformer uses self-attention over the target sentence, and in the authors' words it masks out "all values in the input of the softmax which correspond to illegal connections". Together with shifting the outputs right by one position, this "ensures that the predictions for position $i$ can depend only on the known outputs at positions less than $i$". Radford et al. (GPT, 2018) then dropped the encoder and kept only a stack of causally masked blocks. Every decoder-only language model since, GPT-2, GPT-3 and Llama among them, is built from this one masked operation (see **Decoder-only transformer**).

## The mask

$$
\mathrm{CausalAttn}(Q,K,V)=\mathrm{softmax}\!\Big(\frac{QK^\top}{\sqrt{d_k}}+M\Big)V,\qquad M_{ij}=\begin{cases}0 & j\le i\\ -\infty & j>i\end{cases}\qquad\Longrightarrow\qquad a_{ij}=\frac{\exp(q_i^\top k_j/\sqrt{d_k})}{\sum_{j'\le i}\exp(q_i^\top k_{j'}/\sqrt{d_k})}\ \ (j\le i)
$$

- $Q,K,V\in\mathbb R^{n\times d}$: queries, keys and values of the same sequence of length $n$ (see **Self-attention** and **Scaled dot-product attention**)
- $M$: additive mask: strictly upper triangle $-\infty$, diagonal and below 0. Position $i$ always sees itself, so every row has at least one finite score
- $a_{ij}$: a softmax over the prefix $j\le i$ only; $e^{-\infty}=0$ removes the future from both the numerator and the normaliser

The mask goes on the **scores**, not on the weights. Zeroing $a_{ij}$ after an unmasked softmax would leave rows that no longer sum to 1, and the normaliser would still contain the future scores, so information would leak through it. Adding $-\infty$ first is the same as running the softmax over the prefix only. Implementations use `-inf` or the most negative finite value of the dtype. Row $i$ has $i$ live entries (counting from 1), so the matrix has $n(n+1)/2$ of $n^2$. A dense kernel still computes all $n^2$ scores and throws half away; fused kernels such as **FlashAttention** skip the fully masked tiles.

## Training: every prefix in one pass

$$
\mathcal L=-\sum_{i=1}^{n}\log p_\theta(x_{i+1}\mid x_{\le i}),\qquad p_\theta(\cdot\mid x_{\le i})=\mathrm{softmax}\big(W_{\text{out}}\,h^{(L)}_i\big),\qquad h^{(\ell)}_i=f^{(\ell)}\big(h^{(\ell-1)}_{\le i}\big)
$$

- $x_{i+1}$: the target at position $i$ is the input shifted left by one token (teacher forcing)
- $h^{(\ell)}_i$: hidden state of position $i$ after block $\ell$; it is a function of $h^{(\ell-1)}_{1..i}$ only
- $f^{(\ell)}$: causal attention followed by per-position layers (feed-forward, normalisation, residual), none of which mix positions

Attention is the only sublayer that mixes positions. If it is causal in every block, then by induction over the layers $h^{(L)}_i$ depends only on $x_{\le i}$. One forward pass over a sequence of $n$ tokens therefore gives $n$ valid next-token distributions, one per prefix, all computed with large matrix products. An RNN needs $n$ sequential steps for the same $n$ losses. This parallelism over the sequence is the main reason Transformers displaced RNNs for language modelling. Without the mask the model would learn to copy $x_{i+1}$ from the input, and its training loss would say nothing about generation.

## Inference: the KV cache

$$
K_{1:t}=\begin{bmatrix}K_{1:t-1}\\ k_t^\top\end{bmatrix},\quad V_{1:t}=\begin{bmatrix}V_{1:t-1}\\ v_t^\top\end{bmatrix},\qquad o_t=\mathrm{softmax}\!\Big(\frac{q_t^\top K_{1:t}^\top}{\sqrt{d_k}}\Big)V_{1:t},\qquad \text{bytes}=2\cdot L\cdot h_{kv}\cdot d_k\cdot n\cdot b
$$

- $q_t,k_t,v_t$: projections of the newest token only; earlier rows of $K$ and $V$ are read from the cache
- $o_t$: one query against $t$ keys. No mask is needed: the cache holds only the past
- $L,\ h_{kv},\ d_k$: layers, key/value heads and head size; $2$ counts $K$ and $V$
- $n,\ b$: cached tokens and bytes per value (2 for fp16/bf16)

Generation is sequential: token $t+1$ is sampled from the output at $t$. Because of the mask, appending a token never changes the keys, values or outputs of earlier positions in any layer. Each layer can therefore keep its $K$ and $V$ from previous steps. A step then costs one row of attention, $O(t\,d)$ per layer, plus projections for one token. Recomputing the whole prefix would cost $O(t^2d)$ per step. Over $n$ generated tokens the cache turns $\Theta(n^3)$ score products into $\Theta(n^2)$. The price is memory that grows linearly with context and batch, and every step must read the whole cache from GPU memory. Decoding is therefore usually limited by memory bandwidth, not by arithmetic. **Multi-query attention (MQA)** and **Grouped-query attention (GQA)** shrink the cache by sharing key/value heads.

**Causal attention in well-known models**

- **Transformer (2017):** decoder self-attention masked, 6 layers, $d_{\text{model}}=512$, 8 heads; encoder self-attention and cross-attention unmasked
- **GPT (2018):** 12 masked self-attention blocks, 768 dims, 12 heads, context of 512 tokens
- **GPT-2 (2019):** context 1024 tokens; largest model 48 layers, 1600 dims, 1.5B parameters
- **GPT-3 (2020):** context 2048; 175B model 96 layers, $d_{\text{model}}=12288$, 96 heads; layers alternate dense and locally banded sparse causal patterns (see **Sparse attention**)
- **Llama 2 7B (2023):** 32 layers, 32 heads of size 128, context 4096. fp16 KV cache: $2\cdot32\cdot32\cdot128\cdot2=524{,}288$ bytes = 0.5 MiB per token, 2 GiB for one 4096-token sequence
- **Llama 2 70B:** grouped-query attention with 8 key/value heads, which cuts the cache 8× relative to one K/V pair per query head

> Figure: Ten tokens with fixed random queries and keys (d = 8). Training view: with the causal mask the weights are lower-triangular. Row 4 ('on') puts 0.699 on 'cat', and row 5 (the second 'the') puts 0.756 on 'on'. Switch the mask off and row 5 puts 0.973 of its weight on future tokens, and row 0 ('<s>') puts 0.935 there. Decoding view: step t computes q, k, v for one token, appends (k, v) to the cache and scores one query against t + 1 keys. With the mask, every row produced this way equals the training row exactly (max |Δ| = 0.000 over all 10 rows). Without it, training rows saw the future, decoding cannot, and the rows differ by up to 0.935, so no cache can reproduce them. The counters show the work for steps 0…t. Generating all 10 tokens takes 55 query–key products and 10 K/V projections with the cache, against 220 products and 55 projections if every step recomputes the prefix. At the bottom right, the fp16 cache of a 32-layer, 32-head, d_head = 128 model: 0.5 MiB per token, 2 GiB at 4096 tokens.

## Worked example: three tokens

Take $d_k=2$, queries $q_1=(1,0)$, $q_2=(0.5,1)$, $q_3=(1,1)$, keys $k_1=(1,0)$, $k_2=(0,1)$, $k_3=(1,-1)$ and values $v_1=(1,0)$, $v_2=(0,1)$, $v_3=(2,2)$. Scores are $q_i^\top k_j/\sqrt2$.

Scores, causal weights and outputs (computed in Node). Masked scores are shown as −∞ with the raw score in parentheses.

| i | scores (j = 1, 2, 3) | causal aᵢ | oᵢ (causal) | aᵢ without mask |
| --- | --- | --- | --- | --- |
| 1 | 0.7071, −∞ (0), −∞ (0.7071) | 1, 0, 0 | (1, 0) | 0.4011, 0.1978, 0.4011 |
| 2 | 0.3536, 0.7071, −∞ (−0.3536) | 0.4125, 0.5875, 0 | (0.4125, 0.5875) | 0.3428, 0.4882, 0.1690 |
| 3 | 0.7071, 0.7071, 0 | 0.4011, 0.4011, 0.1978 | (0.7967, 0.7967) | 0.4011, 0.4011, 0.1978 |

**Reading the table**

1. Row 1: only $k_1$ survives, so $a_{11}=1$ and $o_1=v_1=(1,0)$. Unmasked, row 1 would put $0.4011$ on $k_3$ and output $(1.2033,\,1.0000)$: it would use $v_3=(2,2)$, a token it is supposed to predict later.
2. Row 2: $\mathrm{softmax}(0.3536,\,0.7071)=(0.4125,\,0.5875)$, so $o_2=0.4125\,v_1+0.5875\,v_2=(0.4125,\,0.5875)$. Unmasked it would be $(0.6808,\,0.8262)$.
3. Row 3 is the last position, so it has no future and the mask changes nothing: $o_3=(0.7967,\,0.7967)$ in both cases.
4. Decoding: at step 2 the cache holds $k_1,v_1$; we append $k_2,v_2$ and score $q_2$ against two keys. The result, $(0.4125,\,0.5875)$, equals training row 2. At step 3, appending $k_3,v_3$ does not change $o_1$ or $o_2$, which is why they never need recomputing.

```python
import torch, torch.nn.functional as F

torch.manual_seed(0)
B, H, T, d = 2, 4, 6, 16
q, k, v = (torch.randn(B, H, T, d) for _ in range(3))

# 1) explicit additive mask on the scores
allowed = torch.ones(T, T, dtype=torch.bool).tril()          # (T, T), True where j <= i
s = q @ k.transpose(-2, -1) / d ** 0.5                       # (B, H, T, T)
s = s.masked_fill(~allowed, float("-inf"))
out = s.softmax(-1) @ v                                      # (B, H, T, d)

# 2) fused kernel
out2 = F.scaled_dot_product_attention(q, k, v, is_causal=True)
print(torch.allclose(out, out2, atol=1e-5))                  # True

# 3) token-by-token decoding with a KV cache
k_cache = torch.empty(B, H, 0, d)
v_cache = torch.empty(B, H, 0, d)
rows = []
for t in range(T):
    k_cache = torch.cat([k_cache, k[:, :, t:t + 1]], dim=2)  # (B, H, t+1, d)
    v_cache = torch.cat([v_cache, v[:, :, t:t + 1]], dim=2)
    o_t = F.scaled_dot_product_attention(q[:, :, t:t + 1], k_cache, v_cache)  # (B, H, 1, d), no mask
    rows.append(o_t)
print(torch.allclose(torch.cat(rows, dim=2), out, atol=1e-5))  # True

# pitfall: is_causal=True with 1 query and 6 keys uses a top-left aligned mask
bad = F.scaled_dot_product_attention(q[:, :, 5:6], k, v, is_causal=True)
print(torch.allclose(bad, v[:, :, 0:1]))                      # True: the query only sees key 0
```

Causal attention three ways: explicit mask, fused kernel, and KV-cache decoding (all agree)

> **Non-square masks: top-left versus bottom-right:** With a KV cache or chunked prefill there are fewer queries $L$ than keys $S$, and the $L$ queries are the **last** $L$ positions. The correct mask allows key $j$ for query $i$ when $j\le i+(S-L)$ (bottom-right aligned). PyTorch's `is_causal=True` in `F.scaled_dot_product_attention` builds `ones(L, S).tril()`, which is top-left aligned. A single decode query then sees only key 0, as the last line of the code shows. Pass no mask for a single new token, or use `torch.nn.attention.bias.causal_lower_right` or an explicit boolean mask.

> **Padding and the causal mask:** Batched sequences of different lengths need a padding mask as well, combined with the causal mask by logical AND. With left padding, a pad position can end up with every key masked. Its softmax is then $0/0=$ NaN, and the NaN can spread through later layers. Implementations guard against this, for example by zeroing the outputs of fully masked rows, and they also shift position indices so that the first real token gets position 0.

> **The mask is what makes caching exact:** Training and inference compute the same function only because of the mask. Row $i$ of the parallel pass and step $i$ of decoding see the same keys and values, so they produce the same outputs up to floating-point rounding. In a bidirectional encoder, appending a token changes every earlier output, so nothing can be cached. The viz shows this: unmasked, the training row and the decoding row differ by up to 0.935.

> **Other masks:** The causal mask is one point in a family. A **prefix LM** (used in UniLM and T5's prefix-LM variant) lets a prompt attend bidirectionally and keeps the continuation causal. Sliding-window masks keep only $i-w<j\le i$ (see **Sparse attention** and **Longformer**). Cross-attention in an encoder–decoder model is never causally masked, because the whole source is known (see **Cross-attention**).

**Strengths**

- All $n$ next-token losses from one parallel pass; no recurrence
- Training and generation compute exactly the same function, so a KV cache is exact
- Cheap to implement: one additive mask, and fused kernels skip masked tiles
- Any prefix length works at inference time, up to the trained context

**Limitations**

- Each token sees only its left context; tasks that need both sides (tagging, retrieval encoders) do better with a bidirectional encoder
- Score cost is still $O(n^2)$ per layer during training
- KV cache grows as $2L\,h_{kv}\,d_k\,n$ values per sequence; decoding is memory-bandwidth bound
- Generation remains sequential: one token per forward step

**Question:** Why is the causal mask added to the scores as −∞ instead of zeroing the weights after the softmax?

- A. Because −∞ makes the softmax run faster
- B. Zeroing afterwards leaves rows that do not sum to 1 and whose normaliser still contains the future scores; −∞ before the softmax is exactly a softmax over the prefix
- C. Because zero weights would break the gradient of the values
- D. There is no difference; both give the same weights

**Answer:** B. Zeroing afterwards leaves rows that do not sum to 1 and whose normaliser still contains the future scores; −∞ before the softmax is exactly a softmax over the prefix — $e^{-\infty}=0$ removes future terms from the numerator and the denominator. Masking after the softmax divides by $\sum_{j'}e^{s_{ij'}}$ over all $j'$, which depends on future keys, and the surviving weights sum to less than 1.

**Question:** A 32-layer model with 32 key/value heads of size 128 runs in fp16. How large is the KV cache for one sequence of 4096 tokens?

- A. 256 MiB
- B. 1 GiB
- C. 2 GiB
- D. 4 GiB

**Answer:** C. 2 GiB — $2\,(K,V)\times32\times32\times128\times2$ bytes $=524{,}288$ bytes $=0.5$ MiB per token. Times 4096 tokens this is $2^{31}$ bytes $=2$ GiB. Grouped-query attention with 8 KV heads would make it 0.5 GiB.

**Key points**

- Causal attention: $\mathrm{softmax}(QK^\top/\sqrt{d_k}+M)V$ with $M_{ij}=-\infty$ for $j>i$, so $a_{ij}=0$ for every future key and row $i$ is a softmax over $j\le i$.
- KV cache: step $t$ computes $q_t,k_t,v_t$, appends $(k_t,v_t)$ and scores one query against $t$ keys; memory $2\,L\,h_{kv}\,d_k\,n\,b$ bytes (0.5 MiB/token for 32 layers × 32 heads × 128 in fp16).
- Training: targets are the inputs shifted by one, so one masked pass gives $n$ next-token losses in parallel.
- Only attention mixes positions, so a causal mask in every block makes $h_i$ depend on $x_{\le i}$ alone. Past outputs never change, which makes the cache exact.
- With fewer queries than keys, the mask must be bottom-right aligned; PyTorch's `is_causal` is top-left aligned.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
