Causal attention

Causal attention adds −∞ to every score where a key comes after its query, so position i attends only to positions j ≤ i. One parallel pass then trains all next-token predictions at once, and at inference past keys and values never change, so they can be cached.

12 min read1067 wordscore0 viewsupdated 2026-10-12

Definition

Causal attention (also masked or autoregressive self-attention) is self-attention in which query position ii may attend only to key positions j≤ij\le i. Before the softmax, a mask adds −∞-\infty to every score with j>ij>i, so the weights on future tokens are exactly 0. The weights form a lower-triangular matrix. The output at position ii then depends only on the prefix x1,…,xix_1,\dots,x_i. This is the property an autoregressive model p(x)=∏tp(xt∣x<t)p(x)=\prod_t p(x_t\mid x_{<t}) needs.

An RNN language model is causal by construction, because its state at step tt has seen only x≤tx_{\le t}. It is also sequential: step tt cannot start before step t−1t-1 ends. Vaswani et al., "Attention Is All You Need" (NeurIPS 2017), removed the recurrence. The decoder of their Transformer uses self-attention over the target sentence, and in the authors' words it masks out "all values in the input of the softmax which correspond to illegal connections". Together with shifting the outputs right by one position, this "ensures that the predictions for position ii can depend only on the known outputs at positions less than ii". Radford et al. (GPT, 2018) then dropped the encoder and kept only a stack of causally masked blocks. Every decoder-only language model since, GPT-2, GPT-3 and Llama among them, is built from this one masked operation (see Decoder-only transformer).

The mask

CausalAttn(Q,K,V)=softmax ⁣(QK⊤dk+M)V,Mij={0j≤i−∞j>i⟹aij=exp⁡(qi⊤kj/dk)∑j′≤iexp⁡(qi⊤kj′/dk)  (j≤i)\mathrm{CausalAttn}(Q,K,V)=\mathrm{softmax}\!\Big(\frac{QK^\top}{\sqrt{d_k}}+M\Big)V,\qquad M_{ij}=\begin{cases}0 & j\le i\\ -\infty & j>i\end{cases}\qquad\Longrightarrow\qquad a_{ij}=\frac{\exp(q_i^\top k_j/\sqrt{d_k})}{\sum_{j'\le i}\exp(q_i^\top k_{j'}/\sqrt{d_k})}\ \ (j\le i)
Q,K,V∈Rn×dQ,K,V\in\mathbb R^{n\times d}
queries, keys and values of the same sequence of length nn (see Self-attention and Scaled dot-product attention)
MM
additive mask: strictly upper triangle −∞-\infty, diagonal and below 0. Position ii always sees itself, so every row has at least one finite score
aija_{ij}
a softmax over the prefix j≤ij\le i only; e−∞=0e^{-\infty}=0 removes the future from both the numerator and the normaliser

The mask goes on the scores, not on the weights. Zeroing aija_{ij} after an unmasked softmax would leave rows that no longer sum to 1, and the normaliser would still contain the future scores, so information would leak through it. Adding −∞-\infty first is the same as running the softmax over the prefix only. Implementations use -inf or the most negative finite value of the dtype. Row ii has ii live entries (counting from 1), so the matrix has n(n+1)/2n(n+1)/2 of n2n^2. A dense kernel still computes all n2n^2 scores and throws half away; fused kernels such as FlashAttention skip the fully masked tiles.

Training: every prefix in one pass

L=−∑i=1nlog⁡pθ(xi+1∣x≤i),pθ(⋅∣x≤i)=softmax(Wout hi(L)),hi(ℓ)=f(ℓ)(h≤i(ℓ−1))\mathcal L=-\sum_{i=1}^{n}\log p_\theta(x_{i+1}\mid x_{\le i}),\qquad p_\theta(\cdot\mid x_{\le i})=\mathrm{softmax}\big(W_{\text{out}}\,h^{(L)}_i\big),\qquad h^{(\ell)}_i=f^{(\ell)}\big(h^{(\ell-1)}_{\le i}\big)
xi+1x_{i+1}
the target at position ii is the input shifted left by one token (teacher forcing)
hi(ℓ)h^{(\ell)}_i
hidden state of position ii after block ℓ\ell; it is a function of h1..i(ℓ−1)h^{(\ell-1)}_{1..i} only
f(ℓ)f^{(\ell)}
causal attention followed by per-position layers (feed-forward, normalisation, residual), none of which mix positions

Attention is the only sublayer that mixes positions. If it is causal in every block, then by induction over the layers hi(L)h^{(L)}_i depends only on x≤ix_{\le i}. One forward pass over a sequence of nn tokens therefore gives nn valid next-token distributions, one per prefix, all computed with large matrix products. An RNN needs nn sequential steps for the same nn losses. This parallelism over the sequence is the main reason Transformers displaced RNNs for language modelling. Without the mask the model would learn to copy xi+1x_{i+1} from the input, and its training loss would say nothing about generation.

Inference: the KV cache

K1:t=[K1:t−1kt⊤],V1:t=[V1:t−1vt⊤],ot=softmax ⁣(qt⊤K1:t⊤dk)V1:t,bytes=2⋅L⋅hkv⋅dk⋅n⋅bK_{1:t}=\begin{bmatrix}K_{1:t-1}\\ k_t^\top\end{bmatrix},\quad V_{1:t}=\begin{bmatrix}V_{1:t-1}\\ v_t^\top\end{bmatrix},\qquad o_t=\mathrm{softmax}\!\Big(\frac{q_t^\top K_{1:t}^\top}{\sqrt{d_k}}\Big)V_{1:t},\qquad \text{bytes}=2\cdot L\cdot h_{kv}\cdot d_k\cdot n\cdot b
qt,kt,vtq_t,k_t,v_t
projections of the newest token only; earlier rows of KK and VV are read from the cache
oto_t
one query against tt keys. No mask is needed: the cache holds only the past
L, hkv, dkL,\ h_{kv},\ d_k
layers, key/value heads and head size; 22 counts KK and VV
n, bn,\ b
cached tokens and bytes per value (2 for fp16/bf16)

Generation is sequential: token t+1t+1 is sampled from the output at tt. Because of the mask, appending a token never changes the keys, values or outputs of earlier positions in any layer. Each layer can therefore keep its KK and VV from previous steps. A step then costs one row of attention, O(t d)O(t\,d) per layer, plus projections for one token. Recomputing the whole prefix would cost O(t2d)O(t^2d) per step. Over nn generated tokens the cache turns Θ(n3)\Theta(n^3) score products into Θ(n2)\Theta(n^2). The price is memory that grows linearly with context and batch, and every step must read the whole cache from GPU memory. Decoding is therefore usually limited by memory bandwidth, not by arithmetic. Multi-query attention (MQA) and Grouped-query attention (GQA) shrink the cache by sharing key/value heads.

Causal attention in well-known models

Transformer (2017)
decoder self-attention masked, 6 layers, dmodel=512d_{\text{model}}=512, 8 heads; encoder self-attention and cross-attention unmasked
GPT (2018)
12 masked self-attention blocks, 768 dims, 12 heads, context of 512 tokens
GPT-2 (2019)
context 1024 tokens; largest model 48 layers, 1600 dims, 1.5B parameters
GPT-3 (2020)
context 2048; 175B model 96 layers, dmodel=12288d_{\text{model}}=12288, 96 heads; layers alternate dense and locally banded sparse causal patterns (see Sparse attention)
Llama 2 7B (2023)
32 layers, 32 heads of size 128, context 4096. fp16 KV cache: 2⋅32⋅32⋅128⋅2=524,2882\cdot32\cdot32\cdot128\cdot2=524{,}288 bytes = 0.5 MiB per token, 2 GiB for one 4096-token sequence
Llama 2 70B
grouped-query attention with 8 key/value heads, which cuts the cache 8× relative to one K/V pair per query head
A = softmax(QKᵀ/√d + M), M_ij = −∞ for j > i (causal)key j →<s>thecatsatonthematandslept.0 <s>1 the2 cat3 sat4 on5 the6 mat7 and8 slept9 .■ weight · grey: masked (−∞) · click a rowK cacheV cachetraining: all 10 rows in one pass; K and V for every position computed at oncemode training (parallel)mask causalrow i 5 “the”weight on j > i 0 (masked)train row vs decode max |Δ| 0.000all 10 rows max |Δ| 0.000past rows never change: the cache is validgenerating tokens 0…5 (6 steps)q·k products, with cache 21q·k products, recompute 56k,v projections, cache 6k,v projections, recompute 21KV cache, 32 layers × 32 heads × 128, fp16= 0.5 MiB/token: 3.0 MiB at 6 tokens,2 GiB at 4096 tokens per sequencecausal pass: 55 of 100 pairs scored · 10 next-token losses per pass
view
mask
fig.Ten tokens with fixed random queries and keys (d = 8). Training view: with the causal mask the weights are lower-triangular. Row 4 ('on') puts 0.699 on 'cat', and row 5 (the second 'the') puts 0.756 on 'on'. Switch the mask off and row 5 puts 0.973 of its weight on future tokens, and row 0 ('<s>') puts 0.935 there. Decoding view: step t computes q, k, v for one token, appends (k, v) to the cache and scores one query against t + 1 keys. With the mask, every row produced this way equals the training row exactly (max |Δ| = 0.000 over all 10 rows). Without it, training rows saw the future, decoding cannot, and the rows differ by up to 0.935, so no cache can reproduce them. The counters show the work for steps 0…t. Generating all 10 tokens takes 55 query–key products and 10 K/V projections with the cache, against 220 products and 55 projections if every step recomputes the prefix. At the bottom right, the fp16 cache of a 32-layer, 32-head, d_head = 128 model: 0.5 MiB per token, 2 GiB at 4096 tokens.

Worked example: three tokens

Take dk=2d_k=2, queries q1=(1,0)q_1=(1,0), q2=(0.5,1)q_2=(0.5,1), q3=(1,1)q_3=(1,1), keys k1=(1,0)k_1=(1,0), k2=(0,1)k_2=(0,1), k3=(1,−1)k_3=(1,-1) and values v1=(1,0)v_1=(1,0), v2=(0,1)v_2=(0,1), v3=(2,2)v_3=(2,2). Scores are qi⊤kj/2q_i^\top k_j/\sqrt2.

iscores (j = 1, 2, 3)causal aᵢoᵢ (causal)aᵢ without mask
10.7071, −∞ (0), −∞ (0.7071)1, 0, 0(1, 0)0.4011, 0.1978, 0.4011
20.3536, 0.7071, −∞ (−0.3536)0.4125, 0.5875, 0(0.4125, 0.5875)0.3428, 0.4882, 0.1690
30.7071, 0.7071, 00.4011, 0.4011, 0.1978(0.7967, 0.7967)0.4011, 0.4011, 0.1978
Scores, causal weights and outputs (computed in Node). Masked scores are shown as −∞ with the raw score in parentheses.

Reading the table

  1. 1Row 1: only k1k_1 survives, so a11=1a_{11}=1 and o1=v1=(1,0)o_1=v_1=(1,0). Unmasked, row 1 would put 0.40110.4011 on k3k_3 and output (1.2033, 1.0000)(1.2033,\,1.0000): it would use v3=(2,2)v_3=(2,2), a token it is supposed to predict later.
  2. 2Row 2: softmax(0.3536, 0.7071)=(0.4125, 0.5875)\mathrm{softmax}(0.3536,\,0.7071)=(0.4125,\,0.5875), so o2=0.4125 v1+0.5875 v2=(0.4125, 0.5875)o_2=0.4125\,v_1+0.5875\,v_2=(0.4125,\,0.5875). Unmasked it would be (0.6808, 0.8262)(0.6808,\,0.8262).
  3. 3Row 3 is the last position, so it has no future and the mask changes nothing: o3=(0.7967, 0.7967)o_3=(0.7967,\,0.7967) in both cases.
  4. 4Decoding: at step 2 the cache holds k1,v1k_1,v_1; we append k2,v2k_2,v_2 and score q2q_2 against two keys. The result, (0.4125, 0.5875)(0.4125,\,0.5875), equals training row 2. At step 3, appending k3,v3k_3,v_3 does not change o1o_1 or o2o_2, which is why they never need recomputing.
python
1import torch, torch.nn.functional as F
2
3torch.manual_seed(0)
4B, H, T, d = 2, 4, 6, 16
5q, k, v = (torch.randn(B, H, T, d) for _ in range(3))
6
7# 1) explicit additive mask on the scores
8allowed = torch.ones(T, T, dtype=torch.bool).tril() # (T, T), True where j <= i
9s = q @ k.transpose(-2, -1) / d ** 0.5 # (B, H, T, T)
10s = s.masked_fill(~allowed, float("-inf"))
11out = s.softmax(-1) @ v # (B, H, T, d)
12
13# 2) fused kernel
14out2 = F.scaled_dot_product_attention(q, k, v, is_causal=True)
15print(torch.allclose(out, out2, atol=1e-5)) # True
16
17# 3) token-by-token decoding with a KV cache
18k_cache = torch.empty(B, H, 0, d)
19v_cache = torch.empty(B, H, 0, d)
20rows = []
21for t in range(T):
22 k_cache = torch.cat([k_cache, k[:, :, t:t + 1]], dim=2) # (B, H, t+1, d)
23 v_cache = torch.cat([v_cache, v[:, :, t:t + 1]], dim=2)
24 o_t = F.scaled_dot_product_attention(q[:, :, t:t + 1], k_cache, v_cache) # (B, H, 1, d), no mask
25 rows.append(o_t)
26print(torch.allclose(torch.cat(rows, dim=2), out, atol=1e-5)) # True
27
28# pitfall: is_causal=True with 1 query and 6 keys uses a top-left aligned mask
29bad = F.scaled_dot_product_attention(q[:, :, 5:6], k, v, is_causal=True)
30print(torch.allclose(bad, v[:, :, 0:1])) # True: the query only sees key 0
Causal attention three ways: explicit mask, fused kernel, and KV-cache decoding (all agree)

Use when

  • +All nn next-token losses from one parallel pass; no recurrence
  • +Training and generation compute exactly the same function, so a KV cache is exact
  • +Cheap to implement: one additive mask, and fused kernels skip masked tiles
  • +Any prefix length works at inference time, up to the trained context

Avoid when

  • −Each token sees only its left context; tasks that need both sides (tagging, retrieval encoders) do better with a bidirectional encoder
  • −Score cost is still O(n2)O(n^2) per layer during training
  • −KV cache grows as 2L hkv dk n2L\,h_{kv}\,d_k\,n values per sequence; decoding is memory-bandwidth bound
  • −Generation remains sequential: one token per forward step

Check yourself

Why is the causal mask added to the scores as −∞ instead of zeroing the weights after the softmax?

Check yourself

A 32-layer model with 32 key/value heads of size 128 runs in fp16. How large is the KV cache for one sequence of 4096 tokens?

Remember

  • ✓Causal attention: softmax(QK⊤/dk+M)V\mathrm{softmax}(QK^\top/\sqrt{d_k}+M)V with Mij=−∞M_{ij}=-\infty for j>ij>i, so aij=0a_{ij}=0 for every future key and row ii is a softmax over j≤ij\le i.
  • ✓KV cache: step tt computes qt,kt,vtq_t,k_t,v_t, appends (kt,vt)(k_t,v_t) and scores one query against tt keys; memory 2 L hkv dk n b2\,L\,h_{kv}\,d_k\,n\,b bytes (0.5 MiB/token for 32 layers × 32 heads × 128 in fp16).
  • ✓Training: targets are the inputs shifted by one, so one masked pass gives nn next-token losses in parallel.
  • ✓Only attention mixes positions, so a causal mask in every block makes hih_i depend on x≤ix_{\le i} alone. Past outputs never change, which makes the cache exact.
  • ✓With fewer queries than keys, the mask must be bottom-right aligned; PyTorch's is_causal is top-left aligned.
← back to Deep Learning roadmap