Definition
Causal attention (also masked or autoregressive self-attention) is self-attention in which query position may attend only to key positions . Before the softmax, a mask adds to every score with , so the weights on future tokens are exactly 0. The weights form a lower-triangular matrix. The output at position then depends only on the prefix . This is the property an autoregressive model needs.
An RNN language model is causal by construction, because its state at step has seen only . It is also sequential: step cannot start before step ends. Vaswani et al., "Attention Is All You Need" (NeurIPS 2017), removed the recurrence. The decoder of their Transformer uses self-attention over the target sentence, and in the authors' words it masks out "all values in the input of the softmax which correspond to illegal connections". Together with shifting the outputs right by one position, this "ensures that the predictions for position can depend only on the known outputs at positions less than ". Radford et al. (GPT, 2018) then dropped the encoder and kept only a stack of causally masked blocks. Every decoder-only language model since, GPT-2, GPT-3 and Llama among them, is built from this one masked operation (see Decoder-only transformer).
The mask
- queries, keys and values of the same sequence of length (see Self-attention and Scaled dot-product attention)
- additive mask: strictly upper triangle , diagonal and below 0. Position always sees itself, so every row has at least one finite score
- a softmax over the prefix only; removes the future from both the numerator and the normaliser
The mask goes on the scores, not on the weights. Zeroing after an unmasked softmax would leave rows that no longer sum to 1, and the normaliser would still contain the future scores, so information would leak through it. Adding first is the same as running the softmax over the prefix only. Implementations use -inf or the most negative finite value of the dtype. Row has live entries (counting from 1), so the matrix has of . A dense kernel still computes all scores and throws half away; fused kernels such as FlashAttention skip the fully masked tiles.
Training: every prefix in one pass
- the target at position is the input shifted left by one token (teacher forcing)
- hidden state of position after block ; it is a function of only
- causal attention followed by per-position layers (feed-forward, normalisation, residual), none of which mix positions
Attention is the only sublayer that mixes positions. If it is causal in every block, then by induction over the layers depends only on . One forward pass over a sequence of tokens therefore gives valid next-token distributions, one per prefix, all computed with large matrix products. An RNN needs sequential steps for the same losses. This parallelism over the sequence is the main reason Transformers displaced RNNs for language modelling. Without the mask the model would learn to copy from the input, and its training loss would say nothing about generation.
Inference: the KV cache
- projections of the newest token only; earlier rows of and are read from the cache
- one query against keys. No mask is needed: the cache holds only the past
- layers, key/value heads and head size; counts and
- cached tokens and bytes per value (2 for fp16/bf16)
Generation is sequential: token is sampled from the output at . Because of the mask, appending a token never changes the keys, values or outputs of earlier positions in any layer. Each layer can therefore keep its and from previous steps. A step then costs one row of attention, per layer, plus projections for one token. Recomputing the whole prefix would cost per step. Over generated tokens the cache turns score products into . The price is memory that grows linearly with context and batch, and every step must read the whole cache from GPU memory. Decoding is therefore usually limited by memory bandwidth, not by arithmetic. Multi-query attention (MQA) and Grouped-query attention (GQA) shrink the cache by sharing key/value heads.
Causal attention in well-known models
- Transformer (2017)
- decoder self-attention masked, 6 layers, , 8 heads; encoder self-attention and cross-attention unmasked
- GPT (2018)
- 12 masked self-attention blocks, 768 dims, 12 heads, context of 512 tokens
- GPT-2 (2019)
- context 1024 tokens; largest model 48 layers, 1600 dims, 1.5B parameters
- GPT-3 (2020)
- context 2048; 175B model 96 layers, , 96 heads; layers alternate dense and locally banded sparse causal patterns (see Sparse attention)
- Llama 2 7B (2023)
- 32 layers, 32 heads of size 128, context 4096. fp16 KV cache: bytes = 0.5 MiB per token, 2 GiB for one 4096-token sequence
- Llama 2 70B
- grouped-query attention with 8 key/value heads, which cuts the cache 8× relative to one K/V pair per query head
Worked example: three tokens
Take , queries , , , keys , , and values , , . Scores are .
| i | scores (j = 1, 2, 3) | causal aᵢ | oᵢ (causal) | aᵢ without mask |
|---|---|---|---|---|
| 1 | 0.7071, −∞ (0), −∞ (0.7071) | 1, 0, 0 | (1, 0) | 0.4011, 0.1978, 0.4011 |
| 2 | 0.3536, 0.7071, −∞ (−0.3536) | 0.4125, 0.5875, 0 | (0.4125, 0.5875) | 0.3428, 0.4882, 0.1690 |
| 3 | 0.7071, 0.7071, 0 | 0.4011, 0.4011, 0.1978 | (0.7967, 0.7967) | 0.4011, 0.4011, 0.1978 |
Reading the table
- 1Row 1: only survives, so and . Unmasked, row 1 would put on and output : it would use , a token it is supposed to predict later.
- 2Row 2: , so . Unmasked it would be .
- 3Row 3 is the last position, so it has no future and the mask changes nothing: in both cases.
- 4Decoding: at step 2 the cache holds ; we append and score against two keys. The result, , equals training row 2. At step 3, appending does not change or , which is why they never need recomputing.
1import torch, torch.nn.functional as F23torch.manual_seed(0)4B, H, T, d = 2, 4, 6, 165q, k, v = (torch.randn(B, H, T, d) for _ in range(3))67# 1) explicit additive mask on the scores8allowed = torch.ones(T, T, dtype=torch.bool).tril() # (T, T), True where j <= i9s = q @ k.transpose(-2, -1) / d ** 0.5 # (B, H, T, T)10s = s.masked_fill(~allowed, float("-inf"))11out = s.softmax(-1) @ v # (B, H, T, d)1213# 2) fused kernel14out2 = F.scaled_dot_product_attention(q, k, v, is_causal=True)15print(torch.allclose(out, out2, atol=1e-5)) # True1617# 3) token-by-token decoding with a KV cache18k_cache = torch.empty(B, H, 0, d)19v_cache = torch.empty(B, H, 0, d)20rows = []21for t in range(T):22 k_cache = torch.cat([k_cache, k[:, :, t:t + 1]], dim=2) # (B, H, t+1, d)23 v_cache = torch.cat([v_cache, v[:, :, t:t + 1]], dim=2)24 o_t = F.scaled_dot_product_attention(q[:, :, t:t + 1], k_cache, v_cache) # (B, H, 1, d), no mask25 rows.append(o_t)26print(torch.allclose(torch.cat(rows, dim=2), out, atol=1e-5)) # True2728# pitfall: is_causal=True with 1 query and 6 keys uses a top-left aligned mask29bad = F.scaled_dot_product_attention(q[:, :, 5:6], k, v, is_causal=True)30print(torch.allclose(bad, v[:, :, 0:1])) # True: the query only sees key 0Use when
- +All next-token losses from one parallel pass; no recurrence
- +Training and generation compute exactly the same function, so a KV cache is exact
- +Cheap to implement: one additive mask, and fused kernels skip masked tiles
- +Any prefix length works at inference time, up to the trained context
Avoid when
- −Each token sees only its left context; tasks that need both sides (tagging, retrieval encoders) do better with a bidirectional encoder
- −Score cost is still per layer during training
- −KV cache grows as values per sequence; decoding is memory-bandwidth bound
- −Generation remains sequential: one token per forward step
Check yourself
Why is the causal mask added to the scores as −∞ instead of zeroing the weights after the softmax?
Check yourself
A 32-layer model with 32 key/value heads of size 128 runs in fp16. How large is the KV cache for one sequence of 4096 tokens?
Remember
- ✓Causal attention: with for , so for every future key and row is a softmax over .
- ✓KV cache: step computes , appends and scores one query against keys; memory bytes (0.5 MiB/token for 32 layers × 32 heads × 128 in fp16).
- ✓Training: targets are the inputs shifted by one, so one masked pass gives next-token losses in parallel.
- ✓Only attention mixes positions, so a causal mask in every block makes depend on alone. Past outputs never change, which makes the cache exact.
- ✓With fewer queries than keys, the mask must be bottom-right aligned; PyTorch's
is_causalis top-left aligned.