Definition
Scaled dot-product attention is the attention function of the Transformer: . Every query is compared with every key by a dot product, the scores are divided by the square root of the key dimension , an optional mask sets forbidden scores to , a row-wise softmax turns the scores into weights, and the weights average the values. It has no parameters of its own; the projections that produce , and belong to the surrounding layer.
Vaswani et al. named and defined it in "Attention Is All You Need" (NeurIPS 2017). Two score functions were in common use (see Attention mechanism). Additive attention (Bahdanau et al., 2014) scores with a one-hidden-layer MLP. Dot-product attention (Luong et al., 2015) scores with . The authors chose the dot product because "it can be implemented using highly optimized matrix multiplication code", which makes it much faster and more space-efficient in practice. They also noted a problem: for small the two perform similarly, but additive attention beats unscaled dot-product attention for larger , as the large NMT architecture study of Britz et al. (EMNLP 2017) had found. Their fix is one constant factor.
The formula
- matrix of dot products , one matrix multiply
- scaling factor; equivalently a softmax temperature
- additive mask: where query may see key , where it may not; , so masked keys get exactly zero weight
- applied to each row separately, so every query's weights are non-negative and sum to 1
- one value per key; output row is
The shapes say what the function does and does not assume. Queries and keys must share ; values may have any . The number of outputs equals the number of queries , and the number of keys may differ, which is what Cross-attention uses. With , , from one sequence it is Self-attention. Batches and heads are leading dimensions: one call computes independent attentions, which is how Multi-head attention is implemented.
Why divide by √d_k
The paper's footnote gives the argument. Suppose the components of and are independent with mean 0 and variance 1, roughly what layer normalisation and standard initialisation produce. Then each product has mean 0 and variance 1, and the dot product is a sum of of them:
- unscaled scores have standard deviation 8: typical gaps of ±8 between keys make differ by factors of thousands
- the standard deviation, not the variance, is divided out, so the scores have unit scale whatever the head size
Large scores matter because of the softmax Jacobian. With , . When one score is far above the others, is close to one-hot, every entry of is close to 0, and the scores receive almost no gradient. In the authors' words, large dot products push "the softmax function into regions where it has extremely small gradients". After scaling, the spread of the scores, and with it the sharpness of the attention at initialisation, no longer depends on the head size. Training can still sharpen attention by growing the norms of and ; the scale only removes the dependence on at the start.
Masking
A mask decides which keys each query may see, and it must be applied before the softmax. Setting gives , and the remaining weights renormalise among the allowed keys. Zeroing weights after the softmax would leave rows that no longer sum to 1. Two masks cover almost all uses. A padding mask removes key positions that only exist because sequences in a batch were padded to the same length; it depends on the key index only. A causal mask removes keys , so position cannot see the future; it is what makes a decoder autoregressive (see Causal attention). They are combined with a logical AND, or by adding their matrices. Masks can also encode other patterns: blocks for packed sequences, local windows (see Sparse attention), or prefix-LM masks that are bidirectional over a prompt and causal afterwards.
| mask | allowed when | used in |
|---|---|---|
| none | always | encoder self-attention on unpadded input |
| padding | key is a real token | any layer over a padded batch; cross-attention to a padded encoder output |
| causal | decoder self-attention, GPT-style language models | |
| causal + padding | and real | batched decoder training on sequences of different lengths |
| block-diagonal | and in the same packed document | sequence packing without cross-document attention |
Worked example: three queries, d_k = 4
Queries , , ; keys , , ; values , , . The scale is .
| query | qᵀk₁, qᵀk₂, qᵀk₃ | A (÷√dₖ) | A (unscaled) | A (causal, ÷√dₖ) | output (causal) |
|---|---|---|---|---|---|
| q₁ | 1, 0, 2 | 0.3072, 0.1863, 0.5065 | 0.2447, 0.0900, 0.6652 | 1, 0, 0 | (1, 0) |
| q₂ | 2, 2, 0 | 0.4223, 0.4223, 0.1554 | 0.4683, 0.4683, 0.0634 | 0.5, 0.5, 0 | (0.5, 0.5) |
| q₃ | 2, 2, 2 | 1/3, 1/3, 1/3 | 1/3, 1/3, 1/3 | 1/3, 1/3, 1/3 | (0.6667, 0.6667) |
Reading the table
- 1Row 1: scaled scores ; , , , sum , so .
- 2Without the scale the same row is : already sharper at . The gap grows with .
- 3Row 3 has equal scores, so scaling changes nothing: a softmax only sees differences between scores, and dividing equal scores leaves them equal.
- 4Causal mask: row 1 keeps only , giving weight 1 and output . Row 2 keeps with equal scores: and output . Row 3 is unchanged.
F.scaled_dot_product_attention(Q, K, V, is_causal=True)returns the same three rows.
1import math, torch, torch.nn.functional as F23def sdpa(q, k, v, mask=None, scale=None):4 """softmax(q kᵀ · scale) v. q (..., n_q, d_k), k (..., n_k, d_k), v (..., n_k, d_v).5 mask: bool, broadcastable to (..., n_q, n_k), True = may attend (PyTorch's convention)."""6 scale = 1 / math.sqrt(q.size(-1)) if scale is None else scale7 scores = q @ k.transpose(-2, -1) * scale # (..., n_q, n_k)8 if mask is not None:9 scores = scores.masked_fill(~mask, float("-inf")) # exp(-inf) = 0 after the softmax10 a = scores.softmax(dim=-1)11 return a @ v, a1213torch.manual_seed(0)14N, h, n, d_k = 2, 4, 6, 6415q, k, v = (torch.randn(N, h, n, d_k) for _ in range(3))1617# causal mask (query i sees keys j <= i) combined with key padding (sequence 1 has 4 real tokens)18lengths = torch.tensor([6, 4])19causal = torch.ones(n, n, dtype=torch.bool).tril() # (n, n)20keep = torch.arange(n)[None, :] < lengths[:, None] # (N, n)21mask = causal & keep[:, None, None, :] # (N, 1, n, n)2223y, a = sdpa(q, k, v, mask)24print(y.shape, a.shape) # torch.Size([2, 4, 6, 64]) torch.Size([2, 4, 6, 6])25print(torch.allclose(y, F.scaled_dot_product_attention(q, k, v, attn_mask=mask), atol=1e-5)) # True26print(a[1, 0, 5]) # tensor([0.4806, 0.0242, 0.3463, 0.1488, 0.0000, 0.0000])2728# Why the 1/sqrt(d_k): score variance and softmax saturation29raw = q @ k.transpose(-2, -1) # 288 scores with i.i.d. N(0, 1) entries30print(f"{raw.var():.1f} {(raw / math.sqrt(d_k)).var():.2f}") # 75.2 1.17 (theory: 64 and 1)31for scale in (1.0, 1 / math.sqrt(d_k)):32 s = (q @ k.transpose(-2, -1) * scale).requires_grad_()33 s.softmax(-1)[..., 0].sum().backward() # gradient of the weight on key 034 print(f"scale {scale:.3f}: max weight {s.softmax(-1).max(-1).values.mean():.2f}, "35 f"mean |grad| {s.grad.abs().mean():.4f}")36# scale 1.000: max weight 0.88, mean |grad| 0.0113 <- near one-hot, small gradients37# scale 0.125: max weight 0.44, mean |grad| 0.03403839# A query whose keys are all masked gives NaN: guard padded query rows.40bad = torch.zeros(n, n, dtype=torch.bool)41print(sdpa(q[0, 0], k[0, 0], v[0, 0], bad)[0].isnan().all().item()) # TrueScaling choices in practice
- Transformer (2017)
- with in both base and big models, so scores are divided by 8
- T5
- no explicit factor in the attention; the is folded into the initialisation of the query projection (Mesh TensorFlow convention)
- μP (Yang et al., Tensor Programs V)
- instead of , so that attention logits stay well-scaled as width grows and hyperparameters transfer across model sizes
- QK-norm
- normalise and (LayerNorm or ) before the dot product; Henry et al. (2020) for translation, Dehghani et al. (2023) to stop attention-logit growth in the 22B-parameter ViT
- PyTorch
F.scaled_dot_product_attention(q, k, v, attn_mask=None, dropout_p=0.0, is_causal=False, scale=None);scale=Nonemeans ; dispatches to fused kernels such as FlashAttention when it can
Use when
- +Two matrix multiplies and a softmax: the fastest score function on GPUs and TPUs
- +No parameters, so it works for any , and any
- +The factor makes initial attention sharpness independent of head size
- +Masks express causality, padding and sparsity patterns with no change to the function
Avoid when
- − time and an score matrix: quadratic for self-attention
- −A purely bilinear similarity; any non-linear matching must come from the projections
- −Logits can still grow during training; large models add QK-norm or logit capping
- −Mask polarity and fully masked rows are a frequent source of silent bugs
Check yourself
Query and key entries are i.i.d. with mean 0 and variance 1, and d_k = 256. What is the standard deviation of qᵀk, and what is it after scaling?
Check yourself
Why is a causal mask applied by setting scores to −∞ before the softmax rather than zeroing weights after it?
Remember
- ✓; removes key for query before the row softmax.
- ✓If have i.i.d. unit-variance entries, ; dividing by restores unit variance and keeps the softmax out of saturation.
- ✓The softmax Jacobian vanishes as becomes one-hot, which is why large unscaled scores stall learning.
- ✓Padding masks depend on the key only; causal masks allow ; combine them with AND. Fully masked rows give NaN.
- ✓Output has one row per query: ; may differ from (cross-attention).