Scaled dot-product attention

Attention(Q, K, V) = softmax(QKᵀ/√d_k + M)V. Dividing by √d_k keeps scores at unit variance for any key size, so the softmax does not saturate, and a mask M of 0 and −∞ entries removes keys (future tokens, padding) before the softmax.

13 min read986 wordscore0 viewsupdated 2026-10-12

Definition

Scaled dot-product attention is the attention function of the Transformer: Attention(Q,K,V)=softmax(QK⊤/dk)V\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\big(QK^\top/\sqrt{d_k}\big)V. Every query is compared with every key by a dot product, the scores are divided by the square root of the key dimension dkd_k, an optional mask sets forbidden scores to −∞-\infty, a row-wise softmax turns the scores into weights, and the weights average the values. It has no parameters of its own; the projections that produce QQ, KK and VV belong to the surrounding layer.

Vaswani et al. named and defined it in "Attention Is All You Need" (NeurIPS 2017). Two score functions were in common use (see Attention mechanism). Additive attention (Bahdanau et al., 2014) scores with a one-hidden-layer MLP. Dot-product attention (Luong et al., 2015) scores with q⊤kq^\top k. The authors chose the dot product because "it can be implemented using highly optimized matrix multiplication code", which makes it much faster and more space-efficient in practice. They also noted a problem: for small dkd_k the two perform similarly, but additive attention beats unscaled dot-product attention for larger dkd_k, as the large NMT architecture study of Britz et al. (EMNLP 2017) had found. Their fix is one constant factor.

The formula

Attention(Q,K,V)=softmax ⁣(QK⊤dk+M)V,Q∈Rnq×dk, K∈Rnk×dk, V∈Rnk×dv, Mij∈{0,−∞}\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\Big(\frac{QK^\top}{\sqrt{d_k}}+M\Big)V,\qquad Q\in\mathbb R^{n_q\times d_k},\ K\in\mathbb R^{n_k\times d_k},\ V\in\mathbb R^{n_k\times d_v},\ M_{ij}\in\{0,-\infty\}
QK⊤QK^\top
nq×nkn_q\times n_k matrix of dot products qi⊤kjq_i^\top k_j, one matrix multiply
1/dk1/\sqrt{d_k}
scaling factor; equivalently a softmax temperature τ=dk\tau=\sqrt{d_k}
MM
additive mask: 00 where query ii may see key jj, −∞-\infty where it may not; e−∞=0e^{-\infty}=0, so masked keys get exactly zero weight
softmax\mathrm{softmax}
applied to each row separately, so every query's weights are non-negative and sum to 1
VV
one value per key; output row ii is ∑jAijvj∈Rdv\sum_j A_{ij}v_j\in\mathbb R^{d_v}

The shapes say what the function does and does not assume. Queries and keys must share dkd_k; values may have any dvd_v. The number of outputs equals the number of queries nqn_q, and the number of keys nkn_k may differ, which is what Cross-attention uses. With QQ, KK, VV from one sequence it is Self-attention. Batches and heads are leading dimensions: one call computes B×hB\times h independent attentions, which is how Multi-head attention is implemented.

Why divide by √d_k

The paper's footnote gives the argument. Suppose the components of qq and kk are independent with mean 0 and variance 1, roughly what layer normalisation and standard initialisation produce. Then each product qikiq_ik_i has mean 0 and variance 1, and the dot product is a sum of dkd_k of them:

E[q⊤k]=∑i=1dkE[qi] E[ki]=0,Var⁡(q⊤k)=∑i=1dkVar⁡(qiki)=∑i=1dkE[qi2] E[ki2]=dk,Var⁡ ⁣(q⊤kdk)=1\mathbb E[q^\top k]=\sum_{i=1}^{d_k}\mathbb E[q_i]\,\mathbb E[k_i]=0,\qquad \operatorname{Var}(q^\top k)=\sum_{i=1}^{d_k}\operatorname{Var}(q_ik_i)=\sum_{i=1}^{d_k}\mathbb E[q_i^2]\,\mathbb E[k_i^2]=d_k,\qquad \operatorname{Var}\!\Big(\frac{q^\top k}{\sqrt{d_k}}\Big)=1
dk=64d_k=64
unscaled scores have standard deviation 8: typical gaps of ±8 between keys make ese^{s} differ by factors of thousands
dk\sqrt{d_k}
the standard deviation, not the variance, is divided out, so the scores have unit scale whatever the head size

Large scores matter because of the softmax Jacobian. With a=softmax(s)a=\mathrm{softmax}(s), ∂a/∂s=diag(a)−aa⊤\partial a/\partial s=\mathrm{diag}(a)-aa^\top. When one score is far above the others, aa is close to one-hot, every entry of diag(a)−aa⊤\mathrm{diag}(a)-aa^\top is close to 0, and the scores receive almost no gradient. In the authors' words, large dot products push "the softmax function into regions where it has extremely small gradients". After scaling, the spread of the scores, and with it the sharpness of the attention at initialisation, no longer depends on the head size. Training can still sharpen attention by growing the norms of WQW_Q and WKW_K; the scale only removes the dependence on dkd_k at the start.

a = softmax(qᵀK / √dₖ) · q, kⱼ ~ N(0, I) in dₖ = 64 dimsone query, 8 keys, draw 1: aⱼ above, sⱼ below0.080.50.472.30.080.50.04−0.10.01−1.40.110.80.161.20.050.1max aⱼ 0.468 · score sd 1.01 · √dₖ 8.00mean max aⱼ over 200 draws: — qᵀk — qᵀk/√dₖ00.51uniform 1/81416642561024dₖ (log scale)mask: causal · rows = queriesthecatsatdown<pad><pad>the1.00−∞−∞−∞−∞−∞cat.62.38−∞−∞−∞−∞sat.20.12.68−∞−∞−∞down.10.32.42.16−∞−∞<pad>.26.13.17.36.09−∞<pad>.09.06.15.58.02.10masked 15/36 · every row sums to 1−∞ before the softmax → exactly 0 after itdₖ = 64, mean over 200 drawsqᵀkqᵀk/√dₖsd of scores8.091.01max aⱼ0.8860.370entropy (nats)0.2931.708‖∂a/∂s‖_F0.1370.365max entropy ln 8 = 2.079scaling on: scores ÷ 8.00 · score variance grows like dₖ unless divided by √dₖ
scores
mask
fig.One query and 8 keys with i.i.d. N(0, 1) entries. Slide d_k from 1 to 1024, toggle the scaling, and step through 200 seeded draws. Statistics over the 200 draws at d_k = 64: unscaled scores have standard deviation 8.09, the largest weight averages 0.886, the entropy is 0.293 nats (the maximum is ln 8 = 2.079), and the softmax Jacobian ‖diag(a) − aaᵀ‖_F averages 0.137. Divided by √64 = 8, the same scores have standard deviation 1.01, largest weight 0.370, entropy 1.708 and Jacobian norm 0.365. In the first draw the unscaled scores reach 18.2 and the weights are one-hot (1.000). Scaled, that key gets 0.468. At d_k = 1024 the unscaled largest weight is 0.975 and the Jacobian norm 0.034, while the scaled curve stays between 0.31 and 0.38 for every d_k. The right panel applies masks to a fixed 6 × 6 score matrix whose last two tokens are padding. Causal masking removes 15 of 36 entries, padding removes 12, and both together remove 18. The first query then attends only to itself (weight 1.00). Every row still sums to 1 because −∞ becomes an exact 0 before the normalisation.

Masking

A mask decides which keys each query may see, and it must be applied before the softmax. Setting sij=−∞s_{ij}=-\infty gives esij=0e^{s_{ij}}=0, and the remaining weights renormalise among the allowed keys. Zeroing weights after the softmax would leave rows that no longer sum to 1. Two masks cover almost all uses. A padding mask removes key positions that only exist because sequences in a batch were padded to the same length; it depends on the key index only. A causal mask removes keys j>ij>i, so position ii cannot see the future; it is what makes a decoder autoregressive (see Causal attention). They are combined with a logical AND, or by adding their 0/−∞0/-\infty matrices. Masks can also encode other patterns: blocks for packed sequences, local windows (see Sparse attention), or prefix-LM masks that are bidirectional over a prompt and causal afterwards.

maskallowed whenused in
nonealwaysencoder self-attention on unpadded input
paddingkey jj is a real tokenany layer over a padded batch; cross-attention to a padded encoder output
causalj≤ij\le idecoder self-attention, GPT-style language models
causal + paddingj≤ij\le i and jj realbatched decoder training on sequences of different lengths
block-diagonalii and jj in the same packed documentsequence packing without cross-document attention
Common masks M (query i, key j)

Worked example: three queries, d_k = 4

Queries q1=(1,0,1,0)q_1=(1,0,1,0), q2=(0,2,0,0)q_2=(0,2,0,0), q3=(1,1,1,1)q_3=(1,1,1,1); keys k1=(1,1,0,0)k_1=(1,1,0,0), k2=(0,1,0,1)k_2=(0,1,0,1), k3=(2,0,0,0)k_3=(2,0,0,0); values v1=(1,0)v_1=(1,0), v2=(0,1)v_2=(0,1), v3=(1,1)v_3=(1,1). The scale is 1/4=1/21/\sqrt4=1/2.

queryqᵀk₁, qᵀk₂, qᵀk₃A (÷√dₖ)A (unscaled)A (causal, ÷√dₖ)output (causal)
q₁1, 0, 20.3072, 0.1863, 0.50650.2447, 0.0900, 0.66521, 0, 0(1, 0)
q₂2, 2, 00.4223, 0.4223, 0.15540.4683, 0.4683, 0.06340.5, 0.5, 0(0.5, 0.5)
q₃2, 2, 21/3, 1/3, 1/31/3, 1/3, 1/31/3, 1/3, 1/3(0.6667, 0.6667)
Raw scores, scaled weights, unscaled weights and causal weights (checked in PyTorch)

Reading the table

  1. 1Row 1: scaled scores (0.5, 0, 1)(0.5,\,0,\,1); e0.5=1.6487e^{0.5}=1.6487, e0=1e^{0}=1, e1=2.7183e^{1}=2.7183, sum 5.36705.3670, so A1=(0.3072, 0.1863, 0.5065)A_1=(0.3072,\,0.1863,\,0.5065).
  2. 2Without the scale the same row is softmax(1,0,2)=(0.2447, 0.0900, 0.6652)\mathrm{softmax}(1,0,2)=(0.2447,\,0.0900,\,0.6652): already sharper at dk=4d_k=4. The gap grows with dk\sqrt{d_k}.
  3. 3Row 3 has equal scores, so scaling changes nothing: a softmax only sees differences between scores, and dividing equal scores leaves them equal.
  4. 4Causal mask: row 1 keeps only k1k_1, giving weight 1 and output v1=(1,0)v_1=(1,0). Row 2 keeps k1,k2k_1,k_2 with equal scores: (0.5,0.5)(0.5,0.5) and output (0.5,0.5)(0.5,0.5). Row 3 is unchanged. F.scaled_dot_product_attention(Q, K, V, is_causal=True) returns the same three rows.
python
1import math, torch, torch.nn.functional as F
2
3def sdpa(q, k, v, mask=None, scale=None):
4 """softmax(q kᵀ · scale) v. q (..., n_q, d_k), k (..., n_k, d_k), v (..., n_k, d_v).
5 mask: bool, broadcastable to (..., n_q, n_k), True = may attend (PyTorch's convention)."""
6 scale = 1 / math.sqrt(q.size(-1)) if scale is None else scale
7 scores = q @ k.transpose(-2, -1) * scale # (..., n_q, n_k)
8 if mask is not None:
9 scores = scores.masked_fill(~mask, float("-inf")) # exp(-inf) = 0 after the softmax
10 a = scores.softmax(dim=-1)
11 return a @ v, a
12
13torch.manual_seed(0)
14N, h, n, d_k = 2, 4, 6, 64
15q, k, v = (torch.randn(N, h, n, d_k) for _ in range(3))
16
17# causal mask (query i sees keys j <= i) combined with key padding (sequence 1 has 4 real tokens)
18lengths = torch.tensor([6, 4])
19causal = torch.ones(n, n, dtype=torch.bool).tril() # (n, n)
20keep = torch.arange(n)[None, :] < lengths[:, None] # (N, n)
21mask = causal & keep[:, None, None, :] # (N, 1, n, n)
22
23y, a = sdpa(q, k, v, mask)
24print(y.shape, a.shape) # torch.Size([2, 4, 6, 64]) torch.Size([2, 4, 6, 6])
25print(torch.allclose(y, F.scaled_dot_product_attention(q, k, v, attn_mask=mask), atol=1e-5)) # True
26print(a[1, 0, 5]) # tensor([0.4806, 0.0242, 0.3463, 0.1488, 0.0000, 0.0000])
27
28# Why the 1/sqrt(d_k): score variance and softmax saturation
29raw = q @ k.transpose(-2, -1) # 288 scores with i.i.d. N(0, 1) entries
30print(f"{raw.var():.1f} {(raw / math.sqrt(d_k)).var():.2f}") # 75.2 1.17 (theory: 64 and 1)
31for scale in (1.0, 1 / math.sqrt(d_k)):
32 s = (q @ k.transpose(-2, -1) * scale).requires_grad_()
33 s.softmax(-1)[..., 0].sum().backward() # gradient of the weight on key 0
34 print(f"scale {scale:.3f}: max weight {s.softmax(-1).max(-1).values.mean():.2f}, "
35 f"mean |grad| {s.grad.abs().mean():.4f}")
36# scale 1.000: max weight 0.88, mean |grad| 0.0113 <- near one-hot, small gradients
37# scale 0.125: max weight 0.44, mean |grad| 0.0340
38
39# A query whose keys are all masked gives NaN: guard padded query rows.
40bad = torch.zeros(n, n, dtype=torch.bool)
41print(sdpa(q[0, 0], k[0, 0], v[0, 0], bad)[0].isnan().all().item()) # True
Scaled dot-product attention with causal and padding masks, checked against torch.nn.functional.scaled_dot_product_attention

Scaling choices in practice

Transformer (2017)
1/dk1/\sqrt{d_k} with dk=64d_k=64 in both base and big models, so scores are divided by 8
T5
no explicit factor in the attention; the 1/dk1/\sqrt{d_k} is folded into the initialisation of the query projection (Mesh TensorFlow convention)
μP (Yang et al., Tensor Programs V)
1/dk1/d_k instead of 1/dk1/\sqrt{d_k}, so that attention logits stay well-scaled as width grows and hyperparameters transfer across model sizes
QK-norm
normalise qq and kk (LayerNorm or ℓ2\ell_2) before the dot product; Henry et al. (2020) for translation, Dehghani et al. (2023) to stop attention-logit growth in the 22B-parameter ViT
PyTorch
F.scaled_dot_product_attention(q, k, v, attn_mask=None, dropout_p=0.0, is_causal=False, scale=None); scale=None means 1/dk1/\sqrt{d_k}; dispatches to fused kernels such as FlashAttention when it can

Use when

  • +Two matrix multiplies and a softmax: the fastest score function on GPUs and TPUs
  • +No parameters, so it works for any nqn_q, nkn_k and any dvd_v
  • +The 1/dk1/\sqrt{d_k} factor makes initial attention sharpness independent of head size
  • +Masks express causality, padding and sparsity patterns with no change to the function

Avoid when

  • −O(nqnkdk)O(n_qn_kd_k) time and an nq×nkn_q\times n_k score matrix: quadratic for self-attention
  • −A purely bilinear similarity; any non-linear matching must come from the projections
  • −Logits can still grow during training; large models add QK-norm or logit capping
  • −Mask polarity and fully masked rows are a frequent source of silent bugs

Check yourself

Query and key entries are i.i.d. with mean 0 and variance 1, and d_k = 256. What is the standard deviation of qᵀk, and what is it after scaling?

Check yourself

Why is a causal mask applied by setting scores to −∞ before the softmax rather than zeroing weights after it?

Remember

  • ✓Attention(Q,K,V)=softmax(QK⊤/dk+M) V\mathrm{Attention}(Q,K,V)=\mathrm{softmax}(QK^\top/\sqrt{d_k}+M)\,V; Mij=−∞M_{ij}=-\infty removes key jj for query ii before the row softmax.
  • ✓If q,kq,k have i.i.d. unit-variance entries, Var⁡(q⊤k)=dk\operatorname{Var}(q^\top k)=d_k; dividing by dk\sqrt{d_k} restores unit variance and keeps the softmax out of saturation.
  • ✓The softmax Jacobian diag(a)−aa⊤\mathrm{diag}(a)-aa^\top vanishes as aa becomes one-hot, which is why large unscaled scores stall learning.
  • ✓Padding masks depend on the key only; causal masks allow j≤ij\le i; combine them with AND. Fully masked rows give NaN.
  • ✓Output has one row per query: nq×dvn_q\times d_v; nkn_k may differ from nqn_q (cross-attention).
← back to Deep Learning roadmap