# Scaled dot-product attention

> Attention(Q, K, V) = softmax(QKᵀ/√d_k + M)V. Dividing by √d_k keeps scores at unit variance for any key size, so the softmax does not saturate, and a mask M of 0 and −∞ entries removes keys (future tokens, padding) before the softmax.

Source: https://www.mlfoundry.in/learn/scaled-dot-product-attention  
Section: Deep Learning › Attention and Transformer Architectures  
Level: core · 13 min read · updated 2026-10-12

**Scaled dot-product attention:** is the attention function of the Transformer: $\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\big(QK^\top/\sqrt{d_k}\big)V$. Every query is compared with every key by a dot product, the scores are divided by the square root of the key dimension $d_k$, an optional **mask** sets forbidden scores to $-\infty$, a row-wise softmax turns the scores into weights, and the weights average the values. It has no parameters of its own; the projections that produce $Q$, $K$ and $V$ belong to the surrounding layer.

Vaswani et al. named and defined it in "Attention Is All You Need" (NeurIPS 2017). Two score functions were in common use (see **Attention mechanism**). Additive attention (Bahdanau et al., 2014) scores with a one-hidden-layer MLP. Dot-product attention (Luong et al., 2015) scores with $q^\top k$. The authors chose the dot product because "it can be implemented using highly optimized matrix multiplication code", which makes it much faster and more space-efficient in practice. They also noted a problem: for small $d_k$ the two perform similarly, but additive attention beats unscaled dot-product attention for larger $d_k$, as the large NMT architecture study of Britz et al. (EMNLP 2017) had found. Their fix is one constant factor.

## The formula

$$
\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\Big(\frac{QK^\top}{\sqrt{d_k}}+M\Big)V,\qquad Q\in\mathbb R^{n_q\times d_k},\ K\in\mathbb R^{n_k\times d_k},\ V\in\mathbb R^{n_k\times d_v},\ M_{ij}\in\{0,-\infty\}
$$

- $QK^\top$: $n_q\times n_k$ matrix of dot products $q_i^\top k_j$, one matrix multiply
- $1/\sqrt{d_k}$: scaling factor; equivalently a softmax temperature $\tau=\sqrt{d_k}$
- $M$: additive mask: $0$ where query $i$ may see key $j$, $-\infty$ where it may not; $e^{-\infty}=0$, so masked keys get exactly zero weight
- $\mathrm{softmax}$: applied to each row separately, so every query's weights are non-negative and sum to 1
- $V$: one value per key; output row $i$ is $\sum_j A_{ij}v_j\in\mathbb R^{d_v}$

The shapes say what the function does and does not assume. Queries and keys must share $d_k$; values may have any $d_v$. The number of outputs equals the number of queries $n_q$, and the number of keys $n_k$ may differ, which is what **Cross-attention** uses. With $Q$, $K$, $V$ from one sequence it is **Self-attention**. Batches and heads are leading dimensions: one call computes $B\times h$ independent attentions, which is how **Multi-head attention** is implemented.

## Why divide by √d_k

The paper's footnote gives the argument. Suppose the components of $q$ and $k$ are independent with mean 0 and variance 1, roughly what layer normalisation and standard initialisation produce. Then each product $q_ik_i$ has mean 0 and variance 1, and the dot product is a sum of $d_k$ of them:

$$
\mathbb E[q^\top k]=\sum_{i=1}^{d_k}\mathbb E[q_i]\,\mathbb E[k_i]=0,\qquad \operatorname{Var}(q^\top k)=\sum_{i=1}^{d_k}\operatorname{Var}(q_ik_i)=\sum_{i=1}^{d_k}\mathbb E[q_i^2]\,\mathbb E[k_i^2]=d_k,\qquad \operatorname{Var}\!\Big(\frac{q^\top k}{\sqrt{d_k}}\Big)=1
$$

- $d_k=64$: unscaled scores have standard deviation 8: typical gaps of ±8 between keys make $e^{s}$ differ by factors of thousands
- $\sqrt{d_k}$: the standard deviation, not the variance, is divided out, so the scores have unit scale whatever the head size

Large scores matter because of the softmax Jacobian. With $a=\mathrm{softmax}(s)$, $\partial a/\partial s=\mathrm{diag}(a)-aa^\top$. When one score is far above the others, $a$ is close to one-hot, every entry of $\mathrm{diag}(a)-aa^\top$ is close to 0, and the scores receive almost no gradient. In the authors' words, large dot products push "the softmax function into regions where it has extremely small gradients". After scaling, the spread of the scores, and with it the sharpness of the attention at initialisation, no longer depends on the head size. Training can still sharpen attention by growing the norms of $W_Q$ and $W_K$; the scale only removes the dependence on $d_k$ at the start.

> Figure: One query and 8 keys with i.i.d. N(0, 1) entries. Slide d_k from 1 to 1024, toggle the scaling, and step through 200 seeded draws. Statistics over the 200 draws at d_k = 64: unscaled scores have standard deviation 8.09, the largest weight averages 0.886, the entropy is 0.293 nats (the maximum is ln 8 = 2.079), and the softmax Jacobian ‖diag(a) − aaᵀ‖_F averages 0.137. Divided by √64 = 8, the same scores have standard deviation 1.01, largest weight 0.370, entropy 1.708 and Jacobian norm 0.365. In the first draw the unscaled scores reach 18.2 and the weights are one-hot (1.000). Scaled, that key gets 0.468. At d_k = 1024 the unscaled largest weight is 0.975 and the Jacobian norm 0.034, while the scaled curve stays between 0.31 and 0.38 for every d_k. The right panel applies masks to a fixed 6 × 6 score matrix whose last two tokens are padding. Causal masking removes 15 of 36 entries, padding removes 12, and both together remove 18. The first query then attends only to itself (weight 1.00). Every row still sums to 1 because −∞ becomes an exact 0 before the normalisation.

## Masking

A mask decides which keys each query may see, and it must be applied **before** the softmax. Setting $s_{ij}=-\infty$ gives $e^{s_{ij}}=0$, and the remaining weights renormalise among the allowed keys. Zeroing weights after the softmax would leave rows that no longer sum to 1. Two masks cover almost all uses. A **padding mask** removes key positions that only exist because sequences in a batch were padded to the same length; it depends on the key index only. A **causal mask** removes keys $j>i$, so position $i$ cannot see the future; it is what makes a decoder autoregressive (see **Causal attention**). They are combined with a logical AND, or by adding their $0/-\infty$ matrices. Masks can also encode other patterns: blocks for packed sequences, local windows (see **Sparse attention**), or prefix-LM masks that are bidirectional over a prompt and causal afterwards.

Common masks M (query i, key j)

| mask | allowed when | used in |
| --- | --- | --- |
| none | always | encoder self-attention on unpadded input |
| padding | key $j$ is a real token | any layer over a padded batch; cross-attention to a padded encoder output |
| causal | $j\le i$ | decoder self-attention, GPT-style language models |
| causal + padding | $j\le i$ and $j$ real | batched decoder training on sequences of different lengths |
| block-diagonal | $i$ and $j$ in the same packed document | sequence packing without cross-document attention |

## Worked example: three queries, d_k = 4

Queries $q_1=(1,0,1,0)$, $q_2=(0,2,0,0)$, $q_3=(1,1,1,1)$; keys $k_1=(1,1,0,0)$, $k_2=(0,1,0,1)$, $k_3=(2,0,0,0)$; values $v_1=(1,0)$, $v_2=(0,1)$, $v_3=(1,1)$. The scale is $1/\sqrt4=1/2$.

Raw scores, scaled weights, unscaled weights and causal weights (checked in PyTorch)

| query | qᵀk₁, qᵀk₂, qᵀk₃ | A (÷√dₖ) | A (unscaled) | A (causal, ÷√dₖ) | output (causal) |
| --- | --- | --- | --- | --- | --- |
| q₁ | 1, 0, 2 | 0.3072, 0.1863, 0.5065 | 0.2447, 0.0900, 0.6652 | 1, 0, 0 | (1, 0) |
| q₂ | 2, 2, 0 | 0.4223, 0.4223, 0.1554 | 0.4683, 0.4683, 0.0634 | 0.5, 0.5, 0 | (0.5, 0.5) |
| q₃ | 2, 2, 2 | 1/3, 1/3, 1/3 | 1/3, 1/3, 1/3 | 1/3, 1/3, 1/3 | (0.6667, 0.6667) |

**Reading the table**

1. Row 1: scaled scores $(0.5,\,0,\,1)$; $e^{0.5}=1.6487$, $e^{0}=1$, $e^{1}=2.7183$, sum $5.3670$, so $A_1=(0.3072,\,0.1863,\,0.5065)$.
2. Without the scale the same row is $\mathrm{softmax}(1,0,2)=(0.2447,\,0.0900,\,0.6652)$: already sharper at $d_k=4$. The gap grows with $\sqrt{d_k}$.
3. Row 3 has equal scores, so scaling changes nothing: a softmax only sees differences between scores, and dividing equal scores leaves them equal.
4. Causal mask: row 1 keeps only $k_1$, giving weight 1 and output $v_1=(1,0)$. Row 2 keeps $k_1,k_2$ with equal scores: $(0.5,0.5)$ and output $(0.5,0.5)$. Row 3 is unchanged. `F.scaled_dot_product_attention(Q, K, V, is_causal=True)` returns the same three rows.

```python
import math, torch, torch.nn.functional as F

def sdpa(q, k, v, mask=None, scale=None):
    """softmax(q kᵀ · scale) v. q (..., n_q, d_k), k (..., n_k, d_k), v (..., n_k, d_v).
    mask: bool, broadcastable to (..., n_q, n_k), True = may attend (PyTorch's convention)."""
    scale = 1 / math.sqrt(q.size(-1)) if scale is None else scale
    scores = q @ k.transpose(-2, -1) * scale                 # (..., n_q, n_k)
    if mask is not None:
        scores = scores.masked_fill(~mask, float("-inf"))     # exp(-inf) = 0 after the softmax
    a = scores.softmax(dim=-1)
    return a @ v, a

torch.manual_seed(0)
N, h, n, d_k = 2, 4, 6, 64
q, k, v = (torch.randn(N, h, n, d_k) for _ in range(3))

# causal mask (query i sees keys j <= i) combined with key padding (sequence 1 has 4 real tokens)
lengths = torch.tensor([6, 4])
causal = torch.ones(n, n, dtype=torch.bool).tril()                       # (n, n)
keep = torch.arange(n)[None, :] < lengths[:, None]                       # (N, n)
mask = causal & keep[:, None, None, :]                                   # (N, 1, n, n)

y, a = sdpa(q, k, v, mask)
print(y.shape, a.shape)                                   # torch.Size([2, 4, 6, 64]) torch.Size([2, 4, 6, 6])
print(torch.allclose(y, F.scaled_dot_product_attention(q, k, v, attn_mask=mask), atol=1e-5))   # True
print(a[1, 0, 5])                                         # tensor([0.4806, 0.0242, 0.3463, 0.1488, 0.0000, 0.0000])

# Why the 1/sqrt(d_k): score variance and softmax saturation
raw = q @ k.transpose(-2, -1)                             # 288 scores with i.i.d. N(0, 1) entries
print(f"{raw.var():.1f} {(raw / math.sqrt(d_k)).var():.2f}")   # 75.2 1.17  (theory: 64 and 1)
for scale in (1.0, 1 / math.sqrt(d_k)):
    s = (q @ k.transpose(-2, -1) * scale).requires_grad_()
    s.softmax(-1)[..., 0].sum().backward()                # gradient of the weight on key 0
    print(f"scale {scale:.3f}: max weight {s.softmax(-1).max(-1).values.mean():.2f}, "
          f"mean |grad| {s.grad.abs().mean():.4f}")
# scale 1.000: max weight 0.88, mean |grad| 0.0113   <- near one-hot, small gradients
# scale 0.125: max weight 0.44, mean |grad| 0.0340

# A query whose keys are all masked gives NaN: guard padded query rows.
bad = torch.zeros(n, n, dtype=torch.bool)
print(sdpa(q[0, 0], k[0, 0], v[0, 0], bad)[0].isnan().all().item())     # True
```

Scaled dot-product attention with causal and padding masks, checked against torch.nn.functional.scaled_dot_product_attention

**Scaling choices in practice**

- **Transformer (2017):** $1/\sqrt{d_k}$ with $d_k=64$ in both base and big models, so scores are divided by 8
- **T5:** no explicit factor in the attention; the $1/\sqrt{d_k}$ is folded into the initialisation of the query projection (Mesh TensorFlow convention)
- **μP (Yang et al., Tensor Programs V):** $1/d_k$ instead of $1/\sqrt{d_k}$, so that attention logits stay well-scaled as width grows and hyperparameters transfer across model sizes
- **QK-norm:** normalise $q$ and $k$ (LayerNorm or $\ell_2$) before the dot product; Henry et al. (2020) for translation, Dehghani et al. (2023) to stop attention-logit growth in the 22B-parameter ViT
- **PyTorch:** `F.scaled_dot_product_attention(q, k, v, attn_mask=None, dropout_p=0.0, is_causal=False, scale=None)`; `scale=None` means $1/\sqrt{d_k}$; dispatches to fused kernels such as **FlashAttention** when it can

> **Boolean masks have opposite conventions:** In `F.scaled_dot_product_attention` a boolean `attn_mask` is **True where attention is allowed**. In `nn.MultiheadAttention`, `key_padding_mask` and boolean `attn_mask` are **True where attention is blocked**. Hugging Face models take an `attention_mask` of 1 for real tokens and 0 for padding and convert it to an additive mask internally. Passing a mask with the wrong polarity raises no error: the model attends only to padding or only to the future, and the loss looks merely bad.

> **Fully masked rows and −∞ in low precision:** If every key of a query is masked, the softmax computes $0/0$ and returns NaN, which then spreads through the backward pass. This happens for padded *query* positions when the same mask is applied to keys and queries, or with left padding plus a causal mask. Make sure each row keeps at least one key, or zero those rows afterwards. Many codebases use `torch.finfo(dtype).min` instead of $-\infty$: it avoids the NaN (a fully masked row becomes uniform), and it does not overflow in fp16, which a constant such as $-10^{9}$ does.

> **The scale is a temperature:** $\mathrm{softmax}(s/\tau)$ with $\tau=\sqrt{d_k}$: the factor sets how peaked attention is at initialisation, nothing more. A learned temperature, QK-norm with a learnable gain, or a different power of $d_k$ are all reasonable alternatives, and they matter mainly for training stability at scale. What would not work is no factor at all with large heads: then the attention is almost one-hot at initialisation and its gradients are tiny, as the figure shows.

**Strengths**

- Two matrix multiplies and a softmax: the fastest score function on GPUs and TPUs
- No parameters, so it works for any $n_q$, $n_k$ and any $d_v$
- The $1/\sqrt{d_k}$ factor makes initial attention sharpness independent of head size
- Masks express causality, padding and sparsity patterns with no change to the function

**Limitations**

- $O(n_qn_kd_k)$ time and an $n_q\times n_k$ score matrix: quadratic for self-attention
- A purely bilinear similarity; any non-linear matching must come from the projections
- Logits can still grow during training; large models add QK-norm or logit capping
- Mask polarity and fully masked rows are a frequent source of silent bugs

**Question:** Query and key entries are i.i.d. with mean 0 and variance 1, and d_k = 256. What is the standard deviation of qᵀk, and what is it after scaling?

- A. 256, then 1
- B. 16, then 1
- C. 16, then 1/16
- D. 1, then 1/16

**Answer:** B. 16, then 1 — $\operatorname{Var}(q^\top k)=d_k=256$, so the standard deviation is $\sqrt{256}=16$. Dividing by $\sqrt{d_k}=16$ gives standard deviation 1. Dividing by $d_k$ (the μP choice) would give $1/16$ at initialisation.

**Question:** Why is a causal mask applied by setting scores to −∞ before the softmax rather than zeroing weights after it?

- A. Because −∞ is cheaper to store
- B. So that the remaining weights of each row renormalise to sum to 1 over the allowed keys
- C. Because the softmax cannot output zeros otherwise
- D. It makes no difference

**Answer:** B. So that the remaining weights of each row renormalise to sum to 1 over the allowed keys — $e^{-\infty}=0$, so masked keys drop out of both the numerator and the normaliser, and the allowed weights still sum to 1. Zeroing after the softmax would remove the masked weights from the numerator only: the rows would sum to less than 1, and the future would still have influenced the normaliser.

**Key points**

- $\mathrm{Attention}(Q,K,V)=\mathrm{softmax}(QK^\top/\sqrt{d_k}+M)\,V$; $M_{ij}=-\infty$ removes key $j$ for query $i$ before the row softmax.
- If $q,k$ have i.i.d. unit-variance entries, $\operatorname{Var}(q^\top k)=d_k$; dividing by $\sqrt{d_k}$ restores unit variance and keeps the softmax out of saturation.
- The softmax Jacobian $\mathrm{diag}(a)-aa^\top$ vanishes as $a$ becomes one-hot, which is why large unscaled scores stall learning.
- Padding masks depend on the key only; causal masks allow $j\le i$; combine them with AND. Fully masked rows give NaN.
- Output has one row per query: $n_q\times d_v$; $n_k$ may differ from $n_q$ (cross-attention).

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
