# Multi-head attention

> Multi-head attention runs h scaled dot-product attentions in parallel, each on its own d/h-dimensional projection of the queries, keys and values. It concatenates the h outputs and mixes them with W_O, so each position can attend to several places at once for the same cost as one full-width head.

Source: https://www.mlfoundry.in/learn/multi-head-attention  
Section: Deep Learning › Attention and Transformer Architectures  
Level: core · 13 min read · updated 2026-10-12

**Multi-head attention:** is the attention layer of the Transformer. It projects the inputs into $h$ separate query, key and value subspaces of size $d_k=d_v=d_{\text{model}}/h$. It runs **Scaled dot-product attention** independently in each one, concatenates the $h$ results and applies one output projection $W^O$: $\mathrm{MultiHead}(Q,K,V)=\mathrm{Concat}(\mathrm{head}_1,\dots,\mathrm{head}_h)\,W^O$. Each head has its own attention pattern. The total cost is about that of a single head of full width.

Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin introduced multi-head attention in "Attention Is All You Need" (NeurIPS 2017). One attention head returns a single weighted average of the values for each query. If a word needs information from its subject *and* from the previous word, one softmax has to split its mass between them, and the average blurs both. In the paper's words: "Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions. With a single attention head, averaging inhibits this." The base Transformer uses $h=8$ heads of $d_k=64$ in a $d_{\text{model}}=512$ model, in encoder **Self-attention**, decoder **Causal attention** and **Cross-attention** alike.

## The computation

$$
\mathrm{head}_i=\mathrm{softmax}\!\Big(\frac{(QW_i^Q)(KW_i^K)^\top}{\sqrt{d_k}}\Big)\,VW_i^V,\qquad \mathrm{MultiHead}(Q,K,V)=\big[\mathrm{head}_1\;\cdots\;\mathrm{head}_h\big]\,W^O
$$

- $Q,K,V$: query, key and value inputs: $N_q\times d_{\text{model}}$ and $N_k\times d_{\text{model}}$; in self-attention all three are the layer input $X$
- $W_i^Q,W_i^K\in\mathbb R^{d_{\text{model}}\times d_k}$: query and key projections of head $i$
- $W_i^V\in\mathbb R^{d_{\text{model}}\times d_v}$: value projection of head $i$
- $d_k=d_v=d_{\text{model}}/h$: head width; the paper uses $512/8=64$
- $W^O\in\mathbb R^{hd_v\times d_{\text{model}}}$: output projection applied to the concatenated heads

In code nobody builds $h$ separate small matrices. The $h$ query projections side by side form one $d_{\text{model}}\times d_{\text{model}}$ matrix $W^Q=[W_1^Q\cdots W_h^Q]$, and the same holds for $W^K$ and $W^V$. The layer computes $XW^Q$ with one matrix multiply and **reshapes** the result from $(B,N,d_{\text{model}})$ to $(B,N,h,d_k)$, then transposes to $(B,h,N,d_k)$. Head $i$ is simply columns $i\,d_k,\dots,(i+1)d_k-1$. The attention products then run as one batched matmul over the $h$ axis. Concatenation is the inverse reshape back to $(B,N,d_{\text{model}})$.

## What W_O does

$$
\big[\mathrm{head}_1\;\cdots\;\mathrm{head}_h\big]\,W^O=\sum_{i=1}^{h}\mathrm{head}_i\,W^O_{(i)},\qquad W^O=\begin{bmatrix}W^O_{(1)}\\ \vdots\\ W^O_{(h)}\end{bmatrix},\quad W^O_{(i)}\in\mathbb R^{d_v\times d_{\text{model}}}
$$

- $W^O_{(i)}$: the block of $d_v$ rows of $W^O$ that multiplies the output of head $i$
- $\mathrm{head}_i\,W^O_{(i)}$: the contribution of head $i$, written back into the full $d_{\text{model}}$-dimensional residual space

Concatenating and then multiplying by $W^O$ is the same as letting each head write its own $d_{\text{model}}$-dimensional update and **adding** the $h$ updates. So $W^O$ is not a mere reshaping step. Without it, head $i$ could only fill its own $d_v$ coordinates of the output. With it, every head can write in any direction of the residual stream, and heads can cancel or reinforce each other. The per-head factor $W_i^VW^O_{(i)}$ is a $d_{\text{model}}\times d_{\text{model}}$ matrix of rank at most $d_v$, the head's "OV circuit". Likewise $W_i^Q{W_i^K}^\top$ is a rank-$d_k$ bilinear form that decides where head $i$ looks.

## Cost: the same as one wide head

With $d_k=d_{\text{model}}/h$ the four projections hold $4d_{\text{model}}^2$ weights (plus $4d_{\text{model}}$ biases), whatever $h$ is. $QK^\top$ and $AV$ cost $h\cdot2N^2d_k=2N^2d_{\text{model}}$ multiply–adds, also independent of $h$. What grows with $h$ is the number of attention matrices: $h$ maps of size $N\times N$ per layer. That costs memory during training and memory bandwidth in naive kernels (see **FlashAttention**), and the softmax runs $h$ times. What shrinks is $d_k$. Each head's score matrix $Q_iK_i^\top$ has rank at most $d_k$, and with very small heads that limits which patterns a head can express (the "low-rank bottleneck" of Bhojanapalli et al., ICML 2020).

**Configurations in Vaswani et al. (2017)**

- **Transformer base:** $d_{\text{model}}=512$, $h=8$, $d_k=d_v=64$, $d_{ff}=2048$, 6+6 layers, 65M parameters
- **Transformer big:** $d_{\text{model}}=1024$, $h=16$, $d_k=d_v=64$, $d_{ff}=4096$, 213M parameters
- **Per layer, base:** $4\cdot512^2+4\cdot512=1{,}050{,}624$ attention parameters; 8 attention maps per layer
- **Uses:** encoder self-attention, masked decoder self-attention, and encoder–decoder (cross) attention, all multi-head

Varying the number of heads at constant computation (Vaswani et al. 2017, Table 3, rows A; English→German newstest2013 dev set)

| h | dₖ = dᵥ | dev perplexity | dev BLEU |
| --- | --- | --- | --- |
| 1 | 512 | 5.29 | 24.9 |
| 4 | 128 | 5.00 | 25.5 |
| 8 (base) | 64 | 4.92 | 25.8 |
| 16 | 32 | 4.91 | 25.8 |
| 32 | 16 | 5.01 | 25.4 |

A single head is 0.9 BLEU worse than the best setting. Too many heads also hurt: at $h=32$ each head has only 16 dimensions. The parameter count and FLOPs are the same in every row, so the gain comes from splitting the capacity into several independent attention patterns, not from more capacity.

> Figure: One self-attention layer over the 8-token sentence 'the cat sat on the mat and slept' with d = 16. The four projection matrices are fixed, and only their split into heads changes, so the parameter count stays 4d² = 1024 and the QKᵀ + AV cost stays 2N²d = 2048 multiply–adds for every h. With h = 1 (dₖ = 16) the query 'mat' puts 0.60 on 'sat', and every query gets exactly one attention pattern. With h = 4 the same query looks at three different keys (heads 2 and 3 at 'sat', head 1 at 'the', head 4 at 'slept' with weight 0.76). The mean over heads, the grey outline, peaks at only 0.28, which shows how averaging blurs patterns that are sharp per head. Head 3 carries 46% of the output norm after W_O. Averaged over all 8 queries, the number of distinct keys picked by the heads grows from 1 (h = 1) to 1.9, 3.5, 4.75 and 7 (h = 16). At h = 16 each head has dₖ = 1, so its score matrix has rank 1, and the mean row entropy rises from 1.27 to 1.48 nats (uniform would be ln 8 = 2.08). Click a head or move the sliders.

## Worked example: two heads versus one

Three tokens with $d_{\text{model}}=4$: $x_1=(1,0,1,0)$, $x_2=(0,1,0,1)$, $x_3=(1,1,0,0)$. To keep the arithmetic visible take $W^Q=W^K=W^V=I$, so with $h=2$ head 1 sees coordinates 1–2 and head 2 sees coordinates 3–4, each with $d_k=2$. Look at query $x_3$.

Scores and weights for query x₃ (checked numerically in Node)

| row | key x₁ | key x₂ | key x₃ | output |
| --- | --- | --- | --- | --- |
| head 1 score qᵀk/√2 | 0.7071 | 0.7071 | 1.4142 |  |
| head 1 weight | 0.2483 | 0.2483 | 0.5035 | (0.7517, 0.7517) |
| head 2 score | 0 | 0 | 0 |  |
| head 2 weight | 0.3333 | 0.3333 | 0.3333 | (0.3333, 0.3333) |
| single head (dₖ = 4) score qᵀk/2 | 0.5 | 0.5 | 1.0 |  |
| single head weight | 0.2741 | 0.2741 | 0.4519 | (0.7259, 0.7259, 0.2741, 0.2741) |

**From heads to the layer output**

1. Head 1 sees the nonzero half of $x_3$ and attends mostly to itself (0.5035). Head 2 sees its zero half, so $q=0$, every score is 0, and it attends uniformly. Two different patterns from one query: a single head cannot produce both.
2. Concatenate: $[\mathrm{head}_1;\mathrm{head}_2]=(0.7517,\,0.7517,\,0.3333,\,0.3333)$. The single head instead gives one average, $(0.7259,\,0.7259,\,0.2741,\,0.2741)$.
3. Take $W^O=\begin{pmatrix}1&0&0&1\\0&1&1&0\\0&1&-1&0\\1&0&0&-1\end{pmatrix}$. Head 1's block (rows 1–2) gives $0.7517\cdot(1,0,0,1)+0.7517\cdot(0,1,1,0)=(0.7517,0.7517,0.7517,0.7517)$. Head 2's block (rows 3–4) gives $0.3333\cdot(0,1,-1,0)+0.3333\cdot(1,0,0,-1)=(0.3333,0.3333,-0.3333,-0.3333)$.
4. Sum: $(1.0851,\,1.0851,\,0.4184,\,0.4184)$, which equals $[\mathrm{head}_1;\mathrm{head}_2]\,W^O$ computed directly. Head 2 has written into coordinates 1–2 as well, which a plain concatenation could not do.

```python
import torch, torch.nn as nn

class MultiHeadAttention(nn.Module):
    def __init__(self, d_model, n_heads, dropout=0.0):
        super().__init__()
        assert d_model % n_heads == 0
        self.h, self.dk = n_heads, d_model // n_heads
        self.W_q = nn.Linear(d_model, d_model)       # [W_1^Q ... W_h^Q] side by side
        self.W_k = nn.Linear(d_model, d_model)
        self.W_v = nn.Linear(d_model, d_model)
        self.W_o = nn.Linear(d_model, d_model)
        self.drop = nn.Dropout(dropout)

    def split(self, x):                               # (B, N, d) -> (B, h, N, dk)
        B, N, _ = x.shape
        return x.view(B, N, self.h, self.dk).transpose(1, 2)

    def forward(self, q_in, kv_in, mask=None):
        q, k, v = self.split(self.W_q(q_in)), self.split(self.W_k(kv_in)), self.split(self.W_v(kv_in))
        s = q @ k.transpose(-2, -1) / self.dk ** 0.5  # (B, h, Nq, Nk): one score matrix per head
        if mask is not None:                          # mask: True = may attend
            s = s.masked_fill(~mask, float("-inf"))
        a = self.drop(s.softmax(-1))
        o = (a @ v).transpose(1, 2).reshape(q_in.size(0), q_in.size(1), -1)  # concat heads: (B, Nq, d)
        return self.W_o(o), a

torch.manual_seed(0)
B, N, d, h = 2, 10, 512, 8
m = MultiHeadAttention(d, h).eval()
x = torch.randn(B, N, d)
y, a = m(x, x)
print(y.shape, a.shape)                     # torch.Size([2, 10, 512]) torch.Size([2, 8, 10, 10])
print(sum(p.numel() for p in m.parameters()))   # 1050624 = 4*512*512 + 4*512, for any h

# same weights in PyTorch's built-in layer
ref = nn.MultiheadAttention(d, h, batch_first=True).eval()
with torch.no_grad():
    ref.in_proj_weight.copy_(torch.cat([m.W_q.weight, m.W_k.weight, m.W_v.weight]))
    ref.in_proj_bias.copy_(torch.cat([m.W_q.bias, m.W_k.bias, m.W_v.bias]))
    ref.out_proj.weight.copy_(m.W_o.weight)
    ref.out_proj.bias.copy_(m.W_o.bias)
y_ref, a_ref = ref(x, x, x, average_attn_weights=False)
print(torch.allclose(y, y_ref, atol=1e-5), torch.allclose(a, a_ref, atol=1e-5))   # True True

# concat(heads) @ W_O  ==  sum over heads of head_i @ W_O^(i)
with torch.no_grad():
    heads = a @ m.split(m.W_v(x))                     # (B, h, N, dk)
    Wo = m.W_o.weight.view(d, h, d // h)              # (out, head, dk): row block i of W_O^T
    y_sum = torch.einsum("bhnk,ohk->bno", heads, Wo) + m.W_o.bias
print(torch.allclose(y_sum, y, atol=1e-5))            # True
```

Multi-head attention from scratch, checked against torch.nn.MultiheadAttention and against the sum-over-heads form of W_O

> **Reshape, then transpose; not the other way round:** The projected tensor $(B,N,d_{\text{model}})$ must be viewed as $(B,N,h,d_k)$ and **then** transposed to $(B,h,N,d_k)$. Writing `x.view(B, h, N, dk)` directly gives the right shape and silently mixes positions into heads. The model still trains, only worse. The concat step must mirror it: `transpose(1, 2)` followed by `reshape` (or `contiguous().view`), because the transposed tensor is no longer contiguous.

> **Heads are a parameter-free change of structure:** Going from 1 to $h$ heads adds no weights. It replaces one rank-$d_{\text{model}}$ bilinear score $W^Q{W^K}^\top$ by $h$ rank-$d_k$ ones, each with its own softmax. The extra expressiveness comes entirely from the $h$ separate normalizations: a query can now put most of its mass on several different keys at once.

> **Many heads can be removed after training:** Michel, Levy and Neubig, "Are Sixteen Heads Really Better than One?" (NeurIPS 2019), and Voita et al. (ACL 2019) pruned trained Transformers head by head. Many layers keep most of their quality with one or a few heads, and a minority of specialised heads (positional, syntactic, rare-word) do most of the work. Many heads seem to help optimisation more than the final function.

> **At inference the cost is the K/V heads:** During autoregressive decoding every head's keys and values for all past tokens are cached, $2\cdot N\cdot h\cdot d_k$ numbers per layer, and reading that cache dominates each step at long context. Sharing one K/V head across all query heads (**Multi-query attention (MQA)**) or one per group (**Grouped-query attention (GQA)**) cuts it by $h$ or by the group size.

**Strengths**

- Several attention patterns per query for the same parameters and FLOPs as one wide head
- One batched matmul over the head axis; maps directly onto GPU kernels
- Heads specialise (positional, syntactic, copying) and are individually interpretable
- $W^O$ lets every head write anywhere in the residual stream

**Limitations**

- $h$ attention maps of $N\times N$ per layer: memory grows with $h$ unless the kernel is fused
- Small $d_k$ limits each head's score matrix to rank $d_k$; too many heads hurts (Table 3)
- The KV cache stores $h$ key and value heads per layer, which dominates decoding memory
- Many heads are redundant after training

**Question:** A layer has d_model = 768. Going from h = 12 heads to h = 24 heads (dₖ = d_model/h) changes which of these?

- A. The number of projection parameters doubles
- B. The QKᵀ multiply–add count doubles
- C. The number of N × N attention maps doubles and each head's dₖ halves to 32
- D. Nothing at all; the layer computes the same function

**Answer:** C. The number of N × N attention maps doubles and each head's dₖ halves to 32 — $W^Q,W^K,W^V,W^O$ stay $768\times768$ and $h\cdot2N^2d_k=2N^2\cdot768$ is fixed. What changes is the split: 24 softmaxes over rank-32 scores instead of 12 over rank-64 scores. The function changes, so the last option is wrong.

**Question:** Why is concat(head₁ … head_h)·W_O equal to Σᵢ headᵢ·W_O^(i)?

- A. Because softmax is linear
- B. Because a row vector split into blocks times a matrix split into matching row blocks is the sum of the block products
- C. Because the heads are orthogonal
- D. It is only approximately equal

**Answer:** B. Because a row vector split into blocks times a matrix split into matching row blocks is the sum of the block products — $[a_1\;a_2]\begin{bmatrix}B_1\\B_2\end{bmatrix}=a_1B_1+a_2B_2$ is block matrix multiplication. It holds exactly for any heads. This is why each head can be read as writing its own update $\mathrm{head}_iW^O_{(i)}$ into the residual stream.

**Key points**

- $\mathrm{MultiHead}=[\mathrm{head}_1\cdots\mathrm{head}_h]W^O$, $\mathrm{head}_i=\mathrm{softmax}\big(QW_i^Q(KW_i^K)^\top/\sqrt{d_k}\big)VW_i^V$, $d_k=d_{\text{model}}/h$.
- Same cost as one wide head: $4d_{\text{model}}^2$ weights and $2N^2d_{\text{model}}$ MACs for $QK^\top$ and $AV$, for any $h$; base Transformer $h=8$, $d_k=64$.
- Heads are a reshape $(B,N,d)\to(B,h,N,d_k)$ of one big projection; concatenation is the inverse reshape.
- $[\mathrm{head}_1\cdots\mathrm{head}_h]W^O=\sum_i\mathrm{head}_iW^O_{(i)}$: each head writes its own update to the residual stream.
- Vaswani et al. Table 3: h = 1 gives 24.9 dev BLEU vs 25.8 for h = 8; h = 32 (dₖ = 16) drops to 25.4.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
