Multi-head attention

Multi-head attention runs h scaled dot-product attentions in parallel, each on its own d/h-dimensional projection of the queries, keys and values. It concatenates the h outputs and mixes them with W_O, so each position can attend to several places at once for the same cost as one full-width head.

13 min read927 wordscore0 viewsupdated 2026-10-12

Definition

Multi-head attention is the attention layer of the Transformer. It projects the inputs into hh separate query, key and value subspaces of size dk=dv=dmodel/hd_k=d_v=d_{\text{model}}/h. It runs Scaled dot-product attention independently in each one, concatenates the hh results and applies one output projection WOW^O: MultiHead(Q,K,V)=Concat(head1,…,headh) WO\mathrm{MultiHead}(Q,K,V)=\mathrm{Concat}(\mathrm{head}_1,\dots,\mathrm{head}_h)\,W^O. Each head has its own attention pattern. The total cost is about that of a single head of full width.

Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin introduced multi-head attention in "Attention Is All You Need" (NeurIPS 2017). One attention head returns a single weighted average of the values for each query. If a word needs information from its subject and from the previous word, one softmax has to split its mass between them, and the average blurs both. In the paper's words: "Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions. With a single attention head, averaging inhibits this." The base Transformer uses h=8h=8 heads of dk=64d_k=64 in a dmodel=512d_{\text{model}}=512 model, in encoder Self-attention, decoder Causal attention and Cross-attention alike.

The computation

headi=softmax ⁣((QWiQ)(KWiK)⊤dk) VWiV,MultiHead(Q,K,V)=[head1  ⋯  headh] WO\mathrm{head}_i=\mathrm{softmax}\!\Big(\frac{(QW_i^Q)(KW_i^K)^\top}{\sqrt{d_k}}\Big)\,VW_i^V,\qquad \mathrm{MultiHead}(Q,K,V)=\big[\mathrm{head}_1\;\cdots\;\mathrm{head}_h\big]\,W^O
Q,K,VQ,K,V
query, key and value inputs: Nq×dmodelN_q\times d_{\text{model}} and Nk×dmodelN_k\times d_{\text{model}}; in self-attention all three are the layer input XX
WiQ,WiK∈Rdmodel×dkW_i^Q,W_i^K\in\mathbb R^{d_{\text{model}}\times d_k}
query and key projections of head ii
WiV∈Rdmodel×dvW_i^V\in\mathbb R^{d_{\text{model}}\times d_v}
value projection of head ii
dk=dv=dmodel/hd_k=d_v=d_{\text{model}}/h
head width; the paper uses 512/8=64512/8=64
WO∈Rhdv×dmodelW^O\in\mathbb R^{hd_v\times d_{\text{model}}}
output projection applied to the concatenated heads

In code nobody builds hh separate small matrices. The hh query projections side by side form one dmodel×dmodeld_{\text{model}}\times d_{\text{model}} matrix WQ=[W1Q⋯WhQ]W^Q=[W_1^Q\cdots W_h^Q], and the same holds for WKW^K and WVW^V. The layer computes XWQXW^Q with one matrix multiply and reshapes the result from (B,N,dmodel)(B,N,d_{\text{model}}) to (B,N,h,dk)(B,N,h,d_k), then transposes to (B,h,N,dk)(B,h,N,d_k). Head ii is simply columns i dk,…,(i+1)dk−1i\,d_k,\dots,(i+1)d_k-1. The attention products then run as one batched matmul over the hh axis. Concatenation is the inverse reshape back to (B,N,dmodel)(B,N,d_{\text{model}}).

What W_O does

[head1  ⋯  headh] WO=∑i=1hheadi W(i)O,WO=[W(1)O⋮W(h)O],W(i)O∈Rdv×dmodel\big[\mathrm{head}_1\;\cdots\;\mathrm{head}_h\big]\,W^O=\sum_{i=1}^{h}\mathrm{head}_i\,W^O_{(i)},\qquad W^O=\begin{bmatrix}W^O_{(1)}\\ \vdots\\ W^O_{(h)}\end{bmatrix},\quad W^O_{(i)}\in\mathbb R^{d_v\times d_{\text{model}}}
W(i)OW^O_{(i)}
the block of dvd_v rows of WOW^O that multiplies the output of head ii
headi W(i)O\mathrm{head}_i\,W^O_{(i)}
the contribution of head ii, written back into the full dmodeld_{\text{model}}-dimensional residual space

Concatenating and then multiplying by WOW^O is the same as letting each head write its own dmodeld_{\text{model}}-dimensional update and adding the hh updates. So WOW^O is not a mere reshaping step. Without it, head ii could only fill its own dvd_v coordinates of the output. With it, every head can write in any direction of the residual stream, and heads can cancel or reinforce each other. The per-head factor WiVW(i)OW_i^VW^O_{(i)} is a dmodel×dmodeld_{\text{model}}\times d_{\text{model}} matrix of rank at most dvd_v, the head's "OV circuit". Likewise WiQWiK⊤W_i^Q{W_i^K}^\top is a rank-dkd_k bilinear form that decides where head ii looks.

Cost: the same as one wide head

With dk=dmodel/hd_k=d_{\text{model}}/h the four projections hold 4dmodel24d_{\text{model}}^2 weights (plus 4dmodel4d_{\text{model}} biases), whatever hh is. QK⊤QK^\top and AVAV cost h⋅2N2dk=2N2dmodelh\cdot2N^2d_k=2N^2d_{\text{model}} multiply–adds, also independent of hh. What grows with hh is the number of attention matrices: hh maps of size N×NN\times N per layer. That costs memory during training and memory bandwidth in naive kernels (see FlashAttention), and the softmax runs hh times. What shrinks is dkd_k. Each head's score matrix QiKi⊤Q_iK_i^\top has rank at most dkd_k, and with very small heads that limits which patterns a head can express (the "low-rank bottleneck" of Bhojanapalli et al., ICML 2020).

Configurations in Vaswani et al. (2017)

Transformer base
dmodel=512d_{\text{model}}=512, h=8h=8, dk=dv=64d_k=d_v=64, dff=2048d_{ff}=2048, 6+6 layers, 65M parameters
Transformer big
dmodel=1024d_{\text{model}}=1024, h=16h=16, dk=dv=64d_k=d_v=64, dff=4096d_{ff}=4096, 213M parameters
Per layer, base
4⋅5122+4⋅512=1,050,6244\cdot512^2+4\cdot512=1{,}050{,}624 attention parameters; 8 attention maps per layer
Uses
encoder self-attention, masked decoder self-attention, and encoder–decoder (cross) attention, all multi-head
hdₖ = dᵥdev perplexitydev BLEU
15125.2924.9
41285.0025.5
8 (base)644.9225.8
16324.9125.8
32165.0125.4
Varying the number of heads at constant computation (Vaswani et al. 2017, Table 3, rows A; English→German newstest2013 dev set)

A single head is 0.9 BLEU worse than the best setting. Too many heads also hurt: at h=32h=32 each head has only 16 dimensions. The parameter count and FLOPs are the same in every row, so the gain comes from splitting the capacity into several independent attention patterns, not from more capacity.

headᵢ = softmax(QWᵢ^Q (KWᵢ^K)ᵀ/√dₖ)·VWᵢ^V · out = [head₁ … head_h]·W_Ohead 1head 2head 3head 4rows = queries, cols = keys · ▭ query · □ argmax · click a head“mat”: ■ head 1 · □ mean of 4 heads · top = 0.32thecatsatonthematandsleptheads h 4dₖ = d/h 4params 4d² 1024MACs QKᵀ+AV 2048rank(QᵢKᵢᵀ) ≤ 4query “mat”:distinct argmax 3 of 4max weight, head 1 0.32max weight, mean 0.28mean entropy 1.39 natsshare ‖headᵢ·W_O⁽ⁱ⁾‖h10.18h20.19h30.46h40.16d = 16, N = 8 · same W_Q, W_K, W_V, W_O for every h; only the head split changes
heads h
fig.One self-attention layer over the 8-token sentence 'the cat sat on the mat and slept' with d = 16. The four projection matrices are fixed, and only their split into heads changes, so the parameter count stays 4d² = 1024 and the QKᵀ + AV cost stays 2N²d = 2048 multiply–adds for every h. With h = 1 (dₖ = 16) the query 'mat' puts 0.60 on 'sat', and every query gets exactly one attention pattern. With h = 4 the same query looks at three different keys (heads 2 and 3 at 'sat', head 1 at 'the', head 4 at 'slept' with weight 0.76). The mean over heads, the grey outline, peaks at only 0.28, which shows how averaging blurs patterns that are sharp per head. Head 3 carries 46% of the output norm after W_O. Averaged over all 8 queries, the number of distinct keys picked by the heads grows from 1 (h = 1) to 1.9, 3.5, 4.75 and 7 (h = 16). At h = 16 each head has dₖ = 1, so its score matrix has rank 1, and the mean row entropy rises from 1.27 to 1.48 nats (uniform would be ln 8 = 2.08). Click a head or move the sliders.

Worked example: two heads versus one

Three tokens with dmodel=4d_{\text{model}}=4: x1=(1,0,1,0)x_1=(1,0,1,0), x2=(0,1,0,1)x_2=(0,1,0,1), x3=(1,1,0,0)x_3=(1,1,0,0). To keep the arithmetic visible take WQ=WK=WV=IW^Q=W^K=W^V=I, so with h=2h=2 head 1 sees coordinates 1–2 and head 2 sees coordinates 3–4, each with dk=2d_k=2. Look at query x3x_3.

rowkey x₁key x₂key x₃output
head 1 score qᵀk/√20.70710.70711.4142
head 1 weight0.24830.24830.5035(0.7517, 0.7517)
head 2 score000
head 2 weight0.33330.33330.3333(0.3333, 0.3333)
single head (dₖ = 4) score qᵀk/20.50.51.0
single head weight0.27410.27410.4519(0.7259, 0.7259, 0.2741, 0.2741)
Scores and weights for query x₃ (checked numerically in Node)

From heads to the layer output

  1. 1Head 1 sees the nonzero half of x3x_3 and attends mostly to itself (0.5035). Head 2 sees its zero half, so q=0q=0, every score is 0, and it attends uniformly. Two different patterns from one query: a single head cannot produce both.
  2. 2Concatenate: [head1;head2]=(0.7517, 0.7517, 0.3333, 0.3333)[\mathrm{head}_1;\mathrm{head}_2]=(0.7517,\,0.7517,\,0.3333,\,0.3333). The single head instead gives one average, (0.7259, 0.7259, 0.2741, 0.2741)(0.7259,\,0.7259,\,0.2741,\,0.2741).
  3. 3Take WO=(1001011001−10100−1)W^O=\begin{pmatrix}1&0&0&1\\0&1&1&0\\0&1&-1&0\\1&0&0&-1\end{pmatrix}. Head 1's block (rows 1–2) gives 0.7517⋅(1,0,0,1)+0.7517⋅(0,1,1,0)=(0.7517,0.7517,0.7517,0.7517)0.7517\cdot(1,0,0,1)+0.7517\cdot(0,1,1,0)=(0.7517,0.7517,0.7517,0.7517). Head 2's block (rows 3–4) gives 0.3333⋅(0,1,−1,0)+0.3333⋅(1,0,0,−1)=(0.3333,0.3333,−0.3333,−0.3333)0.3333\cdot(0,1,-1,0)+0.3333\cdot(1,0,0,-1)=(0.3333,0.3333,-0.3333,-0.3333).
  4. 4Sum: (1.0851, 1.0851, 0.4184, 0.4184)(1.0851,\,1.0851,\,0.4184,\,0.4184), which equals [head1;head2] WO[\mathrm{head}_1;\mathrm{head}_2]\,W^O computed directly. Head 2 has written into coordinates 1–2 as well, which a plain concatenation could not do.
python
1import torch, torch.nn as nn
2
3class MultiHeadAttention(nn.Module):
4 def __init__(self, d_model, n_heads, dropout=0.0):
5 super().__init__()
6 assert d_model % n_heads == 0
7 self.h, self.dk = n_heads, d_model // n_heads
8 self.W_q = nn.Linear(d_model, d_model) # [W_1^Q ... W_h^Q] side by side
9 self.W_k = nn.Linear(d_model, d_model)
10 self.W_v = nn.Linear(d_model, d_model)
11 self.W_o = nn.Linear(d_model, d_model)
12 self.drop = nn.Dropout(dropout)
13
14 def split(self, x): # (B, N, d) -> (B, h, N, dk)
15 B, N, _ = x.shape
16 return x.view(B, N, self.h, self.dk).transpose(1, 2)
17
18 def forward(self, q_in, kv_in, mask=None):
19 q, k, v = self.split(self.W_q(q_in)), self.split(self.W_k(kv_in)), self.split(self.W_v(kv_in))
20 s = q @ k.transpose(-2, -1) / self.dk ** 0.5 # (B, h, Nq, Nk): one score matrix per head
21 if mask is not None: # mask: True = may attend
22 s = s.masked_fill(~mask, float("-inf"))
23 a = self.drop(s.softmax(-1))
24 o = (a @ v).transpose(1, 2).reshape(q_in.size(0), q_in.size(1), -1) # concat heads: (B, Nq, d)
25 return self.W_o(o), a
26
27torch.manual_seed(0)
28B, N, d, h = 2, 10, 512, 8
29m = MultiHeadAttention(d, h).eval()
30x = torch.randn(B, N, d)
31y, a = m(x, x)
32print(y.shape, a.shape) # torch.Size([2, 10, 512]) torch.Size([2, 8, 10, 10])
33print(sum(p.numel() for p in m.parameters())) # 1050624 = 4*512*512 + 4*512, for any h
34
35# same weights in PyTorch's built-in layer
36ref = nn.MultiheadAttention(d, h, batch_first=True).eval()
37with torch.no_grad():
38 ref.in_proj_weight.copy_(torch.cat([m.W_q.weight, m.W_k.weight, m.W_v.weight]))
39 ref.in_proj_bias.copy_(torch.cat([m.W_q.bias, m.W_k.bias, m.W_v.bias]))
40 ref.out_proj.weight.copy_(m.W_o.weight)
41 ref.out_proj.bias.copy_(m.W_o.bias)
42y_ref, a_ref = ref(x, x, x, average_attn_weights=False)
43print(torch.allclose(y, y_ref, atol=1e-5), torch.allclose(a, a_ref, atol=1e-5)) # True True
44
45# concat(heads) @ W_O == sum over heads of head_i @ W_O^(i)
46with torch.no_grad():
47 heads = a @ m.split(m.W_v(x)) # (B, h, N, dk)
48 Wo = m.W_o.weight.view(d, h, d // h) # (out, head, dk): row block i of W_O^T
49 y_sum = torch.einsum("bhnk,ohk->bno", heads, Wo) + m.W_o.bias
50print(torch.allclose(y_sum, y, atol=1e-5)) # True
Multi-head attention from scratch, checked against torch.nn.MultiheadAttention and against the sum-over-heads form of W_O

Use when

  • +Several attention patterns per query for the same parameters and FLOPs as one wide head
  • +One batched matmul over the head axis; maps directly onto GPU kernels
  • +Heads specialise (positional, syntactic, copying) and are individually interpretable
  • +WOW^O lets every head write anywhere in the residual stream

Avoid when

  • −hh attention maps of N×NN\times N per layer: memory grows with hh unless the kernel is fused
  • −Small dkd_k limits each head's score matrix to rank dkd_k; too many heads hurts (Table 3)
  • −The KV cache stores hh key and value heads per layer, which dominates decoding memory
  • −Many heads are redundant after training

Check yourself

A layer has d_model = 768. Going from h = 12 heads to h = 24 heads (dₖ = d_model/h) changes which of these?

Check yourself

Why is concat(head₁ … head_h)·W_O equal to Σᵢ headᵢ·W_O^(i)?

Remember

  • ✓MultiHead=[head1⋯headh]WO\mathrm{MultiHead}=[\mathrm{head}_1\cdots\mathrm{head}_h]W^O, headi=softmax(QWiQ(KWiK)⊤/dk)VWiV\mathrm{head}_i=\mathrm{softmax}\big(QW_i^Q(KW_i^K)^\top/\sqrt{d_k}\big)VW_i^V, dk=dmodel/hd_k=d_{\text{model}}/h.
  • ✓Same cost as one wide head: 4dmodel24d_{\text{model}}^2 weights and 2N2dmodel2N^2d_{\text{model}} MACs for QK⊤QK^\top and AVAV, for any hh; base Transformer h=8h=8, dk=64d_k=64.
  • ✓Heads are a reshape (B,N,d)→(B,h,N,dk)(B,N,d)\to(B,h,N,d_k) of one big projection; concatenation is the inverse reshape.
  • ✓[head1⋯headh]WO=∑iheadiW(i)O[\mathrm{head}_1\cdots\mathrm{head}_h]W^O=\sum_i\mathrm{head}_iW^O_{(i)}: each head writes its own update to the residual stream.
  • ✓Vaswani et al. Table 3: h = 1 gives 24.9 dev BLEU vs 25.8 for h = 8; h = 32 (dₖ = 16) drops to 25.4.
← back to Deep Learning roadmap