Definition
Multi-head attention is the attention layer of the Transformer. It projects the inputs into separate query, key and value subspaces of size . It runs Scaled dot-product attention independently in each one, concatenates the results and applies one output projection : . Each head has its own attention pattern. The total cost is about that of a single head of full width.
Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin introduced multi-head attention in "Attention Is All You Need" (NeurIPS 2017). One attention head returns a single weighted average of the values for each query. If a word needs information from its subject and from the previous word, one softmax has to split its mass between them, and the average blurs both. In the paper's words: "Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions. With a single attention head, averaging inhibits this." The base Transformer uses heads of in a model, in encoder Self-attention, decoder Causal attention and Cross-attention alike.
The computation
- query, key and value inputs: and ; in self-attention all three are the layer input
- query and key projections of head
- value projection of head
- head width; the paper uses
- output projection applied to the concatenated heads
In code nobody builds separate small matrices. The query projections side by side form one matrix , and the same holds for and . The layer computes with one matrix multiply and reshapes the result from to , then transposes to . Head is simply columns . The attention products then run as one batched matmul over the axis. Concatenation is the inverse reshape back to .
What W_O does
- the block of rows of that multiplies the output of head
- the contribution of head , written back into the full -dimensional residual space
Concatenating and then multiplying by is the same as letting each head write its own -dimensional update and adding the updates. So is not a mere reshaping step. Without it, head could only fill its own coordinates of the output. With it, every head can write in any direction of the residual stream, and heads can cancel or reinforce each other. The per-head factor is a matrix of rank at most , the head's "OV circuit". Likewise is a rank- bilinear form that decides where head looks.
Cost: the same as one wide head
With the four projections hold weights (plus biases), whatever is. and cost multiply–adds, also independent of . What grows with is the number of attention matrices: maps of size per layer. That costs memory during training and memory bandwidth in naive kernels (see FlashAttention), and the softmax runs times. What shrinks is . Each head's score matrix has rank at most , and with very small heads that limits which patterns a head can express (the "low-rank bottleneck" of Bhojanapalli et al., ICML 2020).
Configurations in Vaswani et al. (2017)
- Transformer base
- , , , , 6+6 layers, 65M parameters
- Transformer big
- , , , , 213M parameters
- Per layer, base
- attention parameters; 8 attention maps per layer
- Uses
- encoder self-attention, masked decoder self-attention, and encoder–decoder (cross) attention, all multi-head
| h | dₖ = dᵥ | dev perplexity | dev BLEU |
|---|---|---|---|
| 1 | 512 | 5.29 | 24.9 |
| 4 | 128 | 5.00 | 25.5 |
| 8 (base) | 64 | 4.92 | 25.8 |
| 16 | 32 | 4.91 | 25.8 |
| 32 | 16 | 5.01 | 25.4 |
A single head is 0.9 BLEU worse than the best setting. Too many heads also hurt: at each head has only 16 dimensions. The parameter count and FLOPs are the same in every row, so the gain comes from splitting the capacity into several independent attention patterns, not from more capacity.
Worked example: two heads versus one
Three tokens with : , , . To keep the arithmetic visible take , so with head 1 sees coordinates 1–2 and head 2 sees coordinates 3–4, each with . Look at query .
| row | key x₁ | key x₂ | key x₃ | output |
|---|---|---|---|---|
| head 1 score qᵀk/√2 | 0.7071 | 0.7071 | 1.4142 | |
| head 1 weight | 0.2483 | 0.2483 | 0.5035 | (0.7517, 0.7517) |
| head 2 score | 0 | 0 | 0 | |
| head 2 weight | 0.3333 | 0.3333 | 0.3333 | (0.3333, 0.3333) |
| single head (dₖ = 4) score qᵀk/2 | 0.5 | 0.5 | 1.0 | |
| single head weight | 0.2741 | 0.2741 | 0.4519 | (0.7259, 0.7259, 0.2741, 0.2741) |
From heads to the layer output
- 1Head 1 sees the nonzero half of and attends mostly to itself (0.5035). Head 2 sees its zero half, so , every score is 0, and it attends uniformly. Two different patterns from one query: a single head cannot produce both.
- 2Concatenate: . The single head instead gives one average, .
- 3Take . Head 1's block (rows 1–2) gives . Head 2's block (rows 3–4) gives .
- 4Sum: , which equals computed directly. Head 2 has written into coordinates 1–2 as well, which a plain concatenation could not do.
1import torch, torch.nn as nn23class MultiHeadAttention(nn.Module):4 def __init__(self, d_model, n_heads, dropout=0.0):5 super().__init__()6 assert d_model % n_heads == 07 self.h, self.dk = n_heads, d_model // n_heads8 self.W_q = nn.Linear(d_model, d_model) # [W_1^Q ... W_h^Q] side by side9 self.W_k = nn.Linear(d_model, d_model)10 self.W_v = nn.Linear(d_model, d_model)11 self.W_o = nn.Linear(d_model, d_model)12 self.drop = nn.Dropout(dropout)1314 def split(self, x): # (B, N, d) -> (B, h, N, dk)15 B, N, _ = x.shape16 return x.view(B, N, self.h, self.dk).transpose(1, 2)1718 def forward(self, q_in, kv_in, mask=None):19 q, k, v = self.split(self.W_q(q_in)), self.split(self.W_k(kv_in)), self.split(self.W_v(kv_in))20 s = q @ k.transpose(-2, -1) / self.dk ** 0.5 # (B, h, Nq, Nk): one score matrix per head21 if mask is not None: # mask: True = may attend22 s = s.masked_fill(~mask, float("-inf"))23 a = self.drop(s.softmax(-1))24 o = (a @ v).transpose(1, 2).reshape(q_in.size(0), q_in.size(1), -1) # concat heads: (B, Nq, d)25 return self.W_o(o), a2627torch.manual_seed(0)28B, N, d, h = 2, 10, 512, 829m = MultiHeadAttention(d, h).eval()30x = torch.randn(B, N, d)31y, a = m(x, x)32print(y.shape, a.shape) # torch.Size([2, 10, 512]) torch.Size([2, 8, 10, 10])33print(sum(p.numel() for p in m.parameters())) # 1050624 = 4*512*512 + 4*512, for any h3435# same weights in PyTorch's built-in layer36ref = nn.MultiheadAttention(d, h, batch_first=True).eval()37with torch.no_grad():38 ref.in_proj_weight.copy_(torch.cat([m.W_q.weight, m.W_k.weight, m.W_v.weight]))39 ref.in_proj_bias.copy_(torch.cat([m.W_q.bias, m.W_k.bias, m.W_v.bias]))40 ref.out_proj.weight.copy_(m.W_o.weight)41 ref.out_proj.bias.copy_(m.W_o.bias)42y_ref, a_ref = ref(x, x, x, average_attn_weights=False)43print(torch.allclose(y, y_ref, atol=1e-5), torch.allclose(a, a_ref, atol=1e-5)) # True True4445# concat(heads) @ W_O == sum over heads of head_i @ W_O^(i)46with torch.no_grad():47 heads = a @ m.split(m.W_v(x)) # (B, h, N, dk)48 Wo = m.W_o.weight.view(d, h, d // h) # (out, head, dk): row block i of W_O^T49 y_sum = torch.einsum("bhnk,ohk->bno", heads, Wo) + m.W_o.bias50print(torch.allclose(y_sum, y, atol=1e-5)) # TrueUse when
- +Several attention patterns per query for the same parameters and FLOPs as one wide head
- +One batched matmul over the head axis; maps directly onto GPU kernels
- +Heads specialise (positional, syntactic, copying) and are individually interpretable
- + lets every head write anywhere in the residual stream
Avoid when
- − attention maps of per layer: memory grows with unless the kernel is fused
- −Small limits each head's score matrix to rank ; too many heads hurts (Table 3)
- −The KV cache stores key and value heads per layer, which dominates decoding memory
- −Many heads are redundant after training
Check yourself
A layer has d_model = 768. Going from h = 12 heads to h = 24 heads (dₖ = d_model/h) changes which of these?
Check yourself
Why is concat(head₁ … head_h)·W_O equal to Σᵢ headᵢ·W_O^(i)?
Remember
- ✓, , .
- ✓Same cost as one wide head: weights and MACs for and , for any ; base Transformer , .
- ✓Heads are a reshape of one big projection; concatenation is the inverse reshape.
- ✓: each head writes its own update to the residual stream.
- ✓Vaswani et al. Table 3: h = 1 gives 24.9 dev BLEU vs 25.8 for h = 8; h = 32 (dₖ = 16) drops to 25.4.