Definition
Multi-query attention (MQA) is a variant of Multi-head attention in which the heads keep separate query projections but share a single key projection and a single value projection: . Only one -dimensional key and one -dimensional value are stored per token and layer, instead of of each. It was proposed by Noam Shazeer in "Fast Transformer Decoding: One Write-Head is All You Need" (arXiv, 2019).
Training a Transformer is fast because all positions are processed in parallel and every weight that is loaded from memory is reused across the whole batch and sequence. Autoregressive generation is different. Each step produces one token per sequence, attends to all previous tokens, and must reload the cached keys and values of every past token in every layer. That cache is used for very little arithmetic, so on GPUs and TPUs the decoder is limited by memory bandwidth, not by FLOPs. Shazeer's observation was that the cache, not the query side, causes this. The queries are recomputed each step anyway, while the keys and values must be kept and reread. Sharing one key/value head across all query heads divides that traffic by .
The layer
- one query projection per head, as in multi-head attention
- the single key projection shared by all heads (no index )
- the single shared value projection
- unchanged output projection over the h concatenated heads
Each head still has its own attention pattern, because its queries differ. All heads score against the same keys and average the same values. The key and value projections shrink from to . In Shazeer's experiments the parameters saved were given back to the feed-forward layers, so the multi-head and multi-query models had the same size.
KV-cache arithmetic
- number of decoder layers (each caches its own K and V)
- number of key/value heads stored per layer
- tokens so far (context length) and batch size
- 2 for fp16/bf16
For a model shaped like Llama 2 7B (, , ) in fp16 this is bytes, i.e. 512 KiB per token with multi-head attention, and 16 KiB with one shared head. At batch 8 and 4096 tokens the multi-head cache is 16 GiB, more than the 12.6 GiB of weights. Every decoding step reads all of it.
- the term from rereading the cached keys and values; it does not shrink with batch size, because every sequence has its own cache
- the term from reading the weights, amortised over the batch
These are Shazeer's ratios for incremental decoding. A large batch amortises the weight reads, but not the cache reads. Each cached element is loaded once to serve one query head, two FLOPs per two bytes in fp16. With MQA the same element serves all heads, so the term is divided by .
Shazeer (2019) setup
- Task
- WMT 2014 English→German translation; also a decoder-only language model on the Billion-Word benchmark
- Baseline
- encoder–decoder, 6+6 layers, , , ,
- Multi-query
- same, with one K and one V head in every attention layer and to restore the parameter count
- Parameters
- adds FFN weights, exactly the removed from the 18 attention layers' K/V projections
- Reported
- slightly worse quality than the multi-head baseline, better than baselines that shrink h, dₖ or dᵥ to save the same memory, and much faster incremental decoding with nearly unchanged training speed
Worked example: one decoding step
| quantity | multi-head | multi-query |
|---|---|---|
| K/V heads per layer | 32 | 1 |
| cache per token, all layers | 524,288 B (512 KiB) | 16,384 B (16 KiB) |
| cache for 8 × 4096 tokens | 16.0 GiB | 512 MiB |
| bytes read per step | 28.6 GiB | 13.1 GiB |
| lower bound on step time | 15.3 ms | 7.0 ms |
| memory/arithmetic ratio n/d + 1/b vs 1/d + n/(dh) + 1/b | 1.125 | 0.156 |
Where the numbers come from
- 1Per token: bytes for MHA and for MQA, a factor .
- 2For the batch: bytes GiB, versus MiB.
- 3A step reads the weights once ( bytes GiB) plus the whole cache: GiB versus GiB. At 2.0 TB/s that is at least 15.3 ms versus 7.0 ms. The FLOPs, about plus attention, take 0.4 ms at 312 TFLOP/s, so the step is memory-bound in both cases.
- 4Shazeer's ratio with , , : for MHA, and for MQA. After the change, the weight term dominates, so larger batches now help.
1import torch, torch.nn as nn23class MultiQueryAttention(nn.Module):4 def __init__(self, d_model, n_heads):5 super().__init__()6 self.h, self.dk = n_heads, d_model // n_heads7 self.W_q = nn.Linear(d_model, d_model, bias=False) # h query heads8 self.W_kv = nn.Linear(d_model, 2 * self.dk, bias=False) # ONE key head and ONE value head9 self.W_o = nn.Linear(d_model, d_model, bias=False)1011 def forward(self, x, cache=None):12 # x (B, T, d): T new tokens; cache = (K, V), each (B, n_past, dk)13 B, T, _ = x.shape14 q = self.W_q(x).view(B, T, self.h, self.dk).transpose(1, 2) # (B, h, T, dk)15 k, v = self.W_kv(x).split(self.dk, dim=-1) # (B, T, dk) each16 if cache is not None:17 k, v = torch.cat([cache[0], k], 1), torch.cat([cache[1], v], 1)18 n = k.size(1)19 s = q @ k.unsqueeze(1).transpose(-2, -1) / self.dk ** 0.5 # (B, h, T, n): K broadcast over heads20 causal = torch.ones(T, n, dtype=torch.bool, device=x.device).tril(n - T)21 a = s.masked_fill(~causal, float("-inf")).softmax(-1)22 o = (a @ v.unsqueeze(1)).transpose(1, 2).reshape(B, T, -1) # (B, T, d)23 return self.W_o(o), (k, v)2425torch.manual_seed(0)26m = MultiQueryAttention(512, 8).eval()27x = torch.randn(2, 16, 512)28with torch.no_grad():29 full, _ = m(x) # all 16 tokens at once (training-style)30 y, cache = m(x[:, :12]) # prefill 12 tokens31 outs = [y]32 for t in range(12, 16): # then decode one token at a time33 y, cache = m(x[:, t:t + 1], cache)34 outs.append(y)35print(torch.allclose(torch.cat(outs, 1), full, atol=1e-5)) # True36print(cache[0].shape) # torch.Size([2, 16, 64]); multi-head would cache (2, 8, 16, 64)37print(sum(p.numel() for p in m.parameters())) # 589824 = 2*512*512 + 512*128, vs 1048576 for MHAUse when
- +KV cache and per-step cache traffic divided by (32× in a 32-head model)
- +Much larger batches or contexts fit in the same memory
- +Same attention FLOPs and nearly the same parameters; a drop-in change to the projections
- +Decoding becomes bound by weight reads, which batching amortises
Avoid when
- −Some quality loss, and reported training instability
- −No speed-up for training or prompt prefill (compute-bound)
- −With tensor parallelism over heads, the single K/V head must be replicated on every device
- −GQA with a few K/V heads recovers most of the quality for a small part of the memory
Check yourself
A 40-layer model with 40 heads of 128 serves batch 4 at 8192 tokens in bf16. How large is the KV cache with MHA and with MQA?
Check yourself
Why does a larger batch not fix the memory-bandwidth problem of multi-head incremental decoding?
Remember
- ✓MQA: query heads, one shared key head and one value head: (Shazeer 2019).
- ✓KV cache bytes; MQA sets , e.g. 512 KiB → 16 KiB per token for 32 layers × 32 heads × 128 in fp16.
- ✓Memory/arithmetic in incremental decoding: for MHA vs for MQA.
- ✓Helps memory-bound decoding only; training and prefill FLOPs are unchanged.
- ✓Small quality loss and possible instability; GQA is the usual compromise. Used in PaLM and Gemma 2B.