Multi-query attention (MQA)

Multi-query attention keeps h query heads but gives them one shared key head and one shared value head. The KV cache and the memory read per decoding step shrink by a factor of h, which makes incremental decoding much faster at a small cost in quality.

12 min read784 wordsadvanced0 viewsupdated 2026-10-12

Definition

Multi-query attention (MQA) is a variant of Multi-head attention in which the hh heads keep separate query projections WiQW_i^Q but share a single key projection and a single value projection: headi=softmax(QWiQ(KWK)⊤/dk) VWV\mathrm{head}_i=\mathrm{softmax}\big(QW_i^Q(KW^K)^\top/\sqrt{d_k}\big)\,VW^V. Only one dkd_k-dimensional key and one dvd_v-dimensional value are stored per token and layer, instead of hh of each. It was proposed by Noam Shazeer in "Fast Transformer Decoding: One Write-Head is All You Need" (arXiv, 2019).

Training a Transformer is fast because all positions are processed in parallel and every weight that is loaded from memory is reused across the whole batch and sequence. Autoregressive generation is different. Each step produces one token per sequence, attends to all previous tokens, and must reload the cached keys and values of every past token in every layer. That cache is used for very little arithmetic, so on GPUs and TPUs the decoder is limited by memory bandwidth, not by FLOPs. Shazeer's observation was that the cache, not the query side, causes this. The queries are recomputed each step anyway, while the keys and values must be kept and reread. Sharing one key/value head across all query heads divides that traffic by hh.

The layer

headi=softmax ⁣((QWiQ) (KWK)⊤dk) VWV,MQA(Q,K,V)=[head1⋯headh] WO\mathrm{head}_i=\mathrm{softmax}\!\Big(\frac{(QW_i^Q)\,(KW^K)^\top}{\sqrt{d_k}}\Big)\,VW^V,\qquad \mathrm{MQA}(Q,K,V)=\big[\mathrm{head}_1\cdots\mathrm{head}_h\big]\,W^O
WiQ∈Rdmodel×dkW_i^Q\in\mathbb R^{d_{\text{model}}\times d_k}
one query projection per head, as in multi-head attention
WK∈Rdmodel×dkW^K\in\mathbb R^{d_{\text{model}}\times d_k}
the single key projection shared by all heads (no index ii)
WV∈Rdmodel×dvW^V\in\mathbb R^{d_{\text{model}}\times d_v}
the single shared value projection
WO∈Rhdv×dmodelW^O\in\mathbb R^{hd_v\times d_{\text{model}}}
unchanged output projection over the h concatenated heads

Each head still has its own attention pattern, because its queries differ. All heads score against the same keys and average the same values. The key and value projections shrink from dmodel×hdkd_{\text{model}}\times hd_k to dmodel×dkd_{\text{model}}\times d_k. In Shazeer's experiments the parameters saved were given back to the feed-forward layers, so the multi-head and multi-query models had the same size.

KV-cache arithmetic

KV cache=2⏟K, V⋅L⋅hkv⋅dk⋅n⋅b⋅bytes,hkv={hmulti-head1multi-query\text{KV cache}=\underbrace{2}_{K,\,V}\cdot L\cdot h_{kv}\cdot d_k\cdot n\cdot b\cdot\text{bytes},\qquad h_{kv}=\begin{cases}h & \text{multi-head}\\ 1 & \text{multi-query}\end{cases}
LL
number of decoder layers (each caches its own K and V)
hkvh_{kv}
number of key/value heads stored per layer
n, bn,\ b
tokens so far (context length) and batch size
bytes\text{bytes}
2 for fp16/bf16

For a model shaped like Llama 2 7B (L=32L=32, h=32h=32, dk=128d_k=128) in fp16 this is 2⋅32⋅32⋅128⋅2=524,2882\cdot32\cdot32\cdot128\cdot2=524{,}288 bytes, i.e. 512 KiB per token with multi-head attention, and 16 KiB with one shared head. At batch 8 and 4096 tokens the multi-head cache is 16 GiB, more than the 12.6 GiB of weights. Every decoding step reads all of it.

memory accessarithmetic=Θ ⁣(nd+1b)  (multi-head),Θ ⁣(1d+ndh+1b)  (multi-query)\frac{\text{memory access}}{\text{arithmetic}}=\Theta\!\Big(\frac{n}{d}+\frac1b\Big)\ \ \text{(multi-head)},\qquad \Theta\!\Big(\frac1d+\frac{n}{dh}+\frac1b\Big)\ \ \text{(multi-query)}
dd
dmodeld_{\text{model}}
n/dn/d
the term from rereading the cached keys and values; it does not shrink with batch size, because every sequence has its own cache
1/b1/b
the term from reading the weights, amortised over the batch

These are Shazeer's ratios for incremental decoding. A large batch amortises the weight reads, but not the cache reads. Each cached element is loaded once to serve one query head, two FLOPs per two bytes in fp16. With MQA the same element serves all hh heads, so the n/dn/d term is divided by hh.

Shazeer (2019) setup

Task
WMT 2014 English→German translation; also a decoder-only language model on the Billion-Word benchmark
Baseline
encoder–decoder, 6+6 layers, dmodel=1024d_{\text{model}}=1024, h=8h=8, dk=dv=128d_k=d_v=128, dff=4096d_{ff}=4096
Multi-query
same, with one K and one V head in every attention layer and dff=5440d_{ff}=5440 to restore the parameter count
Parameters
54405440 adds 12⋅2⋅1024⋅1344=33,030,14412\cdot2\cdot1024\cdot1344=33{,}030{,}144 FFN weights, exactly the 33,030,14433{,}030{,}144 removed from the 18 attention layers' K/V projections
Reported
slightly worse quality than the multi-head baseline, better than baselines that shrink h, dₖ or dᵥ to save the same memory, and much faster incremental decoding with nearly unchanged training speed
decode step · 32 layers · 32 Q heads × 128 · fp16 · b = 8 · n = 4,096QK,Vcache per layer: K, V each [b, n, 1, 128] · per token, all layers: 16 KiBmemory: ■ weights 12.6 GiB + ■ KV cache, device holds 80.0 GiBMHA28.6 GiBMQA13.1 GiBlargest batch that fits at n = 4,096: MHA 33 · MQA 1,079KV cache: MHA 16.0 GiB · MQA 512 MiB (÷32)step time ≥ bytes read / 2 TB/s: ■ weights + ■ KV cacheMHA15.3 ms · 522 tok/sMQA7.0 ms · 1,141 tok/sMHA: FLOPs/byte 4.1 · KV share of traffic 56% · compute-bound time 0.4 msMQA: FLOPs/byte 8.9 · KV share of traffic 4% · compute-bound time 0.4 msassumed: 6.74B params kept equal for both, 80.0 GiB, 2 TB/s, 312 TFLOP/s (A100-80GB class)
attention
fig.A 32-layer decoder with 32 query heads of 128, the shape of Llama 2 7B, in fp16, on an assumed GPU with 80 GiB, 2.0 TB/s and 312 TFLOP/s. The parameter count is kept equal for both variants. At batch 8 and 4096 tokens, multi-head attention caches 16.0 GiB of keys and values (512 KiB per token), and multi-query only 512 MiB (16 KiB per token). A decoding step must read 28.6 GiB vs 13.1 GiB, so it takes at least 15.3 ms vs 7.0 ms (522 vs 1,141 tokens/s). The compute-bound time would be only 0.4 ms for both, so both are memory-bound. At 4096 tokens at most 33 sequences fit in memory with MHA and 1,079 with MQA. At batch 128 the MHA cache alone is 256 GiB and does not fit, while MQA needs 20.6 GiB and reaches about 11,600 tokens/s. Lengthening the context hurts MHA the same way: one sequence of 32,768 tokens costs it 16 GiB.

Worked example: one decoding step

quantitymulti-headmulti-query
K/V heads per layer321
cache per token, all layers524,288 B (512 KiB)16,384 B (16 KiB)
cache for 8 × 4096 tokens16.0 GiB512 MiB
bytes read per step28.6 GiB13.1 GiB
lower bound on step time15.3 ms7.0 ms
memory/arithmetic ratio n/d + 1/b vs 1/d + n/(dh) + 1/b1.1250.156
Llama-2-7B-shaped decoder, fp16, batch 8, context 4096, 6.74B parameters (12.6 GiB) for both (computed in Node; times assume 2.0 TB/s)

Where the numbers come from

  1. 1Per token: 2⋅L⋅hkv⋅dk⋅22\cdot L\cdot h_{kv}\cdot d_k\cdot2 bytes =2⋅32⋅32⋅128⋅2=524,288=2\cdot32\cdot32\cdot128\cdot2=524{,}288 for MHA and 2⋅32⋅1⋅128⋅2=16,3842\cdot32\cdot1\cdot128\cdot2=16{,}384 for MQA, a factor h=32h=32.
  2. 2For the batch: 524,288⋅8⋅4096=17.2⋅109524{,}288\cdot8\cdot4096=17.2\cdot10^9 bytes =16=16 GiB, versus 512512 MiB.
  3. 3A step reads the weights once (6.74⋅109⋅26.74\cdot10^9\cdot2 bytes =12.6=12.6 GiB) plus the whole cache: 28.628.6 GiB versus 13.113.1 GiB. At 2.0 TB/s that is at least 15.3 ms versus 7.0 ms. The FLOPs, about 2⋅6.74⋅109⋅8≈1.1⋅10112\cdot6.74\cdot10^9\cdot8\approx1.1\cdot10^{11} plus attention, take 0.4 ms at 312 TFLOP/s, so the step is memory-bound in both cases.
  4. 4Shazeer's ratio with n=d=4096n=d=4096, b=8b=8, h=32h=32: 1+0.125=1.1251+0.125=1.125 for MHA, and 0.0002+0.031+0.125=0.1560.0002+0.031+0.125=0.156 for MQA. After the change, the 1/b1/b weight term dominates, so larger batches now help.
python
1import torch, torch.nn as nn
2
3class MultiQueryAttention(nn.Module):
4 def __init__(self, d_model, n_heads):
5 super().__init__()
6 self.h, self.dk = n_heads, d_model // n_heads
7 self.W_q = nn.Linear(d_model, d_model, bias=False) # h query heads
8 self.W_kv = nn.Linear(d_model, 2 * self.dk, bias=False) # ONE key head and ONE value head
9 self.W_o = nn.Linear(d_model, d_model, bias=False)
10
11 def forward(self, x, cache=None):
12 # x (B, T, d): T new tokens; cache = (K, V), each (B, n_past, dk)
13 B, T, _ = x.shape
14 q = self.W_q(x).view(B, T, self.h, self.dk).transpose(1, 2) # (B, h, T, dk)
15 k, v = self.W_kv(x).split(self.dk, dim=-1) # (B, T, dk) each
16 if cache is not None:
17 k, v = torch.cat([cache[0], k], 1), torch.cat([cache[1], v], 1)
18 n = k.size(1)
19 s = q @ k.unsqueeze(1).transpose(-2, -1) / self.dk ** 0.5 # (B, h, T, n): K broadcast over heads
20 causal = torch.ones(T, n, dtype=torch.bool, device=x.device).tril(n - T)
21 a = s.masked_fill(~causal, float("-inf")).softmax(-1)
22 o = (a @ v.unsqueeze(1)).transpose(1, 2).reshape(B, T, -1) # (B, T, d)
23 return self.W_o(o), (k, v)
24
25torch.manual_seed(0)
26m = MultiQueryAttention(512, 8).eval()
27x = torch.randn(2, 16, 512)
28with torch.no_grad():
29 full, _ = m(x) # all 16 tokens at once (training-style)
30 y, cache = m(x[:, :12]) # prefill 12 tokens
31 outs = [y]
32 for t in range(12, 16): # then decode one token at a time
33 y, cache = m(x[:, t:t + 1], cache)
34 outs.append(y)
35print(torch.allclose(torch.cat(outs, 1), full, atol=1e-5)) # True
36print(cache[0].shape) # torch.Size([2, 16, 64]); multi-head would cache (2, 8, 16, 64)
37print(sum(p.numel() for p in m.parameters())) # 589824 = 2*512*512 + 512*128, vs 1048576 for MHA
Multi-query attention with a KV cache; incremental decoding reproduces the full causal forward pass

Use when

  • +KV cache and per-step cache traffic divided by hh (32× in a 32-head model)
  • +Much larger batches or contexts fit in the same memory
  • +Same attention FLOPs and nearly the same parameters; a drop-in change to the projections
  • +Decoding becomes bound by weight reads, which batching amortises

Avoid when

  • −Some quality loss, and reported training instability
  • −No speed-up for training or prompt prefill (compute-bound)
  • −With tensor parallelism over heads, the single K/V head must be replicated on every device
  • −GQA with a few K/V heads recovers most of the quality for a small part of the memory

Check yourself

A 40-layer model with 40 heads of 128 serves batch 4 at 8192 tokens in bf16. How large is the KV cache with MHA and with MQA?

Check yourself

Why does a larger batch not fix the memory-bandwidth problem of multi-head incremental decoding?

Remember

  • ✓MQA: hh query heads, one shared key head and one value head: headi=softmax(QWiQ(KWK)⊤/dk)VWV\mathrm{head}_i=\mathrm{softmax}(QW_i^Q(KW^K)^\top/\sqrt{d_k})VW^V (Shazeer 2019).
  • ✓KV cache =2⋅L⋅hkv⋅dk⋅n⋅b⋅=2\cdot L\cdot h_{kv}\cdot d_k\cdot n\cdot b\cdotbytes; MQA sets hkv=1h_{kv}=1, e.g. 512 KiB → 16 KiB per token for 32 layers × 32 heads × 128 in fp16.
  • ✓Memory/arithmetic in incremental decoding: Θ(n/d+1/b)\Theta(n/d+1/b) for MHA vs Θ(1/d+n/(dh)+1/b)\Theta(1/d+n/(dh)+1/b) for MQA.
  • ✓Helps memory-bound decoding only; training and prefill FLOPs are unchanged.
  • ✓Small quality loss and possible instability; GQA is the usual compromise. Used in PaLM and Gemma 2B.
← back to Deep Learning roadmap