# Multi-query attention (MQA)

> Multi-query attention keeps h query heads but gives them one shared key head and one shared value head. The KV cache and the memory read per decoding step shrink by a factor of h, which makes incremental decoding much faster at a small cost in quality.

Source: https://www.mlfoundry.in/learn/multi-query-attention-mqa  
Section: Deep Learning › Attention and Transformer Architectures  
Level: advanced · 12 min read · updated 2026-10-12

**Multi-query attention (MQA):** is a variant of **Multi-head attention** in which the $h$ heads keep separate query projections $W_i^Q$ but **share a single key projection and a single value projection**: $\mathrm{head}_i=\mathrm{softmax}\big(QW_i^Q(KW^K)^\top/\sqrt{d_k}\big)\,VW^V$. Only one $d_k$-dimensional key and one $d_v$-dimensional value are stored per token and layer, instead of $h$ of each. It was proposed by Noam Shazeer in "Fast Transformer Decoding: One Write-Head is All You Need" (arXiv, 2019).

Training a Transformer is fast because all positions are processed in parallel and every weight that is loaded from memory is reused across the whole batch and sequence. Autoregressive generation is different. Each step produces one token per sequence, attends to all previous tokens, and must reload the cached keys and values of every past token in every layer. That cache is used for very little arithmetic, so on GPUs and TPUs the decoder is limited by memory bandwidth, not by FLOPs. Shazeer's observation was that the cache, not the query side, causes this. The queries are recomputed each step anyway, while the keys and values must be kept and reread. Sharing one key/value head across all query heads divides that traffic by $h$.

## The layer

$$
\mathrm{head}_i=\mathrm{softmax}\!\Big(\frac{(QW_i^Q)\,(KW^K)^\top}{\sqrt{d_k}}\Big)\,VW^V,\qquad \mathrm{MQA}(Q,K,V)=\big[\mathrm{head}_1\cdots\mathrm{head}_h\big]\,W^O
$$

- $W_i^Q\in\mathbb R^{d_{\text{model}}\times d_k}$: one query projection per head, as in multi-head attention
- $W^K\in\mathbb R^{d_{\text{model}}\times d_k}$: the **single** key projection shared by all heads (no index $i$)
- $W^V\in\mathbb R^{d_{\text{model}}\times d_v}$: the single shared value projection
- $W^O\in\mathbb R^{hd_v\times d_{\text{model}}}$: unchanged output projection over the h concatenated heads

Each head still has its own attention pattern, because its queries differ. All heads score against the same keys and average the same values. The key and value projections shrink from $d_{\text{model}}\times hd_k$ to $d_{\text{model}}\times d_k$. In Shazeer's experiments the parameters saved were given back to the feed-forward layers, so the multi-head and multi-query models had the same size.

## KV-cache arithmetic

$$
\text{KV cache}=\underbrace{2}_{K,\,V}\cdot L\cdot h_{kv}\cdot d_k\cdot n\cdot b\cdot\text{bytes},\qquad h_{kv}=\begin{cases}h & \text{multi-head}\\ 1 & \text{multi-query}\end{cases}
$$

- $L$: number of decoder layers (each caches its own K and V)
- $h_{kv}$: number of key/value heads stored per layer
- $n,\ b$: tokens so far (context length) and batch size
- $\text{bytes}$: 2 for fp16/bf16

For a model shaped like Llama 2 7B ($L=32$, $h=32$, $d_k=128$) in fp16 this is $2\cdot32\cdot32\cdot128\cdot2=524{,}288$ bytes, i.e. 512 KiB per token with multi-head attention, and 16 KiB with one shared head. At batch 8 and 4096 tokens the multi-head cache is 16 GiB, more than the 12.6 GiB of weights. Every decoding step reads all of it.

$$
\frac{\text{memory access}}{\text{arithmetic}}=\Theta\!\Big(\frac{n}{d}+\frac1b\Big)\ \ \text{(multi-head)},\qquad \Theta\!\Big(\frac1d+\frac{n}{dh}+\frac1b\Big)\ \ \text{(multi-query)}
$$

- $d$: $d_{\text{model}}$
- $n/d$: the term from rereading the cached keys and values; it does not shrink with batch size, because every sequence has its own cache
- $1/b$: the term from reading the weights, amortised over the batch

These are Shazeer's ratios for incremental decoding. A large batch amortises the weight reads, but not the cache reads. Each cached element is loaded once to serve one query head, two FLOPs per two bytes in fp16. With MQA the same element serves all $h$ heads, so the $n/d$ term is divided by $h$.

**Shazeer (2019) setup**

- **Task:** WMT 2014 English→German translation; also a decoder-only language model on the Billion-Word benchmark
- **Baseline:** encoder–decoder, 6+6 layers, $d_{\text{model}}=1024$, $h=8$, $d_k=d_v=128$, $d_{ff}=4096$
- **Multi-query:** same, with one K and one V head in every attention layer and $d_{ff}=5440$ to restore the parameter count
- **Parameters:** $5440$ adds $12\cdot2\cdot1024\cdot1344=33{,}030{,}144$ FFN weights, exactly the $33{,}030{,}144$ removed from the 18 attention layers' K/V projections
- **Reported:** slightly worse quality than the multi-head baseline, better than baselines that shrink h, dₖ or dᵥ to save the same memory, and much faster incremental decoding with nearly unchanged training speed

> Figure: A 32-layer decoder with 32 query heads of 128, the shape of Llama 2 7B, in fp16, on an assumed GPU with 80 GiB, 2.0 TB/s and 312 TFLOP/s. The parameter count is kept equal for both variants. At batch 8 and 4096 tokens, multi-head attention caches 16.0 GiB of keys and values (512 KiB per token), and multi-query only 512 MiB (16 KiB per token). A decoding step must read 28.6 GiB vs 13.1 GiB, so it takes at least 15.3 ms vs 7.0 ms (522 vs 1,141 tokens/s). The compute-bound time would be only 0.4 ms for both, so both are memory-bound. At 4096 tokens at most 33 sequences fit in memory with MHA and 1,079 with MQA. At batch 128 the MHA cache alone is 256 GiB and does not fit, while MQA needs 20.6 GiB and reaches about 11,600 tokens/s. Lengthening the context hurts MHA the same way: one sequence of 32,768 tokens costs it 16 GiB.

## Worked example: one decoding step

Llama-2-7B-shaped decoder, fp16, batch 8, context 4096, 6.74B parameters (12.6 GiB) for both (computed in Node; times assume 2.0 TB/s)

| quantity | multi-head | multi-query |
| --- | --- | --- |
| K/V heads per layer | 32 | 1 |
| cache per token, all layers | 524,288 B (512 KiB) | 16,384 B (16 KiB) |
| cache for 8 × 4096 tokens | 16.0 GiB | 512 MiB |
| bytes read per step | 28.6 GiB | 13.1 GiB |
| lower bound on step time | 15.3 ms | 7.0 ms |
| memory/arithmetic ratio n/d + 1/b vs 1/d + n/(dh) + 1/b | 1.125 | 0.156 |

**Where the numbers come from**

1. Per token: $2\cdot L\cdot h_{kv}\cdot d_k\cdot2$ bytes $=2\cdot32\cdot32\cdot128\cdot2=524{,}288$ for MHA and $2\cdot32\cdot1\cdot128\cdot2=16{,}384$ for MQA, a factor $h=32$.
2. For the batch: $524{,}288\cdot8\cdot4096=17.2\cdot10^9$ bytes $=16$ GiB, versus $512$ MiB.
3. A step reads the weights once ($6.74\cdot10^9\cdot2$ bytes $=12.6$ GiB) plus the whole cache: $28.6$ GiB versus $13.1$ GiB. At 2.0 TB/s that is at least 15.3 ms versus 7.0 ms. The FLOPs, about $2\cdot6.74\cdot10^9\cdot8\approx1.1\cdot10^{11}$ plus attention, take 0.4 ms at 312 TFLOP/s, so the step is memory-bound in both cases.
4. Shazeer's ratio with $n=d=4096$, $b=8$, $h=32$: $1+0.125=1.125$ for MHA, and $0.0002+0.031+0.125=0.156$ for MQA. After the change, the $1/b$ weight term dominates, so larger batches now help.

```python
import torch, torch.nn as nn

class MultiQueryAttention(nn.Module):
    def __init__(self, d_model, n_heads):
        super().__init__()
        self.h, self.dk = n_heads, d_model // n_heads
        self.W_q = nn.Linear(d_model, d_model, bias=False)       # h query heads
        self.W_kv = nn.Linear(d_model, 2 * self.dk, bias=False)  # ONE key head and ONE value head
        self.W_o = nn.Linear(d_model, d_model, bias=False)

    def forward(self, x, cache=None):
        # x (B, T, d): T new tokens; cache = (K, V), each (B, n_past, dk)
        B, T, _ = x.shape
        q = self.W_q(x).view(B, T, self.h, self.dk).transpose(1, 2)   # (B, h, T, dk)
        k, v = self.W_kv(x).split(self.dk, dim=-1)                     # (B, T, dk) each
        if cache is not None:
            k, v = torch.cat([cache[0], k], 1), torch.cat([cache[1], v], 1)
        n = k.size(1)
        s = q @ k.unsqueeze(1).transpose(-2, -1) / self.dk ** 0.5      # (B, h, T, n): K broadcast over heads
        causal = torch.ones(T, n, dtype=torch.bool, device=x.device).tril(n - T)
        a = s.masked_fill(~causal, float("-inf")).softmax(-1)
        o = (a @ v.unsqueeze(1)).transpose(1, 2).reshape(B, T, -1)    # (B, T, d)
        return self.W_o(o), (k, v)

torch.manual_seed(0)
m = MultiQueryAttention(512, 8).eval()
x = torch.randn(2, 16, 512)
with torch.no_grad():
    full, _ = m(x)                         # all 16 tokens at once (training-style)
    y, cache = m(x[:, :12])                # prefill 12 tokens
    outs = [y]
    for t in range(12, 16):                # then decode one token at a time
        y, cache = m(x[:, t:t + 1], cache)
        outs.append(y)
print(torch.allclose(torch.cat(outs, 1), full, atol=1e-5))   # True
print(cache[0].shape)        # torch.Size([2, 16, 64]); multi-head would cache (2, 8, 16, 64)
print(sum(p.numel() for p in m.parameters()))   # 589824 = 2*512*512 + 512*128, vs 1048576 for MHA
```

Multi-query attention with a KV cache; incremental decoding reproduces the full causal forward pass

> **MQA does not make training or prefill faster:** The FLOPs of attention are unchanged: $h$ query heads still score against $n$ keys. During training and during the prefill of a prompt the layer is compute-bound, and MQA saves only the small K/V projections. The gains appear in incremental decoding, where the step is memory-bound and the cache is large, i.e. at long context or large batch. Benchmark MQA on generation throughput, not on training step time.

> **Quality and stability cost:** One key/value head for all queries is a real restriction: every head must find what it needs in the same $d_k$-dimensional keys and values. Shazeer reported a small quality drop. Ainslie et al. (2023) also observed training instability with MQA in their T5 experiments. That motivated the middle ground, **Grouped-query attention (GQA)**, which keeps a few K/V heads instead of one.

> **Why the cache, not the queries:** The query of the new token is computed once and used once, so it costs no extra memory traffic. Keys and values are written once but read at every later step: the token at position 1 is reread $n-1$ times. Shazeer's title, "one write-head", refers to this asymmetry. Many read (query) heads can share a single write (key/value) head.

> **Where MQA was used:** PaLM (Chowdhery et al. 2022) used multi-query attention to make sampling cheaper. So did Gemma 2B (2024), while Gemma 7B kept multi-head attention. Most recent open models use GQA instead. Converting a trained multi-head model into MQA is possible with the mean-pooling and uptraining recipe described in **Grouped-query attention (GQA)**.

**Strengths**

- KV cache and per-step cache traffic divided by $h$ (32× in a 32-head model)
- Much larger batches or contexts fit in the same memory
- Same attention FLOPs and nearly the same parameters; a drop-in change to the projections
- Decoding becomes bound by weight reads, which batching amortises

**Limitations**

- Some quality loss, and reported training instability
- No speed-up for training or prompt prefill (compute-bound)
- With tensor parallelism over heads, the single K/V head must be replicated on every device
- GQA with a few K/V heads recovers most of the quality for a small part of the memory

**Question:** A 40-layer model with 40 heads of 128 serves batch 4 at 8192 tokens in bf16. How large is the KV cache with MHA and with MQA?

- A. 25 GiB and 640 MiB
- B. 12.5 GiB and 320 MiB
- C. 25 GiB and 25 GiB
- D. 800 KiB and 20 KiB

**Answer:** A. 25 GiB and 640 MiB — Per token, MHA stores $2\cdot40\cdot40\cdot128\cdot2=819{,}200$ bytes (800 KiB) over all layers. Times $4\cdot8192=32{,}768$ tokens that is $26{,}843{,}545{,}600$ bytes, exactly 25 GiB. MQA keeps one K/V head instead of 40, so it needs $25/40$ GiB $=640$ MiB. The last option gives the per-token figures, not the cache.

**Question:** Why does a larger batch not fix the memory-bandwidth problem of multi-head incremental decoding?

- A. Because the weights must be reloaded for each sequence
- B. Because each sequence has its own KV cache, so cache traffic grows with b as fast as the arithmetic does
- C. Because softmax cannot be batched
- D. Because larger batches need more FLOPs per token

**Answer:** B. Because each sequence has its own KV cache, so cache traffic grows with b as fast as the arithmetic does — Weight reads are shared by the batch (the $1/b$ term), but every sequence has its own cache, which must be read for that sequence alone. The cache term $n/d$ in Shazeer's ratio does not depend on $b$. MQA shrinks it to $n/(dh)$.

**Key points**

- MQA: $h$ query heads, **one** shared key head and one value head: $\mathrm{head}_i=\mathrm{softmax}(QW_i^Q(KW^K)^\top/\sqrt{d_k})VW^V$ (Shazeer 2019).
- KV cache $=2\cdot L\cdot h_{kv}\cdot d_k\cdot n\cdot b\cdot$bytes; MQA sets $h_{kv}=1$, e.g. 512 KiB → 16 KiB per token for 32 layers × 32 heads × 128 in fp16.
- Memory/arithmetic in incremental decoding: $\Theta(n/d+1/b)$ for MHA vs $\Theta(1/d+n/(dh)+1/b)$ for MQA.
- Helps memory-bound decoding only; training and prefill FLOPs are unchanged.
- Small quality loss and possible instability; GQA is the usual compromise. Used in PaLM and Gemma 2B.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
