# Grouped-query attention (GQA)

> Grouped-query attention splits the h query heads into g groups, each sharing one key and one value head: g = h is multi-head and g = 1 is multi-query attention. The KV cache shrinks by h/g, and a multi-head checkpoint can be converted by mean-pooling its K/V heads and briefly uptraining.

Source: https://www.mlfoundry.in/learn/grouped-query-attention-gqa  
Section: Deep Learning › Attention and Transformer Architectures  
Level: advanced · 13 min read · updated 2026-10-12

**Grouped-query attention (GQA):** is an attention layer with $h$ query heads and $g$ key/value heads, $1\le g\le h$, $g$ dividing $h$. Query head $i$ uses key/value head $\lfloor i/(h/g)\rfloor$, so each K/V head serves a **group** of $h/g$ query heads. $g=h$ is **Multi-head attention**, $g=1$ is **Multi-query attention (MQA)**. It was introduced by Ainslie, Lee-Thorp, de Jong, Zemlyanskiy, Lebrón and Sanghai in "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints" (EMNLP 2023).

Multi-query attention divides the KV cache by $h$ and speeds up decoding, but it costs quality and was reported to make training less stable. Ainslie et al. made two points. First, the choice is not binary: with $g$ K/V heads the cache is $g/h$ of multi-head attention, and a handful of groups recovers almost all of the quality. Second, there is no need to pretrain such a model from scratch. A trained multi-head checkpoint can be **converted** by mean-pooling the key and value heads within each group, then **uptrained** with the original pretraining objective for a small fraction ($\alpha=5\%$) of the original steps. GQA spread quickly. Llama 2 used it in its 34B and 70B models (the 7B and 13B models kept multi-head attention), and Llama 3 (8B and 70B) and Mistral 7B use it.

## The layer

$$
\mathrm{head}_i=\mathrm{softmax}\!\Big(\frac{(QW_i^Q)\,(KW_{\gamma(i)}^K)^\top}{\sqrt{d_k}}\Big)\,VW_{\gamma(i)}^V,\qquad \gamma(i)=\Big\lfloor\frac{i}{h/g}\Big\rfloor,\qquad \mathrm{GQA}=\big[\mathrm{head}_0\cdots\mathrm{head}_{h-1}\big]\,W^O
$$

- $W_i^Q$: $h$ query projections, one per head, $i=0,\dots,h-1$
- $W_j^K,\,W_j^V$: $g$ key and value projections, $j=0,\dots,g-1$, each $d_{\text{model}}\times d_k$
- $\gamma(i)$: group of query head $i$; consecutive query heads share a K/V head
- $h/g$: group size: query heads per K/V head (8 in Llama 2 70B, 4 in Llama 3 8B and Mistral 7B)

In code the keys and values are computed and cached with $g$ heads, shape $(B,g,n,d_k)$. Just before the score computation each one is repeated $h/g$ times along the head axis (`repeat_interleave`, or Llama's `repeat_kv`) to line up with the $h$ query heads. Fused kernels skip the copy and index the shared head directly. The attention FLOPs are those of multi-head attention. Only the K/V projections ($2\,d_{\text{model}}\,g\,d_k$ weights instead of $2d_{\text{model}}^2$) and the cache shrink.

## Memory: the cache scales with g

$$
\text{KV bytes per token}=2\cdot L\cdot g\cdot d_k\cdot\text{bytes},\qquad \frac{\text{GQA}}{\text{MHA}}=\frac{g}{h},\qquad \frac{\text{GQA}}{\text{MQA}}=g
$$

- $L$: decoder layers
- $g$: key/value heads per layer (the number of groups)
- $\text{bytes}$: 2 in fp16/bf16

Published configurations (d_k = 128 in all of them) and their fp16 KV cache per token, computed from the formula above

| model | layers | query heads h | KV heads g | group size | KV per token | if MHA |
| --- | --- | --- | --- | --- | --- | --- |
| Llama 2 7B | 32 | 32 | 32 (MHA) | 1 | 512 KiB | 512 KiB |
| Llama 2 70B | 80 | 64 | 8 | 8 | 320 KiB | 2.5 MiB |
| Llama 3 8B | 32 | 32 | 8 | 4 | 128 KiB | 512 KiB |
| Llama 3 70B | 80 | 64 | 8 | 8 | 320 KiB | 2.5 MiB |
| Mistral 7B | 32 | 32 | 8 | 4 | 128 KiB | 512 KiB |

Eight K/V heads is the common choice. For a 64-head model it divides the cache by 8. With 8-way tensor parallelism it also gives each device exactly one K/V head, so nothing has to be replicated, which a single MQA head would require. Mistral 7B additionally uses sliding-window attention with a window of 4096 and a rolling buffer, which caps its cache at 4096 positions (see **Sparse attention**).

## Uptraining a multi-head checkpoint

$$
W_j^K\leftarrow\frac{g}{h}\sum_{i:\,\gamma(i)=j}W_i^{K,\text{MHA}},\qquad W_j^V\leftarrow\frac{g}{h}\sum_{i:\,\gamma(i)=j}W_i^{V,\text{MHA}},\qquad W_i^Q,\ W^O\ \text{copied unchanged}
$$

- $W_i^{K,\text{MHA}}$: key projection of head $i$ in the trained multi-head model
- $g/h\sum$: the mean over the $h/g$ heads of group $j$

After the conversion the model is trained further on its original pretraining data and objective for a fraction $\alpha$ of the original number of steps, here $\alpha=0.05$. In the paper's ablation, mean-pooling worked better than keeping the first head of each group, which in turn beat random initialisation of the new K/V heads. Mean-pooling keeps the most information from the pretrained heads. GQA was also less dependent on the uptraining than MQA: converted GQA models were already reasonable before it, while converted MQA models needed it.

**Ainslie et al. (2023) setup**

- **Models:** T5.1.1 Large and XXL encoder–decoders, uptrained from public multi-head checkpoints
- **Where:** MQA/GQA applied to decoder self-attention and cross-attention; encoder self-attention left multi-head (it is not cached step by step)
- **Uptraining:** $\alpha=5\%$ of the original pretraining steps, same recipe
- **Main setting:** GQA-8 (8 key/value heads) for XXL
- **Tasks:** summarisation (CNN/Daily Mail, arXiv, PubMed, MediaSum, Multi-News), WMT translation, TriviaQA

Inference time per sample and average score over the tasks (Ainslie et al. 2023, Table 1)

| model | inference time (s) | average score |
| --- | --- | --- |
| MHA-Large | 0.37 | 46.0 |
| MHA-XXL | 1.51 | 47.2 |
| MQA-XXL (uptrained) | 0.24 | 46.6 |
| GQA-8-XXL (uptrained) | 0.28 | 47.1 |

GQA-8-XXL is nearly as good as multi-head XXL (47.1 vs 47.2) at close to MQA speed (0.28 s vs 0.24 s), and both uptrained XXL models are faster and better than multi-head Large. In their sweep over the number of groups, decoding time grew only slowly from 1 to 8 groups and faster beyond that.

> Figure: Top: a toy multi-head layer with 8 heads (d_model 32, dₖ 4, 10 tokens) whose K/V projections share a common part, standing in for a trained checkpoint. It is converted to g K/V heads and compared with the original before any uptraining. Mean-pooling gives a relative output error of 0.686 for MQA (g = 1), 0.592 for GQA-2, 0.475 for GQA-4 and 0 for g = 8. Taking the first head of each group gives 0.882, 0.745 and 0.548, and random K/V heads give 1.2 to 1.5 at every g. More groups and mean-pooling give the better starting point for uptraining, as in the paper's ablation. Click a query head to compare its attention map before and after conversion. Bottom: the fp16 KV cache of real configurations. At 4096 tokens Llama 2 70B needs 1.25 GiB per sequence with its 8 K/V heads, against 10 GiB for multi-head and 160 MiB for one shared head. At 131,072 tokens the three become 40 GiB, 320 GiB and 5 GiB. Llama 3 8B and Mistral 7B need 128 KiB per token.

## Worked example: pooling four heads into two groups

Four heads with $d_k=1$ over $d_{\text{model}}=2$, so each key projection is a 2-vector: $w_1=(0.8,0.2)$, $w_2=(0.6,0.4)$, $w_3=(-0.2,0.9)$, $w_4=(0.1,1.3)$. Two tokens $x_a=(1,0)$ and $x_b=(0,1)$ have keys $k=x\cdot w$, and every head uses the query $q=2$. The weight on $x_a$ is $\sigma\big(2(k_a-k_b)\big)$.

Attention weight on xₐ for each query head (checked numerically in Node)

| head | MHA (own w) | GQA-2, mean-pool | GQA-2, first head | MQA, mean of all 4 |
| --- | --- | --- | --- | --- |
| 1 | 0.7685 | 0.6900 | 0.7685 | 0.3208 |
| 2 | 0.5987 | 0.6900 | 0.7685 | 0.3208 |
| 3 | 0.0998 | 0.0911 | 0.0998 | 0.3208 |
| 4 | 0.0832 | 0.0911 | 0.0998 | 0.3208 |

**Conversion and cache size**

1. Groups $\{1,2\}$ and $\{3,4\}$: mean-pooled keys $(0.7,0.3)$ and $(-0.05,1.1)$; one shared key for all four heads: $(0.325,0.7)$.
2. Mean-pooling misses by $0.0785,\,0.0913,\,0.0087,\,0.0079$; keeping the first head misses by $0,\,0.1698,\,0,\,0.0166$. The average error is the same here (0.0466), but the worst head is off by 0.091 instead of 0.170. MQA flattens heads 1 and 3, which wanted opposite things, to a common 0.32. A single K/V head cannot serve both.
3. Llama 2 70B per token: $2\cdot80\cdot8\cdot128\cdot2=327{,}680$ bytes (320 KiB). With $g=h=64$ it would be $2{,}621{,}440$ (2.5 MiB), with $g=1$ only 40 KiB.
4. At 4096 tokens: 1.25 GiB per sequence, against 10 GiB for MHA. A batch of 16 such sequences takes 20 GiB instead of 160 GiB.

```python
import torch, torch.nn as nn

class GroupedQueryAttention(nn.Module):
    def __init__(self, d_model, n_heads, n_kv_heads):
        super().__init__()
        assert n_heads % n_kv_heads == 0
        self.h, self.g, self.dk = n_heads, n_kv_heads, d_model // n_heads
        self.W_q = nn.Linear(d_model, n_heads * self.dk, bias=False)
        self.W_k = nn.Linear(d_model, n_kv_heads * self.dk, bias=False)
        self.W_v = nn.Linear(d_model, n_kv_heads * self.dk, bias=False)
        self.W_o = nn.Linear(n_heads * self.dk, d_model, bias=False)

    def forward(self, x):
        B, N, _ = x.shape
        q = self.W_q(x).view(B, N, self.h, self.dk).transpose(1, 2)   # (B, h, N, dk)
        k = self.W_k(x).view(B, N, self.g, self.dk).transpose(1, 2)   # (B, g, N, dk)  <- this is what gets cached
        v = self.W_v(x).view(B, N, self.g, self.dk).transpose(1, 2)
        rep = self.h // self.g                                         # group size h/g
        k, v = k.repeat_interleave(rep, dim=1), v.repeat_interleave(rep, dim=1)   # (B, h, N, dk); head i -> group i // rep
        s = q @ k.transpose(-2, -1) / self.dk ** 0.5
        s = s.masked_fill(torch.ones(N, N, dtype=torch.bool, device=x.device).triu(1), float("-inf"))  # causal
        o = (s.softmax(-1) @ v).transpose(1, 2).reshape(B, N, -1)
        return self.W_o(o)

@torch.no_grad()
def convert_from_mha(mha, n_kv_heads):
    """Ainslie et al. (2023): mean-pool the K and V heads of each group; copy W_q and W_o."""
    h, dk, d = mha.h, mha.dk, mha.W_q.in_features
    gqa = GroupedQueryAttention(d, h, n_kv_heads)
    gqa.W_q.weight.copy_(mha.W_q.weight)
    gqa.W_o.weight.copy_(mha.W_o.weight)
    for name in ("W_k", "W_v"):
        W = getattr(mha, name).weight.view(n_kv_heads, h // n_kv_heads, dk, d)   # (group, member, dk, d)
        getattr(gqa, name).weight.copy_(W.mean(1).reshape(n_kv_heads * dk, d))
    return gqa

torch.manual_seed(0)
d, h = 512, 8
mha = GroupedQueryAttention(d, h, n_kv_heads=h)     # g = h: ordinary multi-head attention
x = torch.randn(2, 64, d)
with torch.no_grad():
    y = mha(x)
    for g in (8, 4, 2, 1):                          # g = 1 is multi-query attention
        gqa = convert_from_mha(mha, g)
        err = ((gqa(x) - y).norm() / y.norm()).item()
        kv_params = gqa.W_k.weight.numel() + gqa.W_v.weight.numel()
        print(g, kv_params, round(err, 3))
# 8 524288 0.0     (identical to the MHA layer)
# 4 262144 0.74
# 2 131072 0.894
# 1 65536 0.953    (random weights share nothing across heads; uptraining is what closes the gap)
```

GQA layer (g K/V heads repeated to h query heads) and the mean-pooling conversion from a multi-head checkpoint

> **Group order must match between conversion and repetition:** `repeat_interleave` maps query head $i$ to K/V head $i//(h/g)$, i.e. **consecutive** heads share. `k.repeat(1, h // g, 1, 1)` maps head $i$ to $i\bmod g$ instead. Either convention works for a model trained from scratch, but the mean-pooling must group the same heads that the forward pass groups. Loading Llama weights with the wrong one gives a model that runs and produces garbage.

> **Why GQA scales better than MQA:** A larger model has more heads, so a single K/V head is shared by more queries and the relative cut in memory bandwidth from MQA gets more aggressive. Keeping $g$ fixed (often 8) as the model grows keeps the cache a constant small fraction of the model, $g/h$, while each K/V head is still shared by only a few queries. With $g$ equal to the tensor-parallel degree, each device holds its own K/V head and nothing is duplicated.

> **Library support:** Hugging Face configs store $g$ as `num_key_value_heads`. Recent PyTorch versions (2.5+) let `F.scaled_dot_product_attention` take keys and values with fewer heads through `enable_gqa=True`, and FlashAttention kernels accept them natively, so the repeat is never materialised (see **FlashAttention**).

**Strengths**

- KV cache and decode bandwidth divided by the group size $h/g$
- Quality close to MHA (47.1 vs 47.2 for GQA-8-XXL in the paper), speed close to MQA
- Can be obtained from an existing MHA checkpoint with about 5% extra pretraining compute
- $g$ matching the tensor-parallel degree avoids replicating K/V heads

**Limitations**

- One more hyperparameter $g$; the cache is still $g\times$ that of MQA
- No saving in training or prefill FLOPs
- Converted models need uptraining; the conversion alone loses quality
- Group conventions must match exactly between weights, conversion and kernels

**Question:** A model has 48 layers, 48 query heads of 128 and uses GQA with g = 8, in bf16. What is its KV cache per token, and how does it compare with MHA?

- A. 192 KiB, 6× smaller than MHA
- B. 1.125 MiB, the same as MHA
- C. 24 KiB, 48× smaller than MHA
- D. 192 KiB, 8× smaller than MHA

**Answer:** A. 192 KiB, 6× smaller than MHA — $2\cdot48\cdot8\cdot128\cdot2=196{,}608$ bytes $=192$ KiB. MHA would store 48 K/V heads: $2\cdot48\cdot48\cdot128\cdot2=1{,}179{,}648$ bytes $=1.125$ MiB. The ratio is $h/g=48/8=6$, not $g$.

**Question:** When converting an MHA checkpoint to GQA, which initialisation of the shared K/V heads did Ainslie et al. find best?

- A. Random initialisation
- B. The first head of each group
- C. The mean of the heads in each group
- D. Training the new K/V heads from scratch with the rest frozen

**Answer:** C. The mean of the heads in each group — Mean-pooling performed best, then selecting the first head, then random initialisation. Mean-pooling keeps information from every pretrained head of the group, and the short uptraining phase (5% of pretraining) recovers most of the remaining gap.

**Key points**

- GQA: $h$ query heads, $g$ K/V heads, query head $i$ uses K/V head $\lfloor i/(h/g)\rfloor$; $g=h$ is MHA, $g=1$ is MQA (Ainslie et al., EMNLP 2023).
- KV cache per token $=2\cdot L\cdot g\cdot d_k\cdot$bytes, i.e. $g/h$ of MHA; Llama 2 70B ($L=80$, $h=64$, $g=8$): 320 KiB vs 2.5 MiB in fp16.
- Conversion: mean-pool each group's $W^K$ and $W^V$, copy $W^Q$ and $W^O$, uptrain for $\alpha=5\%$ of pretraining steps.
- T5-XXL: GQA-8 reaches 47.1 average vs 47.2 for MHA, at 0.28 s vs 1.51 s per sample (MQA 0.24 s, 46.6).
- Llama 2 70B, Llama 3 8B and 70B, and Mistral 7B all use 8 K/V heads; Llama 2 7B and 13B are plain MHA.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
