Grouped-query attention (GQA)

Grouped-query attention splits the h query heads into g groups, each sharing one key and one value head: g = h is multi-head and g = 1 is multi-query attention. The KV cache shrinks by h/g, and a multi-head checkpoint can be converted by mean-pooling its K/V heads and briefly uptraining.

13 min read932 wordsadvanced0 viewsupdated 2026-10-12

Definition

Grouped-query attention (GQA) is an attention layer with hh query heads and gg key/value heads, 1≤g≤h1\le g\le h, gg dividing hh. Query head ii uses key/value head ⌊i/(h/g)⌋\lfloor i/(h/g)\rfloor, so each K/V head serves a group of h/gh/g query heads. g=hg=h is Multi-head attention, g=1g=1 is Multi-query attention (MQA). It was introduced by Ainslie, Lee-Thorp, de Jong, Zemlyanskiy, Lebrón and Sanghai in "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints" (EMNLP 2023).

Multi-query attention divides the KV cache by hh and speeds up decoding, but it costs quality and was reported to make training less stable. Ainslie et al. made two points. First, the choice is not binary: with gg K/V heads the cache is g/hg/h of multi-head attention, and a handful of groups recovers almost all of the quality. Second, there is no need to pretrain such a model from scratch. A trained multi-head checkpoint can be converted by mean-pooling the key and value heads within each group, then uptrained with the original pretraining objective for a small fraction (α=5%\alpha=5\%) of the original steps. GQA spread quickly. Llama 2 used it in its 34B and 70B models (the 7B and 13B models kept multi-head attention), and Llama 3 (8B and 70B) and Mistral 7B use it.

The layer

headi=softmax ⁣((QWiQ) (KWγ(i)K)⊤dk) VWγ(i)V,γ(i)=⌊ih/g⌋,GQA=[head0⋯headh−1] WO\mathrm{head}_i=\mathrm{softmax}\!\Big(\frac{(QW_i^Q)\,(KW_{\gamma(i)}^K)^\top}{\sqrt{d_k}}\Big)\,VW_{\gamma(i)}^V,\qquad \gamma(i)=\Big\lfloor\frac{i}{h/g}\Big\rfloor,\qquad \mathrm{GQA}=\big[\mathrm{head}_0\cdots\mathrm{head}_{h-1}\big]\,W^O
WiQW_i^Q
hh query projections, one per head, i=0,…,h−1i=0,\dots,h-1
WjK, WjVW_j^K,\,W_j^V
gg key and value projections, j=0,…,g−1j=0,\dots,g-1, each dmodel×dkd_{\text{model}}\times d_k
γ(i)\gamma(i)
group of query head ii; consecutive query heads share a K/V head
h/gh/g
group size: query heads per K/V head (8 in Llama 2 70B, 4 in Llama 3 8B and Mistral 7B)

In code the keys and values are computed and cached with gg heads, shape (B,g,n,dk)(B,g,n,d_k). Just before the score computation each one is repeated h/gh/g times along the head axis (repeat_interleave, or Llama's repeat_kv) to line up with the hh query heads. Fused kernels skip the copy and index the shared head directly. The attention FLOPs are those of multi-head attention. Only the K/V projections (2 dmodel g dk2\,d_{\text{model}}\,g\,d_k weights instead of 2dmodel22d_{\text{model}}^2) and the cache shrink.

Memory: the cache scales with g

KV bytes per token=2⋅L⋅g⋅dk⋅bytes,GQAMHA=gh,GQAMQA=g\text{KV bytes per token}=2\cdot L\cdot g\cdot d_k\cdot\text{bytes},\qquad \frac{\text{GQA}}{\text{MHA}}=\frac{g}{h},\qquad \frac{\text{GQA}}{\text{MQA}}=g
LL
decoder layers
gg
key/value heads per layer (the number of groups)
bytes\text{bytes}
2 in fp16/bf16
modellayersquery heads hKV heads ggroup sizeKV per tokenif MHA
Llama 2 7B323232 (MHA)1512 KiB512 KiB
Llama 2 70B806488320 KiB2.5 MiB
Llama 3 8B323284128 KiB512 KiB
Llama 3 70B806488320 KiB2.5 MiB
Mistral 7B323284128 KiB512 KiB
Published configurations (d_k = 128 in all of them) and their fp16 KV cache per token, computed from the formula above

Eight K/V heads is the common choice. For a 64-head model it divides the cache by 8. With 8-way tensor parallelism it also gives each device exactly one K/V head, so nothing has to be replicated, which a single MQA head would require. Mistral 7B additionally uses sliding-window attention with a window of 4096 and a rolling buffer, which caps its cache at 4096 positions (see Sparse attention).

Uptraining a multi-head checkpoint

WjK←gh∑i: γ(i)=jWiK,MHA,WjV←gh∑i: γ(i)=jWiV,MHA,WiQ, WO copied unchangedW_j^K\leftarrow\frac{g}{h}\sum_{i:\,\gamma(i)=j}W_i^{K,\text{MHA}},\qquad W_j^V\leftarrow\frac{g}{h}\sum_{i:\,\gamma(i)=j}W_i^{V,\text{MHA}},\qquad W_i^Q,\ W^O\ \text{copied unchanged}
WiK,MHAW_i^{K,\text{MHA}}
key projection of head ii in the trained multi-head model
g/h∑g/h\sum
the mean over the h/gh/g heads of group jj

After the conversion the model is trained further on its original pretraining data and objective for a fraction α\alpha of the original number of steps, here α=0.05\alpha=0.05. In the paper's ablation, mean-pooling worked better than keeping the first head of each group, which in turn beat random initialisation of the new K/V heads. Mean-pooling keeps the most information from the pretrained heads. GQA was also less dependent on the uptraining than MQA: converted GQA models were already reasonable before it, while converted MQA models needed it.

Ainslie et al. (2023) setup

Models
T5.1.1 Large and XXL encoder–decoders, uptrained from public multi-head checkpoints
Where
MQA/GQA applied to decoder self-attention and cross-attention; encoder self-attention left multi-head (it is not cached step by step)
Uptraining
α=5%\alpha=5\% of the original pretraining steps, same recipe
Main setting
GQA-8 (8 key/value heads) for XXL
Tasks
summarisation (CNN/Daily Mail, arXiv, PubMed, MediaSum, Multi-News), WMT translation, TriviaQA
modelinference time (s)average score
MHA-Large0.3746.0
MHA-XXL1.5147.2
MQA-XXL (uptrained)0.2446.6
GQA-8-XXL (uptrained)0.2847.1
Inference time per sample and average score over the tasks (Ainslie et al. 2023, Table 1)

GQA-8-XXL is nearly as good as multi-head XXL (47.1 vs 47.2) at close to MQA speed (0.28 s vs 0.24 s), and both uptrained XXL models are faster and better than multi-head Large. In their sweep over the number of groups, decoding time grew only slowly from 1 to 8 groups and faster beyond that.

toy MHA checkpoint, h = 8 → GQA-2: 4 query heads per K/V headq1q2q3q4q5q6q7q8kv1kv2q2: original MHAafter mean-poolrel. output error before uptraining0.51g = 1 MQAg = 2g = 4g = 8 MHA■ mean-pool ■ first head ■ randomoutput error 0.592 · attention TV 0.299Llama 2 70B: 80 layers, 64 query heads, 8 KV heads (group size 8), dₕ 128, fp16MHA (64 KV)10.0 GiB (2560 KiB/token)as built (8 KV)1.25 GiB (320 KiB/token)MQA (1 KV)160 MiB (40 KiB/token)GQA-8: 8× smaller than MHA, 8× larger than MQAone sequence of 4,096 tokens · bytes/token = 2 · layers · KV heads · 128 · 2
groups ginit
model
fig.Top: a toy multi-head layer with 8 heads (d_model 32, dₖ 4, 10 tokens) whose K/V projections share a common part, standing in for a trained checkpoint. It is converted to g K/V heads and compared with the original before any uptraining. Mean-pooling gives a relative output error of 0.686 for MQA (g = 1), 0.592 for GQA-2, 0.475 for GQA-4 and 0 for g = 8. Taking the first head of each group gives 0.882, 0.745 and 0.548, and random K/V heads give 1.2 to 1.5 at every g. More groups and mean-pooling give the better starting point for uptraining, as in the paper's ablation. Click a query head to compare its attention map before and after conversion. Bottom: the fp16 KV cache of real configurations. At 4096 tokens Llama 2 70B needs 1.25 GiB per sequence with its 8 K/V heads, against 10 GiB for multi-head and 160 MiB for one shared head. At 131,072 tokens the three become 40 GiB, 320 GiB and 5 GiB. Llama 3 8B and Mistral 7B need 128 KiB per token.

Worked example: pooling four heads into two groups

Four heads with dk=1d_k=1 over dmodel=2d_{\text{model}}=2, so each key projection is a 2-vector: w1=(0.8,0.2)w_1=(0.8,0.2), w2=(0.6,0.4)w_2=(0.6,0.4), w3=(−0.2,0.9)w_3=(-0.2,0.9), w4=(0.1,1.3)w_4=(0.1,1.3). Two tokens xa=(1,0)x_a=(1,0) and xb=(0,1)x_b=(0,1) have keys k=x⋅wk=x\cdot w, and every head uses the query q=2q=2. The weight on xax_a is σ(2(ka−kb))\sigma\big(2(k_a-k_b)\big).

headMHA (own w)GQA-2, mean-poolGQA-2, first headMQA, mean of all 4
10.76850.69000.76850.3208
20.59870.69000.76850.3208
30.09980.09110.09980.3208
40.08320.09110.09980.3208
Attention weight on xₐ for each query head (checked numerically in Node)

Conversion and cache size

  1. 1Groups {1,2}\{1,2\} and {3,4}\{3,4\}: mean-pooled keys (0.7,0.3)(0.7,0.3) and (−0.05,1.1)(-0.05,1.1); one shared key for all four heads: (0.325,0.7)(0.325,0.7).
  2. 2Mean-pooling misses by 0.0785, 0.0913, 0.0087, 0.00790.0785,\,0.0913,\,0.0087,\,0.0079; keeping the first head misses by 0, 0.1698, 0, 0.01660,\,0.1698,\,0,\,0.0166. The average error is the same here (0.0466), but the worst head is off by 0.091 instead of 0.170. MQA flattens heads 1 and 3, which wanted opposite things, to a common 0.32. A single K/V head cannot serve both.
  3. 3Llama 2 70B per token: 2⋅80⋅8⋅128⋅2=327,6802\cdot80\cdot8\cdot128\cdot2=327{,}680 bytes (320 KiB). With g=h=64g=h=64 it would be 2,621,4402{,}621{,}440 (2.5 MiB), with g=1g=1 only 40 KiB.
  4. 4At 4096 tokens: 1.25 GiB per sequence, against 10 GiB for MHA. A batch of 16 such sequences takes 20 GiB instead of 160 GiB.
python
1import torch, torch.nn as nn
2
3class GroupedQueryAttention(nn.Module):
4 def __init__(self, d_model, n_heads, n_kv_heads):
5 super().__init__()
6 assert n_heads % n_kv_heads == 0
7 self.h, self.g, self.dk = n_heads, n_kv_heads, d_model // n_heads
8 self.W_q = nn.Linear(d_model, n_heads * self.dk, bias=False)
9 self.W_k = nn.Linear(d_model, n_kv_heads * self.dk, bias=False)
10 self.W_v = nn.Linear(d_model, n_kv_heads * self.dk, bias=False)
11 self.W_o = nn.Linear(n_heads * self.dk, d_model, bias=False)
12
13 def forward(self, x):
14 B, N, _ = x.shape
15 q = self.W_q(x).view(B, N, self.h, self.dk).transpose(1, 2) # (B, h, N, dk)
16 k = self.W_k(x).view(B, N, self.g, self.dk).transpose(1, 2) # (B, g, N, dk) <- this is what gets cached
17 v = self.W_v(x).view(B, N, self.g, self.dk).transpose(1, 2)
18 rep = self.h // self.g # group size h/g
19 k, v = k.repeat_interleave(rep, dim=1), v.repeat_interleave(rep, dim=1) # (B, h, N, dk); head i -> group i // rep
20 s = q @ k.transpose(-2, -1) / self.dk ** 0.5
21 s = s.masked_fill(torch.ones(N, N, dtype=torch.bool, device=x.device).triu(1), float("-inf")) # causal
22 o = (s.softmax(-1) @ v).transpose(1, 2).reshape(B, N, -1)
23 return self.W_o(o)
24
25@torch.no_grad()
26def convert_from_mha(mha, n_kv_heads):
27 """Ainslie et al. (2023): mean-pool the K and V heads of each group; copy W_q and W_o."""
28 h, dk, d = mha.h, mha.dk, mha.W_q.in_features
29 gqa = GroupedQueryAttention(d, h, n_kv_heads)
30 gqa.W_q.weight.copy_(mha.W_q.weight)
31 gqa.W_o.weight.copy_(mha.W_o.weight)
32 for name in ("W_k", "W_v"):
33 W = getattr(mha, name).weight.view(n_kv_heads, h // n_kv_heads, dk, d) # (group, member, dk, d)
34 getattr(gqa, name).weight.copy_(W.mean(1).reshape(n_kv_heads * dk, d))
35 return gqa
36
37torch.manual_seed(0)
38d, h = 512, 8
39mha = GroupedQueryAttention(d, h, n_kv_heads=h) # g = h: ordinary multi-head attention
40x = torch.randn(2, 64, d)
41with torch.no_grad():
42 y = mha(x)
43 for g in (8, 4, 2, 1): # g = 1 is multi-query attention
44 gqa = convert_from_mha(mha, g)
45 err = ((gqa(x) - y).norm() / y.norm()).item()
46 kv_params = gqa.W_k.weight.numel() + gqa.W_v.weight.numel()
47 print(g, kv_params, round(err, 3))
48# 8 524288 0.0 (identical to the MHA layer)
49# 4 262144 0.74
50# 2 131072 0.894
51# 1 65536 0.953 (random weights share nothing across heads; uptraining is what closes the gap)
GQA layer (g K/V heads repeated to h query heads) and the mean-pooling conversion from a multi-head checkpoint

Use when

  • +KV cache and decode bandwidth divided by the group size h/gh/g
  • +Quality close to MHA (47.1 vs 47.2 for GQA-8-XXL in the paper), speed close to MQA
  • +Can be obtained from an existing MHA checkpoint with about 5% extra pretraining compute
  • +gg matching the tensor-parallel degree avoids replicating K/V heads

Avoid when

  • −One more hyperparameter gg; the cache is still g×g\times that of MQA
  • −No saving in training or prefill FLOPs
  • −Converted models need uptraining; the conversion alone loses quality
  • −Group conventions must match exactly between weights, conversion and kernels

Check yourself

A model has 48 layers, 48 query heads of 128 and uses GQA with g = 8, in bf16. What is its KV cache per token, and how does it compare with MHA?

Check yourself

When converting an MHA checkpoint to GQA, which initialisation of the shared K/V heads did Ainslie et al. find best?

Remember

  • ✓GQA: hh query heads, gg K/V heads, query head ii uses K/V head ⌊i/(h/g)⌋\lfloor i/(h/g)\rfloor; g=hg=h is MHA, g=1g=1 is MQA (Ainslie et al., EMNLP 2023).
  • ✓KV cache per token =2⋅L⋅g⋅dk⋅=2\cdot L\cdot g\cdot d_k\cdotbytes, i.e. g/hg/h of MHA; Llama 2 70B (L=80L=80, h=64h=64, g=8g=8): 320 KiB vs 2.5 MiB in fp16.
  • ✓Conversion: mean-pool each group's WKW^K and WVW^V, copy WQW^Q and WOW^O, uptrain for α=5%\alpha=5\% of pretraining steps.
  • ✓T5-XXL: GQA-8 reaches 47.1 average vs 47.2 for MHA, at 0.28 s vs 1.51 s per sample (MQA 0.24 s, 46.6).
  • ✓Llama 2 70B, Llama 3 8B and 70B, and Mistral 7B all use 8 K/V heads; Llama 2 7B and 13B are plain MHA.
← back to Deep Learning roadmap