Definition
Grouped-query attention (GQA) is an attention layer with query heads and key/value heads, , dividing . Query head uses key/value head , so each K/V head serves a group of query heads. is Multi-head attention, is Multi-query attention (MQA). It was introduced by Ainslie, Lee-Thorp, de Jong, Zemlyanskiy, Lebrón and Sanghai in "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints" (EMNLP 2023).
Multi-query attention divides the KV cache by and speeds up decoding, but it costs quality and was reported to make training less stable. Ainslie et al. made two points. First, the choice is not binary: with K/V heads the cache is of multi-head attention, and a handful of groups recovers almost all of the quality. Second, there is no need to pretrain such a model from scratch. A trained multi-head checkpoint can be converted by mean-pooling the key and value heads within each group, then uptrained with the original pretraining objective for a small fraction () of the original steps. GQA spread quickly. Llama 2 used it in its 34B and 70B models (the 7B and 13B models kept multi-head attention), and Llama 3 (8B and 70B) and Mistral 7B use it.
The layer
- query projections, one per head,
- key and value projections, , each
- group of query head ; consecutive query heads share a K/V head
- group size: query heads per K/V head (8 in Llama 2 70B, 4 in Llama 3 8B and Mistral 7B)
In code the keys and values are computed and cached with heads, shape . Just before the score computation each one is repeated times along the head axis (repeat_interleave, or Llama's repeat_kv) to line up with the query heads. Fused kernels skip the copy and index the shared head directly. The attention FLOPs are those of multi-head attention. Only the K/V projections ( weights instead of ) and the cache shrink.
Memory: the cache scales with g
- decoder layers
- key/value heads per layer (the number of groups)
- 2 in fp16/bf16
| model | layers | query heads h | KV heads g | group size | KV per token | if MHA |
|---|---|---|---|---|---|---|
| Llama 2 7B | 32 | 32 | 32 (MHA) | 1 | 512 KiB | 512 KiB |
| Llama 2 70B | 80 | 64 | 8 | 8 | 320 KiB | 2.5 MiB |
| Llama 3 8B | 32 | 32 | 8 | 4 | 128 KiB | 512 KiB |
| Llama 3 70B | 80 | 64 | 8 | 8 | 320 KiB | 2.5 MiB |
| Mistral 7B | 32 | 32 | 8 | 4 | 128 KiB | 512 KiB |
Eight K/V heads is the common choice. For a 64-head model it divides the cache by 8. With 8-way tensor parallelism it also gives each device exactly one K/V head, so nothing has to be replicated, which a single MQA head would require. Mistral 7B additionally uses sliding-window attention with a window of 4096 and a rolling buffer, which caps its cache at 4096 positions (see Sparse attention).
Uptraining a multi-head checkpoint
- key projection of head in the trained multi-head model
- the mean over the heads of group
After the conversion the model is trained further on its original pretraining data and objective for a fraction of the original number of steps, here . In the paper's ablation, mean-pooling worked better than keeping the first head of each group, which in turn beat random initialisation of the new K/V heads. Mean-pooling keeps the most information from the pretrained heads. GQA was also less dependent on the uptraining than MQA: converted GQA models were already reasonable before it, while converted MQA models needed it.
Ainslie et al. (2023) setup
- Models
- T5.1.1 Large and XXL encoder–decoders, uptrained from public multi-head checkpoints
- Where
- MQA/GQA applied to decoder self-attention and cross-attention; encoder self-attention left multi-head (it is not cached step by step)
- Uptraining
- of the original pretraining steps, same recipe
- Main setting
- GQA-8 (8 key/value heads) for XXL
- Tasks
- summarisation (CNN/Daily Mail, arXiv, PubMed, MediaSum, Multi-News), WMT translation, TriviaQA
| model | inference time (s) | average score |
|---|---|---|
| MHA-Large | 0.37 | 46.0 |
| MHA-XXL | 1.51 | 47.2 |
| MQA-XXL (uptrained) | 0.24 | 46.6 |
| GQA-8-XXL (uptrained) | 0.28 | 47.1 |
GQA-8-XXL is nearly as good as multi-head XXL (47.1 vs 47.2) at close to MQA speed (0.28 s vs 0.24 s), and both uptrained XXL models are faster and better than multi-head Large. In their sweep over the number of groups, decoding time grew only slowly from 1 to 8 groups and faster beyond that.
Worked example: pooling four heads into two groups
Four heads with over , so each key projection is a 2-vector: , , , . Two tokens and have keys , and every head uses the query . The weight on is .
| head | MHA (own w) | GQA-2, mean-pool | GQA-2, first head | MQA, mean of all 4 |
|---|---|---|---|---|
| 1 | 0.7685 | 0.6900 | 0.7685 | 0.3208 |
| 2 | 0.5987 | 0.6900 | 0.7685 | 0.3208 |
| 3 | 0.0998 | 0.0911 | 0.0998 | 0.3208 |
| 4 | 0.0832 | 0.0911 | 0.0998 | 0.3208 |
Conversion and cache size
- 1Groups and : mean-pooled keys and ; one shared key for all four heads: .
- 2Mean-pooling misses by ; keeping the first head misses by . The average error is the same here (0.0466), but the worst head is off by 0.091 instead of 0.170. MQA flattens heads 1 and 3, which wanted opposite things, to a common 0.32. A single K/V head cannot serve both.
- 3Llama 2 70B per token: bytes (320 KiB). With it would be (2.5 MiB), with only 40 KiB.
- 4At 4096 tokens: 1.25 GiB per sequence, against 10 GiB for MHA. A batch of 16 such sequences takes 20 GiB instead of 160 GiB.
1import torch, torch.nn as nn23class GroupedQueryAttention(nn.Module):4 def __init__(self, d_model, n_heads, n_kv_heads):5 super().__init__()6 assert n_heads % n_kv_heads == 07 self.h, self.g, self.dk = n_heads, n_kv_heads, d_model // n_heads8 self.W_q = nn.Linear(d_model, n_heads * self.dk, bias=False)9 self.W_k = nn.Linear(d_model, n_kv_heads * self.dk, bias=False)10 self.W_v = nn.Linear(d_model, n_kv_heads * self.dk, bias=False)11 self.W_o = nn.Linear(n_heads * self.dk, d_model, bias=False)1213 def forward(self, x):14 B, N, _ = x.shape15 q = self.W_q(x).view(B, N, self.h, self.dk).transpose(1, 2) # (B, h, N, dk)16 k = self.W_k(x).view(B, N, self.g, self.dk).transpose(1, 2) # (B, g, N, dk) <- this is what gets cached17 v = self.W_v(x).view(B, N, self.g, self.dk).transpose(1, 2)18 rep = self.h // self.g # group size h/g19 k, v = k.repeat_interleave(rep, dim=1), v.repeat_interleave(rep, dim=1) # (B, h, N, dk); head i -> group i // rep20 s = q @ k.transpose(-2, -1) / self.dk ** 0.521 s = s.masked_fill(torch.ones(N, N, dtype=torch.bool, device=x.device).triu(1), float("-inf")) # causal22 o = (s.softmax(-1) @ v).transpose(1, 2).reshape(B, N, -1)23 return self.W_o(o)2425@torch.no_grad()26def convert_from_mha(mha, n_kv_heads):27 """Ainslie et al. (2023): mean-pool the K and V heads of each group; copy W_q and W_o."""28 h, dk, d = mha.h, mha.dk, mha.W_q.in_features29 gqa = GroupedQueryAttention(d, h, n_kv_heads)30 gqa.W_q.weight.copy_(mha.W_q.weight)31 gqa.W_o.weight.copy_(mha.W_o.weight)32 for name in ("W_k", "W_v"):33 W = getattr(mha, name).weight.view(n_kv_heads, h // n_kv_heads, dk, d) # (group, member, dk, d)34 getattr(gqa, name).weight.copy_(W.mean(1).reshape(n_kv_heads * dk, d))35 return gqa3637torch.manual_seed(0)38d, h = 512, 839mha = GroupedQueryAttention(d, h, n_kv_heads=h) # g = h: ordinary multi-head attention40x = torch.randn(2, 64, d)41with torch.no_grad():42 y = mha(x)43 for g in (8, 4, 2, 1): # g = 1 is multi-query attention44 gqa = convert_from_mha(mha, g)45 err = ((gqa(x) - y).norm() / y.norm()).item()46 kv_params = gqa.W_k.weight.numel() + gqa.W_v.weight.numel()47 print(g, kv_params, round(err, 3))48# 8 524288 0.0 (identical to the MHA layer)49# 4 262144 0.7450# 2 131072 0.89451# 1 65536 0.953 (random weights share nothing across heads; uptraining is what closes the gap)Use when
- +KV cache and decode bandwidth divided by the group size
- +Quality close to MHA (47.1 vs 47.2 for GQA-8-XXL in the paper), speed close to MQA
- +Can be obtained from an existing MHA checkpoint with about 5% extra pretraining compute
- + matching the tensor-parallel degree avoids replicating K/V heads
Avoid when
- −One more hyperparameter ; the cache is still that of MQA
- −No saving in training or prefill FLOPs
- −Converted models need uptraining; the conversion alone loses quality
- −Group conventions must match exactly between weights, conversion and kernels
Check yourself
A model has 48 layers, 48 query heads of 128 and uses GQA with g = 8, in bf16. What is its KV cache per token, and how does it compare with MHA?
Check yourself
When converting an MHA checkpoint to GQA, which initialisation of the shared K/V heads did Ainslie et al. find best?
Remember
- ✓GQA: query heads, K/V heads, query head uses K/V head ; is MHA, is MQA (Ainslie et al., EMNLP 2023).
- ✓KV cache per token bytes, i.e. of MHA; Llama 2 70B (, , ): 320 KiB vs 2.5 MiB in fp16.
- ✓Conversion: mean-pool each group's and , copy and , uptrain for of pretraining steps.
- ✓T5-XXL: GQA-8 reaches 47.1 average vs 47.2 for MHA, at 0.28 s vs 1.51 s per sample (MQA 0.24 s, 46.6).
- ✓Llama 2 70B, Llama 3 8B and 70B, and Mistral 7B all use 8 K/V heads; Llama 2 7B and 13B are plain MHA.