Definition
KV cache is the per-sequence store of the key and value vectors that every attention layer of a decoder-only language model has already computed. Under a causal mask the keys and values of past positions never change. Each generation step can therefore compute for the newest token only, append and to the cache, and attend over the cached rows. The technique is as old as incremental transformer decoding. Its systems consequences were analysed by Pope et al. ("Efficiently Scaling Transformer Inference", 2022), and its memory management was reworked by Kwon et al. ("Efficient Memory Management for Large Language Model Serving with PagedAttention", SOSP 2023), the paper behind vLLM.
Causal attention and Decoder-only transformer introduce the cache as a correctness fact: the mask makes token-by-token decoding reproduce the parallel training pass exactly. This article treats the cache as the central object of LLM serving. Its size sets how many requests fit on a GPU. Reading it sets the speed of every decode step. Most inference work since 2019 attacks it, through fewer key/value heads, paged allocation, sharing, quantization or eviction.
Without a cache, step re-runs the whole prefix: projections and query-key products per layer, so generating tokens costs score products. With the cache, step projects one token and scores one query against keys, in total. The trade is compute for memory, and at LLM scale the memory side is large.
Prefill and decode
Serving a request has two phases with very different hardware profiles. Prefill pushes the whole prompt through the model in one causal pass, exactly as in training, and writes rows of and into every layer. It is a matrix–matrix product of about FLOPs for parameters, so it is compute-bound and sets the time to first token. Decode then produces one token per sequence per step. Each step is a matrix–vector product of about FLOPs per sequence, but it has to stream every weight and the sequence's whole cache from HBM. It is memory-bandwidth-bound and sets the time per output token.
- projections of the newest token only, per head; with RoPE the key is cached already rotated
- read from the cache, never recomputed
- one query row against keys; no mask is needed because the cache holds only the past
- every one of the layers keeps its own cache
How big it is
- a key and a value
- layers
- key/value heads: for multi-head attention, 1 for Multi-query attention (MQA), for Grouped-query attention (GQA)
- head size, 128 in every Llama
- tokens held per sequence and sequences in the batch
- bytes per number: 2 for bf16, 1 for fp8 or int8, 0.5 for int4
The formula has no and no query heads, because queries are never cached. It is linear in context and in batch. For Llama 2 7B, bytes, 512 KiB per token. A 4096-token sequence then holds 2 GiB, and 30 such sequences (64.4 GB) fit in the 66.5 GB an 80 GB GPU has left after 13.5 GB of bf16 weights. Kwon et al. quote the same shape for OPT-13B: 800 KB per token and up to 1.6 GB per 2048-token request.
Why decode is bandwidth-bound
- arithmetic intensity, FLOPs per byte read from HBM
- each bf16 weight is read once per step and used once per sequence in the batch
- each sequence has its own cache, so batching does not amortise it: 1 FLOP/byte for MHA, 4 for Llama 3 8B's GQA
- a floor that holds whenever is far below the hardware ridge point
An A100 does 312 dense bf16 TFLOPS against about 2 TB/s of HBM bandwidth, a ridge point of about 156 FLOPs per byte. An H100 SXM's is about 295 (989 TFLOPS, 3.35 TB/s). Decode at batch 1 has an intensity near 1 and leaves almost all the arithmetic idle. Batching raises the weight intensity to but does nothing for attention, whose intensity stays at whatever the batch. As contexts grow, the cache becomes the larger share of the bytes read, and decode speed tracks cache size directly.
Making the cache smaller: fewer or compressed heads
The first lever is . Shazeer's multi-query attention ("Fast Transformer Decoding: One Write-Head is All You Need", 2019) shares one key and one value head among all query heads, a factor of . Ainslie et al.'s grouped-query attention (2023) shares each K/V head among a group of query heads. Llama 2 70B and all Llama 3 models use 8 KV heads, which cuts Llama 3 8B to 128 KiB per token, 4 times less than Llama 2 7B, at nearly the same size.
- multi-head latent attention (DeepSeek-V2, 2024): one shared latent per token and layer,
- per-head up-projections; at inference is absorbed into the query projection and into the output projection, so full keys and values are never materialised
- a separate 64-dim RoPE key shared by all heads, because a position-dependent rotation cannot be absorbed into
DeepSeek-V2 has 60 layers and 128 heads of size 128. It caches numbers per token and layer, , which the paper compares to GQA with 2.25 groups. Full multi-head attention of the same shape would cache per layer, so the latent is 98.2% smaller. Against its predecessor DeepSeek 67B, the paper reports a 93.3% smaller KV cache and 5.76 times the maximum generation throughput.
Per-token KV cache of real models (bf16)
- Llama 2 7B
- 32 layers, 32 KV heads, : 512 KiB/token, 2 GiB at its 4096-token context
- Llama 2 13B
- 40 layers, 40 KV heads: 800 KiB/token (the OPT-13B shape in the vLLM paper)
- Llama 3 8B
- 32 layers, 8 KV heads (GQA): 128 KiB/token, 1 GiB at 8192 tokens, 16 GiB at 128K
- Llama 3 70B
- 80 layers, 8 KV heads for 64 query heads: 320 KiB/token instead of 2.5 MiB with MHA
- Mistral 7B
- 32 layers, 8 KV heads, sliding window 4096 with a rolling buffer (slot ): 128 KiB/token, never more than 0.5 GiB per sequence; the paper cites an 8× saving at 32K tokens
- DeepSeek-V2
- 60 layers, MLA with plus a 64-dim RoPE key: 67.5 KiB/token against 3.75 MiB for MHA of the same shape
- Hardware
- A100 80GB: about 2.0 TB/s and 312 dense bf16 TFLOPS. H100 SXM: 3.35 TB/s and 989 TFLOPS
Allocating it: PagedAttention
Early serving systems stored each request's cache in one contiguous tensor sized for the maximum length, because the final length is unknown when the request arrives. Kwon et al. measured that in such systems only 20.4% to 38.2% of KV memory held actual token states. The rest was reserved slots that were never filled, internal fragmentation inside a reservation, or external fragmentation between reservations.
- block (page) size in tokens; vLLM's default is 16
- the sequence's block table: logical block → physical block in a shared pool
- only the last block of a sequence can be partly empty
PagedAttention borrows virtual memory from operating systems. The cache lives in a pool of fixed-size physical blocks. A sequence takes a new block only when its last one fills, and the attention kernel follows the block table, so the blocks need not be adjacent. Waste drops below 4% in the paper's measurements. Blocks are reference-counted, so parallel samples and beam-search branches share their common prefix and copy a block only when they write to it (copy-on-write). vLLM reported 2–4× the throughput of FasterTransformer and Orca at the same latency, mostly because more sequences fit in a batch.
Sharing it: prefix caching
Many requests start with the same tokens: a system prompt, few-shot examples, a shared document, or the earlier turns of a chat. Their keys and values are identical, because each position depends only on the tokens before it. Prefix caching keeps those blocks after a request finishes and maps new requests onto them. vLLM's automatic prefix caching hashes each full block together with all the tokens before it. SGLang's RadixAttention (Zheng et al., 2024) keeps cached prefixes in a radix tree with LRU eviction. A hit skips the prefill for the shared part, which cuts time to first token, and stores one copy instead of one per request. The match must be exact, token for token from position 0: one changed early token invalidates everything after it.
Shrinking it: quantization
The cache can be stored in fewer bits than the weights. FP8 caches with a per-tensor or per-head scale are standard in vLLM and TensorRT-LLM and halve both memory and bytes read. Lower precisions need care with outliers. KIVI (Liu et al., 2024) found that keys have a few channels with large magnitudes, while values do not. It quantizes keys per channel and values per token, keeps the most recent tokens in full precision, and reaches 2 bits with little loss. KVQuant (Hooper et al., 2024) also quantizes keys per channel, and does it before RoPE is applied. See Quantization for the general machinery.
Bounding it: eviction
A sliding window caps the cache at tokens, as in Mistral's rolling buffer, but a model trained with full attention fails when its first tokens are evicted. Xiao et al. ("Efficient Streaming Language Models with Attention Sinks", ICLR 2024) traced this to attention sinks. Softmax must put its probability mass somewhere, and trained models dump a large share of it on the first few tokens whatever they contain. StreamingLLM keeps the KV of 4 initial tokens plus a recent window, and assigns positions by slot in the cache rather than in the original text. With this, Llama 2, MPT, Falcon and Pythia ran stably over up to 4 million tokens, up to 22.2× faster than a sliding window that recomputes its keys. H2O (Zhang et al., NeurIPS 2023) keeps the recent tokens plus "heavy hitters", the tokens with the largest accumulated attention. A budget of about 20% of the cache preserved accuracy on many tasks. Both are lossy: an evicted token cannot be recalled, which hurts retrieval from long documents.
Worked example: serving Llama 3 8B on one 80 GB GPU
Llama 3 8B: , , , , parameters. Contexts of tokens, batch , bf16 everywhere, 2.0 TB/s and 312 TFLOPS.
| quantity | formula | value |
|---|---|---|
| KV per token | 2·32·8·128·2 bytes | 131,072 B = 128 KiB |
| KV per sequence | × 8192 | 1 GiB (1.07 GB) |
| KV for the batch | × 16 | 16 GiB (17.18 GB) |
| weights | 8.03e9 · 2 bytes | 16.06 GB |
| bytes read per decode step | weights + cache | 33.24 GB |
| decode step floor | 33.24 GB / 2.0 TB/s | 16.62 ms → 963 tokens/s |
| FLOPs per step | 2NB + 4·L·T·d·B | 2.57e11 + 6.87e10 = 3.26e11 |
| achieved compute | 3.26e11 / 16.62 ms | 19.6 TFLOPS, 6.3% of peak |
| same model with 32 KV heads | 4 × cache | 68.7 GB + 16.06 GB > 80 GB |
Reading the budget
- 1Prefill of a 4096-token prompt costs FLOPs, about 0.21 s at full A100 throughput. Its arithmetic intensity is about 3,964 FLOPs per byte, far above the ridge of 156, so it is compute-bound.
- 2A single decode step at and reads 16.06 GB of weights and 0.54 GB of cache, at least 8.30 ms. Nearly all of that time is spent waiting for memory.
- 3At , the floor is 16.62 ms per step, or 963 tokens/s, but the GPU computes at only 6.3% of its peak. The cache is already 52% of the bytes read, and the attention part of the work has intensity no matter how large the batch is.
- 4Free memory after the weights is 63.94 GB, room for 59 sequences of 8192 tokens in bf16. An fp8 cache doubles that to 119 and lowers the step floor to 12.32 ms.
- 5With multi-head attention (32 KV heads) the batch-16 cache would be 68.7 GB, and it would not fit beside the weights. GQA is what makes this configuration possible.
- 6A 2000-token system prompt shared by 100 concurrent requests costs 24.4 GiB if each request stores its own copy and 250 MiB with prefix caching.
1import numpy as np23class PagedKVCache:4 """One layer, n_kv heads. K/V live in a pool of fixed-size blocks; each sequence5 owns a block table (logical block i -> physical block id), as in vLLM."""6 def __init__(self, num_blocks, block_size, n_kv, d_head):7 self.bs = block_size8 self.k = np.zeros((num_blocks, block_size, n_kv, d_head), np.float32)9 self.v = np.zeros_like(self.k)10 self.free = list(range(num_blocks))[::-1]11 self.tables, self.lens = {}, {}1213 def append(self, seq, k, v): # k, v: (n_kv, d_head) for one new token14 n = self.lens.get(seq, 0)15 if n % self.bs == 0: # last block full (or none yet): grab a new one16 self.tables.setdefault(seq, []).append(self.free.pop())17 blk = self.tables[seq][n // self.bs]18 self.k[blk, n % self.bs], self.v[blk, n % self.bs] = k, v19 self.lens[seq] = n + 12021 def gather(self, seq): # (T, n_kv, d_head), in logical order22 n, t = self.lens[seq], self.tables[seq]23 return self.k[t].reshape(-1, *self.k.shape[2:])[:n], self.v[t].reshape(-1, *self.v.shape[2:])[:n]2425def decode_attention(q, K, V): # q (H, d), K/V (T, n_kv, d); GQA by repeat26 g = q.shape[0] // K.shape[1]27 K, V = np.repeat(K, g, axis=1), np.repeat(V, g, axis=1) # (T, H, d)28 s = np.einsum("hd,thd->ht", q, K) / np.sqrt(q.shape[1])29 p = np.exp(s - s.max(1, keepdims=True)); p /= p.sum(1, keepdims=True)30 return np.einsum("ht,thd->hd", p, V)3132rng = np.random.default_rng(0)33H, n_kv, d, bs = 8, 2, 64, 1634cache = PagedKVCache(num_blocks=64, block_size=bs, n_kv=n_kv, d_head=d)35ref = {"a": ([], []), "b": ([], [])}36for step in range(70): # two sequences interleaved, as a server batches them37 for seq, length in (("a", 70), ("b", 37)):38 if step < length:39 k, v = rng.standard_normal((2, n_kv, d)).astype(np.float32)40 cache.append(seq, k, v); ref[seq][0].append(k); ref[seq][1].append(v)4142q = rng.standard_normal((H, d)).astype(np.float32)43for seq in ("a", "b"):44 o_paged = decode_attention(q, *cache.gather(seq))45 o_ref = decode_attention(q, np.stack(ref[seq][0]), np.stack(ref[seq][1]))46 print(seq, cache.lens[seq], cache.tables[seq], np.abs(o_paged - o_ref).max())47# a 70 [0, 2, 4, 6, 7] 0.0 <- blocks of the two sequences interleave in the pool48# b 37 [1, 3, 5] 0.04950used = sum(cache.lens.values()); alloc = sum(len(t) for t in cache.tables.values()) * bs51print(f"tokens {used}, slots allocated {alloc}, utilisation {used / alloc:.1%}")52# tokens 107, slots allocated 128, utilisation 83.6%5354def kv_bytes(L, n_kv, d_head, T, B, bytes_per=2):55 return 2 * L * n_kv * d_head * T * B * bytes_per56print(kv_bytes(32, 32, 128, 4096, 1) / 2**30, kv_bytes(80, 8, 128, 8192, 1) / 2**30) # Llama 2 7B, Llama 3 70B57# 2.0 2.5Use when
- +Exact: under a causal mask, cached decoding reproduces the full recomputation bit for bit, up to rounding
- +Turns generation from Θ(n³) into Θ(n²) score products and one token's projections per step
- +Paged allocation wastes under one block per sequence and enables prefix sharing and copy-on-write
- +Composable savings: GQA/MQA/MLA, fp8 or 2-bit storage, sliding windows and eviction multiply
Avoid when
- −Memory grows linearly in context × batch and can exceed the weights (16 GiB vs 13.5 GB for Llama 2 7B at 8 × 4096)
- −Every decode step re-reads the whole cache, so long-context decode is bandwidth-bound
- −Eviction (StreamingLLM, H2O) is lossy; evicted facts cannot be retrieved later
- −Prefix reuse requires an exact token match from the start, and RoPE keys are tied to absolute positions
Where it's used
LLM serving engines
vLLM, SGLang and TensorRT-LLM are built around paged KV blocks, prefix reuse and continuous batching
Chat and agents
multi-turn conversations and tool loops reuse the cached prefix of earlier turns instead of re-running prefill
Long-context inference
fp8 or lower-bit caches and GQA/MLA decide whether a 128K-token request fits at all
Speculative decoding
the target model verifies drafted tokens in one pass over its cache and rolls the cache back on rejection (see Speculative decoding)
Check yourself
Llama 3 70B (80 layers, 64 query heads, 8 KV heads, d_head = 128) serves batch 4 at 32,768 tokens with a bf16 cache. How large is the cache?
Check yourself
A model trained with full attention is run with a pure sliding-window cache that evicts the first tokens, and perplexity explodes once they are gone. What does StreamingLLM keep to fix this?
Remember
- ✓KV cache bytes: 512 KiB per token for Llama 2 7B, 128 KiB for Llama 3 8B (GQA), 67.5 KiB for DeepSeek-V2 (MLA).
- ✓Prefill is one compute-bound causal pass that writes T rows per layer; decode is one token per step, bandwidth-bound, re-reading all weights plus the whole cache.
- ✓Decode intensity is FLOPs/byte for weights but only for attention, so long-context throughput tracks cache size, not batch size.
- ✓PagedAttention (Kwon et al., SOSP 2023) allocates fixed blocks through block tables: waste < 1 block per sequence instead of 60–80% in contiguous systems, plus sharing and copy-on-write.
- ✓Prefix caching reuses exact-prefix blocks; KV quantization (fp8, KIVI's 2-bit) halves or quarters memory; StreamingLLM's attention sinks and H2O's heavy hitters bound it by eviction.