KV cache

The KV cache stores each layer's keys and values so a decode step computes only the new token. It costs 2·L·n_kv·d_head·bytes per token (512 KiB for Llama 2 7B, 128 KiB for Llama 3 8B with GQA) and dominates serving memory and decode bandwidth.

15 min read1791 wordscore0 viewsupdated 2026-10-13

Definition

KV cache is the per-sequence store of the key and value vectors that every attention layer of a decoder-only language model has already computed. Under a causal mask the keys and values of past positions never change. Each generation step can therefore compute q,k,vq,k,v for the newest token only, append kk and vv to the cache, and attend over the cached rows. The technique is as old as incremental transformer decoding. Its systems consequences were analysed by Pope et al. ("Efficiently Scaling Transformer Inference", 2022), and its memory management was reworked by Kwon et al. ("Efficient Memory Management for Large Language Model Serving with PagedAttention", SOSP 2023), the paper behind vLLM.

Causal attention and Decoder-only transformer introduce the cache as a correctness fact: the mask makes token-by-token decoding reproduce the parallel training pass exactly. This article treats the cache as the central object of LLM serving. Its size sets how many requests fit on a GPU. Reading it sets the speed of every decode step. Most inference work since 2019 attacks it, through fewer key/value heads, paged allocation, sharing, quantization or eviction.

Without a cache, step tt re-runs the whole prefix: O(t)O(t) projections and O(t2)O(t^2) query-key products per layer, so generating nn tokens costs Θ(n3)\Theta(n^3) score products. With the cache, step tt projects one token and scores one query against tt keys, Θ(n2)\Theta(n^2) in total. The trade is compute for memory, and at LLM scale the memory side is large.

Prefill and decode

Serving a request has two phases with very different hardware profiles. Prefill pushes the whole prompt through the model in one causal pass, exactly as in training, and writes TT rows of KK and VV into every layer. It is a matrix–matrix product of about 2NT2NT FLOPs for NN parameters, so it is compute-bound and sets the time to first token. Decode then produces one token per sequence per step. Each step is a matrix–vector product of about 2N2N FLOPs per sequence, but it has to stream every weight and the sequence's whole cache from HBM. It is memory-bandwidth-bound and sets the time per output token.

one layer's cache, one KV head: prefill writes T rows, each decode step appends oneo6 = softmax(q6·[k1…k6]ᵀ/√d_h)·[v1…v6]KVk1k2k3k4k5k6··v1v2v3v4v5v6··futureslotsprefill: x1…x5one causal passdecode: x6 onlyq6, k6, v6 per layerwrite 5 rowsappendx1x2x3x4x5x6← sampled from o5prompt tokens (prefill) · generated token (decode)PagedAttention (vLLM)block table, 4 tokens per blockseq A→7→1→3seq B→7→5physical KV blocks in HBM0 free12 free34 free56 free77: shared prefix, ref count 23, 5: last blocks, part emptyprefill (prompt of T tokens)matrix–matrix: ≈ 2N·T FLOPs, one passcompute-bound → time to first tokenwrites T rows of K, V in every layerdecode (1 new token per sequence)matrix–vector: ≈ 2N FLOPs per tokenreads all weights + whole cachebandwidth-bound → time per output tokenLlama 2 7B: 2 · 32 layers · 32 KV heads · 128 · 2 bytes = 512 KiB per token → 2 GiB at 4,096 tokens
fig.Left: one layer's cache for one KV head. Prefill computes keys and values for the five prompt tokens in one causal pass and writes five rows. The decode step computes q₆, k₆, v₆ for the sampled token x₆ only, appends one row, and attends with q₆ over all six cached keys and values. Right: PagedAttention stores the rows in fixed-size physical blocks, here 4 tokens. Each sequence's block table maps its logical blocks to arbitrary physical ones, and two sequences with the same prompt prefix point at the same block 7. Bottom: the two phases and the Llama 2 7B per-token cost. Hover a phase to highlight the rows it touches.
K1:t(ℓ)=[K1:t−1(ℓ)kt(ℓ)⊤],V1:t(ℓ)=[V1:t−1(ℓ)vt(ℓ)⊤],ot(ℓ)=softmax ⁣(qt(ℓ)K1:t(ℓ)⊤dh)V1:t(ℓ)K^{(\ell)}_{1:t}=\begin{bmatrix}K^{(\ell)}_{1:t-1}\\ k^{(\ell)\top}_t\end{bmatrix},\quad V^{(\ell)}_{1:t}=\begin{bmatrix}V^{(\ell)}_{1:t-1}\\ v^{(\ell)\top}_t\end{bmatrix},\qquad o^{(\ell)}_t=\mathrm{softmax}\!\Big(\frac{q^{(\ell)}_t K^{(\ell)\top}_{1:t}}{\sqrt{d_h}}\Big)V^{(\ell)}_{1:t}
qt,kt,vtq_t,k_t,v_t
projections of the newest token only, per head; with RoPE the key is cached already rotated
K1:t−1,V1:t−1K_{1:t-1},V_{1:t-1}
read from the cache, never recomputed
oto_t
one query row against tt keys; no mask is needed because the cache holds only the past
ℓ\ell
every one of the LL layers keeps its own cache

How big it is

KV bytes=2⋅L⋅nkv⋅dh⋅T⋅B⋅b\text{KV bytes}=2\cdot L\cdot n_{kv}\cdot d_h\cdot T\cdot B\cdot b
22
a key and a value
LL
layers
nkvn_{kv}
key/value heads: HH for multi-head attention, 1 for Multi-query attention (MQA), H/gH/g for Grouped-query attention (GQA)
dhd_h
head size, 128 in every Llama
T, BT,\,B
tokens held per sequence and sequences in the batch
bb
bytes per number: 2 for bf16, 1 for fp8 or int8, 0.5 for int4

The formula has no dmodeld_{model} and no query heads, because queries are never cached. It is linear in context and in batch. For Llama 2 7B, 2⋅32⋅32⋅128⋅2=524,2882\cdot32\cdot32\cdot128\cdot2=524{,}288 bytes, 512 KiB per token. A 4096-token sequence then holds 2 GiB, and 30 such sequences (64.4 GB) fit in the 66.5 GB an 80 GB GPU has left after 13.5 GB of bf16 weights. Kwon et al. quote the same shape for OPT-13B: 800 KB per token and up to 1.6 GB per 2048-token request.

Why decode is bandwidth-bound

Iweights=2NB FLOPs2N bytes=B,Iattn=4 T H dh2⋅2 T nkv dh=Hnkv,tstep ≥ 2N+KV bytesHBM bandwidthI_{\text{weights}}=\frac{2NB\ \text{FLOPs}}{2N\ \text{bytes}}=B,\qquad I_{\text{attn}}=\frac{4\,T\,H\,d_h}{2\cdot2\,T\,n_{kv}\,d_h}=\frac{H}{n_{kv}},\qquad t_{\text{step}}\ \ge\ \frac{2N+\text{KV bytes}}{\text{HBM bandwidth}}
II
arithmetic intensity, FLOPs per byte read from HBM
Iweights=BI_{\text{weights}}=B
each bf16 weight is read once per step and used once per sequence in the batch
Iattn=H/nkvI_{\text{attn}}=H/n_{kv}
each sequence has its own cache, so batching does not amortise it: 1 FLOP/byte for MHA, 4 for Llama 3 8B's GQA
tstept_{\text{step}}
a floor that holds whenever II is far below the hardware ridge point

An A100 does 312 dense bf16 TFLOPS against about 2 TB/s of HBM bandwidth, a ridge point of about 156 FLOPs per byte. An H100 SXM's is about 295 (989 TFLOPS, 3.35 TB/s). Decode at batch 1 has an intensity near 1 and leaves almost all the arithmetic idle. Batching raises the weight intensity to BB but does nothing for attention, whose intensity stays at H/nkvH/n_{kv} whatever the batch. As contexts grow, the cache becomes the larger share of the bytes read, and decode speed tracks cache size directly.

Making the cache smaller: fewer or compressed heads

The first lever is nkvn_{kv}. Shazeer's multi-query attention ("Fast Transformer Decoding: One Write-Head is All You Need", 2019) shares one key and one value head among all query heads, a factor of HH. Ainslie et al.'s grouped-query attention (2023) shares each K/V head among a group of query heads. Llama 2 70B and all Llama 3 models use 8 KV heads, which cuts Llama 3 8B to 128 KiB per token, 4 times less than Llama 2 7B, at nearly the same size.

ct=WDKVht∈Rdc,kt,iC=WiUKct,vt,iC=WiUVct,cache=[ct; ktR]∈Rdc+dhRc_t=W^{DKV}h_t\in\mathbb R^{d_c},\qquad k^{C}_{t,i}=W^{UK}_i c_t,\quad v^{C}_{t,i}=W^{UV}_i c_t,\qquad \text{cache}=\big[c_t;\ k^R_t\big]\in\mathbb R^{d_c+d^R_h}
ctc_t
multi-head latent attention (DeepSeek-V2, 2024): one shared latent per token and layer, dc=512d_c=512
WiUK, WiUVW^{UK}_i,\,W^{UV}_i
per-head up-projections; at inference WUKW^{UK} is absorbed into the query projection and WUVW^{UV} into the output projection, so full keys and values are never materialised
ktRk^R_t
a separate 64-dim RoPE key shared by all heads, because a position-dependent rotation cannot be absorbed into WUKW^{UK}

DeepSeek-V2 has 60 layers and 128 heads of size 128. It caches 512+64=576512+64=576 numbers per token and layer, (dc+dhR)L=4.5 dhL(d_c+d^R_h)L=4.5\,d_hL, which the paper compares to GQA with 2.25 groups. Full multi-head attention of the same shape would cache 2⋅128⋅128=32,7682\cdot128\cdot128=32{,}768 per layer, so the latent is 98.2% smaller. Against its predecessor DeepSeek 67B, the paper reports a 93.3% smaller KV cache and 5.76 times the maximum generation throughput.

Per-token KV cache of real models (bf16)

Llama 2 7B
32 layers, 32 KV heads, dh=128d_h=128: 512 KiB/token, 2 GiB at its 4096-token context
Llama 2 13B
40 layers, 40 KV heads: 800 KiB/token (the OPT-13B shape in the vLLM paper)
Llama 3 8B
32 layers, 8 KV heads (GQA): 128 KiB/token, 1 GiB at 8192 tokens, 16 GiB at 128K
Llama 3 70B
80 layers, 8 KV heads for 64 query heads: 320 KiB/token instead of 2.5 MiB with MHA
Mistral 7B
32 layers, 8 KV heads, sliding window 4096 with a rolling buffer (slot i mod 4096i \bmod 4096): 128 KiB/token, never more than 0.5 GiB per sequence; the paper cites an 8× saving at 32K tokens
DeepSeek-V2
60 layers, MLA with dc=512d_c=512 plus a 64-dim RoPE key: 67.5 KiB/token against 3.75 MiB for MHA of the same shape
Hardware
A100 80GB: about 2.0 TB/s and 312 dense bf16 TFLOPS. H100 SXM: 3.35 TB/s and 989 TFLOPS

Allocating it: PagedAttention

Early serving systems stored each request's cache in one contiguous tensor sized for the maximum length, because the final length is unknown when the request arrives. Kwon et al. measured that in such systems only 20.4% to 38.2% of KV memory held actual token states. The rest was reserved slots that were never filled, internal fragmentation inside a reservation, or external fragmentation between reservations.

token i ↦ (table[⌊i/bs⌋], i mod bs),waste per sequence < bs slots\text{token } i\ \mapsto\ \big(\text{table}[\lfloor i/b_s\rfloor],\ i \bmod b_s\big),\qquad \text{waste per sequence}\ <\ b_s\ \text{slots}
bsb_s
block (page) size in tokens; vLLM's default is 16
table\text{table}
the sequence's block table: logical block → physical block in a shared pool
<bs<b_s
only the last block of a sequence can be partly empty

PagedAttention borrows virtual memory from operating systems. The cache lives in a pool of fixed-size physical blocks. A sequence takes a new block only when its last one fills, and the attention kernel follows the block table, so the blocks need not be adjacent. Waste drops below 4% in the paper's measurements. Blocks are reference-counted, so parallel samples and beam-search branches share their common prefix and copy a block only when they write to it (copy-on-write). vLLM reported 2–4× the throughput of FasterTransformer and Orca at the same latency, mostly because more sequences fit in a batch.

KV bytes = 2·L·n_kv·d_head·T·B·bytes · Llama 2 7B: L = 32, MHAper token (bf16): 512 KiB · every query head has its own K/V headper sequence at T = 4,096: 2.00 GiB · batch 8: 16.0 GiBHBM of 1 × 80 GB GPU■ bf16 weights 13 GB■ KV cache 17 GBfree 49 GBsequences of 4,096 tokens that fit next to the weights: 30decode step ≥ (weights read + cache) / (1 × 2 TB/s) = 15.33 ms → 522 tokens/scache share of bytes read per step: 56.0%12 requests, final lengths 3273, 341, 169, 383, … tokens▭ contiguous: reserve T = 4,096 each · ■ paged: 16-token blocks on demand · ■ tokens held04,096r1r2r3r4r5r6r7r8r9r10r11r12tokens held8,749contiguous reserve49,152 (17.8% used)paged allocation8,832 (99.1% used)wasted, paged83 tokens (≤ 15/seq)fit in free HBM30 contig. vs 172 pagedfit in free HBM: sequences like these 12 (mean 729 tokens); 1 GB = 10⁹ bytes, 1 GiB = 2³⁰
model
KV dtypeGPUs
page size
fig.Top: the cache of the selected model against the HBM of 1–8 80 GB GPUs, with the bf16 weights in orange. At the defaults (Llama 2 7B, bf16, 4096 tokens, batch 8) the cache is 512 KiB per token, 2.00 GiB per sequence and 16.00 GiB for the batch, more than the 13 GB of weights. 30 sequences of 4096 tokens fit. A decode step must read weights plus cache, at least 15.33 ms at 2 TB/s, which gives 522 tokens/s, and 56.0% of those bytes are cache. At batch 30 the cache is 60 GiB, the step 38.95 ms and the cache share 82.7%. Batch 64 no longer fits. Switch to Llama 3 8B and GQA cuts the cache 75% to 128 KiB per token, and 119 sequences fit. At 128K one sequence still needs 16 GiB. Mistral 7B's rolling buffer caps a sequence at 0.5 GiB at any length. DeepSeek-V2's latent caches 67.5 KiB per token, 98.2% less than MHA of its shape. An fp8 cache halves Llama 2 7B's to 8 GiB and the step to 11.03 ms, and int4 brings it to 4 GiB and 8.89 ms. Bottom: twelve requests whose final lengths (prompt plus output) are 3273, 341, 169, 383 … tokens, 8,749 in all. Reserving 4096 slots each, a contiguous allocator uses 17.8% of 49,152 slots. 16-token pages allocate 8,832 slots (99.1% used, 83 wasted), so 172 such sequences fit in the free memory instead of 30. 64-token pages waste more (95.6%). At a 512-token maximum the contiguous reservation is 68.7% used.

Sharing it: prefix caching

Many requests start with the same tokens: a system prompt, few-shot examples, a shared document, or the earlier turns of a chat. Their keys and values are identical, because each position depends only on the tokens before it. Prefix caching keeps those blocks after a request finishes and maps new requests onto them. vLLM's automatic prefix caching hashes each full block together with all the tokens before it. SGLang's RadixAttention (Zheng et al., 2024) keeps cached prefixes in a radix tree with LRU eviction. A hit skips the prefill for the shared part, which cuts time to first token, and stores one copy instead of one per request. The match must be exact, token for token from position 0: one changed early token invalidates everything after it.

Shrinking it: quantization

The cache can be stored in fewer bits than the weights. FP8 caches with a per-tensor or per-head scale are standard in vLLM and TensorRT-LLM and halve both memory and bytes read. Lower precisions need care with outliers. KIVI (Liu et al., 2024) found that keys have a few channels with large magnitudes, while values do not. It quantizes keys per channel and values per token, keeps the most recent tokens in full precision, and reaches 2 bits with little loss. KVQuant (Hooper et al., 2024) also quantizes keys per channel, and does it before RoPE is applied. See Quantization for the general machinery.

Bounding it: eviction

A sliding window caps the cache at WW tokens, as in Mistral's rolling buffer, but a model trained with full attention fails when its first tokens are evicted. Xiao et al. ("Efficient Streaming Language Models with Attention Sinks", ICLR 2024) traced this to attention sinks. Softmax must put its probability mass somewhere, and trained models dump a large share of it on the first few tokens whatever they contain. StreamingLLM keeps the KV of 4 initial tokens plus a recent window, and assigns positions by slot in the cache rather than in the original text. With this, Llama 2, MPT, Falcon and Pythia ran stably over up to 4 million tokens, up to 22.2× faster than a sliding window that recomputes its keys. H2O (Zhang et al., NeurIPS 2023) keeps the recent tokens plus "heavy hitters", the tokens with the largest accumulated attention. A budget of about 20% of the cache preserved accuracy on many tasks. Both are lossy: an evicted token cannot be recalled, which hurts retrieval from long documents.

Worked example: serving Llama 3 8B on one 80 GB GPU

Llama 3 8B: L=32L=32, nkv=8n_{kv}=8, dh=128d_h=128, d=4096d=4096, N=8.03×109N=8.03\times10^9 parameters. Contexts of T=8192T=8192 tokens, batch B=16B=16, bf16 everywhere, 2.0 TB/s and 312 TFLOPS.

quantityformulavalue
KV per token2·32·8·128·2 bytes131,072 B = 128 KiB
KV per sequence× 81921 GiB (1.07 GB)
KV for the batch× 1616 GiB (17.18 GB)
weights8.03e9 · 2 bytes16.06 GB
bytes read per decode stepweights + cache33.24 GB
decode step floor33.24 GB / 2.0 TB/s16.62 ms → 963 tokens/s
FLOPs per step2NB + 4·L·T·d·B2.57e11 + 6.87e10 = 3.26e11
achieved compute3.26e11 / 16.62 ms19.6 TFLOPS, 6.3% of peak
same model with 32 KV heads4 × cache68.7 GB + 16.06 GB > 80 GB
Memory and step-time budget (computed in Python)

Reading the budget

  1. 1Prefill of a 4096-token prompt costs 2NT=2⋅8.03×109⋅4096=6.58×10132NT=2\cdot8.03\times10^9\cdot4096=6.58\times10^{13} FLOPs, about 0.21 s at full A100 throughput. Its arithmetic intensity is about 3,964 FLOPs per byte, far above the ridge of 156, so it is compute-bound.
  2. 2A single decode step at B=1B=1 and T=4096T=4096 reads 16.06 GB of weights and 0.54 GB of cache, at least 8.30 ms. Nearly all of that time is spent waiting for memory.
  3. 3At B=16B=16, T=8192T=8192 the floor is 16.62 ms per step, or 963 tokens/s, but the GPU computes at only 6.3% of its peak. The cache is already 52% of the bytes read, and the attention part of the work has intensity H/nkv=4H/n_{kv}=4 no matter how large the batch is.
  4. 4Free memory after the weights is 63.94 GB, room for 59 sequences of 8192 tokens in bf16. An fp8 cache doubles that to 119 and lowers the step floor to 12.32 ms.
  5. 5With multi-head attention (32 KV heads) the batch-16 cache would be 68.7 GB, and it would not fit beside the weights. GQA is what makes this configuration possible.
  6. 6A 2000-token system prompt shared by 100 concurrent requests costs 24.4 GiB if each request stores its own copy and 250 MiB with prefix caching.
python
1import numpy as np
2
3class PagedKVCache:
4 """One layer, n_kv heads. K/V live in a pool of fixed-size blocks; each sequence
5 owns a block table (logical block i -> physical block id), as in vLLM."""
6 def __init__(self, num_blocks, block_size, n_kv, d_head):
7 self.bs = block_size
8 self.k = np.zeros((num_blocks, block_size, n_kv, d_head), np.float32)
9 self.v = np.zeros_like(self.k)
10 self.free = list(range(num_blocks))[::-1]
11 self.tables, self.lens = {}, {}
12
13 def append(self, seq, k, v): # k, v: (n_kv, d_head) for one new token
14 n = self.lens.get(seq, 0)
15 if n % self.bs == 0: # last block full (or none yet): grab a new one
16 self.tables.setdefault(seq, []).append(self.free.pop())
17 blk = self.tables[seq][n // self.bs]
18 self.k[blk, n % self.bs], self.v[blk, n % self.bs] = k, v
19 self.lens[seq] = n + 1
20
21 def gather(self, seq): # (T, n_kv, d_head), in logical order
22 n, t = self.lens[seq], self.tables[seq]
23 return self.k[t].reshape(-1, *self.k.shape[2:])[:n], self.v[t].reshape(-1, *self.v.shape[2:])[:n]
24
25def decode_attention(q, K, V): # q (H, d), K/V (T, n_kv, d); GQA by repeat
26 g = q.shape[0] // K.shape[1]
27 K, V = np.repeat(K, g, axis=1), np.repeat(V, g, axis=1) # (T, H, d)
28 s = np.einsum("hd,thd->ht", q, K) / np.sqrt(q.shape[1])
29 p = np.exp(s - s.max(1, keepdims=True)); p /= p.sum(1, keepdims=True)
30 return np.einsum("ht,thd->hd", p, V)
31
32rng = np.random.default_rng(0)
33H, n_kv, d, bs = 8, 2, 64, 16
34cache = PagedKVCache(num_blocks=64, block_size=bs, n_kv=n_kv, d_head=d)
35ref = {"a": ([], []), "b": ([], [])}
36for step in range(70): # two sequences interleaved, as a server batches them
37 for seq, length in (("a", 70), ("b", 37)):
38 if step < length:
39 k, v = rng.standard_normal((2, n_kv, d)).astype(np.float32)
40 cache.append(seq, k, v); ref[seq][0].append(k); ref[seq][1].append(v)
41
42q = rng.standard_normal((H, d)).astype(np.float32)
43for seq in ("a", "b"):
44 o_paged = decode_attention(q, *cache.gather(seq))
45 o_ref = decode_attention(q, np.stack(ref[seq][0]), np.stack(ref[seq][1]))
46 print(seq, cache.lens[seq], cache.tables[seq], np.abs(o_paged - o_ref).max())
47# a 70 [0, 2, 4, 6, 7] 0.0 <- blocks of the two sequences interleave in the pool
48# b 37 [1, 3, 5] 0.0
49
50used = sum(cache.lens.values()); alloc = sum(len(t) for t in cache.tables.values()) * bs
51print(f"tokens {used}, slots allocated {alloc}, utilisation {used / alloc:.1%}")
52# tokens 107, slots allocated 128, utilisation 83.6%
53
54def kv_bytes(L, n_kv, d_head, T, B, bytes_per=2):
55 return 2 * L * n_kv * d_head * T * B * bytes_per
56print(kv_bytes(32, 32, 128, 4096, 1) / 2**30, kv_bytes(80, 8, 128, 8192, 1) / 2**30) # Llama 2 7B, Llama 3 70B
57# 2.0 2.5
A paged KV cache in numpy: a block pool, per-sequence block tables, GQA decode attention over gathered blocks, and a check against a contiguous cache (run with numpy; printed output shown)

Use when

  • +Exact: under a causal mask, cached decoding reproduces the full recomputation bit for bit, up to rounding
  • +Turns generation from Θ(n³) into Θ(n²) score products and one token's projections per step
  • +Paged allocation wastes under one block per sequence and enables prefix sharing and copy-on-write
  • +Composable savings: GQA/MQA/MLA, fp8 or 2-bit storage, sliding windows and eviction multiply

Avoid when

  • −Memory grows linearly in context × batch and can exceed the weights (16 GiB vs 13.5 GB for Llama 2 7B at 8 × 4096)
  • −Every decode step re-reads the whole cache, so long-context decode is bandwidth-bound
  • −Eviction (StreamingLLM, H2O) is lossy; evicted facts cannot be retrieved later
  • −Prefix reuse requires an exact token match from the start, and RoPE keys are tied to absolute positions

Where it's used

LLM serving engines

vLLM, SGLang and TensorRT-LLM are built around paged KV blocks, prefix reuse and continuous batching

Chat and agents

multi-turn conversations and tool loops reuse the cached prefix of earlier turns instead of re-running prefill

Long-context inference

fp8 or lower-bit caches and GQA/MLA decide whether a 128K-token request fits at all

Speculative decoding

the target model verifies drafted tokens in one pass over its cache and rolls the cache back on rejection (see Speculative decoding)

Check yourself

Llama 3 70B (80 layers, 64 query heads, 8 KV heads, d_head = 128) serves batch 4 at 32,768 tokens with a bf16 cache. How large is the cache?

Check yourself

A model trained with full attention is run with a pure sliding-window cache that evicts the first tokens, and perplexity explodes once they are gone. What does StreamingLLM keep to fix this?

Remember

  • ✓KV cache =2⋅L⋅nkv⋅dh⋅T⋅B⋅b=2\cdot L\cdot n_{kv}\cdot d_h\cdot T\cdot B\cdot b bytes: 512 KiB per token for Llama 2 7B, 128 KiB for Llama 3 8B (GQA), 67.5 KiB for DeepSeek-V2 (MLA).
  • ✓Prefill is one compute-bound causal pass that writes T rows per layer; decode is one token per step, bandwidth-bound, re-reading all weights plus the whole cache.
  • ✓Decode intensity is BB FLOPs/byte for weights but only H/nkvH/n_{kv} for attention, so long-context throughput tracks cache size, not batch size.
  • ✓PagedAttention (Kwon et al., SOSP 2023) allocates fixed blocks through block tables: waste < 1 block per sequence instead of 60–80% in contiguous systems, plus sharing and copy-on-write.
  • ✓Prefix caching reuses exact-prefix blocks; KV quantization (fp8, KIVI's 2-bit) halves or quarters memory; StreamingLLM's attention sinks and H2O's heavy hitters bound it by eviction.
← back to Deep Learning roadmap