# KV cache

> The KV cache stores each layer's keys and values so a decode step computes only the new token. It costs 2·L·n_kv·d_head·bytes per token (512 KiB for Llama 2 7B, 128 KiB for Llama 3 8B with GQA) and dominates serving memory and decode bandwidth.

Source: https://www.mlfoundry.in/learn/kv-cache  
Section: Deep Learning › Large Language Models (LLMs)  
Level: core · 15 min read · updated 2026-10-13

**KV cache:** is the per-sequence store of the key and value vectors that every attention layer of a decoder-only language model has already computed. Under a causal mask the keys and values of past positions never change. Each generation step can therefore compute $q,k,v$ for the newest token only, append $k$ and $v$ to the cache, and attend over the cached rows. The technique is as old as incremental transformer decoding. Its systems consequences were analysed by Pope et al. ("Efficiently Scaling Transformer Inference", 2022), and its memory management was reworked by Kwon et al. ("Efficient Memory Management for Large Language Model Serving with PagedAttention", SOSP 2023), the paper behind vLLM.

**Causal attention** and **Decoder-only transformer** introduce the cache as a correctness fact: the mask makes token-by-token decoding reproduce the parallel training pass exactly. This article treats the cache as the central object of LLM serving. Its size sets how many requests fit on a GPU. Reading it sets the speed of every decode step. Most inference work since 2019 attacks it, through fewer key/value heads, paged allocation, sharing, quantization or eviction.

Without a cache, step $t$ re-runs the whole prefix: $O(t)$ projections and $O(t^2)$ query-key products per layer, so generating $n$ tokens costs $\Theta(n^3)$ score products. With the cache, step $t$ projects one token and scores one query against $t$ keys, $\Theta(n^2)$ in total. The trade is compute for memory, and at LLM scale the memory side is large.

## Prefill and decode

Serving a request has two phases with very different hardware profiles. **Prefill** pushes the whole prompt through the model in one causal pass, exactly as in training, and writes $T$ rows of $K$ and $V$ into every layer. It is a matrix–matrix product of about $2NT$ FLOPs for $N$ parameters, so it is compute-bound and sets the **time to first token**. **Decode** then produces one token per sequence per step. Each step is a matrix–vector product of about $2N$ FLOPs per sequence, but it has to stream every weight and the sequence's whole cache from HBM. It is memory-bandwidth-bound and sets the **time per output token**.

> Figure: Left: one layer's cache for one KV head. Prefill computes keys and values for the five prompt tokens in one causal pass and writes five rows. The decode step computes q₆, k₆, v₆ for the sampled token x₆ only, appends one row, and attends with q₆ over all six cached keys and values. Right: PagedAttention stores the rows in fixed-size physical blocks, here 4 tokens. Each sequence's block table maps its logical blocks to arbitrary physical ones, and two sequences with the same prompt prefix point at the same block 7. Bottom: the two phases and the Llama 2 7B per-token cost. Hover a phase to highlight the rows it touches.

$$
K^{(\ell)}_{1:t}=\begin{bmatrix}K^{(\ell)}_{1:t-1}\\ k^{(\ell)\top}_t\end{bmatrix},\quad V^{(\ell)}_{1:t}=\begin{bmatrix}V^{(\ell)}_{1:t-1}\\ v^{(\ell)\top}_t\end{bmatrix},\qquad o^{(\ell)}_t=\mathrm{softmax}\!\Big(\frac{q^{(\ell)}_t K^{(\ell)\top}_{1:t}}{\sqrt{d_h}}\Big)V^{(\ell)}_{1:t}
$$

- $q_t,k_t,v_t$: projections of the newest token only, per head; with RoPE the key is cached already rotated
- $K_{1:t-1},V_{1:t-1}$: read from the cache, never recomputed
- $o_t$: one query row against $t$ keys; no mask is needed because the cache holds only the past
- $\ell$: every one of the $L$ layers keeps its own cache

## How big it is

$$
\text{KV bytes}=2\cdot L\cdot n_{kv}\cdot d_h\cdot T\cdot B\cdot b
$$

- $2$: a key and a value
- $L$: layers
- $n_{kv}$: key/value heads: $H$ for multi-head attention, 1 for **Multi-query attention (MQA)**, $H/g$ for **Grouped-query attention (GQA)**
- $d_h$: head size, 128 in every Llama
- $T,\,B$: tokens held per sequence and sequences in the batch
- $b$: bytes per number: 2 for bf16, 1 for fp8 or int8, 0.5 for int4

The formula has no $d_{model}$ and no query heads, because queries are never cached. It is linear in context and in batch. For Llama 2 7B, $2\cdot32\cdot32\cdot128\cdot2=524{,}288$ bytes, 512 KiB per token. A 4096-token sequence then holds 2 GiB, and 30 such sequences (64.4 GB) fit in the 66.5 GB an 80 GB GPU has left after 13.5 GB of bf16 weights. Kwon et al. quote the same shape for OPT-13B: 800 KB per token and up to 1.6 GB per 2048-token request.

## Why decode is bandwidth-bound

$$
I_{\text{weights}}=\frac{2NB\ \text{FLOPs}}{2N\ \text{bytes}}=B,\qquad I_{\text{attn}}=\frac{4\,T\,H\,d_h}{2\cdot2\,T\,n_{kv}\,d_h}=\frac{H}{n_{kv}},\qquad t_{\text{step}}\ \ge\ \frac{2N+\text{KV bytes}}{\text{HBM bandwidth}}
$$

- $I$: arithmetic intensity, FLOPs per byte read from HBM
- $I_{\text{weights}}=B$: each bf16 weight is read once per step and used once per sequence in the batch
- $I_{\text{attn}}=H/n_{kv}$: each sequence has its own cache, so batching does not amortise it: 1 FLOP/byte for MHA, 4 for Llama 3 8B's GQA
- $t_{\text{step}}$: a floor that holds whenever $I$ is far below the hardware ridge point

An A100 does 312 dense bf16 TFLOPS against about 2 TB/s of HBM bandwidth, a ridge point of about 156 FLOPs per byte. An H100 SXM's is about 295 (989 TFLOPS, 3.35 TB/s). Decode at batch 1 has an intensity near 1 and leaves almost all the arithmetic idle. Batching raises the weight intensity to $B$ but does nothing for attention, whose intensity stays at $H/n_{kv}$ whatever the batch. As contexts grow, the cache becomes the larger share of the bytes read, and decode speed tracks cache size directly.

## Making the cache smaller: fewer or compressed heads

The first lever is $n_{kv}$. Shazeer's multi-query attention ("Fast Transformer Decoding: One Write-Head is All You Need", 2019) shares one key and one value head among all query heads, a factor of $H$. Ainslie et al.'s grouped-query attention (2023) shares each K/V head among a group of query heads. Llama 2 70B and all Llama 3 models use 8 KV heads, which cuts Llama 3 8B to 128 KiB per token, 4 times less than Llama 2 7B, at nearly the same size.

$$
c_t=W^{DKV}h_t\in\mathbb R^{d_c},\qquad k^{C}_{t,i}=W^{UK}_i c_t,\quad v^{C}_{t,i}=W^{UV}_i c_t,\qquad \text{cache}=\big[c_t;\ k^R_t\big]\in\mathbb R^{d_c+d^R_h}
$$

- $c_t$: multi-head latent attention (DeepSeek-V2, 2024): one shared latent per token and layer, $d_c=512$
- $W^{UK}_i,\,W^{UV}_i$: per-head up-projections; at inference $W^{UK}$ is absorbed into the query projection and $W^{UV}$ into the output projection, so full keys and values are never materialised
- $k^R_t$: a separate 64-dim RoPE key shared by all heads, because a position-dependent rotation cannot be absorbed into $W^{UK}$

DeepSeek-V2 has 60 layers and 128 heads of size 128. It caches $512+64=576$ numbers per token and layer, $(d_c+d^R_h)L=4.5\,d_hL$, which the paper compares to GQA with 2.25 groups. Full multi-head attention of the same shape would cache $2\cdot128\cdot128=32{,}768$ per layer, so the latent is 98.2% smaller. Against its predecessor DeepSeek 67B, the paper reports a 93.3% smaller KV cache and 5.76 times the maximum generation throughput.

**Per-token KV cache of real models (bf16)**

- **Llama 2 7B:** 32 layers, 32 KV heads, $d_h=128$: **512 KiB**/token, 2 GiB at its 4096-token context
- **Llama 2 13B:** 40 layers, 40 KV heads: **800 KiB**/token (the OPT-13B shape in the vLLM paper)
- **Llama 3 8B:** 32 layers, 8 KV heads (GQA): **128 KiB**/token, 1 GiB at 8192 tokens, 16 GiB at 128K
- **Llama 3 70B:** 80 layers, 8 KV heads for 64 query heads: **320 KiB**/token instead of 2.5 MiB with MHA
- **Mistral 7B:** 32 layers, 8 KV heads, sliding window 4096 with a rolling buffer (slot $i \bmod 4096$): 128 KiB/token, never more than 0.5 GiB per sequence; the paper cites an 8× saving at 32K tokens
- **DeepSeek-V2:** 60 layers, MLA with $d_c=512$ plus a 64-dim RoPE key: **67.5 KiB**/token against 3.75 MiB for MHA of the same shape
- **Hardware:** A100 80GB: about 2.0 TB/s and 312 dense bf16 TFLOPS. H100 SXM: 3.35 TB/s and 989 TFLOPS

## Allocating it: PagedAttention

Early serving systems stored each request's cache in one contiguous tensor sized for the maximum length, because the final length is unknown when the request arrives. Kwon et al. measured that in such systems only 20.4% to 38.2% of KV memory held actual token states. The rest was reserved slots that were never filled, internal fragmentation inside a reservation, or external fragmentation between reservations.

$$
\text{token } i\ \mapsto\ \big(\text{table}[\lfloor i/b_s\rfloor],\ i \bmod b_s\big),\qquad \text{waste per sequence}\ <\ b_s\ \text{slots}
$$

- $b_s$: block (page) size in tokens; vLLM's default is 16
- $\text{table}$: the sequence's block table: logical block → physical block in a shared pool
- $<b_s$: only the last block of a sequence can be partly empty

PagedAttention borrows virtual memory from operating systems. The cache lives in a pool of fixed-size physical blocks. A sequence takes a new block only when its last one fills, and the attention kernel follows the block table, so the blocks need not be adjacent. Waste drops below 4% in the paper's measurements. Blocks are reference-counted, so parallel samples and beam-search branches share their common prefix and copy a block only when they write to it (copy-on-write). vLLM reported 2–4× the throughput of FasterTransformer and Orca at the same latency, mostly because more sequences fit in a batch.

> Figure: Top: the cache of the selected model against the HBM of 1–8 80 GB GPUs, with the bf16 weights in orange. At the defaults (Llama 2 7B, bf16, 4096 tokens, batch 8) the cache is 512 KiB per token, 2.00 GiB per sequence and 16.00 GiB for the batch, more than the 13 GB of weights. 30 sequences of 4096 tokens fit. A decode step must read weights plus cache, at least 15.33 ms at 2 TB/s, which gives 522 tokens/s, and 56.0% of those bytes are cache. At batch 30 the cache is 60 GiB, the step 38.95 ms and the cache share 82.7%. Batch 64 no longer fits. Switch to Llama 3 8B and GQA cuts the cache 75% to 128 KiB per token, and 119 sequences fit. At 128K one sequence still needs 16 GiB. Mistral 7B's rolling buffer caps a sequence at 0.5 GiB at any length. DeepSeek-V2's latent caches 67.5 KiB per token, 98.2% less than MHA of its shape. An fp8 cache halves Llama 2 7B's to 8 GiB and the step to 11.03 ms, and int4 brings it to 4 GiB and 8.89 ms. Bottom: twelve requests whose final lengths (prompt plus output) are 3273, 341, 169, 383 … tokens, 8,749 in all. Reserving 4096 slots each, a contiguous allocator uses 17.8% of 49,152 slots. 16-token pages allocate 8,832 slots (99.1% used, 83 wasted), so 172 such sequences fit in the free memory instead of 30. 64-token pages waste more (95.6%). At a 512-token maximum the contiguous reservation is 68.7% used.

## Sharing it: prefix caching

Many requests start with the same tokens: a system prompt, few-shot examples, a shared document, or the earlier turns of a chat. Their keys and values are identical, because each position depends only on the tokens before it. **Prefix caching** keeps those blocks after a request finishes and maps new requests onto them. vLLM's automatic prefix caching hashes each full block together with all the tokens before it. SGLang's RadixAttention (Zheng et al., 2024) keeps cached prefixes in a radix tree with LRU eviction. A hit skips the prefill for the shared part, which cuts time to first token, and stores one copy instead of one per request. The match must be exact, token for token from position 0: one changed early token invalidates everything after it.

## Shrinking it: quantization

The cache can be stored in fewer bits than the weights. FP8 caches with a per-tensor or per-head scale are standard in vLLM and TensorRT-LLM and halve both memory and bytes read. Lower precisions need care with outliers. KIVI (Liu et al., 2024) found that keys have a few channels with large magnitudes, while values do not. It quantizes keys per channel and values per token, keeps the most recent tokens in full precision, and reaches 2 bits with little loss. KVQuant (Hooper et al., 2024) also quantizes keys per channel, and does it before RoPE is applied. See **Quantization** for the general machinery.

## Bounding it: eviction

A sliding window caps the cache at $W$ tokens, as in Mistral's rolling buffer, but a model trained with full attention fails when its first tokens are evicted. Xiao et al. ("Efficient Streaming Language Models with Attention Sinks", ICLR 2024) traced this to **attention sinks**. Softmax must put its probability mass somewhere, and trained models dump a large share of it on the first few tokens whatever they contain. StreamingLLM keeps the KV of 4 initial tokens plus a recent window, and assigns positions by slot in the cache rather than in the original text. With this, Llama 2, MPT, Falcon and Pythia ran stably over up to 4 million tokens, up to 22.2× faster than a sliding window that recomputes its keys. H2O (Zhang et al., NeurIPS 2023) keeps the recent tokens plus "heavy hitters", the tokens with the largest accumulated attention. A budget of about 20% of the cache preserved accuracy on many tasks. Both are lossy: an evicted token cannot be recalled, which hurts retrieval from long documents.

## Worked example: serving Llama 3 8B on one 80 GB GPU

Llama 3 8B: $L=32$, $n_{kv}=8$, $d_h=128$, $d=4096$, $N=8.03\times10^9$ parameters. Contexts of $T=8192$ tokens, batch $B=16$, bf16 everywhere, 2.0 TB/s and 312 TFLOPS.

Memory and step-time budget (computed in Python)

| quantity | formula | value |
| --- | --- | --- |
| KV per token | 2·32·8·128·2 bytes | 131,072 B = 128 KiB |
| KV per sequence | × 8192 | 1 GiB (1.07 GB) |
| KV for the batch | × 16 | 16 GiB (17.18 GB) |
| weights | 8.03e9 · 2 bytes | 16.06 GB |
| bytes read per decode step | weights + cache | 33.24 GB |
| decode step floor | 33.24 GB / 2.0 TB/s | 16.62 ms → 963 tokens/s |
| FLOPs per step | 2NB + 4·L·T·d·B | 2.57e11 + 6.87e10 = 3.26e11 |
| achieved compute | 3.26e11 / 16.62 ms | 19.6 TFLOPS, 6.3% of peak |
| same model with 32 KV heads | 4 × cache | 68.7 GB + 16.06 GB > 80 GB |

**Reading the budget**

1. Prefill of a 4096-token prompt costs $2NT=2\cdot8.03\times10^9\cdot4096=6.58\times10^{13}$ FLOPs, about 0.21 s at full A100 throughput. Its arithmetic intensity is about 3,964 FLOPs per byte, far above the ridge of 156, so it is compute-bound.
2. A single decode step at $B=1$ and $T=4096$ reads 16.06 GB of weights and 0.54 GB of cache, at least 8.30 ms. Nearly all of that time is spent waiting for memory.
3. At $B=16$, $T=8192$ the floor is 16.62 ms per step, or 963 tokens/s, but the GPU computes at only 6.3% of its peak. The cache is already 52% of the bytes read, and the attention part of the work has intensity $H/n_{kv}=4$ no matter how large the batch is.
4. Free memory after the weights is 63.94 GB, room for 59 sequences of 8192 tokens in bf16. An fp8 cache doubles that to 119 and lowers the step floor to 12.32 ms.
5. With multi-head attention (32 KV heads) the batch-16 cache would be 68.7 GB, and it would not fit beside the weights. GQA is what makes this configuration possible.
6. A 2000-token system prompt shared by 100 concurrent requests costs 24.4 GiB if each request stores its own copy and 250 MiB with prefix caching.

```python
import numpy as np

class PagedKVCache:
    """One layer, n_kv heads. K/V live in a pool of fixed-size blocks; each sequence
    owns a block table (logical block i -> physical block id), as in vLLM."""
    def __init__(self, num_blocks, block_size, n_kv, d_head):
        self.bs = block_size
        self.k = np.zeros((num_blocks, block_size, n_kv, d_head), np.float32)
        self.v = np.zeros_like(self.k)
        self.free = list(range(num_blocks))[::-1]
        self.tables, self.lens = {}, {}

    def append(self, seq, k, v):                      # k, v: (n_kv, d_head) for one new token
        n = self.lens.get(seq, 0)
        if n % self.bs == 0:                          # last block full (or none yet): grab a new one
            self.tables.setdefault(seq, []).append(self.free.pop())
        blk = self.tables[seq][n // self.bs]
        self.k[blk, n % self.bs], self.v[blk, n % self.bs] = k, v
        self.lens[seq] = n + 1

    def gather(self, seq):                            # (T, n_kv, d_head), in logical order
        n, t = self.lens[seq], self.tables[seq]
        return self.k[t].reshape(-1, *self.k.shape[2:])[:n], self.v[t].reshape(-1, *self.v.shape[2:])[:n]

def decode_attention(q, K, V):                        # q (H, d), K/V (T, n_kv, d); GQA by repeat
    g = q.shape[0] // K.shape[1]
    K, V = np.repeat(K, g, axis=1), np.repeat(V, g, axis=1)          # (T, H, d)
    s = np.einsum("hd,thd->ht", q, K) / np.sqrt(q.shape[1])
    p = np.exp(s - s.max(1, keepdims=True)); p /= p.sum(1, keepdims=True)
    return np.einsum("ht,thd->hd", p, V)

rng = np.random.default_rng(0)
H, n_kv, d, bs = 8, 2, 64, 16
cache = PagedKVCache(num_blocks=64, block_size=bs, n_kv=n_kv, d_head=d)
ref = {"a": ([], []), "b": ([], [])}
for step in range(70):                                # two sequences interleaved, as a server batches them
    for seq, length in (("a", 70), ("b", 37)):
        if step < length:
            k, v = rng.standard_normal((2, n_kv, d)).astype(np.float32)
            cache.append(seq, k, v); ref[seq][0].append(k); ref[seq][1].append(v)

q = rng.standard_normal((H, d)).astype(np.float32)
for seq in ("a", "b"):
    o_paged = decode_attention(q, *cache.gather(seq))
    o_ref = decode_attention(q, np.stack(ref[seq][0]), np.stack(ref[seq][1]))
    print(seq, cache.lens[seq], cache.tables[seq], np.abs(o_paged - o_ref).max())
# a 70 [0, 2, 4, 6, 7] 0.0       <- blocks of the two sequences interleave in the pool
# b 37 [1, 3, 5] 0.0

used = sum(cache.lens.values()); alloc = sum(len(t) for t in cache.tables.values()) * bs
print(f"tokens {used}, slots allocated {alloc}, utilisation {used / alloc:.1%}")
# tokens 107, slots allocated 128, utilisation 83.6%

def kv_bytes(L, n_kv, d_head, T, B, bytes_per=2):
    return 2 * L * n_kv * d_head * T * B * bytes_per
print(kv_bytes(32, 32, 128, 4096, 1) / 2**30, kv_bytes(80, 8, 128, 8192, 1) / 2**30)  # Llama 2 7B, Llama 3 70B
# 2.0 2.5
```

A paged KV cache in numpy: a block pool, per-sequence block tables, GQA decode attention over gathered blocks, and a check against a contiguous cache (run with numpy; printed output shown)

> **Batching does not amortise the cache:** Weights are shared by every sequence in a batch, so their cost per token falls as $1/B$. Each sequence's cache is its own, so its read cost per token stays fixed. Throughput plans that assume "bigger batch, cheaper tokens" fail at long contexts. In the viz, Llama 2 7B at batch 30 and 4096 tokens spends 82.7% of each step's bytes on the cache. GQA, MLA, quantization and shorter retained contexts help here. A bigger batch does not.

> **The cache is position-dependent:** With RoPE the cached keys are already rotated by their absolute position. That is why prefix caching needs an exact match from position 0: the same paragraph at a different offset has different keys. It is also why StreamingLLM re-assigns positions by slot in the cache, and why some systems cache keys before rotation and apply RoPE when reading them.

> **Prefill and decode are now scheduled separately:** Because prefill is compute-bound and decode is bandwidth-bound, a long prompt arriving in a running batch stalls everyone's next token. Chunked prefill (Sarathi, Agrawal et al., 2023) splits prompts into pieces that are mixed into decode batches. Disaggregated serving (DistServe, Splitwise, 2024) runs the two phases on different GPUs and ships the KV cache between them. Either way, the cache is the state that moves.

**Strengths**

- Exact: under a causal mask, cached decoding reproduces the full recomputation bit for bit, up to rounding
- Turns generation from Θ(n³) into Θ(n²) score products and one token's projections per step
- Paged allocation wastes under one block per sequence and enables prefix sharing and copy-on-write
- Composable savings: GQA/MQA/MLA, fp8 or 2-bit storage, sliding windows and eviction multiply

**Limitations**

- Memory grows linearly in context × batch and can exceed the weights (16 GiB vs 13.5 GB for Llama 2 7B at 8 × 4096)
- Every decode step re-reads the whole cache, so long-context decode is bandwidth-bound
- Eviction (StreamingLLM, H2O) is lossy; evicted facts cannot be retrieved later
- Prefix reuse requires an exact token match from the start, and RoPE keys are tied to absolute positions

- **LLM serving engines:** vLLM, SGLang and TensorRT-LLM are built around paged KV blocks, prefix reuse and continuous batching
- **Chat and agents:** multi-turn conversations and tool loops reuse the cached prefix of earlier turns instead of re-running prefill
- **Long-context inference:** fp8 or lower-bit caches and GQA/MLA decide whether a 128K-token request fits at all
- **Speculative decoding:** the target model verifies drafted tokens in one pass over its cache and rolls the cache back on rejection (see **Speculative decoding**)

**Question:** Llama 3 70B (80 layers, 64 query heads, 8 KV heads, d_head = 128) serves batch 4 at 32,768 tokens with a bf16 cache. How large is the cache?

- A. 10 GiB
- B. 40 GiB
- C. 320 GiB
- D. 1.25 GiB

**Answer:** B. 40 GiB — Per token $2\cdot80\cdot8\cdot128\cdot2=327{,}680$ bytes = 320 KiB. Times 32,768 tokens that is 10 GiB per sequence, and 40 GiB for 4 sequences. The 64 query heads do not appear in the formula. With 64 KV heads it would be 320 GiB.

**Question:** A model trained with full attention is run with a pure sliding-window cache that evicts the first tokens, and perplexity explodes once they are gone. What does StreamingLLM keep to fix this?

- A. The tokens with the highest accumulated attention
- B. The KV of a few initial tokens, which act as attention sinks, plus a recent window
- C. A 2-bit copy of every evicted token
- D. Nothing; it recomputes the window's keys at every step

**Answer:** B. The KV of a few initial tokens, which act as attention sinks, plus a recent window — Trained models put a large share of attention mass on the first tokens regardless of their content, so softmax has somewhere to put probability it does not need. Removing them distorts every attention distribution. Keeping 4 sink tokens plus the recent window restores stable perplexity. Option A is H2O's heavy-hitter policy, and option D is the slow baseline StreamingLLM beats by up to 22.2×.

**Key points**

- KV cache $=2\cdot L\cdot n_{kv}\cdot d_h\cdot T\cdot B\cdot b$ bytes: 512 KiB per token for Llama 2 7B, 128 KiB for Llama 3 8B (GQA), 67.5 KiB for DeepSeek-V2 (MLA).
- Prefill is one compute-bound causal pass that writes T rows per layer; decode is one token per step, bandwidth-bound, re-reading all weights plus the whole cache.
- Decode intensity is $B$ FLOPs/byte for weights but only $H/n_{kv}$ for attention, so long-context throughput tracks cache size, not batch size.
- PagedAttention (Kwon et al., SOSP 2023) allocates fixed blocks through block tables: waste < 1 block per sequence instead of 60–80% in contiguous systems, plus sharing and copy-on-write.
- Prefix caching reuses exact-prefix blocks; KV quantization (fp8, KIVI's 2-bit) halves or quarters memory; StreamingLLM's attention sinks and H2O's heavy hitters bound it by eviction.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
