# Prompt caching

> Prompt caching keeps the KV cache of a shared prompt prefix so later requests with byte-identical prefixes skip that prefill; a hit cuts the prefix's input cost (to 0.1× on some APIs) and its prefill time, but only exact prefixes match.

Source: https://www.mlfoundry.in/learn/prompt-caching  
Section: Generative AI › Serving Generative Models  
Level: core · 8 min read · updated 2026-10-21

**Prompt caching:** (prefix caching, context caching) stores the attention keys and values computed for the start of a prompt and reuses them when a later request begins with exactly the same tokens, so prefill runs only on the new suffix. The output is still generated fresh. Research systems include Prompt Cache (Gim et al., MLSys 2024), SGLang's RadixAttention (Zheng et al., 2024) and vLLM's automatic prefix caching, built on **PagedAttention** (Kwon et al., SOSP 2023). Hosted APIs added it in 2024 (Anthropic in August, OpenAI in October, Google as Gemini context caching).

Real prompts repeat themselves. An assistant sends the same 2,000-token system prompt and tool list on every turn. A document Q&A app sends the same 50-page contract with each question. An agent resends its whole transcript every step. Without caching the server recomputes the **KV cache** for all of it each time. That costs prefill compute and adds hundreds of milliseconds to seconds of time to first token.

The key fact is causal attention: the key and value vectors at position $i$ depend on tokens $1\dots i$ and nothing after. So the cached state for a prefix is valid for any continuation of that prefix, and invalid the moment a single earlier token differs. Prompt caching is therefore prefix matching, not similarity matching. Reusing *answers* to similar questions is a different technique, **Semantic caching**.

> Figure: Four requests with cache breakpoints after a 2,048-token system prompt and an 8,192-token document (5-minute TTL, writes 1.25×, reads 0.1×). R1 writes both entries: 12,950 units instead of 10,390. R2, three minutes later, reads all 10,240 cached tokens: 1,204 instead of 10,420. R3 shares only the system prompt and writes doc B: 10,604.8, slightly more than uncached. R4 changed a timestamp near the top of the system prompt, so nothing matches.

## Cost, hit rate and break-even

$$
C(h)=S+P\,\big[h\,\rho_r+(1-h)\,\rho_w\big],\qquad h^{*}=\frac{\rho_w-1}{\rho_w-\rho_r}
$$

- $P,\ S$: cached prefix tokens and uncached suffix tokens per request
- $h$: fraction of requests that find the prefix in cache
- $\rho_r,\ \rho_w$: read and write price as multiples of the base input price
- $h^{*}$: break-even hit rate: caching beats no caching ($C=P+S$) when $h>h^{*}$

With $\rho_w=1.25$ and $\rho_r=0.1$, $h^*=0.25/1.15=21.7\%$. With a 1-hour entry at $\rho_w=2$, $h^*=1/1.9=52.6\%$. With automatic caching that charges no write premium, $h^*=0$: caching can only save.

$$
h=1-e^{-\lambda T}\qquad\text{(Poisson arrivals at rate }\lambda\text{, TTL }T\text{, each hit refreshes the entry)}
$$

- $\lambda T$: expected requests per TTL window; $\lambda T=1$ gives $h=63.2\%$

A request hits when the previous request with the same prefix started less than $T$ ago. The gap between Poisson arrivals is exponential, so the hit probability is $1-e^{-\lambda T}$. Traffic, not the prompt, decides whether caching pays. Latency follows the same logic: a hit removes roughly $P/R_\text{prefill}$ from TTFT, which is why caching and **Streaming generation** together make long-context apps feel responsive.

**How the main implementations behave (as documented in 2024–2025; check current docs)**

- **Anthropic Messages:** Explicit `"cache_control": {"type": "ephemeral"}` breakpoints on content blocks, at most 4 per request, or one top-level automatic breakpoint. Prefix order is tools → system → messages. Default TTL 5 minutes, refreshed free on every hit; optional `"ttl": "1h"`. Writes cost 1.25× (5 min) or 2× (1 h) the base input price, reads about 0.1×. Usage reports `cache_creation_input_tokens` and `cache_read_input_tokens`. Prefixes below a model-dependent minimum (hundreds to a few thousand tokens) are not cached.
- **OpenAI:** Automatic, no code change, for prompts of at least 1,024 tokens, matched in 128-token increments; cached tokens appear in `usage.prompt_tokens_details.cached_tokens` and were billed at 50% off at the October 2024 launch. Entries typically expire after 5–10 minutes of inactivity.
- **Gemini context caching:** Explicit cache objects with a chosen TTL, billed for storage per token-hour on top of discounted reads.
- **vLLM:** Automatic prefix caching (`--enable-prefix-caching`): each full 16-token KV block is keyed by a hash of its parent block's key and its own token ids, so shared prefixes share physical blocks; unused blocks are evicted LRU.
- **SGLang:** RadixAttention keeps cached prefixes in a radix tree over token sequences, with LRU eviction and a cache-aware scheduler that runs requests sharing long prefixes together.

> Figure: 200 Poisson requests sharing a 10,000-token prefix with a 200-token suffix. Defaults (12 requests/h, 5-minute TTL, writes 1.25×, reads 0.1×): 61.5% simulated hits against 1 − e^(−1) = 63.2% analytic, input cost 55.2% of no caching, mean prefill 0.81 s instead of 2.04 s (0.04 s on a hit). Drop to 2/h and the 5-minute cache costs about 109% of no caching; the 1-hour TTL at the same rate hits 86.5% and costs 36.9%. The automatic scheme (no write premium, 50% reads) at 12/h costs 69.9%.

## Designing prompts for cache hits

1. Order the prompt from most to least stable: tool definitions, system instructions, few-shot examples, long documents, conversation history, then the new user message.
2. Make the stable part byte-identical: no timestamps or request ids, deterministic tool order, sorted JSON keys, fixed whitespace.
3. Place a breakpoint at the end of each stable section (explicit APIs), or keep the shared prefix above the provider's minimum length (automatic APIs).
4. In multi-turn chats, move the last breakpoint forward to the newest turn so each request reuses the whole history.
5. Log the cached-token counter per route and alert when it drops; a drop usually means someone edited the top of the prompt.

## Worked example

P = 10,000 cached tokens, S = 200 uncached; costs in units of the base input-token price per request

| Case | Hit rate h | Computation | Cost | vs 10,200 |
| --- | --- | --- | --- | --- |
| No caching | — | 10,000 + 200 | 10,200 | 100% |
| Single miss (write) | 0 | 1.25 × 10,000 + 200 | 12,700 | 124.5% |
| Single hit (read) | 1 | 0.1 × 10,000 + 200 | 1,200 | 11.8% |
| 5-min TTL, 12 req/h | 1 − e^(−1) = 0.632 | 200 + 10,000 × (0.632 × 0.1 + 0.368 × 1.25) | 5,431 | 53.2% |
| 5-min TTL, 2 req/h | 1 − e^(−1/6) = 0.154 | 200 + 10,000 × (0.154 × 0.1 + 0.846 × 1.25) | 10,935 | 107.2% |
| 1-h TTL, 2 req/h | 1 − e^(−2) = 0.865 | 200 + 10,000 × (0.865 × 0.1 + 0.135 × 2) | 3,771 | 37.0% |

At 2 requests per hour the 5-minute cache loses money: 15.4% hits is below the 21.7% break-even. The 1-hour TTL wins at that rate because $\lambda T$ rises from 1/6 to 2. If the requests arrive more often than every 5 minutes anyway, the 1-hour TTL only adds the higher write price.

```python
import anthropic

client = anthropic.Anthropic()
MANUAL = open("product_manual.md").read()   # ~10k tokens, byte-identical every call

def ask(question: str):
    resp = client.messages.create(
        model="claude-opus-5-5",
        max_tokens=1024,
        system=[
            {"type": "text", "text": "Answer only from the manual below."},
            {"type": "text", "text": MANUAL,
             "cache_control": {"type": "ephemeral"}},   # breakpoint: end of shared prefix
        ],
        messages=[{"role": "user", "content": question}],  # varying part goes last
    )
    u = resp.usage
    print(f"write {u.cache_creation_input_tokens}  read {u.cache_read_input_tokens}  "
          f"uncached {u.input_tokens}")
    return resp

ask("How do I reset the device?")    # first call: write ≈ 10k, read 0
ask("What does error E42 mean?")     # within 5 min: write 0, read ≈ 10k
```

Cache a large, unchanging document with the Anthropic Python SDK and check the usage counters. Needs an API key; not run here

> **Silent invalidators:** Anything that changes early bytes breaks every later match: a timestamp or request id in the system prompt, tools serialised in a different order, JSON with unsorted keys, a retrieved passage inserted *before* the instructions, or switching model or tool set mid-conversation. The symptom is `cache_read_input_tokens` (or `cached_tokens`) stuck at zero. Put stable content first, volatile content after the last breakpoint, and log the read counter.

> **Isolation:** A cache shared across tenants is a timing side channel: a fast response reveals that someone else recently sent the same prefix. Hosted APIs scope caches to an organisation or workspace; self-hosted servers serving many customers should do the same, for example by salting block hashes per tenant.

- **Long system prompts and tools:** Agents and assistants resend thousands of tokens of instructions and tool schemas every turn; one breakpoint after them is the cheapest win.
- **Multi-turn conversations:** Each turn extends the previous prompt, so moving the breakpoint to the latest turn reuses the whole history.
- **Document Q&A:** Many questions over one contract, codebase or manual; the document is the prefix, the question is the suffix.
- **Few-shot and batch jobs:** Thousands of requests sharing the same instructions and examples, differing only in the item to classify.

**Question:** A RAG app builds prompts as [retrieved passages][system instructions][question]. Cache reads stay at zero. Why?

- A. The prompt is below the minimum length
- B. The retrieved passages differ per query and come first, so no two prompts share a prefix
- C. The TTL is too short
- D. Caching only works for the system role

**Answer:** B. The retrieved passages differ per query and come first, so no two prompts share a prefix — Matching is by exact prefix. Varying passages at the start change the very first tokens, so the stable instructions after them never match. Reorder to [instructions][passages][question] and put the breakpoint after the instructions.

**Question:** Writes cost 1.25× and reads 0.1× the base price. Requests arrive as a Poisson process at λ T = 0.2 per TTL window. Does caching save money?

- A. Yes, any hit saves money
- B. No: h = 1 − e^(−0.2) = 18.1%, below the 21.7% break-even
- C. Yes: h = 0.2 is above break-even
- D. It depends only on prompt length

**Answer:** B. No: h = 1 − e^(−0.2) = 18.1%, below the 21.7% break-even — $h=1-e^{-0.2}=0.181$ and $h^*=(1.25-1)/(1.25-0.1)=0.217$. Below break-even the 25% write premium on 82% of requests outweighs the 90% saving on the other 18%. A longer TTL or batching requests closer together fixes it.

**Key points**

- Reuses the prefix's KV cache, so prefill runs only on the new suffix; the answer is still generated fresh.
- Exact prefix match only: one changed early token invalidates everything after it. Stable content first.
- Cost $C(h)=S+P[h\rho_r+(1-h)\rho_w]$; break-even $h^*=(\rho_w-1)/(\rho_w-\rho_r)$, 21.7% for 1.25×/0.1×.
- With Poisson traffic $h=1-e^{-\lambda T}$: request rate times TTL decides whether caching pays.
- Self-hosted: vLLM hashes 16-token blocks, SGLang keeps a radix tree; hosted APIs expose breakpoints, TTLs and read/write counters.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
