Definition
Prompt caching (prefix caching, context caching) stores the attention keys and values computed for the start of a prompt and reuses them when a later request begins with exactly the same tokens, so prefill runs only on the new suffix. The output is still generated fresh. Research systems include Prompt Cache (Gim et al., MLSys 2024), SGLang's RadixAttention (Zheng et al., 2024) and vLLM's automatic prefix caching, built on PagedAttention (Kwon et al., SOSP 2023). Hosted APIs added it in 2024 (Anthropic in August, OpenAI in October, Google as Gemini context caching).
Real prompts repeat themselves. An assistant sends the same 2,000-token system prompt and tool list on every turn. A document Q&A app sends the same 50-page contract with each question. An agent resends its whole transcript every step. Without caching the server recomputes the KV cache for all of it each time. That costs prefill compute and adds hundreds of milliseconds to seconds of time to first token.
The key fact is causal attention: the key and value vectors at position depend on tokens and nothing after. So the cached state for a prefix is valid for any continuation of that prefix, and invalid the moment a single earlier token differs. Prompt caching is therefore prefix matching, not similarity matching. Reusing answers to similar questions is a different technique, Semantic caching.
Cost, hit rate and break-even
- cached prefix tokens and uncached suffix tokens per request
- fraction of requests that find the prefix in cache
- read and write price as multiples of the base input price
- break-even hit rate: caching beats no caching () when
With and , . With a 1-hour entry at , . With automatic caching that charges no write premium, : caching can only save.
- expected requests per TTL window; gives
A request hits when the previous request with the same prefix started less than ago. The gap between Poisson arrivals is exponential, so the hit probability is . Traffic, not the prompt, decides whether caching pays. Latency follows the same logic: a hit removes roughly from TTFT, which is why caching and Streaming generation together make long-context apps feel responsive.
How the main implementations behave (as documented in 2024–2025; check current docs)
- Anthropic Messages
- Explicit
"cache_control": {"type": "ephemeral"}breakpoints on content blocks, at most 4 per request, or one top-level automatic breakpoint. Prefix order is tools → system → messages. Default TTL 5 minutes, refreshed free on every hit; optional"ttl": "1h". Writes cost 1.25× (5 min) or 2× (1 h) the base input price, reads about 0.1×. Usage reportscache_creation_input_tokensandcache_read_input_tokens. Prefixes below a model-dependent minimum (hundreds to a few thousand tokens) are not cached. - OpenAI
- Automatic, no code change, for prompts of at least 1,024 tokens, matched in 128-token increments; cached tokens appear in
usage.prompt_tokens_details.cached_tokensand were billed at 50% off at the October 2024 launch. Entries typically expire after 5–10 minutes of inactivity. - Gemini context caching
- Explicit cache objects with a chosen TTL, billed for storage per token-hour on top of discounted reads.
- vLLM
- Automatic prefix caching (
--enable-prefix-caching): each full 16-token KV block is keyed by a hash of its parent block's key and its own token ids, so shared prefixes share physical blocks; unused blocks are evicted LRU. - SGLang
- RadixAttention keeps cached prefixes in a radix tree over token sequences, with LRU eviction and a cache-aware scheduler that runs requests sharing long prefixes together.
Designing prompts for cache hits
- 1Order the prompt from most to least stable: tool definitions, system instructions, few-shot examples, long documents, conversation history, then the new user message.
- 2Make the stable part byte-identical: no timestamps or request ids, deterministic tool order, sorted JSON keys, fixed whitespace.
- 3Place a breakpoint at the end of each stable section (explicit APIs), or keep the shared prefix above the provider's minimum length (automatic APIs).
- 4In multi-turn chats, move the last breakpoint forward to the newest turn so each request reuses the whole history.
- 5Log the cached-token counter per route and alert when it drops; a drop usually means someone edited the top of the prompt.
Worked example
| Case | Hit rate h | Computation | Cost | vs 10,200 |
|---|---|---|---|---|
| No caching | — | 10,000 + 200 | 10,200 | 100% |
| Single miss (write) | 0 | 1.25 × 10,000 + 200 | 12,700 | 124.5% |
| Single hit (read) | 1 | 0.1 × 10,000 + 200 | 1,200 | 11.8% |
| 5-min TTL, 12 req/h | 1 − e^(−1) = 0.632 | 200 + 10,000 × (0.632 × 0.1 + 0.368 × 1.25) | 5,431 | 53.2% |
| 5-min TTL, 2 req/h | 1 − e^(−1/6) = 0.154 | 200 + 10,000 × (0.154 × 0.1 + 0.846 × 1.25) | 10,935 | 107.2% |
| 1-h TTL, 2 req/h | 1 − e^(−2) = 0.865 | 200 + 10,000 × (0.865 × 0.1 + 0.135 × 2) | 3,771 | 37.0% |
At 2 requests per hour the 5-minute cache loses money: 15.4% hits is below the 21.7% break-even. The 1-hour TTL wins at that rate because rises from 1/6 to 2. If the requests arrive more often than every 5 minutes anyway, the 1-hour TTL only adds the higher write price.
1import anthropic23client = anthropic.Anthropic()4MANUAL = open("product_manual.md").read() # ~10k tokens, byte-identical every call56def ask(question: str):7 resp = client.messages.create(8 model="claude-opus-5-5",9 max_tokens=1024,10 system=[11 {"type": "text", "text": "Answer only from the manual below."},12 {"type": "text", "text": MANUAL,13 "cache_control": {"type": "ephemeral"}}, # breakpoint: end of shared prefix14 ],15 messages=[{"role": "user", "content": question}], # varying part goes last16 )17 u = resp.usage18 print(f"write {u.cache_creation_input_tokens} read {u.cache_read_input_tokens} "19 f"uncached {u.input_tokens}")20 return resp2122ask("How do I reset the device?") # first call: write ≈ 10k, read 023ask("What does error E42 mean?") # within 5 min: write 0, read ≈ 10kWhere it's used
Long system prompts and tools
Agents and assistants resend thousands of tokens of instructions and tool schemas every turn; one breakpoint after them is the cheapest win.
Multi-turn conversations
Each turn extends the previous prompt, so moving the breakpoint to the latest turn reuses the whole history.
Document Q&A
Many questions over one contract, codebase or manual; the document is the prefix, the question is the suffix.
Few-shot and batch jobs
Thousands of requests sharing the same instructions and examples, differing only in the item to classify.
Check yourself
A RAG app builds prompts as [retrieved passages][system instructions][question]. Cache reads stay at zero. Why?
Check yourself
Writes cost 1.25× and reads 0.1× the base price. Requests arrive as a Poisson process at λ T = 0.2 per TTL window. Does caching save money?
Remember
- ✓Reuses the prefix's KV cache, so prefill runs only on the new suffix; the answer is still generated fresh.
- ✓Exact prefix match only: one changed early token invalidates everything after it. Stable content first.
- ✓Cost ; break-even , 21.7% for 1.25×/0.1×.
- ✓With Poisson traffic : request rate times TTL decides whether caching pays.
- ✓Self-hosted: vLLM hashes 16-token blocks, SGLang keeps a radix tree; hosted APIs expose breakpoints, TTLs and read/write counters.