Prompt caching

Prompt caching keeps the KV cache of a shared prompt prefix so later requests with byte-identical prefixes skip that prefill; a hit cuts the prefix's input cost (to 0.1× on some APIs) and its prefill time, but only exact prefixes match.

8 min read664 wordscore0 viewsupdated 2026-10-21

Definition

Prompt caching (prefix caching, context caching) stores the attention keys and values computed for the start of a prompt and reuses them when a later request begins with exactly the same tokens, so prefill runs only on the new suffix. The output is still generated fresh. Research systems include Prompt Cache (Gim et al., MLSys 2024), SGLang's RadixAttention (Zheng et al., 2024) and vLLM's automatic prefix caching, built on PagedAttention (Kwon et al., SOSP 2023). Hosted APIs added it in 2024 (Anthropic in August, OpenAI in October, Google as Gemini context caching).

Real prompts repeat themselves. An assistant sends the same 2,000-token system prompt and tool list on every turn. A document Q&A app sends the same 50-page contract with each question. An agent resends its whole transcript every step. Without caching the server recomputes the KV cache for all of it each time. That costs prefill compute and adds hundreds of milliseconds to seconds of time to first token.

The key fact is causal attention: the key and value vectors at position ii depend on tokens 1…i1\dots i and nothing after. So the cached state for a prefix is valid for any continuation of that prefix, and invalid the moment a single earlier token differs. Prompt caching is therefore prefix matching, not similarity matching. Reusing answers to similar questions is a different technique, Semantic caching.

breakpoints after the system prompt and the document · 5-min TTL · write 1.25×, read 0.1×segment widths not to scale; costs in units of the base input-token priceR1 0:00tools+system 2,048doc A 8,192Q1 150write 10,240 · input 150 → 1.25 × 10,240 + 150 = 12,950 (vs 10,390)R2 0:03tools+system 2,048doc A 8,192Q2 180read 10,240 · input 180 → 0.1 × 10,240 + 180 = 1,204 (vs 10,420)R3 0:05tools+system 2,048doc B 8,192Q3 160read 2,048 · write 8,192 · input 160 → 10,604.8 (vs 10,400)R4 0:09system′ 2,048doc A 8,192Q4 150a timestamp at token 12 changed → no prefix match, all 10,240 rewritten▼ =cache_controlprefilled and writtenKV read from cacheafter last breakpoint: full priceK, V at position i depend on tokens 1…i, so only an identical prefix can be reused.vLLM: each 16-token block is keyed by hash(parent block key, its token ids).SGLang RadixAttention: cached prefixes form a radix tree, evicted LRU.Order: tools → system → messages; stable content first, volatile content last.
fig.Four requests with cache breakpoints after a 2,048-token system prompt and an 8,192-token document (5-minute TTL, writes 1.25×, reads 0.1×). R1 writes both entries: 12,950 units instead of 10,390. R2, three minutes later, reads all 10,240 cached tokens: 1,204 instead of 10,420. R3 shares only the system prompt and writes doc B: 10,604.8, slightly more than uncached. R4 changed a timestamp near the top of the system prompt, so nothing matches.

Cost, hit rate and break-even

C(h)=S+P [h ρr+(1−h) ρw],h∗=ρw−1ρw−ρrC(h)=S+P\,\big[h\,\rho_r+(1-h)\,\rho_w\big],\qquad h^{*}=\frac{\rho_w-1}{\rho_w-\rho_r}
P, SP,\ S
cached prefix tokens and uncached suffix tokens per request
hh
fraction of requests that find the prefix in cache
ρr, ρw\rho_r,\ \rho_w
read and write price as multiples of the base input price
h∗h^{*}
break-even hit rate: caching beats no caching (C=P+SC=P+S) when h>h∗h>h^{*}

With ρw=1.25\rho_w=1.25 and ρr=0.1\rho_r=0.1, h∗=0.25/1.15=21.7%h^*=0.25/1.15=21.7\%. With a 1-hour entry at ρw=2\rho_w=2, h∗=1/1.9=52.6%h^*=1/1.9=52.6\%. With automatic caching that charges no write premium, h∗=0h^*=0: caching can only save.

h=1−e−λT(Poisson arrivals at rate λ, TTL T, each hit refreshes the entry)h=1-e^{-\lambda T}\qquad\text{(Poisson arrivals at rate }\lambda\text{, TTL }T\text{, each hit refreshes the entry)}
λT\lambda T
expected requests per TTL window; λT=1\lambda T=1 gives h=63.2%h=63.2\%

A request hits when the previous request with the same prefix started less than TT ago. The gap between Poisson arrivals is exponential, so the hit probability is 1−e−λT1-e^{-\lambda T}. Traffic, not the prompt, decides whether caching pays. Latency follows the same logic: a hit removes roughly P/RprefillP/R_\text{prefill} from TTFT, which is why caching and Streaming generation together make long-context apps feel responsive.

How the main implementations behave (as documented in 2024–2025; check current docs)

Anthropic Messages
Explicit "cache_control": {"type": "ephemeral"} breakpoints on content blocks, at most 4 per request, or one top-level automatic breakpoint. Prefix order is tools → system → messages. Default TTL 5 minutes, refreshed free on every hit; optional "ttl": "1h". Writes cost 1.25× (5 min) or 2× (1 h) the base input price, reads about 0.1×. Usage reports cache_creation_input_tokens and cache_read_input_tokens. Prefixes below a model-dependent minimum (hundreds to a few thousand tokens) are not cached.
OpenAI
Automatic, no code change, for prompts of at least 1,024 tokens, matched in 128-token increments; cached tokens appear in usage.prompt_tokens_details.cached_tokens and were billed at 50% off at the October 2024 launch. Entries typically expire after 5–10 minutes of inactivity.
Gemini context caching
Explicit cache objects with a chosen TTL, billed for storage per token-hour on top of discounted reads.
vLLM
Automatic prefix caching (--enable-prefix-caching): each full 16-token KV block is keyed by a hash of its parent block's key and its own token ids, so shared prefixes share physical blocks; unused blocks are evicted LRU.
SGLang
RadixAttention keeps cached prefixes in a radix tree over token sequences, with LRU eviction and a cache-aware scheduler that runs requests sharing long prefixes together.
12 req/h · prefix 10,000 + suffix 200 · explicit, 5-min TTL (write 1.25×, read 0.1×)first 60 requests over 5.1 h: 36 hits, 24 misses (prefix written)expected input cost / no caching vs request rate■ 5-min TTL ■ 1-h TTL ■ automatic00.511101001000requests per hour (log)hit rate (simulated)61.5%1 − e^(−λ·TTL)63.2%break-even hit rate21.7%input cost vs none55.2%prefill miss / hit2.04 / 0.04 smean prefill time0.81 stotal input cost of 200 requests (units of base price × 10⁶)no cache2.04cached1.13■ suffix 0.04 ■ prefix writes 0.96 ■ prefix reads 0.12
pricing
fig.200 Poisson requests sharing a 10,000-token prefix with a 200-token suffix. Defaults (12 requests/h, 5-minute TTL, writes 1.25×, reads 0.1×): 61.5% simulated hits against 1 − e^(−1) = 63.2% analytic, input cost 55.2% of no caching, mean prefill 0.81 s instead of 2.04 s (0.04 s on a hit). Drop to 2/h and the 5-minute cache costs about 109% of no caching; the 1-hour TTL at the same rate hits 86.5% and costs 36.9%. The automatic scheme (no write premium, 50% reads) at 12/h costs 69.9%.

Designing prompts for cache hits

  1. 1Order the prompt from most to least stable: tool definitions, system instructions, few-shot examples, long documents, conversation history, then the new user message.
  2. 2Make the stable part byte-identical: no timestamps or request ids, deterministic tool order, sorted JSON keys, fixed whitespace.
  3. 3Place a breakpoint at the end of each stable section (explicit APIs), or keep the shared prefix above the provider's minimum length (automatic APIs).
  4. 4In multi-turn chats, move the last breakpoint forward to the newest turn so each request reuses the whole history.
  5. 5Log the cached-token counter per route and alert when it drops; a drop usually means someone edited the top of the prompt.

Worked example

CaseHit rate hComputationCostvs 10,200
No caching—10,000 + 20010,200100%
Single miss (write)01.25 × 10,000 + 20012,700124.5%
Single hit (read)10.1 × 10,000 + 2001,20011.8%
5-min TTL, 12 req/h1 − e^(−1) = 0.632200 + 10,000 × (0.632 × 0.1 + 0.368 × 1.25)5,43153.2%
5-min TTL, 2 req/h1 − e^(−1/6) = 0.154200 + 10,000 × (0.154 × 0.1 + 0.846 × 1.25)10,935107.2%
1-h TTL, 2 req/h1 − e^(−2) = 0.865200 + 10,000 × (0.865 × 0.1 + 0.135 × 2)3,77137.0%
P = 10,000 cached tokens, S = 200 uncached; costs in units of the base input-token price per request

At 2 requests per hour the 5-minute cache loses money: 15.4% hits is below the 21.7% break-even. The 1-hour TTL wins at that rate because λT\lambda T rises from 1/6 to 2. If the requests arrive more often than every 5 minutes anyway, the 1-hour TTL only adds the higher write price.

python
1import anthropic
2
3client = anthropic.Anthropic()
4MANUAL = open("product_manual.md").read() # ~10k tokens, byte-identical every call
5
6def ask(question: str):
7 resp = client.messages.create(
8 model="claude-opus-5-5",
9 max_tokens=1024,
10 system=[
11 {"type": "text", "text": "Answer only from the manual below."},
12 {"type": "text", "text": MANUAL,
13 "cache_control": {"type": "ephemeral"}}, # breakpoint: end of shared prefix
14 ],
15 messages=[{"role": "user", "content": question}], # varying part goes last
16 )
17 u = resp.usage
18 print(f"write {u.cache_creation_input_tokens} read {u.cache_read_input_tokens} "
19 f"uncached {u.input_tokens}")
20 return resp
21
22ask("How do I reset the device?") # first call: write ≈ 10k, read 0
23ask("What does error E42 mean?") # within 5 min: write 0, read ≈ 10k
Cache a large, unchanging document with the Anthropic Python SDK and check the usage counters. Needs an API key; not run here

Where it's used

Long system prompts and tools

Agents and assistants resend thousands of tokens of instructions and tool schemas every turn; one breakpoint after them is the cheapest win.

Multi-turn conversations

Each turn extends the previous prompt, so moving the breakpoint to the latest turn reuses the whole history.

Document Q&A

Many questions over one contract, codebase or manual; the document is the prefix, the question is the suffix.

Few-shot and batch jobs

Thousands of requests sharing the same instructions and examples, differing only in the item to classify.

Check yourself

A RAG app builds prompts as [retrieved passages][system instructions][question]. Cache reads stay at zero. Why?

Check yourself

Writes cost 1.25× and reads 0.1× the base price. Requests arrive as a Poisson process at λ T = 0.2 per TTL window. Does caching save money?

Remember

  • ✓Reuses the prefix's KV cache, so prefill runs only on the new suffix; the answer is still generated fresh.
  • ✓Exact prefix match only: one changed early token invalidates everything after it. Stable content first.
  • ✓Cost C(h)=S+P[hρr+(1−h)ρw]C(h)=S+P[h\rho_r+(1-h)\rho_w]; break-even h∗=(ρw−1)/(ρw−ρr)h^*=(\rho_w-1)/(\rho_w-\rho_r), 21.7% for 1.25×/0.1×.
  • ✓With Poisson traffic h=1−e−λTh=1-e^{-\lambda T}: request rate times TTL decides whether caching pays.
  • ✓Self-hosted: vLLM hashes 16-token blocks, SGLang keeps a radix tree; hosted APIs expose breakpoints, TTLs and read/write counters.
← back to Generative AI roadmap