Definition
Semantic caching stores past (query embedding, response) pairs and, for a new query, returns the stored response of the most similar past query if their similarity exceeds a threshold . Unlike an exact-match cache it also catches paraphrases. Unlike Prompt caching it skips the whole model call, so it can return an answer that does not fit the new question. GPTCache (Zilliz, 2023; described by Bang at the NLP-OSS workshop, 2023) is a widely used open-source implementation.
Support bots, internal help desks and search boxes see the same questions over and over in different words. "How do I reset my password?" and "I forgot my password, how can I change it?" need the same answer. An exact-match cache keyed on the text hash misses the second one. A semantic cache embeds both with an Embedding models encoder and compares vectors, so the second costs one embedding and one vector lookup instead of a full generation.
It sits in front of the model, usually in an LLM gateway: scope the request, try an exact-match lookup, embed, search a Vector databases index, then either return the cached answer or call the model and store the new pair.
Decision rule and economics
- embedding model; vectors are usually L2-normalised so cosine is a dot product
- cached queries with the same scope key (tenant, model, prompt and index versions)
- similarity threshold, tuned per embedding model on labelled query pairs
- hit rate: fraction of queries served from cache
- embedding plus vector search, paid on every query, hits and misses alike
- fraction of hits whose cached answer is correct for the new query
Both and the wrong-answer rate rise as falls. The threshold is a classifier decision on "same question or not", so tune it like one: label a few hundred real query pairs, plot precision and recall against and pick the point your product can tolerate. The ceiling on is the fraction of queries whose intent has been seen before; no threshold beats it.
| Concern | Why it matters | Typical handling |
|---|---|---|
| Tenant / user | Answers can contain private or account-specific data | Scope by tenant; never share personalised answers across users |
| Model and prompt version | A new system prompt or model changes the right answer | Put versions in the scope key; flush on deploy |
| Retrieval index version | RAG answers go stale when documents change | Version the index or invalidate entries citing changed documents |
| Time-sensitive facts | Prices, stock, weather, news | Short TTL or exclude by route or classifier |
| Conversation context | "And for the US?" means different things in different chats | Cache only first turns, or embed a rewritten standalone query |
| Side effects | Tool calls that send email or move money must run each time | Never cache actions, only read-only answers |
Practical facts
- Cosine is not calibrated
- Absolute similarity values differ between embedding models; with some models unrelated sentences score above 0.7. A τ tuned for one model is meaningless for another.
- Near misses
- Questions that differ in one word (EU vs US, password vs username, cancel vs pause) often score as high as true paraphrases. These pairs drive most false hits.
- Lookup cost
- One embedding call plus an approximate top-1 search (see Approximate nearest neighbor search (HNSW, IVF, PQ)): milliseconds, against hundreds of milliseconds to seconds for a generation.
- GPTCache structure
- Separates the embedding function, the vector store, the store for answers, the eviction policy and a similarity evaluator, so each can be swapped.
Tuning the threshold
- 1Sample a few thousand real queries and, for each, its nearest earlier query under your embedding model.
- 2Label each pair: would the earlier answer be correct for the new query? Oversample pairs in the 0.8–0.95 band, where decisions flip.
- 3Sweep τ and plot hit rate and false-hit rate (or precision and recall of "same answer").
- 4Pick τ from the cost of a wrong answer, not from a default; re-run after changing the embedding model, the prompt or the traffic mix.
- 5Monitor in production: sample served hits for review and track the user re-ask rate after a cache hit.
Worked example: choosing τ
| τ | Hits | Wrong answers/day | Cost (units) | vs 100,000 |
|---|---|---|---|---|
| 0.85 | 324/400 = 81.0% | 24/400 → 6,000 | 19,000 + 200 = 19,200 | 19.2% |
| 0.88 | 279/400 = 69.75% | 8/400 → 2,000 | 30,250 + 200 = 30,450 | 30.45% |
| 0.90 | 237/400 = 59.25% | 1/400 → 250 | 40,750 + 200 = 40,950 | 40.95% |
| 0.95 | 166/400 = 41.5% | 0 | 58,500 + 200 = 58,700 | 58.7% |
Going from 0.90 to 0.85 saves another 21,750 units a day but serves 5,750 more wrong answers. For a support bot that is usually the wrong trade; for suggestions where a near answer is still useful it may be fine. At 0.95 the semantic cache does no better than an exact-match cache.
1import time2import numpy as np34class SemanticCache:5 def __init__(self, embed, tau=0.88, ttl_s=86_400):6 self.embed, self.tau, self.ttl = embed, tau, ttl_s7 self.vecs, self.entries = {}, {} # scope -> (n, d) matrix / [(answer, t)]89 def lookup(self, scope, query):10 q = np.asarray(self.embed(query), dtype=np.float32)11 q /= np.linalg.norm(q)12 V = self.vecs.get(scope)13 if V is not None:14 sims = V @ q # cosine: rows are unit vectors15 j = int(np.argmax(sims))16 answer, t = self.entries[scope][j]17 if sims[j] >= self.tau and time.time() - t < self.ttl:18 return answer, float(sims[j]), q19 return None, None, q2021 def store(self, scope, q, answer):22 V = self.vecs.get(scope)23 self.vecs[scope] = q[None] if V is None else np.vstack([V, q])24 self.entries.setdefault(scope, []).append((answer, time.time()))2526def answer(cache, llm, scope, query):27 hit, sim, q = cache.lookup(scope, query)28 if hit is not None:29 return hit # no LLM call30 out = llm(query)31 cache.store(scope, q, out)32 return out3334# scope = "tenant/model/prompt-version"; never mix tenants in one scopeUse when
- +Removes the whole generation for repeated intents: latency in milliseconds, cost of one embedding.
- +Catches paraphrases that an exact-match cache misses.
- +Sits outside the model; works with any provider.
Avoid when
- −False hits return a confident answer to a different question.
- −Stale or personalised answers need scoping, TTLs and invalidation.
- −Little benefit for long, unique or multi-turn prompts, where repeats are rare.
| Property | Exact-match response cache | Semantic cache | Prompt (prefix) caching |
|---|---|---|---|
| Key | hash of normalised request | query embedding, cos ≥ τ | exact token prefix |
| Skips | the whole call | the whole call | prefill of the shared prefix only |
| Can be wrong | only if stale | yes, on near-miss questions | no: output is regenerated |
| Hits on paraphrases | no | yes | not applicable |
| Typical saving per hit | ≈ 100% | ≈ 100% minus embedding cost | most of the prefix's input cost |
Check yourself
In the viz, τ drops from 0.90 to 0.85. Hit rate rises from 59.3% to 81.0%. What else happens?
Check yourself
Which pair of requests is safest to serve from a shared semantic cache?
Remember
- ✓Hit if the nearest cached query in scope has ; then return its stored answer and skip the LLM.
- ✓Cost ; errors . Lowering raises both.
- ✓Near-miss questions with different answers cause most false hits; tune τ per embedding model on labelled pairs.
- ✓Scope keys by tenant, model, prompt and index version; add TTLs; never cache personalised answers or actions.
- ✓Prompt caching is exact and always safe; semantic caching is approximate and can be wrong. Many systems use both.