Semantic caching

A semantic cache embeds each query and returns a stored answer when the nearest cached query has cosine similarity ≥ τ, skipping the LLM call; τ trades hit rate against wrong answers for similar-looking questions.

7 min read619 wordscore0 viewsupdated 2026-10-21

Definition

Semantic caching stores past (query embedding, response) pairs and, for a new query, returns the stored response of the most similar past query if their similarity exceeds a threshold τ\tau. Unlike an exact-match cache it also catches paraphrases. Unlike Prompt caching it skips the whole model call, so it can return an answer that does not fit the new question. GPTCache (Zilliz, 2023; described by Bang at the NLP-OSS workshop, 2023) is a widely used open-source implementation.

Support bots, internal help desks and search boxes see the same questions over and over in different words. "How do I reset my password?" and "I forgot my password, how can I change it?" need the same answer. An exact-match cache keyed on the text hash misses the second one. A semantic cache embeds both with an Embedding models encoder and compares vectors, so the second costs one embedding and one vector lookup instead of a full generation.

It sits in front of the model, usually in an LLM gateway: scope the request, try an exact-match lookup, embed, search a Vector databases index, then either return the cached answer or call the model and store the new pair.

semantic cache in front of an LLM (illustrative similarities, τ = 0.88)query“How can I reset my password?”1. scope keytenant · model · prompt version2. exact-match cache (hash)normalised text never seen → miss3. embed the queryunit vector q, same model as the index4. ANN search, top-1 in scopenearest cached query: cos 0.93cos ≥ τ = 0.88 ?yes: return cached answerno LLM call, millisecondsno: call the LLMstore entry, return answernearest cached queries (cosine)I forgot my password, how do I change it?0.93How do I reset my username?0.86How do I change my email address?0.71τ0.86 is close, but a username reset is adifferent answer: τ must sit above it.cache entry (vector index + key-value store){"scope": "acme/model-a/prompt-v3", "query": "I forgot my password, how do…", "vector": [0.021, -0.113, …], "answer": "Open Settings → Security → …", "created": "…", "ttl_s": 86400}invalidate on TTL, documentor prompt change; never sharepersonal answers across users
fig.The lookup path. The cache key is scoped by tenant, model and prompt version before any similarity search; an exact-match hash catches verbatim repeats; otherwise the query is embedded and matched to its nearest cached query. With τ = 0.88 the paraphrase at cosine 0.93 is served from cache, while "reset my username" at 0.86 must stay below the threshold because its answer differs. Similarities here are illustrative.

Decision rule and economics

hit(q)  ⟺  max⁡j∈Cscopecos⁡ ⁣(e(q),e(qj))≥τ,cos⁡(a,b)=a⋅b∥a∥ ∥b∥\text{hit}(q)\iff\max_{j\in\mathcal C_\text{scope}}\cos\!\big(e(q),e(q_j)\big)\ge\tau,\qquad \cos(a,b)=\frac{a\cdot b}{\lVert a\rVert\,\lVert b\rVert}
e(⋅)e(\cdot)
embedding model; vectors are usually L2-normalised so cosine is a dot product
Cscope\mathcal C_\text{scope}
cached queries with the same scope key (tenant, model, prompt and index versions)
τ\tau
similarity threshold, tuned per embedding model on labelled query pairs
E[C]=cemb+(1−h) cLLM,wrong-answer rate=h (1−precision)\mathbb E[C]=c_\text{emb}+(1-h)\,c_\text{LLM},\qquad \text{wrong-answer rate}=h\,(1-\text{precision})
hh
hit rate: fraction of queries served from cache
cembc_\text{emb}
embedding plus vector search, paid on every query, hits and misses alike
precision\text{precision}
fraction of hits whose cached answer is correct for the new query

Both hh and the wrong-answer rate rise as τ\tau falls. The threshold is a classifier decision on "same question or not", so tune it like one: label a few hundred real query pairs, plot precision and recall against τ\tau and pick the point your product can tolerate. The ceiling on hh is the fraction of queries whose intent has been seen before; no threshold beats it.

ConcernWhy it mattersTypical handling
Tenant / userAnswers can contain private or account-specific dataScope by tenant; never share personalised answers across users
Model and prompt versionA new system prompt or model changes the right answerPut versions in the scope key; flush on deploy
Retrieval index versionRAG answers go stale when documents changeVersion the index or invalidate entries citing changed documents
Time-sensitive factsPrices, stock, weather, newsShort TTL or exclude by route or classifier
Conversation context"And for the US?" means different things in different chatsCache only first turns, or embed a rewritten standalone query
Side effectsTool calls that send email or move money must run each timeNever cache actions, only read-only answers
What the cache key and policy must include (practice, not a standard)
τ = 0.88 · 400 queries · 24 intents, 12 in near-miss pairs · synthetic 24-d embeddingscosine to nearest cached query; shaded = hit≤0.50.60.70.80.91.0■ same intent ■ different intentshare of all queries vs τ00.510.70.80.91.0hits wrong oracle exact-matchhit rate69.8%wrong answers2.0%hit precision97.1%exact-match cache41.5%cost vs no cache30.4%incoming query ↔ cached querycosHow do I reset my password? ↔ I forgot my password, how can I change it?0.914hit ✓Reset password please ↔ How do I reset my username?0.870missWhat is your refund policy in the EU? ↔ What is your refund policy in the US?0.909WRONG hitHow do I cancel my subscription? ↔ cancel my plan0.950hit ✓oracle = intent already seen (94.0%), the ceiling for any cache
near-miss intents
fig.400 synthetic support queries over 24 intents (20 phrasings each, 24-d embeddings), 12 of them in near-miss pairs with different answers. At the default τ = 0.88: 69.8% hits, 2.0% wrong answers (precision 97.1%), cost 30.4% of always calling the LLM. τ = 0.85 gives 81.0% hits but 6.0% wrong; τ = 0.95 falls back to the exact-match level of 41.5% with no errors. Merge the near-miss pairs into one answer and the wrong answers vanish at every τ: the risk comes from distinct intents that embed close together. The oracle ceiling is 94.0%. Example pairs at τ = 0.88: a password paraphrase (0.914) hits correctly, EU vs US refund policy (0.909) is a wrong hit, password vs username (0.870) misses.

Practical facts

Cosine is not calibrated
Absolute similarity values differ between embedding models; with some models unrelated sentences score above 0.7. A τ tuned for one model is meaningless for another.
Near misses
Questions that differ in one word (EU vs US, password vs username, cancel vs pause) often score as high as true paraphrases. These pairs drive most false hits.
Lookup cost
One embedding call plus an approximate top-1 search (see Approximate nearest neighbor search (HNSW, IVF, PQ)): milliseconds, against hundreds of milliseconds to seconds for a generation.
GPTCache structure
Separates the embedding function, the vector store, the store for answers, the eviction policy and a similarity evaluator, so each can be swapped.

Tuning the threshold

  1. 1Sample a few thousand real queries and, for each, its nearest earlier query under your embedding model.
  2. 2Label each pair: would the earlier answer be correct for the new query? Oversample pairs in the 0.8–0.95 band, where decisions flip.
  3. 3Sweep τ and plot hit rate and false-hit rate (or precision and recall of "same answer").
  4. 4Pick τ from the cost of a wrong answer, not from a default; re-run after changing the embedding model, the prompt or the traffic mix.
  5. 5Monitor in production: sample served hits for review and track the user re-ask rate after a cache hit.

Worked example: choosing τ

τHitsWrong answers/dayCost (units)vs 100,000
0.85324/400 = 81.0%24/400 → 6,00019,000 + 200 = 19,20019.2%
0.88279/400 = 69.75%8/400 → 2,00030,250 + 200 = 30,45030.45%
0.90237/400 = 59.25%1/400 → 25040,750 + 200 = 40,95040.95%
0.95166/400 = 41.5%058,500 + 200 = 58,70058.7%
The simulation's numbers scaled to 100,000 queries/day; LLM call = 1 unit, embedding + lookup = 0.002 units on every query

Going from 0.90 to 0.85 saves another 21,750 units a day but serves 5,750 more wrong answers. For a support bot that is usually the wrong trade; for suggestions where a near answer is still useful it may be fine. At 0.95 the semantic cache does no better than an exact-match cache.

python
1import time
2import numpy as np
3
4class SemanticCache:
5 def __init__(self, embed, tau=0.88, ttl_s=86_400):
6 self.embed, self.tau, self.ttl = embed, tau, ttl_s
7 self.vecs, self.entries = {}, {} # scope -> (n, d) matrix / [(answer, t)]
8
9 def lookup(self, scope, query):
10 q = np.asarray(self.embed(query), dtype=np.float32)
11 q /= np.linalg.norm(q)
12 V = self.vecs.get(scope)
13 if V is not None:
14 sims = V @ q # cosine: rows are unit vectors
15 j = int(np.argmax(sims))
16 answer, t = self.entries[scope][j]
17 if sims[j] >= self.tau and time.time() - t < self.ttl:
18 return answer, float(sims[j]), q
19 return None, None, q
20
21 def store(self, scope, q, answer):
22 V = self.vecs.get(scope)
23 self.vecs[scope] = q[None] if V is None else np.vstack([V, q])
24 self.entries.setdefault(scope, []).append((answer, time.time()))
25
26def answer(cache, llm, scope, query):
27 hit, sim, q = cache.lookup(scope, query)
28 if hit is not None:
29 return hit # no LLM call
30 out = llm(query)
31 cache.store(scope, q, out)
32 return out
33
34# scope = "tenant/model/prompt-version"; never mix tenants in one scope
A minimal scoped semantic cache (numpy, brute-force top-1; swap in an ANN index at scale). Ran with a toy embedding; plug in a real embedding model for use

Use when

  • +Removes the whole generation for repeated intents: latency in milliseconds, cost of one embedding.
  • +Catches paraphrases that an exact-match cache misses.
  • +Sits outside the model; works with any provider.

Avoid when

  • −False hits return a confident answer to a different question.
  • −Stale or personalised answers need scoping, TTLs and invalidation.
  • −Little benefit for long, unique or multi-turn prompts, where repeats are rare.
PropertyExact-match response cacheSemantic cachePrompt (prefix) caching
Keyhash of normalised requestquery embedding, cos ≥ τexact token prefix
Skipsthe whole callthe whole callprefill of the shared prefix only
Can be wrongonly if staleyes, on near-miss questionsno: output is regenerated
Hits on paraphrasesnoyesnot applicable
Typical saving per hit≈ 100%≈ 100% minus embedding costmost of the prefix's input cost
Three caches that are often stacked in one gateway

Check yourself

In the viz, τ drops from 0.90 to 0.85. Hit rate rises from 59.3% to 81.0%. What else happens?

Check yourself

Which pair of requests is safest to serve from a shared semantic cache?

Remember

  • ✓Hit if the nearest cached query in scope has cos⁡≥τ\cos\ge\tau; then return its stored answer and skip the LLM.
  • ✓Cost cemb+(1−h)cLLMc_\text{emb}+(1-h)c_\text{LLM}; errors h(1−precision)h(1-\text{precision}). Lowering τ\tau raises both.
  • ✓Near-miss questions with different answers cause most false hits; tune τ per embedding model on labelled pairs.
  • ✓Scope keys by tenant, model, prompt and index version; add TTLs; never cache personalised answers or actions.
  • ✓Prompt caching is exact and always safe; semantic caching is approximate and can be wrong. Many systems use both.
← back to Generative AI roadmap