# Semantic caching

> A semantic cache embeds each query and returns a stored answer when the nearest cached query has cosine similarity ≥ τ, skipping the LLM call; τ trades hit rate against wrong answers for similar-looking questions.

Source: https://www.mlfoundry.in/learn/semantic-caching  
Section: Generative AI › Serving Generative Models  
Level: core · 7 min read · updated 2026-10-21

**Semantic caching:** stores past (query embedding, response) pairs and, for a new query, returns the stored response of the most similar past query if their similarity exceeds a threshold $\tau$. Unlike an exact-match cache it also catches paraphrases. Unlike **Prompt caching** it skips the whole model call, so it can return an answer that does not fit the new question. GPTCache (Zilliz, 2023; described by Bang at the NLP-OSS workshop, 2023) is a widely used open-source implementation.

Support bots, internal help desks and search boxes see the same questions over and over in different words. "How do I reset my password?" and "I forgot my password, how can I change it?" need the same answer. An exact-match cache keyed on the text hash misses the second one. A semantic cache embeds both with an **Embedding models** encoder and compares vectors, so the second costs one embedding and one vector lookup instead of a full generation.

It sits in front of the model, usually in an LLM gateway: scope the request, try an exact-match lookup, embed, search a **Vector databases** index, then either return the cached answer or call the model and store the new pair.

> Figure: The lookup path. The cache key is scoped by tenant, model and prompt version before any similarity search; an exact-match hash catches verbatim repeats; otherwise the query is embedded and matched to its nearest cached query. With τ = 0.88 the paraphrase at cosine 0.93 is served from cache, while "reset my username" at 0.86 must stay below the threshold because its answer differs. Similarities here are illustrative.

## Decision rule and economics

$$
\text{hit}(q)\iff\max_{j\in\mathcal C_\text{scope}}\cos\!\big(e(q),e(q_j)\big)\ge\tau,\qquad \cos(a,b)=\frac{a\cdot b}{\lVert a\rVert\,\lVert b\rVert}
$$

- $e(\cdot)$: embedding model; vectors are usually L2-normalised so cosine is a dot product
- $\mathcal C_\text{scope}$: cached queries with the same scope key (tenant, model, prompt and index versions)
- $\tau$: similarity threshold, tuned per embedding model on labelled query pairs

$$
\mathbb E[C]=c_\text{emb}+(1-h)\,c_\text{LLM},\qquad \text{wrong-answer rate}=h\,(1-\text{precision})
$$

- $h$: hit rate: fraction of queries served from cache
- $c_\text{emb}$: embedding plus vector search, paid on every query, hits and misses alike
- $\text{precision}$: fraction of hits whose cached answer is correct for the new query

Both $h$ and the wrong-answer rate rise as $\tau$ falls. The threshold is a classifier decision on "same question or not", so tune it like one: label a few hundred real query pairs, plot precision and recall against $\tau$ and pick the point your product can tolerate. The ceiling on $h$ is the fraction of queries whose *intent* has been seen before; no threshold beats it.

What the cache key and policy must include (practice, not a standard)

| Concern | Why it matters | Typical handling |
| --- | --- | --- |
| Tenant / user | Answers can contain private or account-specific data | Scope by tenant; never share personalised answers across users |
| Model and prompt version | A new system prompt or model changes the right answer | Put versions in the scope key; flush on deploy |
| Retrieval index version | RAG answers go stale when documents change | Version the index or invalidate entries citing changed documents |
| Time-sensitive facts | Prices, stock, weather, news | Short TTL or exclude by route or classifier |
| Conversation context | "And for the US?" means different things in different chats | Cache only first turns, or embed a rewritten standalone query |
| Side effects | Tool calls that send email or move money must run each time | Never cache actions, only read-only answers |

> Figure: 400 synthetic support queries over 24 intents (20 phrasings each, 24-d embeddings), 12 of them in near-miss pairs with different answers. At the default τ = 0.88: 69.8% hits, 2.0% wrong answers (precision 97.1%), cost 30.4% of always calling the LLM. τ = 0.85 gives 81.0% hits but 6.0% wrong; τ = 0.95 falls back to the exact-match level of 41.5% with no errors. Merge the near-miss pairs into one answer and the wrong answers vanish at every τ: the risk comes from distinct intents that embed close together. The oracle ceiling is 94.0%. Example pairs at τ = 0.88: a password paraphrase (0.914) hits correctly, EU vs US refund policy (0.909) is a wrong hit, password vs username (0.870) misses.

**Practical facts**

- **Cosine is not calibrated:** Absolute similarity values differ between embedding models; with some models unrelated sentences score above 0.7. A τ tuned for one model is meaningless for another.
- **Near misses:** Questions that differ in one word (EU vs US, password vs username, cancel vs pause) often score as high as true paraphrases. These pairs drive most false hits.
- **Lookup cost:** One embedding call plus an approximate top-1 search (see **Approximate nearest neighbor search (HNSW, IVF, PQ)**): milliseconds, against hundreds of milliseconds to seconds for a generation.
- **GPTCache structure:** Separates the embedding function, the vector store, the store for answers, the eviction policy and a similarity evaluator, so each can be swapped.

**Tuning the threshold**

1. Sample a few thousand real queries and, for each, its nearest earlier query under your embedding model.
2. Label each pair: would the earlier answer be correct for the new query? Oversample pairs in the 0.8–0.95 band, where decisions flip.
3. Sweep τ and plot hit rate and false-hit rate (or precision and recall of "same answer").
4. Pick τ from the cost of a wrong answer, not from a default; re-run after changing the embedding model, the prompt or the traffic mix.
5. Monitor in production: sample served hits for review and track the user re-ask rate after a cache hit.

## Worked example: choosing τ

The simulation's numbers scaled to 100,000 queries/day; LLM call = 1 unit, embedding + lookup = 0.002 units on every query

| τ | Hits | Wrong answers/day | Cost (units) | vs 100,000 |
| --- | --- | --- | --- | --- |
| 0.85 | 324/400 = 81.0% | 24/400 → 6,000 | 19,000 + 200 = 19,200 | 19.2% |
| 0.88 | 279/400 = 69.75% | 8/400 → 2,000 | 30,250 + 200 = 30,450 | 30.45% |
| 0.90 | 237/400 = 59.25% | 1/400 → 250 | 40,750 + 200 = 40,950 | 40.95% |
| 0.95 | 166/400 = 41.5% | 0 | 58,500 + 200 = 58,700 | 58.7% |

Going from 0.90 to 0.85 saves another 21,750 units a day but serves 5,750 more wrong answers. For a support bot that is usually the wrong trade; for suggestions where a near answer is still useful it may be fine. At 0.95 the semantic cache does no better than an exact-match cache.

```python
import time
import numpy as np

class SemanticCache:
    def __init__(self, embed, tau=0.88, ttl_s=86_400):
        self.embed, self.tau, self.ttl = embed, tau, ttl_s
        self.vecs, self.entries = {}, {}       # scope -> (n, d) matrix / [(answer, t)]

    def lookup(self, scope, query):
        q = np.asarray(self.embed(query), dtype=np.float32)
        q /= np.linalg.norm(q)
        V = self.vecs.get(scope)
        if V is not None:
            sims = V @ q                        # cosine: rows are unit vectors
            j = int(np.argmax(sims))
            answer, t = self.entries[scope][j]
            if sims[j] >= self.tau and time.time() - t < self.ttl:
                return answer, float(sims[j]), q
        return None, None, q

    def store(self, scope, q, answer):
        V = self.vecs.get(scope)
        self.vecs[scope] = q[None] if V is None else np.vstack([V, q])
        self.entries.setdefault(scope, []).append((answer, time.time()))

def answer(cache, llm, scope, query):
    hit, sim, q = cache.lookup(scope, query)
    if hit is not None:
        return hit                              # no LLM call
    out = llm(query)
    cache.store(scope, q, out)
    return out

# scope = "tenant/model/prompt-version"; never mix tenants in one scope
```

A minimal scoped semantic cache (numpy, brute-force top-1; swap in an ANN index at scale). Ran with a toy embedding; plug in a real embedding model for use

> **Leaking answers across users:** A global cache that stores "Your balance is 1,240 euros" under "what's my balance?" will hand it to the next user who asks. Scope personalised routes by user or do not cache them. Even scoped by tenant, cached answers are a store of past prompts and outputs that falls under the same retention rules as logs (see **Privacy and data leakage**).

> **Two thresholds and a verifier:** Serve directly above a high threshold, call the LLM below a low one, and send the band in between to a cheap check: a cross-encoder (as in **Reranking**) or a small model asked "do these two questions have the same answer?". This keeps most of the hit rate of a low $\tau$ with the precision of a high one, at the cost of some extra latency on borderline queries.

**Strengths**

- Removes the whole generation for repeated intents: latency in milliseconds, cost of one embedding.
- Catches paraphrases that an exact-match cache misses.
- Sits outside the model; works with any provider.

**Limitations**

- False hits return a confident answer to a different question.
- Stale or personalised answers need scoping, TTLs and invalidation.
- Little benefit for long, unique or multi-turn prompts, where repeats are rare.

Three caches that are often stacked in one gateway

| Property | Exact-match response cache | Semantic cache | Prompt (prefix) caching |
| --- | --- | --- | --- |
| Key | hash of normalised request | query embedding, cos ≥ τ | exact token prefix |
| Skips | the whole call | the whole call | prefill of the shared prefix only |
| Can be wrong | only if stale | yes, on near-miss questions | no: output is regenerated |
| Hits on paraphrases | no | yes | not applicable |
| Typical saving per hit | ≈ 100% | ≈ 100% minus embedding cost | most of the prefix's input cost |

**Question:** In the viz, τ drops from 0.90 to 0.85. Hit rate rises from 59.3% to 81.0%. What else happens?

- A. Nothing; more hits are always better
- B. Wrong answers rise from 0.25% to 6.0% of all queries
- C. Embedding cost per query doubles
- D. The oracle ceiling rises

**Answer:** B. Wrong answers rise from 0.25% to 6.0% of all queries — Lower $\tau$ accepts less similar neighbours, including near-miss intents with different answers. The embedding cost is per query and does not depend on $\tau$; the oracle (intent seen before, 94.0%) depends only on the traffic.

**Question:** Which pair of requests is safest to serve from a shared semantic cache?

- A. "What's my order status?" from two different customers
- B. "What are your opening hours?" and "When are you open?" in the same tenant and prompt version
- C. "Send the report to Bob" twice
- D. "And what about the US?" in two different conversations

**Answer:** B. "What are your opening hours?" and "When are you open?" in the same tenant and prompt version — It is a read-only, non-personal, context-free question with one right answer. Order status is personal, sending a report is an action with side effects, and the follow-up depends on its conversation.

**Key points**

- Hit if the nearest cached query in scope has $\cos\ge\tau$; then return its stored answer and skip the LLM.
- Cost $c_\text{emb}+(1-h)c_\text{LLM}$; errors $h(1-\text{precision})$. Lowering $\tau$ raises both.
- Near-miss questions with different answers cause most false hits; tune τ per embedding model on labelled pairs.
- Scope keys by tenant, model, prompt and index version; add TTLs; never cache personalised answers or actions.
- Prompt caching is exact and always safe; semantic caching is approximate and can be wrong. Many systems use both.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
