Semantic Caching for LLM Applications: When It Saves Money and When It Breaks Quality

A semantic cache promises a simple trade: skip a full model call when a new prompt closely resembles one you've already answered, and return the earlier response instead. For a support bot fielding the same handful of questions all day, or a RAG assistant answering the same lookup a hundred different ways, that trade can cut latency and spend hard. It can also ship a wrong answer with total confidence, because the cache never asked the model anything — it asked an embedding index whether two strings sound alike. This guide is for engineers deciding whether to put a semantic cache in front of LLM calls: how to size the similarity threshold, what a cache hit needs to verify before it can be served, and how to make a bad cache decision visible in your traces instead of invisible inside a hit-rate percentage.

At a glance

This guide is for engineers operating an LLM-backed chatbot, RAG assistant, or agent who are deciding whether — and how — to add a semantic cache in front of model calls. It will not teach you how to choose or size a vector index from scratch; the vector database guide already covers that decision. It teaches the part that's specific to caching: when similarity is a good enough proxy for "the same question," how to catch the queries where it isn't, and how to keep one stale or leaked answer from spreading to the next hundred users.

The MVSA operating model

Four moves make a semantic cache decision defensible: match, verify, serve, audit. Match finds a plausible candidate by embedding distance. Verify runs the checks a raw similarity score cannot — do the entities, tenant, and recency line up? Serve returns the response only after Verify clears it, tagged so the decision is visible later. Audit samples served hits against fresh model answers and feeds the result back into the threshold and invalidation rules. Skip Verify and you have a search feature pretending to be a cache; skip Audit and you have a threshold nobody can defend.

Decision rule: A cache hit is a proposal, not an answer. Nothing gets served until Verify passes — a high similarity score only earns a candidate a chance to be checked.

The framework is deliberately narrower than "add caching to your LLM app." It does not cover exact-match prompt caching (hashing an identical prompt string, which has no correctness ambiguity), provider-side context caching, or how to pick an embedding model. It covers the specific judgment call semantic caching introduces: deciding whether two different pieces of text should produce the same answer.

Know when a semantic cache is the right layer

Look for repeated shape, not repeated words

Semantic caching earns its cost when many different users ask questions that are worded differently but mean the same thing, and the correct answer to that meaning is stable. A support FAQ is the clean case: "how do I cancel my subscription," "what's the process to cancel," and "can I stop my plan" are three strings and one answer. A RAG assistant answering repeated questions against a fixed document set is the second clean case — the answer depends on the retrieved passage, not on who is asking.

The trap is assuming that "worded differently, means the same thing" is the only pattern that matters. The harder case — the one this article is built around — is "worded almost identically, means something different." Two queries can sit at 0.94 cosine similarity and still deserve two different answers, because a single swapped entity can flip the correct answer without moving the embedding very far at all: a plan name, an account id, a date range, a region. An embedding model trained to capture topical and syntactic similarity has no obligation to treat "Pro plan" and "Team plan" as maximally distant just because your business logic does.

Where semantic caching pays for itself

Field note: If you can't write down, in one sentence, what makes two queries "the same" for this use case, you're not ready to cache it. "Similar text" is not a definition — "same plan, same account, same policy version" is.

Build the match step: embeddings, index, and a starting threshold

Choose what you embed

Embed the normalized query text, not the raw user turn. Strip greeting boilerplate, lowercase where it's safe to, and — for multi-turn conversations — either exclude them from the cache entirely or embed a resolved, context-complete version of the question rather than the bare follow-up. A follow-up like "what about the other one" carries no meaning without the turn before it, and no amount of threshold tuning fixes a query that was never a complete unit of meaning to begin with.

Pick a distance metric and a conservative starting point

Cosine similarity on normalized embeddings is the standard choice, and it's a reasonable default. The threshold above which you'll even consider a match is a business decision disguised as a math one: it trades recall (how much traffic gets a chance to hit the cache) against precision (how often that chance was a mistake). Start conservative — high enough that you'd be embarrassed by very few of the matches it lets through — and treat the number as a hypothesis you'll revise from Audit evidence, not a constant you set once in a config file.

// Illustrative: replace with your embedding provider and cache store.
type CacheCandidate = {
  similarity: number;
  cachedEntities: Record<string, string>;
  queryEntities: Record<string, string>;
  cachedAt: number;
  maxAgeMs: number;
};

type CacheDecision = "serve" | "verify" | "miss";

function decideCacheHit(candidate: CacheCandidate, now: number): CacheDecision {
  if (candidate.similarity < 0.80) return "miss";

  const stale = now - candidate.cachedAt > candidate.maxAgeMs;
  if (stale) return "miss";

  const entitiesMatch = Object.entries(candidate.queryEntities).every(
    ([key, value]) => candidate.cachedEntities[key] === value,
  );
  if (!entitiesMatch) return "miss";

  return candidate.similarity >= 0.93 ? "serve" : "verify";
}

Field note: Start the threshold high enough that you're comfortable being wrong occasionally, then lower it only after an audit sample shows the false-hit rate holding steady at the new level. Raising a threshold after an incident is easy; explaining a wrong answer that already shipped to a customer is not.

Tune the threshold before you trust it: the worked example

Carry one example through the rest of this article. A support bot has cached the answer to "What's the refund window for a Pro plan?" — the correct, current answer is 30 days from purchase, prorated after day 14. A new user asks "What's the refund window for a Team plan?" The two strings embed at 0.94 cosine similarity: same topic, same structure, one word different. The Team plan's actual refund window is 14 days, no proration — a materially different policy.

A naive semantic cache with a threshold of 0.90 treats 0.94 as an easy auto-serve. It returns the Pro plan's 30-day, prorated answer to a Team plan question. The response reads fluently, cites a real number, and is wrong. Nothing in the interaction signals uncertainty, because from the cache's point of view nothing went wrong — it did exactly what it was built to do: find the nearest neighbor and return it.

A 0.94 match sits well inside most teams' auto-serve range — until a plan-name check reclassifies it as a miss.

Cosine similarity is not correctness; it's a search heuristic for "worth a second look." Treating a threshold crossing as the final decision is the single most common way semantic caching goes wrong in production, and it's rarely caught in testing because the happy-path queries — the ones teams actually write test cases for — don't exercise the entity boundary at all.

Verify before you serve: checks a raw score can't run

Parameter and entity equivalence

The cheapest, highest-value check is a structured comparison: extract the entities that matter for this query class (plan name, account id, region, date range, product SKU) from both the cached query and the new one, using a lightweight slot-filler, a regex, or a small classifier — not another full model call. Verify needs exact entity match, not another similarity score: if the extracted entities don't match exactly, the candidate is a miss, no matter how high the embedding similarity was. This single check would have caught the Pro-versus-Team example without needing a different embedding model or a higher threshold.

Confidence of the cached answer itself

A cache should only ever serve an answer it has reason to trust. If the original response that got cached was itself uncertain — a low-confidence generation, a refusal, an answer flagged by a downstream evaluator, or one produced under a since-superseded prompt version — exclude it from the eligible cache pool at write time rather than trying to catch it at read time.

Recency and the invalidation window

Every cached entry needs an explicit maximum age, chosen per query class rather than globally. A refund policy might be safe to cache for a day; a feature-availability answer during an active rollout might not be safe to cache for an hour. Treat the age window as part of the same decision that set the similarity threshold — both are judgments about how much staleness the business can tolerate for this class of question.

Decision rule: If Verify can't confirm equivalence within a fixed, cheap check — a slot match, a structured entity comparison, or a small classifier — treat the candidate as a miss. A verify step that costs nearly as much as the original model call defeats the reason to cache in the first place.

Guard the failure modes: stale answers, leakage, and poisoning

Staleness: the answer that used to be true

The most common semantic-cache failure isn't a bad similarity match at all — it's a correct match to an answer that has since gone out of date. A pricing page changes, a feature ships, a policy updates, and the cache keeps confidently returning the old truth because nothing about the query changed, only the world. This is why Verify needs a recency check independent of similarity: a perfect semantic match to a stale answer is still a wrong answer.

Cross-tenant leakage: when the cache key forgets who's asking

A semantic cache built as a single shared index across all customers will happily return one tenant's cached answer to a different tenant's structurally similar question, unless the cache key — not just a post-hoc filter — encodes the tenant boundary. This is especially dangerous for RAG-backed caches, where the cached answer may embed content from a specific customer's private documents.

Security note: Scope every cache key to the same tenant, workspace, or account boundary you'd use to scope a database query. A similarity index with no tenant partition is a search engine over everyone's answers, not a private cache.

Tenant scope belongs in the cache key, not in a post-hoc filter; a filter applied after retrieval means the sensitive content already left the trust boundary once, and a future refactor that forgets the filter reintroduces the leak silently.

Cache poisoning: one bad response, many wrong answers

Because a semantic cache serves the same stored response to many structurally similar future queries, a single bad generation that gets cached doesn't fail once — it fails every time a new query matches it, until someone notices and invalidates it. A hallucinated fact, a jailbroken response, or an answer generated during a provider degradation can all become a durable wrong answer if nothing gates what's allowed into the cache at write time. Gate cache writes on the same quality signal you'd use to gate a release: a passing evaluator score, a non-refusal, and no policy flags.

Measure hit rate and quality together, not separately

Sample every hit path, not just the misses

Hit rate alone tells you how much traffic the cache touched, not whether it touched it correctly. Pull a stratified sample of served cache hits — not a random sample of all traffic, specifically the hits — and re-run each one through a fresh model call to compare. The sampling discipline in the RAG evaluation guide applies directly here: instead of scoring retrieval quality, you're scoring whether the served cache answer still agrees with what a fresh call would have produced for the same query.

What a healthy cache dashboard shows

A cache with a 60% hit rate and a 4% sampled false-hit rate is a very different system from one with the same hit rate and a 0.3% false-hit rate, even though the top-line number looks identical. Report them together, always, or the hit-rate number becomes a vanity metric that hides the thing you actually need to know.

// Illustrative: one row from a cache-audit sample, not a live schema.
{
  "trace_id": "run_5c1e",
  "query": "what's the refund window for a team plan",
  "cache_hit": true,
  "similarity": 0.94,
  "verify_passed": false,
  "verify_reason": "entity_mismatch:plan",
  "served_from": "model",
  "fresh_vs_cached_agreement": null
}

Operating note: A verify_reason field like entity_mismatch:plan is worth more than the raw similarity score during an incident review — it tells the reviewer exactly which check should have fired, and whether it did.

Instrument the cache layer so a bad decision is visible in traces

Tag the decision, not just the outcome

Every cache-eligible request should produce trace attributes for each stage of the MVSA loop: the match candidate and its similarity score, whether Verify passed and why or why not, and whether the response was ultimately served from the cache or fell through to the model. This turns "was this response cached" from a question you can only answer by reading application logs into something visible on the same span you'd already open to debug latency or cost — the discipline LLM latency optimization applies to the critical-path span applies equally well here: you want to see the decision on the request that mattered, not only in an aggregate dashboard.

Make the miss path debuggable too

It's tempting to only instrument hits, since misses "just" fall through to the normal model call. But a miss that should have been a hit — because the threshold is too conservative, or Verify is too strict — is exactly the kind of regression an aggregate hit-rate metric will never surface on its own. Log the nearest candidate and its similarity score even on a miss, so a reviewer tuning the threshold has real distribution data instead of guessing from the hits alone.

Audit isn't a report you read once a quarter — it's the input that tightens Verify's entity rules and the input that expires a stale cache entry.

A cache hit should be exactly as visible in a trace as a fresh model call, with the same span structure, the same ability to inspect inputs and outputs, and the same place in the timeline an operator already knows how to open.

Compare the trade-offs

Semantic caching is a real trade, not a free win, along four dimensions that move independently:

• Cost: a cache hit that clears Verify is nearly free compared to a fresh model call, but Verify itself has a cost — a small classifier call or a structured lookup — and a poorly tuned threshold that serves wrong answers can cost more downstream (support tickets, trust, correction traffic) than it ever saved upstream. LLM cost optimization covers the broader playbook for cutting spend without hiding a quality regression inside an average.
• Latency: a served cache hit skips the model call's full round trip, which is the single biggest latency win available for repeated queries. But Verify adds its own latency, and a cache miss that first attempted a match adds a small tax to every request, hit or not.
• Quality: correct hits are indistinguishable from a fresh answer to the user. Incorrect hits are worse than a fresh answer, because they carry the same fluent confidence with none of the underlying reasoning that would have caught the mismatch.
• Operator effort: someone has to own the threshold, the entity rules per query class, the invalidation windows, and the audit cadence. A cache with no owner drifts toward either over-caution (a threshold nobody dares lower, wasting the opportunity) or under-caution (a threshold quietly raised to hit a cost target, reintroducing false hits).

The cheapest wrong answer is still the wrong answer; a false hit that saved $0.002 and cost a customer's trust in the product is not a win by any accounting that includes the second number.

Make the rollout decision

Roll out in stages, same as any other release

Treat enabling or expanding a semantic cache like any other production change: shadow mode first, where the cache computes what it would serve without actually serving it, compared against the real model response for every request. Then a narrow cohort with the cache live, watched closely. Then a broader rollout once the sampled false-hit rate holds steady across a representative traffic mix, not just the traffic mix from launch week.

Set an explicit revert condition

Decide, before turning the cache on for real traffic, what false-hit rate or leakage signal triggers an immediate rollback — and write it down next to the threshold, not in a separate incident-response document nobody reads until the incident.

Release rule: Promote a lower threshold only after a two-week audit sample shows the false-hit rate at or below your target and no unresolved leakage incident. Anything less is a guess wearing a percentage sign.

Ship the practical checklist

• [ ] Every cache-eligible query class is documented, along with the query types deliberately excluded.
• [ ] Verify runs a cheap, explicit equivalence check — never another full model call.
• [ ] Cache keys are scoped to tenant or workspace, matching your existing data-access boundary.
• [ ] A staleness window exists for every cached entry, sized per query class rather than globally.
• [ ] Cache decisions — match, verify, serve — are visible as trace attributes, not just an aggregate hit-rate number.
• [ ] An audit sample compares served cache answers against fresh model answers on a fixed, recurring cadence.

Start with the smallest cache surface that produces real evidence: one query class, a conservative threshold, and a two-week audit before expanding. A semantic cache earns trust the same way any other production change does — through a documented decision, a reversible rollout, and a sampled record of what it actually did, not through the promise of what it should do.

Frequently asked questions