Semantic & exact caching for LLM apps

What it is (intuition first)

An LLM call is the most expensive line in most applications: it costs money, it costs hundreds of milliseconds to seconds of latency, and at scale it costs you a rate limit. Caching is the oldest trick in computing applied to that line — "don't pay twice for the same answer." But "the same" is the whole problem, because two users almost never type byte-identical prompts. "What's your refund policy?" and "how do I get my money back?" want the same answer and share zero characters.

That splits LLM caching into two families that solve genuinely different problems:

Exact caching keys on the literal input (or a hash of it). A repeated request is a dictionary lookup — microseconds, free, perfectly correct. The catch: any byte change is a miss. This is the right model for deterministic, repeated work (idempotent tool calls, re-runs of the same eval, identical system prompts).
Semantic caching keys on meaning. You embed the query into a vector, search a vector store for a previously-answered query that is "close enough" (cosine similarity above a threshold), and if you find one, return its stored answer without ever calling the model. This catches paraphrases — at the price of a new and unfamiliar failure mode: a false hit, where two queries look similar in embedding space but actually want different answers.

A third, lower-level mechanism is easy to confuse with the first two and is arguably the most used in production: prompt-prefix / KV caching at the serving layer. It doesn't cache final answers at all — it caches the model's internal Key/Value tensors for a shared prefix of tokens, so the model skips recomputing attention over a long system prompt or document it has already seen. This is what Anthropic, OpenAI, Google, and DeepSeek mean by "prompt caching" in their pricing pages.

The mental model that keeps these straight:

Exact cache keys on bytes and returns the answer. Semantic cache keys on meaning and returns the answer. Prefix/KV cache keys on a token prefix and returns computation (the model still generates a fresh answer).

Why it matters

It is the cheapest order-of-magnitude win available. Provider prefix caches read cached tokens at ~10% of the input price (Anthropic, DeepSeek) and serve them far faster than recompute. An application-level semantic cache can remove the API call entirely on a hit. Caching is usually the first lever you pull before reaching for a smaller model or quantization.
It changes the latency profile, not just the bill. A semantic-cache hit is one embedding + one vector search (single-digit to low-tens of ms) versus a full generation (hundreds of ms to seconds). For chat, search, and support bots — where the same handful of questions dominate the traffic — this is the difference between "instant" and "thinking…".
It is where correctness quietly goes wrong. Exact and prefix caches are safe by construction (they only reuse identical content). Semantic caches are not: the entire game is the hit-rate ↔ staleness/accuracy tradeoff, and a badly-tuned threshold serves confidently wrong answers. This is the one cache family you must evaluate, not just deploy.
It is a substrate for agents and RAG. A multi-turn agentic loop ([[agentic-loops]]) re-sends a growing prefix every step; prefix caching is what makes that affordable. RAG pipelines ([[graphrag]]) re-send the same retrieved documents across users; prefix caching reuses them. The connection to [[kv-cache-optimization]] is direct — same KV tensors, but reused across requests instead of within one decode.

How it works (real mechanics)

1. Exact caching — the trivial, correct baseline

Hash the normalized request (often model + system + messages + params), look it up in Redis/an LRU, return on hit. The only design decisions are what goes into the key (include temperature, tools, and model ID — changing any of them changes the answer) and normalization (lowercase? strip whitespace? sort JSON keys?). Over-normalize and you risk collisions between prompts that should differ; under-normalize and your hit rate collapses. For sampled (temperature > 0) generations, an exact cache makes outputs deterministic per input, which may or may not be what you want — caching freezes the first sampled answer forever.

2. Semantic caching — the embedding-keyed cache

The canonical open-source implementation is GPTCache (Zilliz, github.com/zilliztech/GPTCache), and the canonical academic write-up is GPT Semantic Cache (Regmi & Pun, 2024, arXiv:2411.05276). The pipeline:

query
  │  1. embed            (sentence embedding → vector; see [[embeddings]])
  ▼
vector store (Redis / Milvus / FAISS)
  │  2. ANN search       (find nearest stored query by cosine similarity)
  ▼
nearest neighbour, similarity s
  │  3. threshold gate   (s ≥ τ ?)
  ├── hit  → return stored answer            (no LLM call)
  └── miss → call LLM, store (embedding, answer), return

Pseudocode for the core loop:

def answer(query, tau=0.8):
    v = embed(query)                          # 1. embed
    hit, sim = cache.search(v, top_k=1)       # 2. ANN search (O(log n) with HNSW)
    if hit is not None and sim >= tau:         # 3. threshold gate
        return hit.answer                      #    CACHE HIT — no model call
    answer = llm(query)                        #    MISS — pay full cost
    cache.add(v, query, answer)                #    populate for next time
    return answer

The reported numbers (GPT Semantic Cache, arXiv:2411.05276, Redis + a hierarchical ANN index, cosine threshold ~0.8): cache hit rates of 61.6% to 68.8% across query categories, reducing API calls by up to 68.8%, with >97% response accuracy on positive hits. The ANN index drops search from O(n) to O(log n). Treat these as one paper's measurements on its own dataset, not a guarantee — hit rate is entirely a function of how repetitive your traffic is, and the >97%-on-hit figure is what their threshold happened to deliver on their data, not a property of semantic caching in general.

The single most important knob is the similarity threshold τ. It is a direct hit-rate ↔ accuracy dial:

τ too high → few hits (most paraphrases miss), low risk of a wrong answer, small savings.
τ too low → many hits, but you start returning a neighbour's answer to a query that isn't actually the same question — a false hit. The paper notes a fixed 0.8 threshold "may not generalize across all use cases."

There is no universal correct τ; it must be tuned per domain against a labelled set of query pairs, and it interacts with the choice of embedding model (a weak embedder puts semantically-different queries close together, widening the false-hit zone). A common refinement is a two-stage check: ANN to shortlist, then a cheaper cross-encoder or an LLM-judge to confirm the match before serving — trading a little latency to cut false hits.

3. Prompt-prefix / KV caching — reuse computation, not answers

This is the provider-side mechanism, and it is exact, not semantic — it matches a token prefix byte-for-byte. When a request shares a leading run of tokens with a recent one (a big system prompt, a few-shot block, a mounted document), the server has already computed the Key/Value tensors for those tokens and reuses them, skipping the prefill compute. The research framing is Prompt Cache (Gim et al., MLSys 2024, arXiv:2311.04934), which generalizes this to modular reusable prompt segments via a "prompt markup language," reporting large time-to-first-token reductions by reusing precomputed attention states for frequently-reused chunks.

The one invariant that governs all prefix caching: it is a prefix match — any byte change anywhere in the prefix invalidates everything after it. The cache key is the exact tokens up to a breakpoint. A timestamp interpolated into the system prompt, an unsorted JSON dump, a reordered tool list — each shifts the bytes and turns a hit into a cold write. The architectural consequence is to put stable content first (frozen system prompt, deterministic tool order) and volatile content last (the per-request question).

Provider semantics (Anthropic, from the official prompt-caching docs) — useful as one concrete, well-documented instance; other providers differ in the knobs but share the prefix-match model:

Minimum cacheable prefix is model-dependent. On Claude Opus 4.6 it is 4096 tokens; on Sonnet 4.6 it is 2048; on Sonnet 4.5 it is 1024. Shorter prefixes silently don't cache — no error, just cache_creation_input_tokens: 0.
TTL options: a default 5-minute ephemeral cache, or an opt-in 1-hour TTL.
Write cost: 1.25× the base input-token price for the 5-minute TTL, 2× for the 1-hour TTL. Read cost: ~0.1× base input — the headline "up to 90% cheaper" figure.
Breakpoints: up to 4 explicit cache_control breakpoints, so you can cache sections that change at different frequencies (tools vs system vs a stable message prefix).
Lookback window: a breakpoint walks back at most 20 content blocks to find a prior cache entry; long agentic turns with many tool blocks can silently miss without an intermediate breakpoint.
Verification: the response usage reports cache_creation_input_tokens (written) and cache_read_input_tokens (served). If reads stay zero across repeated identical-prefix requests, a silent invalidator is in the prefix.

Provider landscape (pricing claims — verify before quoting; these are vendor figures and change): Anthropic uses explicit cache_control breakpoints, reads at ~0.1× input. DeepSeek caches automatically for prefixes over a small token floor and advertises a roughly 90% discount on cached input. OpenAI and Google Gemini also ship prompt/context caching, with their own pricing and (for Gemini) storage-by-duration billing rather than per-cached-token. Flag any specific percentage as a vendor claim that drifts with pricing updates.

4. Economics — when caching actually pays

Provider write premiums make caching a break-even calculation, not a free win. With Anthropic's 5-minute TTL (1.25× write, 0.1× read): a cached prefix breaks even on the second request (1.25 + 0.1 = 1.35× vs 2× uncached). The 1-hour TTL (2× write) needs at least three reads to pay off but survives gaps in bursty traffic. The lesson: cache prefixes you will reuse soon and often; caching a one-shot prefix just pays the write premium for nothing.

Key ideas & tradeoffs

Exact vs semantic is a correctness-vs-coverage choice. Exact is safe and cheap but brittle to wording; semantic covers paraphrases but introduces false hits. Many production systems layer them: exact first (free, certain), semantic second (broader, risk-managed by τ).
Hit rate ↔ staleness/accuracy is the central tension of semantic caching. Raising τ trades coverage for safety; lowering it trades safety for coverage. There is no setting that maximizes both — you pick a point on the curve for your domain's tolerance for a wrong answer.
Prefix caching reuses computation; it is always correct but constrained. It never returns a wrong answer (the model still generates), but it only helps when prompts share an exact leading token run — so prompt layout (stable-first) is the real lever, more than any cache flag.
Invalidation is the hard part, as always in caching. Exact/semantic answer caches can serve stale content when the underlying source of truth changes (a refund policy update makes every cached refund answer wrong) — they need TTLs or event-driven eviction tied to the knowledge source. Prefix caches invalidate themselves on any prefix byte change, which is the opposite problem: they're too eager, and the fix is byte-discipline (no timestamps/UUIDs/unsorted JSON in the prefix).
The embedding model is part of the cache. A semantic cache is only as good as its [[embeddings]] — a weak or mismatched embedder collapses distinct questions together (more false hits) or scatters paraphrases apart (fewer hits). Swapping the embedder silently changes cache behaviour.
Caching sampled output freezes randomness. An answer cache returns the same stored generation forever; for temperature > 0 creative tasks that may be undesirable. Prefix caching doesn't have this issue — it caches input computation, and the model still samples fresh output.

Honest caveats & open questions

Reported hit rates are dataset-specific. The 61.6–68.8% figures (and the >97%-on-hit accuracy) are one paper on one workload. Your hit rate is governed by your query repetitiveness, and your safe τ by your domain — neither transfers. Measure on your own traffic.
False hits are silent and confident. A semantic cache that serves a neighbour's answer produces a fluent, wrong response with no error. Without an offline eval set of near-miss query pairs you won't catch the failure until a user does. This is the strongest reason to gate hits with a second-stage check in high-stakes domains.
Staleness vs freshness is unsolved in general. Time-based TTLs are a blunt instrument; the principled fix — invalidate cached answers when the underlying document/policy changes — requires wiring the cache to your knowledge source, which most deployments skip.
Generative caching is early. "A Generative Caching System for LLMs" (2025, arXiv:2503.17603) proposes synthesizing an answer from multiple stored cache fragments rather than returning one verbatim, claiming higher effective hit rates — promising, but it reintroduces a generation step and its own correctness questions. Treat as a research direction, not settled practice.
Prefix-cache portability is nil. Caches are model-scoped and provider-scoped; a fork/sub-agent that rebuilds system/tools/model with any difference misses the parent's cache entirely. Concurrent identical requests all pay the write (a cache entry is only readable after the first response begins streaming).
Vendor pricing claims drift. Any "90% cheaper" / specific-multiplier figure is a snapshot; re-check the provider's current pricing page before relying on the break-even math.

How it connects to OpenAlice

OpenAlice's agent core is a hardened tool-loop ([[agentic-loops]]) that re-sends a growing conversation prefix on every step — exactly the workload prefix caching exists for. Several of OpenAlice's documented operating rules are caching discipline in disguise:

The "frozen system prompt, byte-identical preservation" rule (Alice's Soul/Identity blocks preserved verbatim; prompt-composer byte-identical for personality content) is, mechanically, what keeps the prefix cache hot — any drift in those leading bytes would cold-write the cache every turn. Personality preservation and cache efficiency point the same way here.
Stable-first prompt layout (deterministic tool order, volatile per-turn context appended last, dynamic context injected as a mid-conversation message rather than edited into the system prompt) is the OpenAlice realization of the prefix-match invariant — it preserves the cached history prefix across turns.
Verify-with-usage maps to the house "always verify" discipline: read cache_read_input_tokens to confirm hits, and treat a persistent zero as a silent invalidator (a datetime.now(), UUID, or unsorted JSON in the prefix) rather than assuming the cache works.
For answer-level caching (semantic), the relevant OpenAlice surfaces are the retrieval/[[embeddings]] paths and [[model-routing]] — a semantic-cache hit is the cheapest "route," short-circuiting the model entirely, and shares the embedding infrastructure already used for memory and search.

The KV tensors reused across requests here are the same ones [[kv-cache-optimization]] shrinks within a single decode — caching across requests and compressing within one are complementary levers on the same data structure.

Semantic & exact caching for LLM apps — embedding-keyed cache, prompt-prefix/KV reuse, invalidation, hit-rate vs staleness