Caching and invalidation strategies for LLM systems
A cache is a bet the input repeats; invalidation is the price of being wrong.

A cache assumes the same input arrives twice. Prompt caching at the API layer and prefix caching inside the serving engine both make that bet: each stores the processed form of a prefix so a later request that repeats it skips the work. RFC 9111 supplies the vocabulary — a response is fresh or stale, validation and invalidation are separate operations — and the same split governs an LLM cache: reuse trades freshness for latency and cost, and invalidation is the price paid to bound how stale that reuse may become.
Two layers, one unit of reuse
An HTTP cache “stores cacheable responses to reduce the response time and network bandwidth consumption on future equivalent requests” (RFC 9111). LLM systems do it twice. At the provider, prompt caching works by “resuming from specific prefixes in your prompts”. At the serving engine, vLLM’s Automatic Prefix Caching “caches the KV cache of existing queries”, so a new query that shares a prefix can “skip the computation of the shared part”.
Both key on the prefix, never the whole request, and both are exact: Anthropic states “Cache hits require 100% identical prompt segments”. Nothing here is semantic — two prompts that mean the same thing but differ by one character are two keys, and the second is a miss.
Freshness: a bet on repetition
RFC 9111 separates freshness from correctness: a stored response is “fresh” when it can be reused without contacting the origin, and when it is “not fresh, it might still be reusable if validation can freshen it”. The LLM equivalent is the time-to-live (TTL). Anthropic’s prompt cache, by default, “has a 5-minute lifetime”, refreshed “no additional cost each time the cached content is used”.
That makes the wager concrete: a TTL is a claim that the same input returns within the window. A prefix used once a day pays the write and rarely reads it; a system prompt attached to every request pays once and reads forever. vLLM states the same precondition from the serving side: prefix caching “does not bring performance gain when” queries “do not share the same prefix with any of existing queries”.
Invalidation: how stale you are allowed to be
When the input changes, the cache must be told. RFC 9111 is blunt about the HTTP case: unsafe request methods “have the potential for changing state on the origin server”, so caches “are required to invalidate stored responses to keep their contents up to date”. To “invalidate” means a cache will “either remove all stored responses” for that target or “mark them as ‘invalid’ and in need of a mandatory validation”. Invalidation is also local: it “does not guarantee that all appropriate responses are invalidated globally”.
LLM caches behave the same way, with a coarser key. Anthropic’s cache “follows the hierarchy: tools → system → messages”, and “Changes at each level invalidate that level and all subsequent levels” — editing one tool definition reaches “the entire cache”. Because the key is a prefix hash, invalidation is positional: a change late in the prompt costs only the suffix after it.
Two consequences matter here. Invalidation is all-or-nothing per breakpoint, so a timestamp hidden inside a cached block silently turns every request into a write. And a hit’s staleness has no bound unless one is set: the TTL is the only thing between a cached prefix and an input that changed beneath it.
What you must never cache
Because a prefix cache bets the bytes are identical, tenant and personalised content must never share one. Anthropic isolates caches between organisations: “Different organizations never share caches, even if they use identical prompts”. A boundary is a property of the key, not something assumed from two prompts looking alike. The risk compounds when a cached prefix carries retrieved documents or earlier turns: the cache turns a per-request read into a shared, long-lived one, and a document meant for one caller can be replayed to the next. That data-bleed surface is why the cached region must be scoped to content you will stand behind for the whole TTL.
The decision: cache the stable prefix
The strategy falls out of one move: place the boundary at the last block identical across the requests you mean to share, never inside content that changes per request. Static content — tools, system instructions, a fixed document — goes before the boundary; the incoming message goes after it. The rest is sizing between two costs: too short a TTL wastes the write, too long a TTL stretches the staleness window. The price and latency sides of that trade-off are a separate sheet’s concern; what this one owns is the boundary, the lifetime, and the rule that decides when a cached prefix stops being true.
Level and prerequisites. L5 — this is a systems decision about reuse and staleness, not a how-to: the reader must choose a cache boundary and a lifetime under a correctness constraint, and must know that a hit on changed content is a silent fault. Prerequisites: the L2 sheets on serving and context, and the L3 sheets on retrieval and context handling that this layer assumes.
Where to go next
- Automation & AI — the macro-area this sheet belongs to.
- Applications — the node this sheet sits in.
References
- IETF — RFC 9111, HTTP Caching — the definition of a cache, freshness versus staleness, validation and invalidation.
- Anthropic — Prompt caching — prefix caching, the 5-minute TTL, the invalidation hierarchy and the isolation guarantee.
- vLLM — Automatic Prefix Caching — KV-cache reuse, its precondition and its limits.
- vLLM — documentation home — prefix caching named as a serving optimisation.