Prompt caching

Mechanically, prompt caching is KV-cache persistence across requests. After a request's prefill, the serving system keeps the key and value tensors for a prefix of the prompt, indexed by a hash of the exact token sequence, often in fixed-size blocks so that any block-aligned prefix can be matched. When a new request arrives, the scheduler looks up the longest cached prefix, skips prefill for those tokens and computes only the remainder. Because every token's keys and values depend on all preceding tokens, a single changed token invalidates the cache from that point onward; identical content in a different position is not reusable. Open-source engines expose the same idea as automatic prefix caching (vLLM) or RadixAttention (SGLang, which organises cached prefixes in a radix tree).

Commercial APIs implement it in two styles, and the details change between model generations, so current documentation is the authority. Anthropic's API uses explicit cache_control breakpoints (up to four per request) or an automatic mode; the prefix is hashed in the order tools, system, messages; the default lifetime is five minutes, refreshed on each hit, with an optional one-hour lifetime; at the time of writing cache writes cost 1.25× the base input price (2× for one hour) and cache reads around 0.1×, with model-dependent minimum prefix lengths. OpenAI applies caching automatically once a prompt passes a minimum length (1,024 tokens on many models), routes requests by a hash of the initial tokens, optionally steered by a prompt_cache_key, and bills cached input tokens at a discount. Google's Gemini API offers both implicit caching and explicit cached-content objects with a configurable TTL.

Getting hits is a prompt-engineering discipline. Stable content goes first (tool definitions, system prompt, reference documents, few-shot examples) and variable content last; timestamps, request IDs, randomised example order or non-deterministic JSON key ordering early in the prompt silently destroy the hit rate. Changing the tool list or model version also invalidates the prefix. Teams should monitor the provider's usage fields for cached versus uncached input tokens instead of assuming savings, since low-traffic endpoints may see entries expire between requests.

The security dimension is a timing side channel. A cache hit makes time to first token measurably shorter, so if a cache is shared between users, an attacker can probe whether a given prefix was recently sent by someone else. Gu et al. (ICML 2025) audited commercial APIs with statistical timing tests and detected global cache sharing across all users at seven providers, OpenAI among them at the time of the study. Current major providers state that caches are isolated per organisation or workspace, and self-hosted multi-tenant deployments should apply the same isolation. Prompt caching is also distinct from response caching, which stores whole answers for identical or semantically similar queries and returns them without running the model at all.