Prompt Caching: Reusing the KV Cache Across Calls

A coding agent may send the same tool catalog, repository rules, and system instructions on every turn. Prompt caching lets the inference service reuse prefill state for an unchanged prefix. The model receives the same tokens, while repeated input can be billed and processed differently according to the provider’s cache policy.

What prompt caching actually is

The KV cache stores attention keys and values computed during prefill. Within one request, it prevents decode from recomputing the full prefix for every output token. Prompt caching reuses provider-managed prefill state across requests that share a prefix. See LLM inference fundamentals for the intra-request mechanism.

Providers identify a reusable token prefix and associate it with cached computation. Pricing, minimum prefix length, storage duration, and placement vary by model and can change independently of the API shape. Measure usage fields and latency for the exact model instead of assuming a discount or time saving from another provider.

Reuse is prefix-based. If a token changes near the start of a request, later tokens cannot use the old prefix entry. Stable content therefore belongs before timestamps, identifiers, retrieved passages, and the current user message.

Provider controls

Anthropic: explicit breakpoints

Anthropic’s prompt-caching API uses cache_control breakpoints. A breakpoint marks the preceding request prefix, following the provider’s tools, system, and messages ordering. TTL choices and minimum cacheable lengths depend on the model. A request below the threshold still runs, so inspect cache_creation_input_tokens and cache_read_input_tokens rather than inferring success from the request syntax.

OpenAI: automatic prefix caching

OpenAI’s prompt caching is automatic on supported models. Responses expose the reused portion in usage.prompt_tokens_details.cached_tokens. The optional prompt_cache_key is a routing hint rather than an isolation boundary. Choose a key that groups genuinely shared, sufficiently frequent prefixes, and never treat it as an authorization control.

Google Gemini: implicit + explicit

Gemini supports implicit and explicit context caching. Explicit caching creates a named resource with a configured lifetime and storage charge. It fits a large context reused by many requests. Implicit caching requires less lifecycle code but gives the application less control. Compare read savings with storage cost at the expected query rate.

Structuring a prompt for cache hits

Place stable content first and dynamic content last. A common order is tool definitions, system policy, session-stable context, conversation history, and the current message. Do not reorder semantically meaningful messages merely for caching; preserve the provider’s request contract.

For read-heavy context, place the cache breakpoint after session-stable material. In a growing conversation, place it before the current user message so prior turns join the reusable prefix. Anthropic’s lookback behavior allows a new turn to extend an earlier cache entry without rewriting the stable blocks at the start.

A timestamp near the start creates a new prefix whenever the time changes. Per-request IDs have the same effect and may also prevent sharing across users. Put volatile values after the reusable portion or supply them through an appropriate tool result.

Tool definitions are usually part of the prefix. Renaming a tool or changing its description can invalidate the system prompt and message cache that follows it. Version tool schemas and system prompts as deployable artifacts, then expect a cold-cache period after each change.

Code: Python with the Anthropic SDK

Install the Anthropic SDK with pip install anthropic. This example marks the tool catalog and long system context explicitly.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47

# pip install anthropic
from anthropic import Anthropic

client = Anthropic()

LONG_SYSTEM_CONTEXT = open("knowledge_base.md").read()  # 30k+ tokens

TOOLS = [
    {"name": "search", "description": "...", "input_schema": {...}},
    {
        "name": "get_document",
        "description": "...",
        "input_schema": {...},
        # Cache breakpoint at the end of the tools array. Everything above
        # (tools) gets cached when this fires.
        "cache_control": {"type": "ephemeral"},
    },
]

def chat(messages: list[dict]) -> dict:
    response = client.messages.create(
        model="claude-opus-4-7",
        max_tokens=1024,
        tools=TOOLS,
        system=[
            {
                "type": "text",
                "text": "You are a research assistant. Cite sources.",
            },
            {
                "type": "text",
                "text": LONG_SYSTEM_CONTEXT,
                # Second breakpoint at the end of system. tools+system are
                # both cached as one prefix.
                "cache_control": {"type": "ephemeral", "ttl": "1h"},
            },
        ],
        messages=messages,
    )
    u = response.usage
    print(
        f"input={u.input_tokens}  "
        f"cache_read={u.cache_read_input_tokens}  "
        f"cache_write={u.cache_creation_input_tokens}  "
        f"output={u.output_tokens}"
    )
    return response.model_dump()

On a cold request, cache_creation_input_tokens should contain the prefix size. A later matching request should move tokens into cache_read_input_tokens. Log all three input counters with model, prompt version, and latency. TTL behavior is provider-specific; use the current documentation when deciding whether pre-warming is economical.

Cost and latency model

Let P be reusable prefix tokens, D_i dynamic input on turn i, W the cache-write rate, R the cache-read rate, and B the uncached input rate. For N matching turns:

1
2

uncached input cost = Σ(B × (P + D_i))
cached input cost   = W × P + Σ(B × D_i) + (N - 1) × R × P

Add explicit-cache storage fees and cache misses where applicable. The break-even point depends on prefix size, write premium, read discount, TTL, and reuse count. Calculate it from current provider prices and observed traffic.

Latency needs an experiment because routing, queueing, and batch load can dominate prefill. Send a cold request followed by several byte-identical-prefix requests, record TTFT and usage counters, then repeat under representative concurrency. Prompt caching has not helped if billed cached tokens rise while user-visible tail latency remains unchanged for the workload that matters.

Invalidation and versioning

A prefix cache is content-addressed, so changing content naturally produces a miss. The application still needs versioning to explain those misses. Record a hash or release ID for tool schemas, system instructions, static context, and model selection. When hit rate changes after a deployment, these fields identify which prefix component moved.

Avoid runtime serialization differences. JSON key order, whitespace, Unicode normalization, and generated schema descriptions can change token IDs while representing the same data. Use deterministic serialization for stable blocks and test the rendered request, not only the source objects. Do not sort message history or tool definitions if their order carries meaning.

Deployments create cold-cache periods. For large shared prefixes, a canary can warm the new version before most traffic moves. Pre-warming is worthwhile only when the expected reuse exceeds the write and storage cost. It should use the same model, rendered prefix, and routing key as production requests.

Cache lifetime also affects rollbacks. Restoring an old prompt may hit an old entry if it remains available, but applications should not rely on that behavior. A rollback must be correct on a cold cache; caching is an optimization rather than stored application state.

Privacy and isolation

Prompt caching does not authorize data sharing. Provider-side implementations may isolate entries according to account or project rules, but the application must still enforce tenant access before constructing a request. A routing hint such as prompt_cache_key is not a security namespace.

The largest reusable prefix is not always the safest prefix. A per-tenant policy document can be reused within that tenant but must never be moved into an application-wide shared block. User profiles and retrieved documents need the same classification. Design cache grouping after establishing data boundaries.

Do not include secrets merely because a provider says cache data is ephemeral. API keys, database credentials, and bearer tokens should stay out of prompts. If sensitive business content is allowed, verify the provider’s retention, regional, and zero-data-retention terms for the selected API and account configuration.

Application logs can leak more than the cache. Usage counters and prompt-version hashes are usually enough to debug hit rate. Avoid logging the full stable prefix, especially when it contains proprietary instructions or customer data.

Measuring cache effectiveness

Track cache-read tokens, cache-write tokens, uncached input tokens, TTFT, end-to-end latency, and request cost. Group them by model, prompt version, tenant class, and endpoint. A global hit ratio can hide one high-volume workload that works well and many low-volume workloads that pay repeated writes.

Useful ratios include:

1
2
3

token hit rate = cache_read_tokens / total_eligible_prefix_tokens
write amplification = cache_write_tokens / cache_read_tokens
effective savings = uncached_estimate - observed_input_and_storage_cost

Define “eligible prefix” from the rendered request so the denominator excludes dynamic input. Monitor p50 and p95 TTFT for hits and misses separately. If the two distributions overlap, queueing or routing may dominate the prefill saved by the cache.

Test failure cases deliberately: one changed tool description, a timestamp inserted near the start, a TTL expiry, a prompt-version rollout, and two tenants with similar content. Confirm both the expected usage counters and isolation behavior. This catches cache regressions before a billing anomaly becomes the first signal.

Self-hosted serving

Self-hosted engines expose the same prefix-reuse opportunity through scheduler and memory-management controls rather than an API discount. Reused prefixes occupy GPU memory, so retention competes with active KV caches and model weights. A policy that keeps every prefix can reduce the number of concurrent sequences and worsen tail latency.

Choose admission and eviction using observed reuse. Prefix length alone is insufficient: a very large prefix used once may displace several smaller prefixes with steady traffic. Track saved prefill tokens per byte of cache memory, entry age, and recomputation cost. Partitioning by model revision and adapter is mandatory because cached tensors depend on the exact weights that produced them.

Distributed serving adds placement concerns. A request receives a hit only when it reaches a worker that holds the compatible prefix, unless the platform implements cache transfer or shared storage. Sticky routing can improve reuse but may create hot workers. Compare routing balance with cache locality under production concurrency.

Further reading

• Anthropic prompt caching
• OpenAI prompt caching guide and Prompt Caching 101
• Google AI context caching
• Anthropic prompt-caching announcement