Prompt Caching Deep Dive: How It Works and How to Improve Hit Rates

1. Why Prompt Caching Is the "First Lever" for Cost Reduction

Among all techniques for reducing LLM costs, including quantization, distillation, switching to smaller models, and batch processing, Prompt Caching is one of the few optimizations that offers "almost no quality loss, minimal changes, and immediate gains." Its core benefit can be summed up in one sentence: Once a stable prefix produces a cache hit, it is billed at roughly one-tenth the input rate.

In a typical Agent application, the system prompt, tool definitions, and long documents often account for most of the input tokens, while this content remains unchanged from turn to turn. Without caching, every turn pays full price for the same prefill computation and the same input tokens. Once the cache hits, those tokens are billed at just one-tenth the price. In scenarios such as long conversations, RAG, and coding Agents, where inputs routinely reach tens of thousands of tokens, reducing total input costs by tens of percent or even more than 50% is common.

But many people "enable caching without saving money." The causes almost always fall into three categories: cache misses, too many cache writes despite successful hits, and caches that expire before low-frequency traffic returns. This article explains each of these pitfalls.

2. How It Works: Cache KV, Save Prefill

Understanding the mechanism explains why the cache-hit requirements are so strict.

Before a Transformer generates its first output token, it must prefill the entire prompt by computing the Key/Value tensors (KV cache) for every token at every layer. This computation scales with input length and is a major source of latency and cost for long prompts. Prompt Caching stores the KV cache corresponding to a prefix on the server. If a later request has the same prefix, the service reuses that cache and skips prefill computation for that segment. Pricing depends on whether the cache is hit:

• cache_read (cache-hit read): roughly one-tenth the base input price;
• cache_creation (initial write): more expensive than the base input price: 1.25x for a 5-minute TTL and 2x for a 1-hour TTL;
• input (uncached remainder): billed at the standard price.

Notice that writes are much more expensive than reads. Caching is not profitable merely because it is enabled; it becomes profitable only after enough cache hits. As a rough break-even estimate, with a 5-minute TTL, net savings begin around the second cache hit. Because a 1-hour TTL costs more to write, it takes about three hits to break even. Enabling caching for low-frequency, one-off requests only makes them more expensive.

3. Cache-Hit Requirement: The Prefix Must Be Byte-for-Byte Identical

This is the most important and most easily overlooked rule in the entire mechanism:

Caching uses prefix matching. If even one byte in the prefix differs, every cached segment after that position is invalidated.

The server computes cache keys byte by byte from the beginning of the rendered prompt through each cache_control breakpoint. A timestamp, reordered JSON key, or reshuffled few-shot example invalidates every breakpoint after that position.

The rendering order is fixed: tools → system → messages. Tool definitions come first, followed by the system prompt, and then messages. This creates an important hierarchy of invalidation:

• Change one tool definition → the entire request cache is invalidated because tools come first;
• Change one character in the system prompt → the entire messages section must be recomputed;
• Append only a new turn to the end of messages → all previous turns remain cached.

Once you understand this hierarchy, every optimization technique that follows is simply a consequence of it.

4. Designing a Prompt Structure That Can Produce Cache Hits

Applying the rule in reverse gives the core design principle for prompts: put stable content first and variable content last. An Agent request that can keep producing cache hits should look like this:

system = [{
    "type": "text",
    "text": FROZEN_SYSTEM_PROMPT,        # Rules, persona, and output format that never change
    "cache_control": {"type": "ephemeral"}  # Place the breakpoint on the "last stable block"
}]
messages = [
    *history,                             # Append conversation history turn by turn; the prefix stays stable
    {"role": "user", "content": this_turn_input}  # Current user input; changes every time
]

Place the breakpoint (cache_control) on the last stable block, not at the end of the entire prompt. If you put the breakpoint after the changing question, every request writes a different cache entry that will never be read. This is the most common form of "fake caching": the marker is present, money is spent, and the hit rate remains 0.

In a multi-turn conversation, place the breakpoint on the last content block of the current final turn. The next request can reuse the entire conversation-history prefix, and cache hits accumulate incrementally as the conversation grows.

Notably, newer models such as Opus 4.8 support "appending a system message during a conversation" by adding {"role": "system", ...} inside messages. This does not modify the top-level system prompt, so it does not invalidate the cached conversation-history prefix. This is the correct way to issue operational instructions mid-conversation, such as switching modes or injecting context, instead of editing the top-level system prompt.

5. Six Common Causes of Cache-Hit Failures

In practice, low hit rates almost always come from the following "silent invalidation checklist." Grep any code that contributes to the prefix for these patterns:

There are only three ways to fix these issues: move dynamic content after the final breakpoint, make it deterministic through sorting or a fixed order, or delete it if it does not carry essential logic.

One point deserves special emphasis: do not put timestamps or the current date in the system prompt. If that information must be injected, put it in the user message for the current turn. A message introduced on turn 5 does not invalidate the cache for the first five turns, while Current time: 14:03:02 at the beginning of the system prompt causes every request to miss.

6. TTL and Invalidation: Low-Frequency Traffic Naturally Has Low Hit Rates

Caches have an expiration period. The default TTL is 5 minutes, and you can explicitly select 1 hour with ttl: "1h". If an entry is not hit again before its TTL expires, it is deleted, and the next request must write it again.

This produces a counterintuitive effect: the lower the traffic, the lower the hit rate. For an internal tool used only occasionally during the day, requests are often more than 5 minutes apart. Nearly every request rewrites the cache, which costs more because of the write premium instead of saving money. In this case, either disable caching or use a 1-hour TTL. The tradeoff is that writes cost twice as much and require more reads to break even.

Another common pitfall is concurrent request ordering: a cache entry becomes readable only after the first request starts streaming output. If you simultaneously fan out N requests with the same prefix, all of them are billed at full price because none can read the cache another request is still writing. The correct approach is to send one request first, wait for its first token, and then send the remaining N-1 requests.

If first-request latency matters, as in chat or voice applications, you can send a warm-up request with max_tokens: 0 at startup. This pays the write fee once and avoids imposing the initial latency on the user's first real request.

7. Monitoring: Measure the Hit Rate Instead of Guessing

You cannot determine whether caching works from the bill alone. You must inspect the usage fields in the response:

• cache_read_input_tokens: the portion served from cache and billed at one-tenth the price;
• cache_creation_input_tokens: the portion written to cache and billed at a premium;
• input_tokens: the uncached remainder, billed at full price.

A simple hit-rate metric is:

Hit rate = cache_read_input_tokens
         / (cache_read + cache_creation + input_tokens)

Evaluation criterion: if you repeatedly send requests with the same prefix but cache_read_input_tokens remains 0, silent invalidation is almost certainly occurring. Diff the rendered bytes of two requests to locate the byte that changes. After the hit rate improves, compare the input cost per thousand requests before and after enabling caching instead of looking at a single request.

A common misconception is that a small input_tokens value means little money was spent. It represents only the "uncached remainder"; total input equals the sum of all three fields. If an Agent runs for an hour and reports only 4K input_tokens, that does not mean it was cheap. Most of the cost may be in cache_creation, including the write premium. Always inspect all three fields together.

8. Limitations and Boundaries

Finally, several hard constraints must be understood:

1. Not every provider or model supports it. Automatic caching (auto caching) is unavailable on many third-party platforms, and some older models do not support it. Confirm your provider and model combination before deployment.
2. Caches are isolated by model. Switching models invalidates the entire cache even if the prompt is unchanged byte for byte. During A/B tests, each model uses its own independent cache.
3. Isolation across requests and users. User A's cache is unavailable to user B unless their prefixes are byte-for-byte identical, such as when they share the same system prompt and document. Using caching for "global sharing" requires designing the prompt without user-level differences.
4. Minimum cacheable prefix length. A prefix that is too short will "silently remain uncached": no error is returned, but cache_creation stays at 0. Thresholds vary by model, ranging from 1024 to 4096 tokens, so do not force caching on short prompts.
5. The number of breakpoints is limited. A request can contain at most 4 cache_control breakpoints, so long conversations require careful breakpoint placement. A single breakpoint can also look back through no more than 20 content blocks, so long turns containing many tool results need an additional breakpoint in the middle.

Summary

• Prompt Caching saves prefill computation, bills cache hits at a discount of roughly one-tenth, and charges a write premium. It is highly effective when prefixes are reused frequently, but makes low-frequency, one-off requests more expensive.
• There is only one cache-hit requirement: the prefix must be byte-for-byte identical. The rendering order is tools → system → messages, and a change at any point invalidates every cached segment after it.
• Design principle: stable content first, variable content last. Place the breakpoint on the last stable block. Dynamic content such as timestamps and user IDs belongs in the current message, never in the system prompt.
• Always check the silent invalidation list: timestamps, UUIDs, unsorted JSON, per-user system prompts, and dynamic tool sets. If the hit rate remains 0, the root cause is almost always here.
• Calculate the hit rate with cache_read_input_tokens instead of relying only on the bill. For low-frequency traffic, either disable caching or use a 1-hour TTL.

In one sentence: Prompt Caching is not a switch that saves money when you add a marker; it is an engineering discipline that treats prompts as immutable structures. Hit rates are designed, not tuned.