Caching Strategies for LLM Applications

LLM calls have two original sins: they are expensive, with both input and output billed by token and a single agent request often triggering dozens of model calls; and they are slow, with the first token taking seconds and making users feel as though they are waiting for dial-up internet. Yet an astonishing share of LLM application traffic consists of repeated requests: the same FAQ, the same document summarized repeatedly, or the same kind of "help me improve this code" request.

Caching is the most direct lever for reducing costs and latency, but it is not as simple as "add Redis and you are done." A poorly designed cache has two possible outcomes: either the hit rate is too low to matter, making the cache pointless, or it returns stale or incorrect answers, which is worse than having no cache at all. The following sections break LLM application caching into three distinct layers.

1. Why LLM Caching Deserves Its Own Chapter

Traditional web caching stores a "computed result," uses request parameters as the key, and relies on TTL for expiration. LLM applications add two dimensions:

1. Cache granularity spans multiple layers. You can cache anything from the KV cache inside a model API to the result of an entire application-level call, but the hit conditions and cost structures differ completely.
2. "Similar" requests can also hit. Traditional caches require exact key equality. An LLM semantic cache can produce a hit when two questions are worded differently but mean the same thing. That is both an advantage and the biggest trap.

Understanding these two points prevents the three cache layers from being conflated.

2. Layer One: Prompt Cache at the Model API Layer, with Discounts for Stable Prefixes

This is the first layer you should enable and the easiest one to overlook. The principle is straightforward: during inference, the model computes a KV cache for the input. If two requests have exactly the same prefix, the second request can reuse that cache, allowing the provider to offer a discount.

• Anthropic prompt caching: explicitly declare cache_control. Reading cached prefix tokens costs about 1/10 of the standard rate, writes carry a 25% premium, and the cache lives for five minutes, with a one-hour option available.
• OpenAI automatic prefix caching: no declaration is required. Prefixes of at least 1024 tokens are cached automatically, with the matching portion costing about 50% less.
• Gemini context caching: explicitly create cached content, which is useful for keeping large documents in context over longer periods.

There is only one hit condition: the prefix must match exactly, token for token. The golden rule is therefore: put stable content first and volatile content last.

// Anthropic:把 system prompt、few-shot、长文档放前面并标记缓存
const res = await anthropic.messages.create({
  model: "claude-opus-4-5",
  system: [
    { type: "text", text: SYSTEM_PROMPT },                    // 稳定指令
    { type: "text", text: PRODUCT_DOCS, cache_control: { type: "ephemeral" } } // 稳定长文档
  ],
  messages: [{ role: "user", content: userQuestion }]         // 易变内容放最后
});

The most common mistake is putting the current time, a random ID, user_id, or request sequence number in the prefix. A change of even one byte invalidates the entire prefix cache. Move those variables to the end of messages, and the hit rate rises immediately.

3. Layer Two: Semantic Cache at the Application Layer, Matching Similar Queries

A prompt cache saves tokens but still calls the model. A semantic cache goes further: a similar question returns a historical answer directly, eliminating the entire model call. The process is query → embedding → find the nearest neighbor among historical query vectors → return the cached answer if similarity exceeds the threshold.

Redis 8 includes vector search, making one instance suitable for both KV and vector storage:

// 写入:把 query、答案、embedding 一起存
await redis.sendCommand([
  "VADD", "semcache", embeddingBytes, `ans:${id}`,        // 向量入队
  "HSET", `ans:${id}`, "answer", answer, "ts", Date.now() // 答案存哈希
]);

// 查询:找最相似的邻居
const [hitId, score] = await redis.sendCommand([
  "VSIM", "semcache", embeddingBytes, "COUNT", 1, "WITHSCORES"
]);
if (score > 0.95) {                       // 阈值,越严越安全
  return await redis.hget(`ans:${hitId}`, "answer");
}
// 未命中 → 真正调模型 → 写回缓存

The threshold is the core of this technique: values above 0.95 are relatively conservative, while false hits increase noticeably below 0.90. The classic semantic-cache failure is similarity in wording without similarity in meaning:

• "How do I request a refund?" vs. "How do I request a refund for someone else?" The embeddings are close, but the answers cannot be reused.
• "How do I reverse a Python list?" vs. "How do I reverse a Python string?" A single word differs, but the APIs are different.

A semantic cache is therefore suitable only for factual, unambiguous, nonpersonalized answers. Never put creative generation, conversations with user context, or time-sensitive content into a semantic cache.

4. Layer Three: Result Cache at the Application Layer, Using Exact Keys

When you can define a stable business key, do not use a semantic cache. Match it exactly for 100% deterministic behavior without the risk of a false similarity hit. A typical key is doc_id + operation(+ version):

const key = `summarize:${doc.id}:v${doc.version}`;
let summary = await redis.get(key);
if (!summary) {
  summary = await llm.summarize(doc.content);
  await redis.set(key, summary, "EX", 7 * 24 * 3600); // 7 天 TTL 兜底
}

This works well for deterministic operations such as summarizing the same document, extracting structured fields, or rewriting text with a fixed template. If 10,000 people open the summary for the same doc_id, the model is called only once. This layer has the highest ROI, with costs approaching zero.

5. Optimizing Hit Rates: Make Sure the Cache Actually Gets Used

A cache that nobody hits is wasted effort. Three levers improve hit rates:

1. Keep the prompt structure stable. Put instructions, roles, few-shot examples, and referenced documents first, then place user input and timestamps last. This benefits both prompt caching and result caching.
2. Normalize queries. Before computing a key or embedding, apply trim + lower + 去标点 + 同义词归一. " How do I build RAG?? " and "how do I build rag" should hit the same entry.
3. Prewarm hot entries. The cache is empty at cold start, while the top 20% of questions often account for 80% of traffic. Before launch, use historical logs to run the top N queries into the cache so the first day's hit rate remains healthy.

6. Consistency Pitfalls: Caching Can Be More Dangerous Than Not Caching

This section determines whether you are engineering the system or planting a hidden failure.

• Stale cache entries: the source document changes, but its summary remains old. Include version or content_hash in the key so any data change produces a new key and naturally invalidates the old entry. When necessary, issue an explicit DEL from the data-update workflow.
• Time-sensitive content should not be cached. Queries such as "today's stock price," "what time is it now," or "the latest announcement" must be short-circuited at the entry point or given only a very short TTL.
• Personalization vs. shared caches. Answers that include private user context, such as user_id, chat history, or permission scope, must never enter a shared semantic cache. Doing so would leak information across users. Either use a per-user namespace or do not cache them at all.

In one sentence: cache answers that are equally correct for everyone, not answers that happen to be correct for one particular user.

7. Combined Strategy and Monitoring

The three layers are not alternatives; they stack. Ranked by the savings from a hit:

结果缓存(精确 key)   → 趋近零成本,优先级最高
语义缓存(相似 query) → 省整次调用,次之
prompt cache(前缀)   → 省 token + 省时间,兜底

Evaluate each incoming request in that order, moving to the next layer only after a miss. Even when the request ultimately reaches the model, prompt caching can still provide a token-level discount.

A cache without monitoring is a black box. Track three metrics:

• Hit rate: measure each layer separately. If a layer underperforms, inspect normalization, thresholds, and key design.
• Incorrect cache hits: semantic caches especially require sampled reviews. Randomly rerun selected hits through the model for comparison, or collect user feedback that an answer missed the question. If quality declines, immediately raise the threshold or disable the cache.
• Invalidation strategy: use the full combination of TTL 兜底 + 版本号主动失效 + 数据更新时 DEL. Relying on TTL alone will eventually cause a stale-data incident.

Summary

• Each of the three layers covers a different part of the stack: prompt caching saves tokens, semantic caching eliminates entire calls, and result caching approaches zero cost. Prioritize hits from highest to lowest precision.
• Prompt caching depends on prefix stability: always move volatile content such as time, user_id, and random strings to the end.
• Semantic caching depends on thresholds and use cases: start at 0.95 or higher and cache only factual, nonpersonalized, non-time-sensitive answers.
• Consistency is the baseline: put version numbers in keys, do not cache time-sensitive content, and never place personalized answers in a shared cache.
• Do not deploy caching without monitoring: measure hit rates by layer and require sampled reviews of semantic-cache hits.

Key takeaway: Caching in an LLM application is not "adding Redis." It is "betting on certainty at three different scales." The more accurate the bet, the more you save. Get it wrong, and you save tokens at the cost of user trust.