Prompt Prefix Cache Correctness: Version Keys, Tenant Isolation, and Hit Acceptance

Prompt Caching: A Deep Dive explains stable prefixes and hit optimization. This article focuses only on whether a given hit is eligible for acceptance. LLM Caching Strategies also covers result and semantic caches, which can return answers directly; prefix caches only reuse prefill computation and must not alter sampling semantics. KV Cache Admission and Eviction manages VRAM block commitments, while this control governs the content identity, versions, and tenant boundaries of blocks or managed cached content.

Memory summaries can become prefix inputs, but the artifact ID, source epoch, and permissions from Memory Consolidation Lineage must be part of the cache identity. Coincidentally identical summary text must not bypass its validity constraints. Prompt engineering determines what to write; this article does not attempt to fix isolation by rewording prompts.

Declare the Cache Scope Before Computing Its Identity

Every prefix must have exactly one scope: PUBLIC, TENANT, or PRINCIPAL. PUBLIC requires every source to be approved through a public-data inventory; content must not be classified as public automatically just because it contains no obvious names. The default is tenant scope. The gateway obtains the scope from trusted authentication context, canonicalizes rendered messages and tool definitions, and then computes the content digest. The tenant namespace is derived from a server-side key using RFC 2104 HMAC. Logs record only a short digest and never expose the raw ID.

{
  "cacheObject": "pc-01J2ZE",
  "scope": "TENANT",
  "namespace": "hmac-sha256:49a1...",
  "contentDigest": "sha256:f861...",
  "modelDigest": "sha256:8d02...",
  "tokenizerDigest": "sha256:2f4a...",
  "runtimeDigest": "sha256:590c...",
  "adapterDigest": "sha256:19bb...",
  "promptBundleDigest": "sha256:80c1...",
  "toolsetDigest": "sha256:91c4...",
  "policyRevision": 18,
  "authorizationEpoch": 42,
  "cacheEpoch": 7
}

The model, tokenizer, runtime, adapter, chat template, tool schema, and any parameters that can change the token sequence or attention computation are all part of the identity. Security policies and authorization epochs must participate in the acceptance decision even when they do not change the text. Placing dynamic timestamps, request IDs, and user questions after the cacheable prefix is not a security control; if the prefix itself contains private data, it may still be reused only within the corresponding scope.

Google Gemini's context caching documentation describes creating and using explicit cached content, while vLLM's Automatic Prefix Caching documentation describes reusing computation for prefix blocks that have already been computed. Their APIs and lifecycles differ, so the application registry stores its own expected contract rather than treating a provider cache name or runtime block hash as an authorization credential.

Repeat the Inexpensive Acceptance Check on Every Hit

A request first completes authorization and version resolution to obtain the expected tuple, then queries the cache. The object must match the namespace, every digest, the policy revision, the authorization epoch, and the cache epoch; any mismatch is treated as a miss. Before physical KV state participates in computation, the request must also obtain a CacheUsePermit from a linearly consistent gate shared with authorization epoch updates. The permit binds the request, cache object, epoch, and fence while registering the operation as in flight. A revocation CAS advances the epoch and closes old permits; it races with concurrent acquisition so only one side can succeed. Asynchronous cleanup is not responsible for immediate rejection. Trusted context propagation from MCP Multi-Tenant Isolation must extend through the gateway, queue, and worker.

Physical lookup must enforce isolation before a block participates in computation, not rely on application-level validation after a hit. Every TENANT/PRINCIPAL request must inject the HMAC namespace, cacheEpoch, and runtime digest into the first-block hash or cache salt; subsequent chained blocks inherit that root. If the runtime does not support per-request salts, use tenant-isolated pools or disable private prefix caching. Only inventory-approved PUBLIC prefixes may use a shared salt. Physical blocks still require strong digests and verification of either the token sequence or a collision-resistant identity. A short hash cannot be accepted without comparison after a hit. Shared blocks must be immutable and released using reference counts or generational fences. Public sharing must also account for hit-latency side channels; highly sensitive deployments can disable cross-tenant physical reuse entirely.

Normal output validation and tool authorization still run after the cache is accepted. A prefix cache is not a result cache. With fixed model parameters, cold and warm requests should satisfy the same business assertions; a hit must not restore revoked tools or skip authorization filtering for the current turn.

Use Tenant Swaps and Collision Doubles in Negative Tests

Create tenants A and B, each using byte-identical public templates, different private documents, and the same user question. Warm the cache with A, then issue a request from B while injecting a missing physical salt, an incorrect namespace, a stale authorization epoch, an identical short hash, adapter and tokenizer switches, policy revocation, concurrent eviction, and worker restarts. Assert that private requests are separated before lookup by their salts or isolated pools. Then replace the hash function in the test runtime to force collisions and confirm that full identity comparison still rejects the wrong block.

Next, give A's cache handle, object name, and asynchronous job ID to B. Every entry point must restrict scope using authenticated context. Insert revocation between lookup and permit acquisition, and assert that either revocation or KV use succeeds, but never both. The revocation workflow must wait for, cancel, or isolate registered computation, and an old worker without a permit must not continue. Compare hot and cold outputs using required claims, citations, tool sets, and schemas rather than requiring randomized generations to match word for word.

Metrics include hits and saved prefill by scope, misses caused by version mismatches, authorization epoch rejections, cross-scope attempts, divergence between hot and cold assertions, visible objects from old generations, and cleanup latency. Labels must not contain raw tenant IDs or prompt text. Cross-scope acceptance, wrong-version acceptance, and actual use of old generations must remain at zero; a high hit rate cannot offset them.

For rollout, first observe candidate hits without using them, then enable tenant-scoped caching, and only then evaluate explicitly approved public prefixes. Incident rollback monotonically advances cacheEpoch, stops public sharing, and bypasses the cache. Cleanup by namespace may be performed when necessary, but correctness must not depend on cleanup completing immediately. During recovery, warm the new epoch again and never re-enable old objects. Performance may temporarily decline, but response and authorization semantics must remain unchanged. That is the acceptance criterion for a cache that can be disabled safely.

Further Reading

• Prompt Caching: A Deep Dive
• KV Cache Admission and Eviction
• Memory Consolidation Lineage
• MCP Multi-Tenant Isolation