Prompt Caching: Reusing KV Computation for Shared Prefixes

Each round of Agent conversation carries nearly identical system rules and tool schemas. If each round has the GPU process all this input from scratch, computation is wasted on exactly the same content. To see where the waste occurs, first unpack the autoregressive serving process: it first processes all input tokens in parallel, generating corresponding K/V intermediate tensors for each layer; this step is called prefill. Then it generates output token by token, which is called decoding. Adjacent requests often share the system prompt, tool definitions, and early conversation history. These token sequences are identical verbatim, but the default implementation still recomputes their K/V for each new request.

Cross-request prefix caching addresses exactly this repeated computation: save the intermediate tensors already computed for a request. When a new request arrives, first find which saved prefix its input matches. Once found, continue prefill directly from the end of the shared prefix, computing K/V only for the new suffix tokens. In this way, the repeated part only needs to be computed once, and all subsequent requests reuse it.

What is stored in the cache is not an archive of natural language answers, nor a semantic retrieval result index. It is valid only for the exact token prefix under a specific model, specific tokenizer, specific weights (including adapters), and specific positional configuration. Change the model version, change the tokenization method, or even have the same text cut into a different token sequence, and the cache cannot be reused directly.

The main benefits are on the input side: reduce repeated input computation, lower first-token latency (TTFT), and release throughput originally occupied by repeated prefill for new requests. The boundary is equally clear: only input prefill is skipped. The token-by-token decoding process for long outputs is completely unchanged—still generating one token at a time. There is no “cache acceleration” for the output part.

From the input-output relationship perspective: the input to prompt caching is the model configuration and the precise token prefix shared by multiple requests; the output is the intermediate KV state that can be reused by subsequent requests. The server-side processing order is—first find the longest available cached prefix, then compute only for the new suffix. A “hit” only means that some input computation was saved; it does not mean the answer was cached, nor does it guarantee that the output is necessarily correct. It only applies to requests whose model, tokenizer, positional configuration, and security domain are consistent; if any dimension is inconsistent, the cache should be considered unavailable.

In prompt processing systems, “cache” refers to at least four mutually distinct mechanisms: intra-request KV cache, cross-request prefix cache, response (semantic) cache, and application data cache. It is dangerous to mix them under a single “cache hit rate” number, because the latter two types no longer store pure computational state but content that may be stale or unauthorized. To distinguish them, you only need to answer three questions: What object is saved? In what scope is it reused? Which computation stage is skipped? Using the answers to these three questions as classification input, the output falls into one of the four types.

Intra-request KV cache stores the key-value state of historical tokens already computed during the same generation process. The reuse scope is strictly limited to the current sequence itself; the hit condition is that these states belong to the sequence being decoded, and what changes is the token-by-token decoding phase. It is the purest form of compute reuse: when decoding each subsequent token, it no longer recomputes the entire prefix, and the saved and reused material never leaves this request, carrying no cross-request semantics.

Cross-request prefix cache stores input state corresponding to the common beginning of different requests. Its reuse scope is across requests; the key is composed of the model plus the exact token prefix; and what changes is the input prefill phase. It still reuses computational state, but generalizes “already computed for this request” to “computed in a previous request and identical token-by-token from the beginning.” Whether it hits is determined by prefix consistency before the answer is produced, so it does not introduce content-level correctness issues.

Response cache, also called semantic cache, stores the final answer to identical or similar questions; its key consists of the request itself or its semantic representation plus version, and it may skip part or all of the generation phase. Application data cache stores retrieval results or tool call results; its key consists of the query plus permission and freshness constraints, and the skipped phase is in an external component. The common point of these two types is that the saved and reused object has become “content.” When question semantics are similar but the asker is different, permissions differ, or external data has changed, directly returning cached content may produce a stale or unauthorized answer; the correctness risk is far higher than the pure computation-state reuse of the first two types.

For this reason, these four mechanisms cannot share the same hit-rate measure: a prefix cache hit means a batch of prefill computations was saved, while a semantic cache hit means an old answer was returned directly; the benefits and risks of the two are completely opposite. To determine which type a specific cache belongs to, look at what it reuses—current generation history, common input state, final answer, or tool result. From this you can explain where its benefits come from and also clarify its correctness boundaries.

How much prompt caching can reuse depends on the length of consecutive identical tokens from the first token in two token sequences—that is, the longest common prefix (LCP). Given two token sequences A and B, the LCP output is the length ℓ, indicating the number of consecutive tokens that are identical token by token from the first token. The system compares token by token and stops at the first divergence; the KV after the divergence point cannot be directly reused. This explains why two prompts that contain the same 8000-token document can have a hit rate of zero if the order differs: in causal attention, the representation at position i depends on all previous tokens. When the document is moved to a different position, the input context for every subsequent position changes, and the entire KV chain from the divergence point onward becomes invalid.

The common prefix must be exact at the token level, not at the semantic level. Semantic equivalence does not mean token equivalence: any difference in JSON field order, whitespace, Unicode normalization, template version, or tool ordering changes the token sequence, shortening or even eliminating the common prefix. Some radix-tree-based systems can search for the longest hit among many stored prefixes, so the cache does not need to match the entire prompt one-to-one, but the search target remains the condition of 'a prefix that is exactly identical from the first token'; the hit rule has not changed.

This rule directly determines prompt layout: do not put dynamic values at the very front. Once a request ID, current time, or user question appears at the beginning, all stable content that follows—system instructions, tool definitions, knowledge documents—will lose reuse because the prefix is broken. Putting static content first and dynamic content later is the basic way to lengthen the common prefix.

Note that a long LCP only indicates that computational state can be reused; it does not imply that the two prompts are semantically closer. Two completely different questions can share a very long common prefix (for example, the same system prompt part), while two semantically almost identical questions may have zero cache hits due to a single token difference. Any change affecting tokens, positions, or model state—changing a character, inserting a line, or switching model versions—will terminate the hit at the corresponding position.

A typical 12k-token request illustrates the whole process of layout and benefit estimation. Assume the prompt consists of five segments: system rules 2k, tool definitions 3k, shared policy 5k, conversation history 1k, current question 1k. Without any layout, dynamic content may be sandwiched in the middle or placed at the beginning, and the common prefix gets truncated; after sorting by stability, large shared content comes first, dynamic history and questions come later, and the longest common prefix has a chance to reach 10k—so the 12k-token prompt is split into a 10k cache-hit region and a 2k new suffix.

The core formula for savings is the expected number of input tokens to avoid processing, namely the hit rate h times the hit length L. h is the proportion of requests that can hit the prefix, and L is the length of the hit tokens; multiplying the two gives the average avoided amount. This formula requires first arranging a sufficiently long L by stability, then weighting with the hit rate h—talking about the hit rate without the hit length, or about the longest prefix value without the actual hit proportion, will overestimate or underestimate the benefit.

Latency benefit can be estimated with the same set of assumptions. Without caching, prefill accounts for 600ms of TTFT; after a hit, the 10k/12k prefill computation is saved, but you pay 60ms for cache lookup and transfer. The rough saving for a single hit request is 600×(10/12)−60=440ms: first convert the saved prefill time according to the ratio of hit length to input, then subtract the fixed overhead of the cache itself. Then, with the hit rate applied, the expected saving across all traffic is about 440×70%=308ms. By the same logic, the average number of saved tokens is 10k×70%=7k.

The 440ms and 308ms here are estimates under given assumptions: prefill 600ms, hit length 10k, hit rate 70%, lookup and transfer 60ms. Real attention computation cost grows nonlinearly with length, and the actual overhead of lookup, transfer, and invalidation checks varies by implementation, so any pre-deployment conclusion must be based on actual measurement. The purpose of this example is to give the estimation sequence: first determine L by arranging for stability, then use h×L to find the average avoided token count, and finally convert to latency and subtract cache overhead.

The goal of layout and normalization is not to maximize hit rate, but to make the common prefix as long as possible without breaking semantic and permission boundaries. The basis for ordering is the stability, permission domain, and semantic order of each segment: stable and public content goes first, dynamic and private content goes later.

The specific order can be organized like this. Put system rules that are stable across model and application versions at the very front; then put tool schemas with a fixed order and shared few-shot examples; after that put versioned public documents or policies. Only after that come conversation history, retrieval evidence, and the current input. Values that change on every request, such as timestamps, random nonces, and trace ids, should either go into metadata that does not participate in model computation or be placed at the very end, to avoid breaking the prefix.

Serialization must also be deterministic: when serializing objects, fix the field order and tool order, and unify whitespace and Unicode normalization. These differences are semantically invisible, but they can change the token sequence enough to make an otherwise identical context miss. The better the normalization, the greater the chance of exact prefix reuse.

But layout has a hard boundary that cannot be crossed: to improve hit rate, you must not promote user-specific permissions or secrets into a shared prefix, nor move content that should be private into a public area. Performance optimization is subject to data boundaries; any move that increases reuse must simultaneously preserve the meaning of security rules and tenant isolation. Semantic order is also constrained: moving security rules earlier is good for caching, but once the overall semantics of the template change, you must re-evaluate security effectiveness rather than only looking at how much hit rate has increased. The output of layout is a sequence with stable public content first and dynamic private content later, and its correctness criterion is always that the meaning of security rules and tenant isolation both hold.

A 10k-token prefix cannot reside on the GPU indefinitely, and the reason can be seen directly from the KV cache size formula. In a rough estimate, the KV byte count MKV is proportional to 2 × number of layers Nlayer × number of KV heads NKVhead × head dimension d × number of tokens Ntoken × bytes per element b. The factor 2 comes from saving K and V each once. Taking 32 layers, 8 KV heads, 128 dimensions, FP16 (2 bytes per element), and 10k tokens as an example: 2×32×8×128×10000×2 = 1,310,720,000 bytes, about 1.31 GB—that is just one prefix segment of a single request. Multiplied by the number of concurrent requests, the footprint competes with model weights and active requests for GPU memory. Different architectures, tensor parallelization methods, and quantization will change the specific numbers, but the magnitude shows that prefix caching is by no means free.

Therefore, the output of capacity planning is not just the byte count, but also the choice of storage tier; the inputs are number of layers, number of KV heads, head dimension, number of tokens, bytes per element, and concurrency. Several typical strategies each have costs: GPU resident caching has the fastest hits but is expensive and squeezes concurrency; moving to CPU or remote storage has large capacity, but the transfer overhead may exceed the recomputation itself; TTL/LRU is simple to implement but does not understand prefix value and may evict the most valuable long prefixes; value-aware eviction based on 'saved compute divided by occupied bytes' can retain long and hot prefixes, but valuation and implementation are both complex. In engineering practice, common approaches include using block partitioning or PagedAttention to manage memory, prefix trees to organize shared structures, and then layering TTL, LRU, or value-based eviction to trade off hit speed, capacity, and concurrency.

If the text prefix has not changed, that does not mean the KV cache can still be reused. Change a LoRA or adjust the RoPE configuration, and hidden states will shift as a whole; at that point, directly reusing the old cache is equivalent to making the new configuration read old state. Any factor that may affect hidden states—weights and adapters, tokenizer, positional encoding, attention implementation, precision, model template, and various inference configurations—must enter the cache namespace, or directly trigger invalidation, whenever it changes. Prompt, tool, and policy version changes also change the token sequence itself, and likewise need to be reflected in the key through version fields.

Cache keys are typically composed of model snapshot, adapter identifier, token sequence hash, tenant or security domain, start position, and so on. After a cache hit, you can also sample and recompute some results for comparison, to catch implementation-level errors. During rolling releases, old and new caches must be isolated: v42 must not read state left by v41; isolation relies on version fields entering the key and namespace.

This design exists because cache pollution propagates silently. If an erroneous state is reused at high frequency, the impact is far greater than a single one-off inference error, and it is difficult to trace. Therefore, in addition to hash and version validation, the system should retain a fast full invalidation switch, so that when signs of pollution appear, it can immediately disable the entire cache.

The inputs to the invalidation design are model snapshot, adapters, tokenizer, positional configuration, templates, and tenant domain; the outputs are cache namespaces, keys, and invalidation actions. If any one of these changes, the same text may produce different KV caches. When version fields are incomplete, the correct approach is to refuse reuse and recompute as a miss, rather than risk reading old state.

KV looks like just numeric tensors, not readable text, but it is a derived representation of the input and may contain sensitive information, so it must be governed as sensitive data. The risk is not limited to the content itself: when sharing caches across tenants, an attacker can infer from hit latency whether a certain prefix is already present in the cache, creating a side channel; if the cache key lacks permission fields, content that should not be shared may also be reused.

The default practice is to isolate by tenant, organization, or data classification, and to keep public system prefixes separate from user private history. Caches need retention periods, encryption, access auditing, and deletion propagation; logs record only the hash and length of the prefix, not the original text. When sharing prefixes such as public policies, first confirm that all tenants hold the same version and have the same access rights; the service provider's caching policy itself is part of the data processing agreement and needs to be specified at the agreement level.

User-controllable prefixes must not be used to directly address shared caches. Such designs must be paired with domain isolation, full hashes, and length or quota limits to prevent hash collisions, prefix probing, and resource exhaustion attacks.

The inputs to security governance are the prefix's data classification, tenant, retention period, and access rights; the outputs are isolation domains, encryption, audit, and deletion policies. The core conclusion is: KV is derived data of the input; hit timing can leak “whether a certain prefix exists”, so cross-tenant sharing is limited to genuinely public, version-consistent, permission-consistent versioned content.

Hit rate rose from 40% to 80% while TTFT did not improve; this is not contradictory. There are three possible explanations: the hits were all just very short prefixes, so the prefill savings were negligible; cache lookup or cross-machine transfer took longer than recomputation itself; requests were routed to cache nodes, and queueing time offset the compute savings. Therefore evaluation cannot look only at hit rate; it must look at request-level hit length distribution, saved prefill tokens and time, lookup and transfer time, GPU KV occupancy, eviction behavior, TTFT p50/p95/p99, throughput, cost, and quality—these are what constitute net benefit.

The premise of A/B comparison is comparability: the prompt semantics and routing load of the two groups must be consistent; otherwise the change in hits is just an artifact of layout differences. Results must also be sliced by tenant, length, and concurrency to avoid averages masking actual degradation for certain traffic. Failure paths must also be included in testing: version mismatch, cache node loss, cache avalanche, cold start, delete propagation, and cross-tenant access—any one of these failures can reduce online benefits to zero or even become an incident.

The value metric truly useful for decision-making is the TTFT or computational savings per GB·hour of cache, and the additional serviceable throughput within SLO. These correspond directly to cache occupancy and business capacity, and are closer to decision-making than raw hit rate.

The inputs to cache evaluation are request-level hit lengths, computation and transfer time, GPU memory usage, latency, throughput, and quality; the output is net benefit sliced by tenant, length, and concurrency. When hit rate rises but TTFT does not drop, it usually means hits are too short, or lookup, transfer, and queueing offset the benefit.

The complete causal chain of prompt caching can be strung into a path from problem to acceptance. The starting point is measurement: first confirm the share of prefill in total latency and how many repeated token prefixes appear in traffic. Only if both are actually present does caching have room for benefit.

The second step is to reorder context by stability and permissions, placing stable public content first and dynamic private content later, so the longest common prefix has a chance to grow. The third step is to construct the cache key, composed of configuration, tenant, and token prefix, ensuring that identical text under different model versions or different tenants is not incorrectly reused. The fourth step is value-aware storage and hit routing, deciding which prefixes stay resident on GPU, which are demoted to CPU or remote storage, and which are evicted based on "saved computation divided by bytes occupied". The fifth step is atomic invalidation on version changes: if any of weights, adapters, tokenizer, position configuration, or template changes, immediately deactivate the corresponding namespace to prevent old state from being silently reused under the new configuration.

The final step is joint acceptance: look at hit length, TTFT, GPU memory usage, and quality together, rather than using a single hit rate as a substitute for the conclusion. Each link in the chain is conditional on the previous one—without measurement we would not know what to reorder, without correct keys and invalidation there is no safe reuse, without value-aware storage capacity cannot be maintained, and without joint acceptance net benefit cannot be proven.