A Shared KV-Cache Between Replicas Without Self-Deception

A shared KV-cache between replicas saves GPU time in only one case: the remote block arrives and becomes available to attention before the replica would have finished recomputing the same prefix. Everything else, including a pretty cache hit-rate graph, is secondary.

In practice, teams often count only the tokens saved by reuse. Then they enable a shared layer and get extra fabric load, eviction of active cache, and a long TTFT tail. The problem is not the idea of a shared cache itself. The problem is treating it as free deduplication when it is really an exchange of compute for bytes, queues, and memory.

A local prefix cache must be separated from a shared one. A local cache barely uses the network and should be the first step. A shared layer is useful when the same long prefix consistently lands on different replicas, after eviction, or on a newly started instance. vLLM documentation explicitly distinguishes KV transfer between instances through a KV connector from local storage. For a failed load, it offers a choice between recomputation and request failure. For an interactive service, recomputation is usually safer than returning an error.

First separate three different kinds of reuse

A shared pool of KV blocks, a local prefix cache, and prefill-to-decode transfer solve different problems. If you combine them into one hit-rate metric, the payback calculation becomes useless.

Local reuse happens when a new request lands on the same replica and its initial blocks match blocks already stored there. This is the cheapest hit: there is no network read, remote catalog, or cross-machine copy. Its effectiveness depends on sticky routing, GPU cache size, and eviction rules.

Shared reuse happens when a replica finds ready blocks on another node or in shared storage. It avoids repeating prefill, but adds a read path. That path may include the source GPU memory, RDMA or TCP, receiver memory, staging buffers, and loading into the GPU.

Prefill-decode disaggregation moves KV from a prefill worker to a decode worker for one request. There may be no cross-request prefix reuse at all. You move the state because you have separated the compute roles. LMCache, for example, describes KV transfer between prefill and decode over NVLink, RDMA, or TCP, as well as separate prefix reuse between instances. These mechanisms are related, but they are not the same.

The consequence is straightforward. For a shared cache, the numerator contains avoided repeated prefill. For disaggregation, the numerator contains the gain from scaling prefill and decode separately. Do not put one architecture's benefit into the other architecture's formula.

Calculate KV size from KV heads, not the model name

The KV volume for a prefix depends on the attention architecture, storage type, and length of the matching part of the request. A model's parameter count says almost nothing about the size of one KV block.

For a dense transformer without special compression schemes, use this estimate:

bytes_per_token = 2 × L × Hkv × D × q
prefix_bytes = bytes_per_token × Tmatched

Here, L is the number of layers, Hkv is the number of KV heads, D is the size of one head, and q is the number of bytes per KV element. The factor 2 accounts for key and value. For FP16 and BF16, q = 2; for FP8 it is usually q = 1, but the actual format and metadata should come from the configuration of the specific engine.

With GQA, the number of query heads and KV heads differs. This is not a cosmetic detail. If you put all query heads into the formula, you may budget several times more network traffic than actually exists. If the model uses MLA, sliding window attention, or a custom cache representation, the standard formula does not apply either. Measure the actual allocated cache size for a short and a long prefix, then calculate the slope per token.

Blocks change the calculation. If the engine stores cache in blocks of B tokens, only complete matching blocks are useful remotely. For a match of Tmatch tokens, use:

Tuseful = floor(Tmatch / B) × B
Sload = bytes_per_token × Tuseful

The tail inside an incomplete block must be recomputed. Do not call it a remote hit just because the string prefix matches.

The minimum measurement to make before designing

Run the same request with several lengths of an unchanged prefix, for example 4K, 16K, 32K, and 64K tokens. Record actual KV allocation, prefill time, and TTFT on a cold start. Repeat at production concurrency, not with a single request.

You need a table like this:

prefix_tokens,kv_bytes,cold_prefill_ms,local_hit_ttft_ms
4096,...,...,...
16384,...,...,...
32768,...,...,...
65536,...,...,...

The difference in kv_bytes between adjacent rows, divided by the difference in token count, gives the real byte slope. The difference in cold_prefill_ms shows the actual cost of rereading the prefix on your GPU and batch size. This is more reliable than any calculator from someone else's cluster.

Remote loading must beat recomputation by a margin

Compare the full time needed to obtain usable KV blocks, not just the number of gigabytes divided by the advertised gigabits per second.

For one remote hit, use:

Tremote = Tlookup + Tqueue + Sload / Beff + Tstage + Tinstall
Trecompute = Tdispatch + Tprefill(Tuseful)

Tlookup includes hash and metadata lookup. Tqueue is the wait for a network channel, staging buffer, or remote server. Beff is the useful throughput specifically for KV transfer at the same level of contention. Tstage accounts for an intermediate copy through CPU memory when the path is not direct. Tinstall includes placing blocks in the receiver's cache and any compatibility checks.

The gain condition looks dull, but it protects against expensive mistakes:

Tremote + Tinterference < Trecompute

Tinterference is the time a foreign load takes away from active requests. It is often invisible in a single-request trace. For example, loading a long remote prefix may occupy a DMA channel or consume free GPU blocks, after which the current decode waits for memory to be released. Formally, the remote hit succeeded. The user still saw worse latency.

Do not use the network card's rated speed. On 100 GbE, the theoretical ceiling looks large after unit conversion, but the application-level stream can easily be limited by PCIe, NUMA, TCP, pinned-memory copies, message size, and competing transfers. For multi-tier offloading, vLLM separately warns that secondary tiers do not access the GPU directly: the transfer goes through the primary CPU tier. Measure that path as two segments, not as one network link.

A practical rule: enable remote loading only when its p95 at the target prefix length is clearly below the p50 of cold prefill. Equal averages are not enough. With queues and occasional spikes, the network almost always loses in the tail.

Network cost is determined by byte volume, not hit rate

Hit rate without prefix size hides the important part. A thousand hits for 256 tokens and ten hits for 64K tokens create completely different loads, even though both cases may appear as “cache hit” in a dashboard.

Count incoming and outgoing traffic separately:

Rread = λ × hremote × E[Sload]
Rwrite = λ × hstore × E[Sstore]
Rtotal = Rread + Rwrite + Rreplica

Here, λ is the incoming request rate, hremote is the share of requests with a successful remote read, and hstore is the share of requests that create blocks worth publishing to the shared pool. Add Rreplica if you copy one block to several locations or perform background replication.

The most harmful setting is publishing every completed KV block to the shared pool. Most user-specific tails are unique. You pay for writing, indexing, and eviction, but there will be no subsequent read. Publish only blocks that are likely to be reused: fixed system instructions, shared documents, stable multi-turn histories, or predefined application prefixes.

Calculate the network budget at peak as well. Take the worst fifteen-minute interval, not the daily average. Then test these load patterns:

• regular remote hits on long prefixes;
• simultaneous startup of new replicas that begin pulling the same set of hot blocks;
• a request spike where a cache miss forces both prefix recomputation and publication of new blocks;
• degradation of one storage node, when traffic shifts to the remaining nodes.

The last scenario is especially unpleasant. If remote loading becomes slower than prefill, the system can saturate the network while also making the GPU repeat the work. The GPU utilization graph may look busy, but useful throughput falls.

Measure eviction with reuse distance, not TTL

A block remains in the pool for as long as its relative popularity and available capacity allow, not for a fixed number of seconds. TTL sets an upper bound on lifetime. It does not guarantee that the cache will survive until the next request.

For each prefix or group of identical prefixes, measure reuse distance: the volume of unique KV blocks written or requested between two accesses to that same prefix. If 300 GB of unique working data passes between accesses, while the shared pool has only 120 GB actually available after reservations, the next read is unlikely under an LRU-like policy.

Effective capacity is not physical capacity:

Ceffective = Cphysical - Cactive_decode - Creserved - Cfragmentation

Do not count Cactive_decode as free. Long active generations hold blocks, and suddenly evicting them can damage service quality right now for the sake of a hypothetical future hit. Cfragmentation comes from block sizes, shard mismatches, incomplete pages, and metadata structures.

Keep separate metrics for three miss causes: the block was never published, it was published but evicted, or it was found but could not be loaded because of incompatibility or a transport error. One “miss” metric mixes a product problem with a capacity problem and a network problem.

The recommendation to “set a longer TTL” is popular because it is simple. It is usually wrong. A long TTL for unique prefixes turns the shared pool into a warehouse of one-time data, pushes out genuinely reusable blocks, and increases catalog pressure. TTL should follow the observed reuse interval, not the hope that the same document will appear again someday.

Starting a new replica changes the math more than it seems

A new replica is useful only when it can accept and process requests without creating another bottleneck. An empty local cache makes it cold. A shared KV-cache can shorten this cold period, but a poor startup policy can turn autoscaling into a mass synchronized load.

Imagine ten new decode replicas after a traffic spike. They all receive similar requests with a long system instruction and a shared context base. If the load balancer distributes requests evenly, every replica tries to read the same blocks. You have created a fan-out of reads even though one older warm replica already held the data locally.

Use different rules during the first minutes of a replica's life:

1. Limit concurrent remote loads both on the new replica and globally for the shared pool.
2. Route the first repeated requests to warm instances until the new replica has received the required base set.
3. Warm only measured hot prefixes, not random recent requests.
4. Define readiness by the ability to serve the target workload, not merely by the process being up.
5. Keep recomputation as a controlled fallback if the remote-load queue grows.

“Warming the cache” also requires calculation. If you move W bytes to N replicas within a window of Δt, the warm-up alone requires roughly N × W / Δt of useful throughput, before normal traffic is counted. If the channel cannot handle it, prefill on some new replicas may be cheaper and faster than coordinated loading.

The Mooncake work describes a KVCache-oriented architecture with separate prefill and decoding clusters and a scheduler that accounts for throughput and SLO. This does not mean every shared cache should be built the same way. It means that cache becomes a scheduler resource, not a transparent library beneath the application.

A partial hit is often better than a full one, but only with the right boundary

A long request may not share its entire prefix. The system instruction may be common, followed by the same RAG set, followed by the user's personalized history. If the engine can take the first complete matching blocks and recompute the tail, a partial hit can remove most of the prefill.

For this case, calculate separately:

Ttotal = Tremote(Tcommon) + Tprefill(Ttail) + Tdecode

Do not transfer personalized blocks just because they are long. They are poor candidates for two reasons. First, they are unlikely to recur on another replica. Second, the shared pool expands the scope of data access. The publication boundary should match the point where the data is genuinely suitable for reuse under your security and tenancy policies.

In banking, healthcare, or government environments, the idea of hashing “the whole prompt and storing it everywhere” is especially risky. A hash does not change the fact that the KV-cache encodes the processed state of the input text. Separate cache namespaces by model, tokenizer version, attention parameters, adapter, tenant, and data class. Do not try to compensate for missing isolation with a long TTL or a random key prefix.

The right boundary for a shared block is usually before user attributes: a shared instruction, an approved template, a knowledge-base version, or an anonymized document. After that point, keep the data in the local cache or recompute it.

Check compatibility before looking for gains

KV blocks are not universal results for a “similar model.” They depend on the exact model, weights, tokenizer, token positions, cache format, attention implementation, and sometimes the connected adapter. Two endpoints with the same marketing model name may be incompatible for KV exchange.

A cache key should include at least the model and revision identifier, the token hash of the prefix, position-construction parameters, KV format, block size, and tenant namespace. If LoRA or another adapter is used, include its version. If you change the system prompt, do not rely on textual similarity: a changed token at the beginning shifts the entire remaining prefix.

Also check prompt-construction determinism. Teams often blame the cache when the real cause is simpler: one service inserts the date, another changes tool order, and a third serializes JSON differently. The prompts look identical. The token sequences differ, so there is no match.

vLLM provides connectors and configuration for producer, consumer, and both roles, while its load-failure policy can be recompute or fail. This is a useful reminder: the transport path must be part of the service contract, with a timeout, limits, and an observable fallback. It should not be a mandatory dependency between generation and a remote cache.

Make the decision using expected request cost

Bring the measurements into one model before deployment. You do not need a perfect cluster simulation. You need an honest estimate that shows where the sign of the benefit changes.

For request class i, write:

E[benefit_i] = hremote_i × (Trecompute_i - Tremote_i)
               - Pstall_i
               - Pwrite_i
               - Peviction_i

E[total_benefit] = Σ λi × E[benefit_i]

Express Pstall in milliseconds of tail latency or the cost of lost throughput, but do not pretend it does not exist. Pwrite includes publishing blocks that nobody later reads. Peviction reflects the cost of evicting more useful data. If you cannot estimate the last two terms, run a load test with limited capacity and a real mix of request lengths.

A useful working experiment takes weeks at most, not months. Choose one class of repeatable prefixes, enable a read-only shared cache without publishing new tails, set a strict limit on remote loads, and collect four distributions: cold prefill, local hit, remote hit, and fallback recompute. Then increase the allowed share of remote reads until p95 TTFT and throughput show the point where the network starts doing harm.

AI Router works well for this experiment as a single OpenAI-compatible gateway: you can keep the client code, change the model route, and evaluate latency profiles separately for hosted models and external providers. But design a shared KV-cache only where you control execution, memory, and the network path, not on top of an opaque remote inference endpoint.

Do not start by buying another storage layer. First take a trace of real prefixes, calculate bytes per token, measure useful throughput, and classify misses by cause. Then it will be clear whether you need a shared KV-cache, smarter routing to warm replicas, or simply a sufficiently large local cache.