Every tenant needs prefix cache isolation

Prefix cache saves expensive prefill when requests start the same way. In a multi-user LLM service, that same optimization can expose an observable signal: a known or guessed prefix was already in the cache, so the response began forming faster.

This is not reading another user's KV cache, nor is it a direct reconstruction of a prompt from GPU memory. The attacker gets narrower but still useful information: whether a specific prefix existed in a recently available cache area. For a bank document, medical record, internal instruction, or system prompt, the fact of such a match may itself reveal too much.

Prefix cache isolation must be a property of request routing, not an option you hope the client SDK will handle. Every KV cache block needs a trust scope. A request from another scope must not get a hit, even when all tokens match.

Fast prefill becomes a presence oracle

An attacker does not need to know another user's entire prompt. They take a likely beginning, such as a project name, contract template, fragment of a system instruction, or identifier found in an email, add enough tokens to reach a block boundary, and send a series of requests.

If one variant consistently produces an earlier time to first token than nearly identical variants, the attacker can form a hypothesis that it hit the prefix cache. They then change fragments and narrow the guess. The most troubling part is that this process is easy to automate and does not require access to another user's responses.

The simplified sequence looks like this:

1. The victim sends a long private context, and the engine stores the full KV blocks.
2. The attacker sends candidates with the same beginning and measures TTFT repeatedly.
3. A candidate with matching blocks skips part of prefill and produces a different latency profile.
4. The attacker compares the variants while accounting for queueing and normal response-time variation.

One fast request proves nothing. Latency is affected by queueing, continuous batching, cold loading, block eviction, output length, and network behavior. But a repeatable difference between a control group and a test group may be enough for a classifier. Do not reassure yourself that network noise makes the attack impossible. Noise makes the experiment harder, but it does not change the type of leakage.

vLLM documentation describes this risk directly: cache reuse can be inferred from latency differences, while cache_salt limits reuse to requests with the same salt. This matters more than the usual claim that the KV cache lives in process memory and is not visible to the user. The channel runs through service behavior.

Block hashing and tenant boundaries solve different problems

Engineers often see SHA-256 in prefix cache keys and consider the problem solved. A strong hash is needed so that different token sequences do not practically receive the same key by chance or because of a weak function. It does not stop two tenants from receiving the same key for identical text.

These are two separate protections:

• The hash binds the specific tokens and the previous block to a KV cache key.
• The cache scope determines who is allowed to reuse that key.
• A salt turns identical tokens into different keys for different scopes.

In a hash-based implementation, a block usually depends on the tokens in the current block and the hash of its parent block. It is therefore enough to add the salt to the first protected block: it changes that block's hash, and the chain changes the hashes of every following block. vLLM documentation describes this approach for cache_salt.

The practical rule follows from this: never add the salt to a log or to the prompt text. It is not model data. It belongs only in the internal cache key. If you insert a tenant ID into the system prompt for isolation, you change the model's behavior, spend extra tokens, and still have no proof that the engine uses that ID in the KV cache key.

The cache scope should follow the data, not the customer name

A tenant scope is not suitable for every request within an organization. One tenant may include dozens of departments, external customers, and users with different permissions. If they all share a prefix cache, one user can check whether another user's context exists inside the same company.

I use four scopes rather than one universal setting.

The policy should choose the scope based on data provenance. For example, a shared assistant instruction published for all customers may live in global-public. An attachment from a personal account should receive at least user. I usually classify context containing an account number, diagnosis, HR information, or contract text as user or request immediately, even when the user works in a corporate tenant.

This distinction is often blurred: “everything can be cached within a customer.” No. A customer agreement does not give every employee the same right to learn which documents their colleagues recently processed. The data access boundary and the prefix cache boundary should match as closely as possible.

The gateway, not calling code, should assign the salt

A client parameter is convenient for a local experiment. In production it is dangerous if the user can choose the salt. An attacker could supply the victim's expected salt, a shared string such as tenant-a, or deliberately move their own requests into a broader cache scope.

The gateway should extract identity from a verified API key, JWT, or mTLS certificate and then build a server-side cache identity. This usually includes the tenant, the selected scope, and a policy version. The version prevents old and new cache semantics from mixing after policy changes.

Here is an example of the logic in Python. It is not a ready-made library, but a minimal template for architecture review.

import base64
import hashlib
import hmac

CACHE_SECRET = b"stored-outside-application-config"


def make_cache_salt(tenant_id: str, scope: str, principal_id: str | None) -> str:
    if scope not in {"tenant", "user", "request"}:
        raise ValueError("scope is not allowed for private content")

    subject = principal_id if scope == "user" else "-"
    material = f"cache-v3|{scope}|{tenant_id}|{subject}".encode()
    digest = hmac.new(CACHE_SECRET, material, hashlib.sha256).digest()
    return base64.urlsafe_b64encode(digest).decode().rstrip("=")

For request, do not use the function above without an additional request or data-object identifier. Otherwise you will get reuse at the tenant level under a different label. It is safer to add a random request_nonce generated by the server and never accepted from the client.

After authentication, the gateway should discard the cache_salt field if the client supplied one and set its own value. It is also useful to record safe audit fields rather than the salt itself: scope=user, the policy identifier, and the fact that a salt was assigned. A raw salt in tracing turns an internal separator into a portable secret.

Portable context requires a narrower boundary

Most leaks do not appear in a simple chat, but in routes where the application assembles a long prompt from several sources. RAG, tools, agent loops, and system templates create context that looks shared even though it is made from private parts.

Separate the prompt by provenance before it reaches the engine:

• A public, immutable instruction can remain without a private salt.
• Tenant documents receive a tenant salt.
• ACL-filtered excerpts and chat history receive a user salt.
• A secret uploaded for one operation receives a request salt.

One cache_salt for the entire request provides simple and reliable isolation, but reduces reuse. If a private document follows a shared instruction, a salt on the first block will also close off the shared portion. That is a reasonable price for a first secure release.

A more complex design places a salt barrier at message or segment boundaries. The shared system instruction can then remain in a common scope, while the chain after the first private message becomes available only to the appropriate group. An RFC from vLLM on cache salting described this hierarchy: an organization can share its document within a tenant, while the next barrier limits reuse to one user. Use this approach only if the engine genuinely supports segment barriers. Otherwise the gateway will create an attractive policy that the runtime does not enforce.

A shared tenant salt must not be guessable

The string acme-prod is convenient for debugging, but it is poor secret material. If the salt is known or derived from a public tenant ID, it still separates scopes when the gateway behaves correctly. But it offers no additional protection if a client-controlled salt path remains somewhere in the system.

It is better to use an HMAC result with a server-side secret. It has several properties needed here:

• The client cannot calculate another tenant's salt.
• The salt does not reveal the tenant ID in a cache-key dump.
• The gateway can reproduce it on any replica without a shared store containing millions of random values.
• Changing the version or secret creates a new scope and naturally invalidates old hits.

Do not confuse HMAC with the primary protection. The most important rule remains the same: only a trusted layer chooses the scope. HMAC simply makes the implementation less fragile.

For multiple replicas, the replicas that can share an external or distributed KV cache need the same secret. If the cache is physically local to each replica, consistent salts are still useful for correct semantics, but they do not by themselves create cross-replica reuse.

Test the channel with an experiment, not a hit-rate chart

The hit rate tells you about cost and performance, but it does not prove isolation. You need a separate test in which one subject warms a known long prefix and another tries to detect it through latency.

Run the test on a setup with the same model, context length, generation parameters, and batching scheme as the production route. The control and test requests should differ only in cache scope. Otherwise you will measure a side effect of the template rather than the tenant boundary.

A practical protocol looks like this:

1. Create two tenants, red and blue, two users in red, and separate API-key sets.
2. Choose a long synthetic prefix that crosses several full blocks and warm it from red/user-1.
3. Send this same prefix in batches from red/user-1, red/user-2, and blue/user-1. Add a similar but non-matching control variant.
4. For each request, save server-side TTFT, prompt length, selected scope, queue information, and a cache-hit indicator without the prefix text.
5. Compare the distributions. Reuse should occur only in groups that the policy explicitly allows to share a scope.

The expected result is straightforward: red/user-1 may show a speedup after warming the user scope. red/user-2 and blue/user-1 should not get their own fast group merely because they repeated the same context. If red/user-2 speeds up, you configured tenant scope where you expected user scope, or the engine is ignoring the salt.

Do not rely on the average. A few requests may land in a quiet queue and artificially improve it. Look at quantiles, run counts, launch order, and repeatability when the group order changes. It is better to run requests on a shuffled schedule so that warming and eviction do not line up with one test group.

Random delay cannot equalize response time

Sometimes a team suggests adding jitter to responses to hide cache hits. It is a popular idea because it is easy to implement in a gateway. It does not solve the problem.

An attacker can average more measurements, while you worsen latency for every legitimate user. With a strong enough signal, random delay reduces accuracy but does not prohibit cross-tenant matches. You are hiding observability while leaving the cause inside the system.

A fixed minimum delay looks better, but it also has a cost: the service waits even when the model is already ready to return the first token. The difference may also appear in prefill duration, queue load, GPU consumption, or metrics available through another interface.

Isolating cache keys removes the reuse that creates the signal. If the risk remains high because of other system properties, disable prefix cache for the relevant data category. But do not present artificial delay as access control.

The leak often starts in a neighboring service

Even a perfect salt in the inference runtime will not help if another layer reuses data more broadly than the policy allows. Check more than the GPU KV cache.

Embeddings caches, retrieval results, prepared message templates, tool-calling responses, task queues, and observability pipelines deserve special attention. For example, the backend may correctly use user scope for the KV cache but store a retrieval result under a key made from the query text without a tenant ID. The user will not see a timing signal. They will receive another user's fragment directly.

A single cache-identity builder reduces the risk of inconsistencies. Each cache chooses its own fields because the data and lifetime differ, but tenant and ACL context must not disappear along the way. Name them explicitly in code: retrieval_scope, response_scope, kv_scope. The term cache_key without a stated scope makes it too easy to hide a mistake during review.

A separate policy for the API gateway is especially useful. AI Router can assign a cache scope before model routing while preserving one OpenAI-compatible interface for the application. But the gateway cannot guess whether a document is shared across a tenant or personal. The application must pass the data classification through a trusted server-side contract, and the policy must reject an unsafe scope.

Measure performance after choosing the trusted scope

Isolation reduces the number of potential hits, and that is expected. You cannot first maximize the global hit rate and then try to cover the consequences. That order leads to a global cache that quickly becomes a hidden database of indicators about other people's data.

Define the allowed scopes first, then measure reuse within each one. The loss is often smaller than expected: multi-turn conversations, repeated requests for the same document, and shared instructions within one tenant already provide plenty of locality. If tenant-wide reuse is barely needed, do not expand the scope just for a better-looking metric.

A good policy sounds boring and specific: public prefixes are shared only after verification; corporate materials do not cross tenants; personal documents do not cross users; one-time secrets do not survive the request. If the team cannot express these rules as tests, it does not yet control who receives a speedup or why.

Open the trace of one long request today and answer one question: which exact component assigned its prefix cache scope? If the answer is not present in the code and audit trail, the salt is not protecting tenants yet. It is simply sitting somewhere in the configuration.