Can an Active Request Migrate Between GPU Replicas?

An active generation cannot be honestly moved to another GPU replica just because the gateway has the same prompt and the same model. Once decode begins, the original replica holds private request state: the KV cache for the context processed so far, the position in the sequence, the sampler state, and, in some configurations, the speculative decoding state. If that replica disappears, the new one does not know which token should have come next.

Active request migration must therefore mean one of two different things: rerunning the computation before the first token is sent, or terminating an already started stream correctly. Trying to hide this boundary behind an automatic retry usually creates a worse failure: the user receives a repeated fragment, an incomplete JSON object, or two different answers to one action.

Decode cannot continue without request-specific state

After prefill, the engine stores attention keys and values for every layer in the KV cache. Decode uses this cache to calculate the next token without processing the entire context again. This is not a shared model snapshot that can be restored from a request_id. It is mutable state belonging to one sequence in the memory of one worker.

For genuine continuation, the new replica must receive a consistent snapshot of at least the following:

• the KV cache for all accepted and generated tokens;
• the exact position and attention masks;
• sampler parameters and random generator state;
• the identifiers of the weights, tokenizer, chat template, and connected adapters;
• the draft-model state if speculative decoding is enabled.

Losing even one part turns «continue» into a new run. Sometimes a new run looks similar, especially with temperature=0 and a short answer. That is not a guarantee. Parallel execution, different kernels, quantization, different engine revisions, and MoE routing can change token selection even when the team expects deterministic behavior.

The vLLM documentation describes automatic prefix caching as reusing the KV cache for new requests with a matching prefix. This is useful for rerunning prefill, but it is not the same as moving a live sequence between workers. The SGLang documentation and RFCs separately address remote and hierarchical KV cache storage because one worker's local cache is not automatically available to the rest of the pool.

Do not call regeneration «continuation» in the API, logs, or user interface. This blurs the contract precisely where you may later need to investigate a disputed payment, a tool call, or an incomplete structured response.

Repeating prefill is safe only before the first client delivery

A gateway can rerun prefill on another replica if it knows two things: the original worker will no longer execute the request, and the client has not received any result fragment yet. At that point, the retry changes latency but not the observable semantics of the request.

The boundary must be based on delivery, not on the internal fact that generation has begun. A worker may already have calculated ten tokens while the gateway has not yet written the first SSE event to the client's socket. In that case, retrying is acceptable. Conversely, one sent delta already makes an automatic retry dangerous, even if the worker fails immediately afterward.

It is useful to store the phase separately from the HTTP status:

{
  "request_id": "req_01J...",
  "phase": "prefill",
  "attempt": 1,
  "first_delta_sent": false,
  "model_revision": "model-x@sha256:...",
  "retry_budget": 1
}

After sending the first fragment, the gateway records the transition:

{
  "request_id": "req_01J...",
  "phase": "decode",
  "attempt": 1,
  "first_delta_sent": true,
  "delivered_output_tokens": 37
}

This log is not needed for a pretty observability dashboard. It answers a specific question during an outage: does the router have the right to send the same input to another replica? If first_delta_sent=false, it does. If it is true, the gateway must end the stream with an error unless it has a tested mechanism for transferring the complete request state.

Do not substitute status=200 for this signal. In streaming HTTP, the server can send 200 headers and then fail before the first meaningful chunk. It can also write the first chunk to a proxy buffer that the client never sees because the connection breaks farther down the chain. Strictly guaranteeing that «the client received nothing» requires more than router telemetry. In most systems, a more conservative rule is sufficient: once the first write to the downstream connection occurs, retries are forbidden.

Distinguish a failure from a slow response

Not every missed heartbeat means that a GPU replica has been lost. Automatic failover after a short timeout often creates two active attempts: the old replica comes out of a pause and continues working while the new one is already retrying the request. With non-idempotent tool calls, this is no longer a text problem but a duplicate business action.

The router should distinguish at least four states.

1. The replica is clearly dead. The process has exited, the orchestrator reported a lost pod, or the upstream connection closed. Before the first token, the request can be retried.
2. The network between the gateway and the replica has failed. The replica may continue decoding. Retry only after cancellation is confirmed or a hard lease expires, at which point the original attempt no longer has the right to publish a result.
3. The replica is overloaded. Long prefill, a queue, or the construction of large batches is not the same as a failure. Moving a request because of a soft latency timeout increases load and can take down the pool.
4. The downstream connection to the client has failed. The result is no longer needed, but the upstream may still be using a GPU. The gateway should send a cancel to the original worker and release the slot instead of starting a backup attempt.

An internal lease makes this scheme manageable. The router gives an attempt an identifier and a publication deadline. Before sending each chunk, the worker must check that the lease is still valid. During a switch, the router revokes the lease from the old attempt and then creates a new one. This does not restore already lost tokens, but it reduces the chance that two replicas will write to the same user stream at once.

Do not confuse a cancellation request with proof that cancellation occurred. An HTTP cancel request sent to an overloaded worker may not arrive or may arrive too late. Until the system receives confirmation or the lease expires, the old request must be treated as potentially alive.

Prefix cache reduces repeated work but does not move a session

Rerunning prefill does not have to be expensive if the new replica can find a shared KV cache for the common prefix. The system prompt, tool descriptions, security policy, and first part of a long history often match across attempts. A prefix cache allows the engine to compute only the missing tail.

But this often leads to the wrong conclusion: «There is a distributed KV cache, so decode can be migrated.» A shared cache usually indexes completed blocks or blocks that can be reused for an input sequence. Active decode changes the sequence after every token. Its cache may be tied to the request, stored in GPU memory, use a format specific to the attention backend, and depend on how the model is split across devices.

SGLang is actively developing prefill/decode disaggregation and KV cache transfer between these roles. Its roadmap describes delta KV transfer for shared prefixes in agentic scenarios, while releases separately mention decode-side prefix cache. This is an execution-architecture optimization, not a promise that the death of a decode worker in the middle of a response will be seamless.

Check cache compatibility before treating it as portable. Matching model names are not enough. The following must be identical:

• weight revision and quantization format;
• tokenizer and chat-template application rules;
• attention architecture and KV cache dtype;
• tensor parallelism, pipeline parallelism, and block layout;
• the set of LoRA adapters and their order.

If even one parameter differs, do not mix the cache in the hope that it will work. A cache error is worse than slow prefill: it can produce a plausible but incorrect answer. Public SGLang reports include an example of KV cache corruption at a block boundary where the same prompt with temperature=0 produced different sequences. That is a good reason to treat cache correctness as part of resilience testing, not merely as a latency optimization.

A streaming protocol needs a contract for partial responses

The client should not have to guess whether the response ended, whether an action can be retried, or what to do with text already shown. The OpenAI-compatible SSE format is convenient for integration, but on its own it does not provide a reliable way to resume generation from token N.

After a decode failure, the gateway should close the stream with an explicit reason. If the format supports a final control event, it should include request_id, partial_output=true, retryable=false, and a reason code. If the connection has already been physically closed, the client will see a network error. The same data must then be available through the request log or included in a client retry request.

The contract can look like this:

{
  "error": {
    "code": "upstream_decode_interrupted",
    "message": "Генерация прервана после отправки части ответа",
    "request_id": "req_01J...",
    "partial_output": true,
    "delivered_output_tokens": 37,
    "retryable": false
  }
}

Here, retryable=false does not mean that the user has to give up. It means that the gateway is not allowed to retry the original request invisibly. A user-facing client can offer a «Continue» action by creating a new request with the displayed text. That is a sensible path for chat. For a JSON response, the client should usually discard the partial object and request generation again with the same input but as a new logical action.

Do not concatenate new text with old text in the router. Even if the retry starts with the same fragment, differences in punctuation, a tool call, or a closing bracket can make the result unreliable. This is especially dangerous with structured output: the first 90 percent of a JSON object may look valid, while one repeated key turns it into an invalid string.

Request idempotency and action idempotency are different

Teams often add an Idempotency-Key and consider the problem solved. This header helps match client retries to one logical request. It does not make generation itself safe to retry after partial output.

You need to distinguish three objects:

• the user's logical request, such as «prepare a payment order»;
• an inference attempt, tied to a specific replica and lease;
• an external action, such as calling a function, sending an email, or charging money.

For plain text completion, the gateway can retry an attempt before the first token. For an agent request with tools, the rules are stricter. If the worker managed to call a tool and then died before producing the final text, the full request cannot be retried without deduplication on the tool side. Otherwise, the model may create a ticket, send a notification, or execute an operation a second time.

The tool must accept its own idempotency key derived from the logical action, not from the GPU attempt. For example:

logical_request_id = req_01J...
tool_call_id = call_07
idempotency_key = req_01J...:call_07

After failover, the new worker can learn that call_07 has already been executed and receive the previous result instead of making a second call. Without this, even perfect KV cache migration cannot protect the system from duplicates.

State transfer is justified only with compatibility designed in advance

Continuation of decode is sometimes possible. But it requires a protocol built into the inference runtime, not a general-purpose router that suddenly tries to outsmart the engine. The runtime must transfer or replicate KV blocks, sequence metadata, and sampler state between nodes known to be compatible, record a consistent snapshot boundary, and recover without racing with decode that is still in progress.

This is expensive. Transferring KV state for a long context requires network capacity, memory, and consistency controls. Synchronous replication of every new token adds work to the most latency-sensitive part of the process. Asynchronous replication leaves a loss window: the primary may have sent tokens to the client while the standby has not yet received the corresponding snapshot.

Therefore, do not put «continuation after a GPU failure» in the SLA until you have answered four questions:

1. Which exact data is transferred between replicas, and when is the snapshot considered consistent?
2. Do the model versions, cache layout, adapters, and decode parameters match?
3. How does the router prevent the old attempt from publishing tokens after failover?
4. How does testing prove that there are no duplicates or gaps when a failure occurs at every point in the token cycle?

If the answer to any of these sounds like «it should usually work», there is no continuation. There is a best-effort retry that must not be presented as reliability.

The router policy should be short and strict

A good failover policy does not try to squeeze a retry out of every failure. It prohibits dangerous actions and leaves a small number of allowed transitions.

on_upstream_failure:
  before_first_delta:
    require: [source_attempt_fenced, retry_budget_available]
    action: retry_prefill_on_healthy_replica
    max_attempts: 2

  after_first_delta:
    action: terminate_stream
    client_error: upstream_decode_interrupted
    automatic_retry: false

  uncertain_source_liveness:
    action: fence_source_attempt
    wait_for: cancel_ack_or_lease_expiry
    then: retry_only_if_no_delta_was_sent

max_attempts: 2 is not a universal number. It shows the shape of the policy: retries need a budget. Without one, a large-scale GPU failure turns into a flood of repeated prefills on the remaining replicas. The router starts spending scarce compute capacity duplicating requests that are already doomed.

For long requests, add routing based on retry cost. If repeated prefill would exceed a set token or time limit, the gateway can return a controlled error before attempting it. This is less pleasant than an invisible delay, but more honest than waiting a minute and then returning a 504 after the second overloaded replica.

Test failures between tokens, not just pod restarts

The test «we restarted the worker and the service responds again» says nothing about live streams. You need a fault-injection test that kills the process at points where the router's decision changes.

The minimum set of scenarios is:

1. Kill the replica during prefill before the first downstream byte is written. Expected result: one complete response after the retry.
2. Kill the replica after calculating a token but before sending the SSE event. Expected result: retry is allowed and the client sees no fragment.
3. Kill the replica immediately after the first delta is sent. Expected result: the stream ends with an error and a second generation does not start.
4. Break the network between the gateway and the replica while keeping the process alive. Expected result: the old attempt is fenced or waits for the lease to expire, with no duplicates.
5. Kill the replica after a tool call. Expected result: the new run does not execute the action a second time.

Check more than the HTTP status. The test should compare the chunk list by request_id, the number of attempts, whether the upstream was canceled, tool logs, and the absence of simultaneous publication by two attempt_id values. Save the input, seed, model revision, and event trace, or rare failures will not be reproducible.

AI Router can preserve an OpenAI-compatible contract for clients, but internal routing must store more than base_url and the model name: the stream phase, attempt identifier, lease, and first-delivery boundary. Without this data, failover remains guesswork.

Do not promise seamless continuation if the infrastructure can only retry the request. It is much more reliable to rerun prefill quickly before the first token and clearly report an interruption afterward. This rule survives changes in models, GPUs, and inference runtimes because it is based on what the client has already seen.