How Chunked Prefill Changes Queue Fairness

A long document rarely breaks a chat in one obvious way. It simply arrives while dozens of users are waiting for the next part of an answer and takes enough continuous GPU work to make the chat respond in bursts or stay silent for a long time before the first token.

Chunked prefill addresses this queueing problem directly. It splits the initial processing of a large prompt into portions and gives the scheduler a chance to serve decode for conversations that are already open between those portions. But there is no universal portion size. A small portion often protects interactivity, a large one pushes a file through faster, and a portion that is too small makes the GPU and scheduler spend time coordinating the work itself.

In a mixed service, the goal is not the highest average speed or an impressive GPU utilization number. You need to decide in advance who is allowed to wait: a user in a short chat, a 100-page PDF importer, or both under different, measurable promises.

Prefill and decode compete for different kinds of GPU work

Prefill builds the KV cache from the input tokens. For a long document, this phase can be a substantial compute task: the model must process the entire context once before it can produce the first token. Decode continues responses that have already started, one or several tokens per iteration, and depends more heavily on reading the accumulated KV cache.

The two phases have different profiles, but they compete for the same schedule on a shared GPU pool. If the server runs a long prefill in one piece, a new short chat message may wait for the large compute run to finish. If it splits prefill endlessly, the long file encounters many pauses and the useful work per run decreases.

The vLLM documentation describes this trade-off directly: when chunked prefill is enabled, the scheduler first places waiting decode requests and then assigns the remaining prefill budget. If a prompt does not fit within max_num_batched_tokens, the server splits it into parts. The documentation also points to the opposing effects: a smaller budget improves inter-token latency, while a larger one helps time to first token for new requests.

Two things are often confused here. Chunk size is not the same as document size. In vLLM, max_num_batched_tokens is the token budget for a scheduled iteration, which may include decode and part of one or more prefills. The actual portion of a long request depends on the remaining budget and the current batch composition.

Fairness cannot be measured with one average latency

Queue fairness means that a request class requiring less work does not receive a random penalty because of one heavy neighbor, while the heavy request is not left in the queue forever. It does not mean that all requests must finish at the same time. Chat and document uploads do different work and should have different expectations.

Two metrics are especially useful for chat. TTFT, or time to first token, shows how long the user waits for the response to begin. ITL, or inter-token latency, shows the pauses between subsequent tokens. A service can have good average TTFT and still frustrate users with long pauses in the middle of a response. For streaming interfaces, P95 and P99 ITL are usually more important than the average.

For a long file, measure time to first token, total prefill time, and useful input-processing speed:

prefill throughput = number of input tokens / time from queueing to KV cache readiness

Do not substitute generation tokens per second for this metric. A file may produce a short summary while requiring a huge prefill. If you measure only output tokens per second, you will not see that long-document imports have effectively stopped.

Finally, count queue wait time separately from model execution time. When TTFT increases, it can indicate three different problems: the request did not reach the GPU because of queueing, the request spent a long time in prefill, or the KV cache did not fit and the server started eviction or preemption. Each case calls for a different response.

A small chunk protects chat, but it is not free

A small token budget limits how long one prefill run can hold the GPU. Decode therefore gets room more often in the next iteration, and chat responses become smoother. When documents compete with chat, this usually reduces the ITL tail.

The cost has three parts. First, a long request goes through more scheduling cycles. Second, the server uses large matrix operations less efficiently if each run contains too few prefill tokens. Third, under a constant decode stream, prefill may receive only tiny pieces of the budget. The document does not disappear from the queue, but its completion stretches out.

A large budget produces the opposite picture. It can significantly improve the TTFT of a new large request and the speed of loading a file because the server processes more input tokens per run. But if one such run takes a noticeable amount of time, users will see an ITL spike. For a voice assistant or operator chat, that is often worse than importing the document more slowly in the background.

The popular advice to «set the largest value that fits in memory» is wrong for mixed workloads. Memory answers whether the KV cache will fit. Fairness depends on how much work one request class can perform before the other class gets the queue again.

The experiment should make requests compete, not average them together

Test chunked prefill by making two streams compete, not with a single request on an empty server. The first stream should imitate chats: a short context, a short answer, and regular arrivals. The second should submit long documents, with some clearly longer than your usual RAG context.

Keep the model, quantization, GPU, context limit, sampling parameters, and number of concurrent clients fixed. In one run, change only the token budget and related long-prefill rules. If you change the model, cache policy, and chunk size at the same time, the results will explain nothing.

A minimal test matrix looks like this:

Do not use identical requests for every document. A real queue has a heavy tail: many moderate contexts and a few extremely long ones. Create at least three groups, such as a short chat request, a typical RAG context, and a large document. You can fill the content with repeated neutral text if the tokenizer produces the required length. For measuring the scheduler, the meaning of the text does not matter; you need a controlled token length.

First run a baseline with chunking disabled or as large as the engine allows. This shows what head-of-line blocking looks like when decode has no opportunity to take the queue. Then compare it with several budgets. Do not look for one universal winner. You need the boundary beyond which chat stops meeting its SLO.

Log request events, not just the server summary

You need timestamps on both the client and server sides. The client should record when the request is sent, when the first token arrives, every subsequent token, and when the response ends. Server metrics should include queue size, the number of active sequences, KV cache usage, and scheduler iteration duration.

The following record format is convenient to save as JSON Lines. It does not depend on a particular SDK and lets you recalculate TTFT, ITL, and total processing time later.

{
  "request_id": "chat-00421",
  "class": "chat",
  "prompt_tokens": 420,
  "requested_output_tokens": 160,
  "sent_at_ms": 1730000000000,
  "first_token_at_ms": 1730000000840,
  "finished_at_ms": 1730000006420,
  "output_tokens": 147,
  "inter_token_ms_p95": 54
}

For a document, change class to document and save prefill_ready_at_ms if the engine or wrapper can provide it. If it cannot, do not pretend that TTFT equals pure prefill time. Name the metric honestly in the report: «time to the first output token of the document». It includes queueing, prefill, and a minimal amount of generation.

The summary table should contain at least the following rows for each run:

• P50, P95, and P99 TTFT for chats.
• P50, P95, and P99 ITL for chats.
• P50 and P95 total time for documents.
• Document input tokens per second.
• The share of time during which the document queue is non-empty.

The last row matters. If the GPU appears busy almost all the time while the document queue grows, you have not found high utilization. You have found a mode in which interactive requests systematically take the remaining budget away from background work.

GPU load shows the cause, not service quality

Average GPU utilization is useful as a diagnostic signal but poor as a final metric. You can keep the GPU almost fully occupied with long prefill and still ruin the chat experience. You can lower average utilization because of small chunks while keeping the promised latency for users. The choice must be based on product cost, not the highest number on a monitoring dashboard.

At a minimum, separate observation into four charts: utilization, GPU memory, KV cache usage, and the duration of one scheduler iteration. If ITL improves and iteration duration falls when you reduce the budget, that is expected. If the document queue grows sharply at the same time, the portion is too small for the current chat intensity.

Track memory separately from compute. Chunked prefill does not reduce the final KV cache of a long request. After prefill finishes, the document still occupies memory until generation ends or the engine frees the sequence. If a large context causes frequent preemption, reducing the chunk may smooth ITL but will not remove the memory shortage.

There is also a less obvious case: GPU utilization falls while latency gets worse. This usually indicates granularity that is too fine when only a few requests are active. Each round contains little useful work, batches do not fill up, and the scheduler transfers control more often. Adding competing requests does not fix the service in this situation; it only hides a poor parameter.

Limiting long prefills matters more than simple FCFS

FCFS seems fair because requests are handled in arrival order. In a mixed queue, this is a weak form of fairness. A document that arrived a millisecond earlier can delay hundreds of short requests even if they require orders of magnitude less computation.

vLLM provides separate parameters for chunked prefill: the long_prefill_token_threshold, the maximum number of partially processed prefills, and the maximum number of long prefills processed at the same time. The documentation states directly that setting max_long_partial_prefills below max_num_partial_prefills can allow short prompts to overtake long ones in some cases and improve latency.

A practical policy for a shared pool usually looks like this:

1. Define the «long» prompt threshold from the distribution of real inputs, not from the model's context-window size.
2. Limit the number of simultaneous long partial prefills to one in the first test.
3. Let several short prefills run in parallel if memory and the model allow it.
4. Check whether documents starve under a high, constant chat load.
5. If they do, give them a time quota, a separate queue, or a separate pool instead of simply increasing the chunk to the maximum.

This does not promise equal completion times. It is a rule that prevents one heavy category from taking over computation without control. The recent paper Fairness-Aware and Latency-Controllable Scheduling for Chunked-Prefill LLM Serving reaches the same conclusion: a static budget and strict FCFS produce an unpredictable latency tail and starvation, so priority should account for accumulated waiting time and the remaining prefill volume.

One example shows where naive tuning breaks

Imagine one GPU pool with 30 chats already running. Each is producing a short response stream. Then a document with a large prompt arrives. With a large budget, the server places most of the document's prefill into the next few iterations. Decode gets priority in the next scheduling step, but users have already experienced a stretched current iteration. Their ITL rises even though the decode queue has formally kept its order.

Now reduce the budget sharply. Chats begin producing tokens more smoothly. But the document receives only a small remainder after decode. If new chats arrive continuously, that remainder is almost always small, and the document may wait for minutes. The chat dashboard looks excellent while the file-upload operation becomes a hidden endless queue.

A workable configuration needs a third condition: a limit on the number of long requests being chunked at the same time and explicit control over their wait time. If a document grows old in the queue, it must periodically receive a sufficiently large portion of work. Otherwise, you have not made the system fair; you have simply shifted the inconvenience to a less visible user type.

Do not confuse this with separating prefill and decode across different GPUs. Separate pools remove direct competition between the phases but introduce new routing and KV cache transfer costs. The P/D-Serve study considers this separation a way to scale different phases, not an automatic replacement for a queue policy. On one shared pool, well-configured chunked prefill is often the simpler first step.

Do not copy someone else's chunk size into your cluster

The number 2048, 8192, or 16384 says nothing without the model, GPU, response length, and workload mix. The current vLLM documentation gives these as guidelines: smaller values, such as 2048, provide better ITL, while values above 8192 are recommended for maximum throughput on large GPUs and small models. This is a direction for searching, not a ready-made production configuration.

Start by choosing a product constraint. For example, chat should stay below its P95 ITL while documents are processed concurrently, and P95 total document time should remain acceptable for a background operation. Then find the largest budget that keeps the chat SLO. This choice usually gives documents more useful throughput than an unnecessarily small chunk.

If you serve different models through one API, test each important «model plus GPU» pair separately. AI Router lets you keep the application-level OpenAI-compatible call unchanged while routing to the required model, but queue parameters and latency results still belong to the specific inference engine and hardware.

After choosing the budget, add a regression test to the release process. Adding a new system prompt, enabling tool calling, or changing the model changes input lengths. A setting that was reasonable for yesterday's token distribution quickly becomes a source of long pauses if nobody watches the tails.

The configuration is ready only after you check for starvation

Enable chunked prefill for controlled competition, not for the switch itself. A good result looks ordinary: short chats keep TTFT and ITL within bounds under load, documents continue to move forward, and the GPU does not suffer constant gaps caused by iterations that are too small.

Check the worst case, not just the typical one. Run a sustained stream of chats, add a queue of large files, and watch the oldest document. If its age grows without an upper bound, your policy protects chat but is not fair. Fix the queue rule, the number of concurrent long prefills, or the pool architecture. Chunk size alone will not solve it.