Do Multiple Teams Need a Fair GPU Queue?

When several teams share the same GPUs or the same set of external APIs, the problem rarely looks like a queueing problem. At first, it looks like a complaint: the chat suddenly takes a minute to respond, the overnight computation is not finished by morning, the provider starts returning 429s, and a team that changed nothing asks why its service has become slow.

The cause is usually simple. All jobs sit in one queue, and the scheduler treats them as equal. But a request to classify one document, a long response generation, fine-tuning, and a large evaluation run differ in runtime, memory use, and the number of accelerators they occupy. A fair GPU queue does not start with choosing an algorithm. It starts with an honest definition of what exactly you are sharing.

FIFO makes interactive services hostage to batch work

FIFO processes jobs in arrival order. The algorithm is transparent, easy to implement, and perfectly adequate for one type of short task. Under mixed load, it provides exactly the guarantee that many people forget to read: whoever arrives first gets the resource first, regardless of the task size or its owner.

Imagine one queue for eight GPU slots. At 09:00, the analytics team submits 400 evaluation jobs. Each job generates a long response and holds a slot for several minutes. At 09:03, the product team starts receiving user requests for its assistant. Their requests take seconds, but they are behind the evaluation jobs that have already been accepted. The accelerators may be 70 percent busy while the user journey is effectively unavailable.

This is not a FIFO error. It is an error in what people expect from FIFO. The queue follows arrival order but does not distinguish between:

• short and long work;
• a request that a person is waiting for now and a background run;
• one task and a batch of a thousand tasks;
• one team's work and the load from everyone else.

FIFO performs especially poorly with multistep jobs. If one dispatcher gives a batch job four slots, it may hold them until the end of a long stage even after dozens of short interactive requests have accumulated in the queue. Adding more GPUs may help sometimes, but it does not change the service order. At the next large launch, you will see the same effect on a more expensive cluster.

FIFO still has a place. You can keep it inside one narrow class where jobs have a predictable cost, such as short operations for a single service. Do not make it the global policy for a shared pool.

Strict priority protects urgent work but can starve everyone else

A strict-priority queue always takes work from the highest nonempty class. It is good at protecting a path that cannot be delayed: an availability check, a cancellation, incident handling, or a human request in an active conversation. It also quickly becomes the source of a hidden fight over the label «urgent».

If class P0 is nonempty most of the time, P1 gets no resources. If P1 is constantly busy, the batch class may wait forever. This is called starvation. In real operations, it often looks not like infinite waiting but like jobs that people start and then cancel manually because their owners have stopped trusting the queue.

The popular recommendation to «give every service a priority» is worse than having no policy. After a few months, almost every service has a high class because its owner can describe its latency as important. When everything has high priority, nothing is protected.

Keep strict priorities for a small set of cases where delay is genuinely more dangerous than fairness. Give every such class three limits:

1. A concurrency limit, so urgent work cannot consume the entire pool.
2. A queue-length or wait-time limit, so stale jobs do not accumulate.
3. An admission rule that cannot be bypassed by changing a request field.

Priority answers the question «Who must not be delayed?» It does not answer «How should the remaining capacity be shared fairly?»

Weighted fair queuing divides occupied capacity, not equal latency

Weighted fair queuing, or WFQ, keeps separate work streams and services them in proportion to their weights. If team A has weight 4 and team B has weight 2, then under constant load A should receive roughly twice as much serviced cost. The important measure is cost, not the number of messages in the queue.

The Kubernetes API Priority and Fairness documentation describes a similar idea for requests to the API server: different priority levels receive separate concurrency limits, while fair queuing within a level prevents one flow from displacing the others. It also calls out the problem of a misbehaving client that floods the server with requests. This is a useful example because queues are built not for elegant mathematics but to isolate neighbors sharing a resource.

In Linux, the fq discipline also separates flows, serves them in rounds, and uses a quantum, meaning the amount of work a flow may submit in one round. A larger quantum means the other flows wait longer for their next opportunity. For GPUs and APIs, this is a direct warning: if one queue is given too large a portion of work at once, a formally fair algorithm will again produce poor interactive latency.

WFQ does not promise that all requests will finish in the same amount of time. A long task still takes longer than a short one. It promises something else: while several flows are active, one flow cannot occupy the entire available share forever.

In practice, compute schedulers use an approximation of WFQ. For each flow, store the virtual finish time of its next job:

finish = max(flow_finish, virtual_time) + estimated_cost / weight

The dispatcher chooses the job with the smallest finish. After issuing the job, it updates flow_finish and the overall virtual time. You do not need to simulate an ideal packet network. You need to apply one formula consistently, store state atomically, and prevent one sender from creating a new flow for every request.

Fairness breaks when you count requests instead of cost

Two requests can have the same HTTP size and completely different costs. One asks to extract a couple of fields from a short text. The other sends a long context, requires a detailed response, launches several tools, and then repeats the call after an error. If both count as «one request», the heavy work gets a subsidized rate.

For GPUs, a useful unit often looks like this:

cost = occupied_slots * measured_seconds + memory_penalty

For LLM inference, the initial estimate is usually made before launch:

estimated_cost = input_tokens + 2 * max_output_tokens

The coefficient before output tokens depends on the model and the mode. Do not treat it as a physical constant. Take a log of completed requests, compare the forecast with actual runtime, and update the coefficients regularly. For batch jobs where the dataset size and generation parameters are known, the estimate is usually more accurate than for free-form user text.

Do not mix three different limits:

• Service share determines how much work a team receives under contention.
• Concurrency limit restricts how many jobs a team can run at the same time.
• Rate limit restricts how quickly a team can create new work.

A WFQ weight does not replace a concurrency limit. A team with a large weight may fairly receive a larger share, but one of its jobs can still occupy the entire GPU if the scheduler admits such a job. A rate limit does not replace a weight either: a team can slowly submit a thousand heavy jobs and occupy the queue for hours.

Instead of one abstract «limit», track resources separately. For a local model, these might be GPU seconds, memory, and slot count. For an external model, they might be requests per minute, tokens per minute, concurrent calls, and a spending budget. Do not combine these values into one number until you can explain what every part of the sum means.

A flow should be a team and workload class, not a user or API key

The flow definition determines whom the scheduler considers a neighbor. A mistake here devalues any algorithm.

If a flow equals a user, one team can get hundreds of shares by creating hundreds of service accounts. If a flow equals an API key, the same effect appears after key rotation or when an application is split into microservices. If a flow equals only a team, its heavy overnight run will compete with its own interactive service, even though the business usually does not want that.

A good starting scheme is:

flow_id = organization_id + workload_class + resource_pool

organization_id identifies the owner of the share. workload_class separates interactive work from batch and service operations. resource_pool prevents a job waiting for an external API from taking a place in the local GPU queue.

Do not create more classes than you can protect with rules. These are usually enough:

• interactive for requests in an active user journey;
• batch for evaluations, indexing, bulk generation, and experiments;
• system for cancellations, health checks, and occasional support operations.

The trusted server, not the client request header, should set the class. A client may state its intent, but the dispatcher must verify it using the route, service account, task type, or a separate permission. Otherwise, every batch call will soon declare itself interactive.

Size estimates and small quanta protect short requests

Flow-based WFQ protects teams from one another, but it does not always protect short work within a single team. If one request with a huge context lands in interactive and ten short requests are behind it, a simple order within the flow creates head-of-line blocking again.

The most aggressive way to solve this problem is shortest remaining processing time: run the work with the least remaining time first. It reduces average latency well, but in pure form it can keep pushing long jobs back if the stream of short jobs never ends. In production, a bounded version is better.

Divide the interactive class into several size buckets, for example by predicted cost. A small job gets a chance to start quickly, but after a limited number of dispatches the scheduler must take work from the next bucket. This is not perfect theory, but the policy survives real traffic and can be explained to service owners.

There is also a simpler option: limit the size of one work portion. For generation, this means that a long response does not reserve its entire expected length until completion. The dispatcher issues a limited number of tokens or a limited amount of compute time, then decides whether to continue the job. This approach requires cancellation and resume support in the worker. If the worker cannot safely continue a task, do not pretend that you are splitting it into parts. Account for the full cost honestly and limit the number of heavy jobs instead.

Short requests cannot be protected by a timeout alone. A timeout fires after the user has already waited. The queue must decide what to admit before execution starts.

GPUs and external APIs need separate budgets and waiting points

A job that calls an external model should not occupy a local GPU slot while waiting for the network or the provider's response. This seems obvious, but workers often violate it: a process receives a job, reserves a shared slot, then calls the external API and holds the slot while waiting.

Split the path into stages. Local preparation takes a short CPU limit. The external API call goes through a queue for the specific provider. Local post-processing receives its own resource again. Each stage gets its own semaphore and cost counter.

For an external API, a combination of a token bucket and a fair queue is useful:

provider: external-model-a
limits:
  requests_per_minute: 900
  tokens_per_minute: 180000
  max_in_flight: 24
scheduler:
  algorithm: wfq
  flow: organization_id + workload_class
  weights:
    interactive: 6
    batch: 1
  max_queue_age_seconds:
    interactive: 45
    batch: 7200
retry:
  max_attempts: 3
  budget_per_flow_percent: 10
  backoff: exponential_with_jitter

This configuration prevents a specific failure: the batch flow cannot create so many simultaneous calls that interactive requests receive 429s along with it. It also prevents retries from taking more than one tenth of the flow's budget. These numbers are not universal. Derive them from the provider's quotas, observed latency, and acceptable wait time instead of copying them into production without measurement.

Google Cloud's quota documentation separately recommends assigning different quotas to heavy and light methods and using delayed retries for 429s. That is sensible, but for a shared dispatcher, backoff alone is not enough: the retry must pass through the same limit and the same fairness policy again. Otherwise, old jobs bypass the queue on every new attempt.

Cancellation and retries determine whether the queue stays fair under failure load

The user closed the page, but their generation continues consuming GPU time for another two minutes. A batch job encountered a temporary error and sent hundreds of retries at once. The client timeout expired, but the server worker is still waiting for the external API. All three situations create work that nobody is willing to pay for anymore, yet the queue continues to count it as equal.

Every job should have a deadline, a cancel_token, and an idempotency identifier. The dispatcher checks the deadline before queueing, before handing the job to a worker, and between long stages. The worker checks cancellation wherever it can actually free a resource: before starting the model, between batch parts, before the next provider call, and after the response returns.

Set separate rules for retries:

1. Retry only errors for which the operation is safe or protected by an idempotency key.
2. Return the retry to its original flow, not to the front of the global queue.
3. Charge the retry against the task owner's attempt budget.
4. Add jitter to the delay so clients do not retry in formation.
5. Stop trying when the deadline has expired, even if the attempt limit has not been reached.

An immediate retry after a 429 looks like a response to a temporary problem. In practice, it often prolongs the overload. If the provider limited the flow by tokens, retrying the same long request after 100 milliseconds only creates a second record of the same violation.

Test the policy under artificial overload, not by average response time

Average latency says almost nothing about fairness. It may look good while one small team waits for an hour and a large team sends many short jobs that pull the average down.

Collect metrics separately by owner, class, and resource pool. A minimal set includes:

• p50, p95, and p99 time waiting before start;
• actual cost of serviced work for each flow;
• number of running jobs and queue length;
• cancellations before start, cancellations during execution, and work after cancellation;
• 429s, timeouts, and the share of retries in the total volume.

Then run a boring but revealing test. Let team A continuously submit heavy batch work. Let team B send short interactive requests at a steady rate. Let team C occasionally create a burst of work. Run the scenario first on FIFO, then with strict priority, and finally with WFQ using the same concurrency limits. Compare not only total completions but also B's request start times, A's resource share, and whether C received at least part of its work.

If WFQ gave teams the right shares but interactive requests still wait unacceptably long, the problem is usually not the weights. Check the quantum size, the maximum job size, the number of simultaneously occupied slots, and the order within each flow. A scheduler cannot restore latency that was already created by a job occupying all available GPUs.

Start with isolation that an on-call engineer can explain

You do not need to build a perfect global scheduler immediately. First remove the one shared queue for everything. Separate interactive work from batch, introduce per-team concurrency limits, measure actual cost, and prevent retries from bypassing the queue. After that, weights begin to mean something useful.

Once the rules are stable, document them as an agreement. State who owns each flow, which classes are allowed, what share is guaranteed under contention, what happens when neighbors are idle, and what wait time counts as expired. Without this, WFQ remains an algorithm in code rather than a policy for a shared resource.

If calls go through AI Router, key-level limits can separate consumers at the API gateway, but the business queue policy and workload classification should still live in the application or a separate dispatcher. Otherwise, the key will limit the rate of entry but will not decide whose long job is allowed to take the next free slot.

Fairness does not mean every team gets the same thing. It means that the agreed shares remain in place precisely when the resource becomes scarce. That is the moment the queue exists for.