LLM Gateway Overhead in a Real Request

An LLM gateway should not be evaluated with the question, “Does it add latency?” It adds several different kinds of work, each with its own profile: a few microseconds of CPU time, a wait for network storage, serialization of a large body, a queue in front of the connection pool, or an unnecessary retry. If you combine them into one gateway_latency metric, the team will see the problem too late.

A direct model call seems like an honest baseline, but it also includes DNS, TLS, connection setup, prompt transmission, waiting in the provider’s queue, and generation. Compare the gateway with that same call under the same conditions, not with an idealized picture in your head. The goal of measurement is simple: separate the cost of useful safeguards, such as authentication, rate limits, and auditing, from accidental pauses that can be removed.

First define what counts as latency

LLM request latency is not one number because the user, client SDK, gateway, and provider see different boundaries of the operation. In a streaming chat, the user judges the system by the time to the first token, or TTFT. For JSON field extraction, moderation, and batch processing, time to the complete response matters more. If you mix these metrics, a fast streaming response can look worse than a short non-streaming call even though the user receives text sooner.

Break every request into non-overlapping intervals:

• client_to_gateway: from the application sending the request to the gateway receiving it;
• gateway_queue: waiting for a free worker, connection, or internal limit;
• gateway_processing: authentication, policy checks, masking, route selection, and request preparation;
• upstream_ttft: from sending the request to the provider to the first byte or first stream event;
• upstream_completion: generating and transmitting the rest of the response;
• gateway_to_client: buffering, filtering, and delivery to the client.

In a non-streaming call, upstream_ttft cannot always be separated through external observation. That is fine. Do not replace an unknown value with a guess. Instrument the gateway so it records when the request is written to the external connection and when the first received byte arrives.

Record time spent on retries separately. A retry is not a slow model and is not ordinary router latency. It is a different operating mode with a different risk: the provider may already have accepted the first attempt, and an automatic retry can create duplicate work for some operations. Metrics need to show both the original attempt and the final user request.

Do not use the average as your main indicator. The average easily hides a queue of ten requests that appears once every few minutes and damages p99. For every interval, collect at least p50, p95, p99, request count, and error rate. For streaming, add a TTFT distribution. For a gateway with limits, add queue depth and the share of rejected requests.

Compare the direct call and the gateway with the same model workload

A comparison is meaningful only when both sides perform the same work with the same model. Changing base_url is not enough: the direct route may use a different geographic location, connection pool, API version, or generation parameters that the SDK adds automatically.

Build a fixed test corpus. It should include a short request, a typical production request, a large context, and a streaming request with a predictably long response. Do not put user data in the corpus. Save each request body as a file and send it byte for byte through the direct route and the gateway.

Before the test, fix the following:

• the exact model identifier and generation parameters;
• one execution region for the test client;
• the same limit on concurrent requests;
• streaming enabled or disabled on both sides;
• timeout and retry rules;
• cold and warmed connections as separate test series.

The last point often invalidates conclusions. In the first series, the client creates TCP and TLS connections. In the second, it reuses them. If you ran the direct call with one long-lived client but called the gateway from a separate process for every request, you measured the cost of your test script.

Start with a simple control series. It will not replace a load test, but it quickly shows a rough difference in time to headers and total response time.

export DIRECT_URL="$DIRECT_URL"
export GATEWAY_URL="$GATEWAY_URL"
export API_KEY="$API_KEY"
export BODY_FILE="request.json"

curl --http1.1 --silent --show-error --output /dev/null \
  --write-out 'route=direct connect=%{time_connect} ttfb=%{time_starttransfer} total=%{time_total} code=%{http_code}\n' \
  --header "Authorization: Bearer $API_KEY" \
  --header 'Content-Type: application/json' \
  --data-binary "@$BODY_FILE" \
  "$DIRECT_URL"

curl --http1.1 --silent --show-error --output /dev/null \
  --write-out 'route=gateway connect=%{time_connect} ttfb=%{time_starttransfer} total=%{time_total} code=%{http_code}\n' \
  --header "Authorization: Bearer $API_KEY" \
  --header 'Content-Type: application/json' \
  --data-binary "@$BODY_FILE" \
  "$GATEWAY_URL"

The output should look like route=gateway connect=0.012 ttfb=0.441 total=1.836 code=200. This measurement does not provide TTFT for Server-Sent Events and does not reveal the internal layers, but it can expose an obvious error: the gateway waits for the complete response before sending the first byte to the client even though the upstream has already started streaming.

Then move to a series with a persistent connection and controlled concurrency. Run several concurrency levels until you see a change in the distribution. A sharp rise in p95 with an almost unchanged p50 usually points not to a slow model, but to saturation in a queue, connection pool, or background telemetry exporter.

The queue in front of routing is usually the first bottleneck

A queue creates long latency tails even when every gateway filter works quickly. This is the most common case mistakenly described as “expensive routing.” A model-selection condition may take fractions of a millisecond, but the request waits tens or hundreds of milliseconds because all workers are busy serializing responses, writing logs, or waiting for a slow upstream.

Look for queues in three places. The first is before the application code: incoming connection limits, the accept backlog, the load balancer, and TLS termination. The second lives inside the process: the worker queue, file descriptor pool, HTTP connection pool, and limit on concurrent streams. The third appears in dependencies: Redis for distributed limits, the audit database, the trace collector, and the policy service.

A poor diagnosis sounds like this: “The router adds 180 ms.” A useful diagnosis sounds like this: “The p99 of the internal queue rises after 24 concurrent requests because the outgoing connection pool to the selected provider is limited, while streaming responses hold connections until completion.” The second diagnosis gives you something to fix.

Check queueing behavior against your own data. When incoming traffic is close to maximum throughput, even small fluctuations in processing time raise the tails. Increasing the number of workers sometimes helps, but sometimes only moves the queue into the CPU, network, or downstream service. You cannot fix a wait in Redis by adding processes if every new process makes even more requests to the same Redis instance.

A queue needs a limit and clear behavior when it overflows. An unlimited queue creates a nice graph of successful responses until users start waiting too long and cancel their requests. A bounded queue with fast rejection produces an unpleasant but manageable signal. The client can show the user a status, move the task to the background, or retry after a delay.

Client cancellation must also reach the upstream. If the user closes the chat while the gateway continues receiving a long stream and writing it to the audit log, you are spending a connection, tokens, and queue capacity. The cancelled-request metric should distinguish cancellation before the model is called, cancellation during generation, and cancellation after a completed response that the client never read.

Authentication is rarely expensive until it goes over the network

Checking an API key signature or looking up a key in a local cache usually does not determine LLM call latency. It becomes expensive when every request triggers remote introspection, a database call with locking, or a synchronous usage-counter update. At that point, the system has an extra network dependency immediately before the model call.

Do not combine three tasks that are often placed in one handler: confirming the key’s identity, loading the policy, and debiting the limit. They have different freshness requirements. A revoked key must stop being accepted quickly. The list of permitted models can be updated less often. The limit counter needs the level of consistency your product promises. When all three operations require the same synchronous database on every request, you are paying for consistency where it is not needed.

A practical design keeps verifiable key data and stable parts of the policy in a cache with a limited lifetime. Revocation or blocking goes through an invalidation channel or a short record lifetime. The limiter uses a fast local path for ordinary traffic and accesses shared state only when the rules require it. Do not create an unlimited cache: it turns a leaked key into a longer-lived problem.

Measure authentication as a separate span with attributes for the result, key type, and rejection reason, but not the key itself or a hash that can be used to match clients outside the required boundary. For diagnostics, a stable internal tenant identifier is enough if your data-processing policy permits it in telemetry.

Check the negative path separately. In practice, a surge of invalid keys, expired tokens, or deliberate attacks can put more load on the database than successful traffic if the gateway looks up a nonexistent record every time. A short-lived negative cache, source-based rate limiting, and inexpensive format validation can reduce this load. But do not return different detailed authorization reasons to the client if that would allow valid keys to be enumerated.

Logging can consume your throughput

Logs rarely add noticeable latency to one quiet request. They can break a system under concurrent load, when the gateway serializes request bodies, masks fields, places a record in a queue, and waits for an external agent to accept it. A large prompt and long response make the problem more expensive precisely when the model is already busy doing useful work.

Separate data by purpose. Auditing answers who called which model, with what result, and under which rules. A technical log helps investigate an error. Metrics show the shape of the load. The full prompt body is needed only for limited debugging cases with an explicit mode, access control, and retention period. If you write it every time, you create a risk of processing sensitive data and add work to every request.

Synchronous auditing is justified when an operation cannot be considered accepted without a recorded entry. This is an expensive choice, and it should be treated as expensive. If the rules do not require this order, write the event asynchronously to a bounded local queue or log instead of waiting for a remote search system in the request handler.

Do not turn every value into a metric label. A model label with a limited set of values is usually useful. A request_id label, the user’s full path, the upstream error text, or a raw session identifier creates high cardinality. At best, this increases memory consumption and storage cost. At worst, metric collection itself becomes a source of latency.

PII masking also has a cost, but you cannot “optimize” it by turning it off. The right question is where in the data representation it is needed. If the rules require content to be masked before entering the audit log, apply masking once to the copy intended for auditing. Do not run the same set of regular expressions separately for the log, trace, error metrics, and client response. A shared inspection result reduces work and lowers the chance that one channel will forget the mask.

Routing should be computable without a second LLM call

Route selection adds little latency when it relies on data already known to the gateway: the requested model, availability, region, tenant permissions, expected context size, budget class, and pool state. It becomes unpredictable when selection requires a separate model call, a series of remote checks, or sequentially trying providers.

A popular but poor idea sounds like this: “Let an intelligent model decide which model is best for every request.” Such a classifier can be useful outside the critical path, for example for offline task labeling and rule building. In online processing, it adds its own TTFT, cost, a new failure point, and a difficult question about how to evaluate the quality of its decision. If the task requires complex classification, cache the result for a recurring context or have the application pass an explicit task class.

Make routing rules observable. For every decision, save a short reason such as tenant_policy, region_requirement, capacity_fallback, or requested_model. Do not record the complete set of internal scores for every request. It is enough to know which rule won and whether the decision was a fallback.

The router should not open a connection to every candidate “just in case.” Use one selected route, one connection pool, and one clear timeout budget. Start a fallback after a classified error: unavailability, overload, a regional restriction, or an explicit provider rejection. Do not switch models after a user error in the prompt, an invalid body format, or after the upstream has already started streaming. Otherwise, the client receives a response from a different model when it expects predictability.

If the task involves data-storage requirements, the router must check them before the first byte leaves the permitted boundary. You cannot send the prompt first and then decide that the selected path is unsuitable. This is not a performance issue, but errors in this order often appear in code that someone tried to speed up.

Streaming changes perceived latency but does not remove the full workload

A streaming response makes the interface feel faster when the gateway forwards the first event immediately after receiving it from the upstream. It does not reduce the time needed to generate the complete text and does not free the connection sooner. In fact, a long stream holds the connection, state memory, and concurrency slot longer than a short non-streaming response.

You need four timestamps: the gateway receiving the request, sending it upstream, receiving the first upstream byte, and sending the first byte to the client. The difference between the third and fourth timestamps shows the cost of processing the stream inside the gateway. If it grows with response length, check buffering, synchronous auditing of every fragment, content filters, and locks during client writes.

Do not rebuild the stream into a complete text just for ordinary logging. That defeats the purpose of streaming and increases memory use. If auditing requires a final hash, length, token count, or result classification, calculate these incrementally. If a complete response is needed for a strictly limited case, set a maximum size and handle an overflow explicitly.

The client may be slower than the model. If the gateway reads from the upstream quickly but writes to the client slowly, buffers grow. If it stops reading from the upstream, the upstream may stop or close the connection. Choose a policy in advance: a bounded buffer with cancellation, backpressure to the upstream, or disconnecting the client after a limit is exceeded. Each option affects cost and audit completeness, so test the policy separately with an intentionally slow client.

One trace should show the cost of every layer

Distributed tracing is useful only when it answers a timing question rather than becoming a collection of nicely named spans. W3C Trace Context defines the traceparent and tracestate headers for carrying context between components and requires context to be handled correctly during propagation. The specification also explicitly says not to put personal or sensitive data in these headers.

For one LLM request, a tree like this is enough:

llm.request
├── gateway.authenticate
├── gateway.policy
├── gateway.rate_limit
├── gateway.route
├── gateway.queue
├── upstream.request
│   ├── upstream.first_byte
│   └── upstream.read_stream
├── gateway.audit_enqueue
└── gateway.client_write

Do not create a span for every token. With a long response, this creates a flood of telemetry and distorts the measurement. For a stream, counters for the number of events, bytes, and time between the first and last event are enough. Use trace events for rare diagnostic details: a fallback, cancellation, limit exceeded, or parsing error.

OpenTelemetry describes context propagation as a mechanism that links spans into one trace even when different components create them. Apply this principle literally: the trace starts in the application, passes through the gateway, and continues in the outgoing HTTP client. If the gateway creates a new trace without a reason, you lose the ability to prove exactly where the pause occurred.

Sampling should not be identical for every request. You can sample successful high-volume traffic probabilistically while retaining errors, cancellations, fallbacks, and slow requests more often. But do not treat a sampling flag from an external client as an unconditional command. An external client may try to make you record too much data. W3C explicitly notes that recording decisions must account for trust, abuse, and the component’s own load.

A load test should break one layer at a time

One large “everything enabled” test is useful for final verification but poor at explaining the result. First record a baseline profile of the direct call. Then enable one layer at a time: authentication, the limiter, auditing, tracing, routing, masking, and fallback. After every step, compare not only total latency but also the individual intervals.

A practical sequence looks like this:

1. Warm up the connections and run a series with a fixed, low level of concurrency.
2. Repeat the series while increasing concurrency until p95 changes or errors appear.
3. Enable one internal layer and identify the difference through its span, CPU, memory, and number of outgoing calls.
4. Run the same test with a long streaming response and a slow client.
5. Repeat the check when one upstream is unavailable to see the cost of fallbacks and retries.

Do not change the SDK version, log format, connection-pool settings, and routing rule at the same time. After such an experiment, you can only argue about the cause. One changed factor takes more time, but it produces an answer you can reproduce.

Treat throughput as the number of useful operations completed within a given SLO, not as the maximum number of accepted requests. A gateway that accepts a thousand requests, puts them in an unlimited queue, and responds a minute later has not become more productive. It has only postponed the failure.

For every test, save the configuration, versions, corpus size, concurrency level, client metrics, and a trace sample. This discipline is tedious, but without it the team will not be able to tell an improvement from a provider that happened to be faster for a while two weeks later.

Only mandatory decisions should remain on the critical path

Every gateway layer should pass a simple test: must it finish before the prompt is sent to the model or before the first token reaches the client? Authentication, access checks, regional restrictions, and mandatory limits usually must. Analytics export, search-index enrichment, and detailed technical logging usually do not.

This is not an argument for a “thin proxy” without controls. Business systems need auditing, PII masking, limits, and managed model selection. But these functions cannot be evaluated as one general latency surcharge because some must be synchronous while others should run outside the user path.

AI Router lets teams keep an OpenAI-compatible client contract by changing the API address, but after connecting, you should still measure your own policies, auditing, and routing scenarios rather than treating the gateway as invisible. The useful result is not “the gateway adds N milliseconds,” but a table showing the cost of every layer, the boundary of its queue, and a decision that can be checked again.

If the trace has no separate time for the queue, time to the first upstream byte, and time to transmit the first byte to the client, you are not measuring overhead yet. You are measuring the whole wait and hoping to guess its cause.