What provides better LLM reliability?

For a conversational service, I divide reliability into four contracts: how soon the system responds again, how similar the new response is, whether the system can handle the traffic, and whether it remains available when a region goes down. Automatic failover usually wins the first contract. A private reserved pool usually wins the second and part of the third. Only an architecture whose fallback path does not share the primary path's control plane and failure domain can win the fourth.

Measure reliability as four separate promises

One availability percentage hides different failures that customers experience in different ways. An HTTP response in two seconds with the wrong format does not make a chat available, just as a perfectly compatible model does not help when its request waits in a queue for a minute. Before choosing infrastructure, define four separate metrics.

The first metric is response recovery time. It begins with the first customer request that the primary path can no longer complete within its deadline, not when an engineer opens an incident. The budget includes failure detection, the router's decision, resubmission, connection warmup, and generation by the fallback model.

The second metric is response compatibility. It includes valid JSON, tool calls, compliance with system instructions, language, style, refusal policy, and the ability to continue a conversation that has already started. You cannot reduce compatibility to whether another provider accepts the same API fields.

The third metric is guaranteed throughput on a bad day. A public API may have spare capacity during a test and rate limit requests alongside many other customers during a major outage. A reserved pool removes competition for the accelerators already assigned to you, but it has a hard ceiling.

The fourth metric is failure-domain independence. Two model names do not mean two independent paths. They may depend on the same cloud region, DNS service, secrets store, gateway, network connection, or conversation history service.

Show these metrics separately on the operations dashboard and in the agreement with the owner of the customer process. Otherwise, the team celebrates transport availability while agents manually correct broken answers.

Each metric needs its own error threshold. For example, a service can meet its target for the share of HTTP responses while failing the target for responses that pass the business schema. A single SLO averages those events and gives the team no clue what to fix. I count transport failures, missed deadlines, invariant violations, and capacity-related handoffs to a human separately.

The threshold must also reflect the channel. A voice assistant handles several seconds of silence poorly, while email permits longer generation. For chat, the time until the first useful fragment matters more than the total time for a long response. One infrastructure SLO for all three channels will make the team overpay or hide a bad experience behind an average.

Failover shortens the pause when it detects the right failure

A router can restore traffic in seconds because it does not need to start GPUs and load weights. But a fast fallback works only when the team has already decided which errors permit a retry, how much time it can spend, and which requests are safe to send again.

A network timeout, HTTP 429, HTTP 500, and a stream that breaks after the first tokens require different actions. RFC 9110 defines the Retry-After header for a 503 response as a date or delay after which the client should retry. It is a useful signal, but I do not give it the whole waiting budget: the provider describes its own state, not the customer's patience. If Retry-After exceeds the remaining customer deadline, the router should choose another allowed path or end the request with a controlled error.

Do not retry every request blindly. If the model managed to call a ticket creation tool and the connection broke before confirmation, a retry can create a second ticket. Operations with side effects need an idempotency identifier in the tool, and the router must store call state separately from response text.

A practical budget looks like this. Suppose a chat channel must show a useful answer within 8 seconds. You can allow the primary attempt 2.5 seconds to the first token, routing and a new connection 0.5 seconds, the fallback model 4 seconds, and reserve the final second for delivery and controlled degradation. These numbers illustrate the calculation rather than define a universal standard. Derive yours from the measured latency of your prompt and channel.

An overly sensitive automation sends traffic to the fallback because of a few slow responses and creates its own load spike. An overly patient one spends the entire deadline on the primary provider. I use two signals: a short window of consecutive transport errors to open the circuit immediately and a longer latency window to divert traffic gradually. A half-open state returns a small share of requests before declaring the primary path healthy.

Sending two identical requests in parallel, often called hedging, reduces tail latency, but it is an expensive emergency measure for customer service. It doubles consumption precisely when providers are already limiting capacity, and it makes tool-call cancellation harder. I allow hedging only for requests without side effects, after a measured latency threshold, and with immediate cancellation of the losing attempt. Sending every request to two models all the time for an attractive p99 is usually the wrong choice.

The design also needs protection from a cascade. When a large provider fails, the entire primary flow reaches a fallback that previously saw only a test percentage. The router should increase the share in stages, watch fallback errors, and preserve some of the limit for conversations already in progress. Otherwise, automation turns one failed path into simultaneous failures across all paths.

Automatic failover wins on recovery time when the fallback provider has already passed testing, connections open in advance, limits are known, and a process outside the failed domain makes the decision. If switching requires a phone call to an engineer or a configuration change after the incident starts, it is automatic only in the architecture diagram.

A compatible API does not guarantee a compatible response

Two OpenAI-compatible endpoints can accept the same request and return answers that affect the business process differently. Transport compatibility means that fields and status codes match. Semantic compatibility means that the fallback model preserves the required response properties. Customer service depends on the second contract.

The difference usually appears at the boundaries rather than in polished free-form text. One model always fills the required intent field, while another sometimes adds commentary before the JSON. One calls create_ticket only after the customer gives explicit consent, while another decides to call the tool sooner. One stays within the channel's tone limit, while another writes more and pushes an important phrase beyond the interface boundary.

Streaming creates another trap. Once the application has sent the first tokens to a customer, it cannot quietly restart the answer with a different model. The customer will see repetition, a language change, or an interrupted thought. Before the first token, a retry is usually transparent. After the first token, it is better to end the stream with a clear message and offer a retry if the application cannot resume at a sentence boundary.

Test the fallback on the same sample that admits the primary model to production. I define mandatory invariants instead of requiring identical wording:

• JSON passes the same schema without repairing free-form text.
• Allowed tools and their calling conditions do not change.
• The response does not expose hidden instructions or personal data.
• Handoff to a human triggers for the same risk classes.
• Length and language fit the service channel.

Every invariant needs a blocking threshold before release and an observable metric afterward. An average quality score is dangerous here: excellent answers to easy questions can hide rare but expensive mistakes involving refunds or medical information.

Tie the prompt version and request adapter to the compatibility group. One provider may support strict response schemas, while another supports only a normal instruction in text. If the adapter silently removes an unsupported parameter, the request remains valid but the guarantee disappears. The router must either apply a previously tested template for that path or declare the model incompatible.

Test context limits on real conversations as well. A fallback model with a smaller usable window may accept the declared volume but lose the beginning after the adapter truncates it. Summarizing before a retry changes facts and consumes the deadline. Decide in advance which messages remain verbatim, which tool results cannot be shortened, and at what length the conversation goes directly to a human.

If models do not pass the same set of invariants, do not place them in one instant-failover group. Split routes by task. A fallback model can handle order-status lookups while sensitive intents go to a human. That is more honest than calling full failover successful because the endpoint returned 200.

A GPU reservation turns external scarcity into an internal queue

Reserved GPUs give the team rights to specific capacity and remove the risk of competing with other customers for that pool. They are particularly useful when the service needs the same weight version, a fixed chat template, local data storage, or predictable latency. But a reservation does not create infinite throughput.

Size capacity for peak flow, not the daily average. A simplified check uses Little's Law: required concurrency is roughly the request rate multiplied by the average service time. If an emergency mode receives 12 requests per second and the model occupies a GPU for an average of 3 seconds, the system needs about 36 concurrent slots before allowances for variance and housekeeping. This illustrates the formula and does not describe a particular model.

The average is still insufficient. Long conversation histories increase prefill time, long answers hold a slot, and batching changes latency nonlinearly. Measure the p95 input length, output length, time to first token, and total time on your inference server. Then set aside capacity for the failure of some nodes and for the traffic previously handled by an external provider.

When a public provider runs out of capacity, it usually returns a rate limit or stretches latency. In your own pool, the queue belongs to you, so it needs an explicit policy. Without one, new chats, internal regenerations, and long low-priority jobs compete equally. I keep a separate quota for the customer channel, limit response length in emergency mode, and drop background summaries before the queue affects a conversation.

Test the flow after failover, not just the normal peak. If the local pool normally handles 30 percent of requests, an external API outage can suddenly add the remaining 70 percent. Those conversations have already accumulated long contexts, and retries have increased the incoming flow. The load model must include that transfer and a retry limit, or the reservation estimate describes a calm day.

It helps to keep some capacity unavailable to the normal scheduler. That reserve looks wasteful before the first node failure, but without it any hardware loss immediately lengthens the queue. Its size depends on the smallest failed unit: if the scheduler loses a whole server with several GPUs, a reserve equal to a fraction of one accelerator fixes nothing.

Hardware failure also has a recovery time. A fallback process on an already running node starts quickly. A new node without loaded weights must obtain its image and weights and reserve memory, so recovery can take minutes. The exact time depends on model size, storage, and the inference server. Do not promise it from a warm test.

A GPU reservation is more reliable than provider failover for compatibility and access to committed capacity. It is weaker when the whole pool sits in one location or has no emergency headroom. Buying accelerators without a queue policy merely moves the failure from someone else's API to your load balancer.

A regional outage tests independence, not replica count

When a region goes down, ten replicas in one zone provide zero external availability. The same applies to two providers if both paths pass through one corporate gateway or shared authorization service. A dependency map proves regional resilience, not the number of rows in a model list.

Trace the request path from the customer channel to the answer. It usually includes public DNS, perimeter protection, an API gateway, API key validation, a prompt store, conversation history, a router, a model, business tools, and telemetry. Every synchronous component must either operate in the second region or have predefined behavior without it.

A common failure looks like this: models at two providers are available, but the router reads rules from a database in the failed region. A new instance in the second region starts, cannot obtain configuration, and rejects every request. The team sees green provider status pages and spends time switching models even though the failure sits in front of them.

Conversation history is another unpleasant dependency. If the fallback region cannot see the latest messages, the model may repeat a question, forget an identity check, or call a tool again. Full synchronous replication sometimes increases latency and couples the regions. For chat, it is often enough to store a compact state snapshot with a version number and permit a tool retry only after checking the operation identifier.

Protect an active fallback region from stale rules as well. Distribute route configuration, the allowed tool list, and the prompt version as a signed package that the process can read locally. If the package has expired, it is safer to leave only informational answers and human handoff than to load unknown configuration from the failed region.

A health check must not depend on the path it tests. A model request made from inside the primary region will not reveal the loss of external DNS or a corporate connection. I run probes from points close to the real traffic entry and separately verify that routing decisions still work when the primary telemetry is unavailable.

An external provider in another region can be a useful emergency exit for a local GPU pool, but only when the data policy permits that route. If personal data cannot leave the country, the router must mask it before sending or forbid the external path for that category. Do not decide this during an incident.

AI Router combines provider routing with its own open-weight models on GPUs in Kazakhstan, along with PII masking and audit logs. This makes it possible to assemble different paths behind one OpenAI-compatible endpoint, but a test of the specific design's regional dependencies still has to prove its independence.

Customer RTO starts before infrastructure RTO

For the service owner, recovery occurs when a customer can complete the task again, not when a GPU metric turns green. Technical RTO must therefore connect to conversation state, tool retries, and interface behavior.

Divide requests into three states. Before sending to a model, the application can safely route a request anywhere. After sending but before the first token, it can retry if tools have not run or use idempotency identifiers. After part of the answer reaches the customer, a hidden retry is dangerous, and the interface should retain the text already shown, mark the failure, and offer a deliberate continuation.

RPO also makes sense for a conversation, although teams rarely use that term here. It asks how many recent messages and tool results may be lost when changing regions. Losing one exchange may be acceptable for an informational bot. For payment confirmation or a plan change, even one lost result requires checking the system of record instead of trusting the text history.

The service channel needs its own degradation behavior. If the fallback model is not approved for financial actions, it can still accept a request, gather required fields, and give a correlation number to a human agent. When capacity is scarce, the service can reduce the maximum response length and disable optional summarization. It must not quietly change its safety policy to improve the response percentage.

You can record selection conditions without giving the technologies one overall ranking. When answers must remain strictly consistent, the primary path is a reserved pool with one version, backed by the same pool in another failure domain. For a sharp and unpredictable spike, multiple providers work better, supplemented by a limited local environment for important requests.

Local storage requirements lead to GPUs in an approved region and a second approved location if one exists. When there is no second location, human handoff is the honest fallback, not an external API that violates the data policy. For a short RTO after an API failure, use previously tested providers and a warm local pool.

Tools with side effects change the choice more than the model type does. Every automated path needs idempotency and result verification. If the system does not know whether an action completed, the fallback becomes a human who checks the system of record. This analysis does not choose a technology for the team, but it prevents the team from confusing text availability with permission to act.

A hybrid design provides two ways to recover

For most high-volume customer services, I choose two paths with different strengths rather than a symmetrical collection of identical backups. Routing across providers absorbs an API failure and an unexpected spike. A reserved pool keeps a compatible model for regulated or sensitive requests. One path does not have to replace the other completely.

A router needs compatibility groups, not merely an ordered list of models. Within a group, models have passed the same JSON, tool, and policy checks. Across groups, the permitted intent set changes: for example, a fallback group answers informational questions but hands a contract change to a human.

Assign minimum guaranteed capacity to each group. Otherwise, the router knows where to send a request, but the chosen path has no obligation to accept it. For an external API, verified limits, multiple accounts, or an agreement with the provider, if one exists, stand in for a capacity guarantee. For a private pool, the figure is the number of available slots after the planned node failure.

Route order must consider the remaining deadline, data category, cost of another attempt, and circuit state. Picking the next provider in a loop is insufficient. It can send personal data to a prohibited region or spend the final two seconds on a model with a long time to first token.

The following shortened policy fragment is not syntax for a specific product. It is a testable contract between application developers and operations:

route: support-chat
deadline_ms: 8000
attempts:
  - pool: local-reserved
    first_token_timeout_ms: 2500
    compatibility_group: support-v3
  - pool: external-multi-provider
    first_token_timeout_ms: 3500
    compatibility_group: support-v3
    pii: masked
after_stream_started: return_controlled_error
on_capacity_exhausted:
  disable_tasks: [conversation_summary, draft_variants]
  preserve_tasks: [customer_reply, human_handoff]

This policy prevents three vague decisions during an incident. It forbids a hidden model change after streaming starts, keeps unmasked data out of the external pool, and gives customer responses capacity ahead of background work. A production configuration also needs version identifiers, an overall attempt limit, and a rule for every tool with side effects.

A hybrid design costs more than trusting one public API and is more complex than one GPU cluster. Choose it when the cost of an unavailable or incorrect answer exceeds the cost of standing capacity and regular tests. At low volume, two tested external providers and an honest human handoff make more sense than an underused cluster without round-the-clock operations.

Compare costs by the emergency outcome, not only token prices or GPU hours. The calculation includes idle reserve, duplicate model evaluations, second-region state storage, and on-call work. On the other side are lost conversations, manual correction of actions, and data-location obligations. If the team cannot name these items, a precise infrastructure choice will still be arbitrary.

A drill must break the path, not substitute a model

Testing a manual switch proves only that someone can change configuration. A useful drill disables a real dependency, sends representative conversations through the fallback, and measures the customer outcome.

I run the exercise in a fixed sequence. First, the team records the expected RTO, acceptable context loss, permitted intents, and maximum queue. It then blocks the primary path at the network level so the application receives the same timeouts it would see during an outage. In a separate streaming test, it breaks the connection after the first tokens. After recovery, it checks for duplicate tool actions, response schema compliance, the handoff rate, and the actual time to useful text.

A regional scenario must prevent configuration, secrets, and history reads from the primary region. If the test leaves these services available, it measures a model failure rather than a regional failure. Fallback telemetry must not disappear with the primary monitoring system either, or the team will lose evidence exactly when it needs it.

For automatic failover, record the number of attempts, the reason for each decision, and the remaining deadline before every attempt. For the GPU pool, record queue depth, dropped background jobs, warmup time, and the share of requests handed to a human. Do not reduce the result to a final 200 status code.

After the drill, change thresholds according to measured errors rather than participants' impressions. If the fallback model violated an invariant, remove that intent from the automatic route until it is fixed. If the local pool overflowed, first correct the priority policy and load model, then decide whether to buy more GPUs.

Automatic failover is more reliable for quickly restoring transport and absorbing external scarcity. Reserved GPUs are more reliable for committed capacity, model version, and local control. Customer service needs both properties only in flows where a mistake or pause is genuinely expensive. Explicit degradation is more useful for the remaining flows than an expensive illusion of complete redundancy.