An SLA for an LLM gateway must measure the full request path

A bank needs an SLA that measures the outcome at the boundary of its application, not whether a process inside the vendor is healthy. If the gateway accepted a request but the model did not answer in time, the business experienced a failure regardless of which component broke.

That is why an availability percentage on its own is almost useless. The procurement team should compare the availability formula, latency threshold, response quality, RTO, support response time, rules for external dependencies, and consequences of a breach. I have seen contracts with an attractive 99.95% where the vendor could exclude almost any real outage. That percentage promises nothing.

Measure availability by successful requests in each calendar month

A workable commitment says that at least an agreed share of eligible requests in every calendar month must finish with a successful response before a set time limit. An eligible request is a valid bank request within agreed limits, sent to an operating endpoint. The unit of measurement is a request, not a minute when an internal server was healthy.

The basic formula should appear directly in the contract schedule:

Availability = (Eligible Requests - Failed Requests) / Eligible Requests * 100%

Failures include 5xx responses, network disconnects after the connection is accepted, gateway timeouts, malformed responses, and responses delivered after the contractual limit. A 200 response with an empty body, a broken stream, or JSON that fails the agreed schema cannot count as success. Otherwise, the vendor optimizes the metric while the bank keeps handling errors.

RFC 9110 separates HTTP status classes and classifies 5xx as server errors. That is a useful starting point, but it is insufficient for an LLM: transport success does not guarantee a usable result. The contract must describe streaming responses separately because the first token and full completion have different success moments. For an ordinary response, success is recorded after receipt of the complete valid body. For a stream, availability can be based on a successful start, but a break before the final event must enter a separate stream completion metric.

Do not replace the monthly window with a quarterly one. Quarterly averaging can hide a serious January outage behind good results in February and March, even though the banking process suffered in January. The reporting time zone and exact start and end of the month must also be fixed in advance, for example in Almaty time. Otherwise, the parties can assign one incident to different periods.

One overall number is not enough when a gateway serves separate environments. Procurement should require separate indicators for the production API, the management console, and sensitive administrative operations such as revoking a key. A console outage does not always stop requests already in flight, but an inability to close a compromised key quickly creates a different risk. Those events should not be mixed into one percentage.

For reference, 99.9% over a 30-day month allows about 43 minutes and 12 seconds of downtime, 99.95% allows about 21 minutes and 36 seconds, and 99.99% allows about 4 minutes and 19 seconds. These figures make sense only for minute-based measurement. With request-based calculation, the actual permitted impact depends on traffic, so the contract should show both the failure share and an equivalent duration for continuous incidents.

Define p95 separately for first token and full response

A p95 threshold must identify a specific operation, model class, input size, output size, and measurement point. The phrase «p95 below two seconds» cannot be tested without those conditions: generating a short classification label is not comparable with generating a thousand-token response.

For streaming generation, a bank will usually need at least two metrics:

• time to first token, from receipt of the complete request by the gateway to the first meaningful fragment;
• time to last token, from receipt of the request to valid stream completion;
• the share of requests that exceed the hard timeout;
• p99 to expose the heavy tail, even when the commitment uses p95.

For nonstreaming calls, measure total duration. Group metrics into agreed buckets, such as up to 2,000 input tokens and up to 500 output tokens, rather than mixing all requests. Otherwise, a large number of short requests will hide latency in long operations. Do not demand the same threshold from every model: a fast classifier and a large reasoning model need different profiles.

Calculate the percentile over all eligible requests in short intervals, then aggregate the month by a documented rule. Daily p95 values must not be averaged: a percentile is not additive, and that calculation distorts the distribution. The vendor should either calculate p95 from the raw monthly sample for every bucket or publish an agreed histogram with bucket boundaries and observation counts.

A testable clause looks like this:

Для запросов класса Interactive с входом <= 2 000 токенов и лимитом
выхода <= 500 токенов не менее 95% подходящих потоковых запросов
за каждые 15 минут получают первый содержательный токен за <= 1,8 с.
Запросы, завершившиеся 5xx или тайм-аутом, включаются в выборку
с длительностью, равной порогу тайм-аута.

The final line matters. If failed requests are removed from latency calculations, p95 can improve during an outage because the slowest calls simply disappear as errors. This familiar observability mistake turns a speed metric into a reward for failure.

The bank should also define where the clocks sit. A practical arrangement uses a server metric at the gateway ingress plus an independent client metric from two bank locations. The server figure helps isolate components, while the client figure shows the application's experience. A difference beyond an agreed tolerance should trigger joint investigation, not automatically make one party's data authoritative.

Routing quality is part of request success

An LLM gateway can return a fast and technically valid response while violating a routing commitment: it can select a prohibited provider, leave the country, substitute an incompatible model, or lose request parameters. For a bank, that is not a «feature degradation» but a failed operation.

The contract needs a registry of routing policies. For every profile, specify permitted models and providers, processing and storage geography, whether a fallback route is allowed, the maximum retry count, and behavior when all options are exhausted. A vendor must not silently send a request to another region to preserve availability if the data policy prohibits it. An explicit failure with a reason code is better than a successful response that violates the processing regime.

It helps to separate transport success from policy success. The first asks whether a valid response arrived. The second confirms that the gateway followed the bank's routing rules. Each operation log should retain the request identifier, selected route, model, result category, duration, and policy applied without exposing request content. Agree the format and retention period for these records separately.

Acceptance testing can use a reproducible sequence:

1. The bank creates a profile that permits two models in one approved geography.
2. The team removes the primary route from the test pool using the agreed method.
3. The gateway sends requests to the permitted fallback and records the switch reason.
4. The team blocks both routes and receives an explicit failure instead of a hidden call to a third provider.
5. The bank matches request identifiers against the routing log and client measurements.

This test exposes a distinction often erased in presentations: fault tolerance means switching to a preapproved path, not any path that happens to be available. The consequence of getting this wrong is measured in data and model requirements, not a few extra seconds of latency.

RTO starts with a confirmed incident, and RPO does not always apply

RTO for an LLM gateway is the maximum time allowed to restore an agreed function after confirmation of an incident at a given severity. It cannot be replaced by the time to the first support response. An engineer may reply in ten minutes while the service remains unavailable for four hours.

Define the start unambiguously as the earlier of the moment the vendor's monitoring detects the breach or the moment the bank supplies sufficient evidence through the emergency channel. Requiring the bank to wait until the vendor «acknowledges» an incident lets the vendor move the starting point. The end is sustained metric recovery for a control window, such as 15 minutes, not one successful request.

Assign RTO by function. Production request execution will usually need a tighter target than historical cost analytics. Key revocation and enforcement of a route ban may need a separate limit because those operations affect security. The contract should distinguish complete restoration from a temporary reduction in function. If emergency mode preserves only one allowed model at reduced throughput, that is partial restoration rather than incident closure.

RPO asks a different question: how much state can be lost after recovery. RPO may not apply to stateless proxying. But a gateway can hold route configurations, key policies, limits, audit records, and usage data. For each kind of state, write either a specific RPO or «not applicable» with a reason. A blanket RPO=0 promise without a state definition sounds strong but cannot be tested.

The recovery plan should be exercised before signature and then tested at an agreed frequency. The bank needs the date, scenario, actual detection and restoration times, deviations, and completed actions. A report saying «test passed» without a timeline does not prove compliance with RTO.

Support must respond, update, and bring in engineers

Support response time is only the first step. For a critical incident, the contract should set four separate timers: ticket acknowledgment, engagement of an on-call engineer, update frequency, and recovery under the RTO.

Describe severity levels by observed effect rather than the vendor's opinion. P1 can mean complete production API unavailability, widespread errors, a confirmed routing-policy breach, or inability to perform an urgent protective action. P2 can mean significant degradation without complete failure when an acceptable workaround exists. If the vendor can unilaterally downgrade priority, it can formally meet the SLA by changing a label. A downgrade should require written reasoning and evidence that the workaround works.

For every severity, the table should state:

• coverage, such as 24x7 for P1 and business hours for P3;
• the contact channel and a fallback if the portal is unavailable;
• acknowledgment time and time to engage a technical specialist;
• the interval between substantive updates;
• deadlines for preliminary and final root-cause reports.

A «substantive update» contains new facts: current impact, action taken, test result, next checkpoint, and any changed forecast. An automated «we are working on it» message does not stop the clock. For P1, the bank needs a phone or another synchronous emergency channel because a ticket inside a failed support console is useless.

The contract should also state what the on-call person can do. If the first line can only forward the ticket to a queue, a 15-minute response promise offers little help. Set a deadline for an engineer who can see gateway telemetry and can switch a route, isolate a faulty component, or start recovery. For a prolonged P1, define when an incident manager and technical owner must join.

A root-cause report must say more than «external provider failure». A sound report includes the timeline, affected requests and policies, immediate cause, reason fallback failed, actions with owners and deadlines, and a way to verify the fix. The bank does not need the vendor's internal secrets, but it does need enough information to assess the chance of recurrence.

An external provider cannot be a universal exclusion

A gateway sells management across several routes, so an external model failure cannot automatically be subtracted from its SLA. That failure is exactly when the gateway has to prove its purpose: detect the problem, select an approved alternative, or return a controlled failure honestly. A blanket provider exclusion leaves the bank with an SLA for a thin proxy layer even though it buys an end-to-end outcome.

A fair rule separates controllable and uncontrollable events. The gateway vendor is responsible for provider selection, health checks, retries, fallback, policy compliance, and correct error forwarding. It does not have to promise that a particular external model will never fail. But if the contract allows a fallback and the gateway did not use it in time, the event counts against gateway availability.

An exclusion is reasonable only if all of these conditions hold:

• the event occurred entirely outside the gateway's control;
• all agreed and permitted fallbacks were also unavailable or prohibited by bank policy;
• the gateway detected the event correctly and performed the defined actions;
• the vendor gave the bank identifiers, a timeline, and evidence of the external cause;
• the exclusion covers only the actual external event, not the whole day.

The vendor should not subtract scheduled maintenance by an external provider if it knew about it and could change the route in advance. Exhausted purchased quota, an expired provider contract, or a bad limit configuration also remain the gateway vendor's responsibility. «Third party» describes an organizational boundary but does not prove lack of control.

Agree a separate mode for pinning, where the bank itself requires one model or provider. In that case, external unavailability can sit in a separate indicator if the gateway proves its own layer remained healthy. Procurement should compare two figures: end-to-end availability for the managed pool and availability of the gateway itself under a pinned route. The first shows the service received, while the second helps establish cause.

Force majeure must not duplicate routine internet failures, provider overload, or capacity shortages. It is an exceptional legal category, not a bucket for technical events. The longer the list of exclusions following a promised percentage, the less that percentage is worth.

Wording about «actions of cloud and telecommunications operators» deserves close scrutiny. If drafted broadly, it excludes nearly all infrastructure on which the service runs. The bank should limit it to events the vendor could not reasonably prevent through redundancy, capacity reserves, or rerouting. A routine failure of one network path does not meet that test when the architecture promises a second path.

Proof of an external cause is more than a screenshot of a public status page. Such a page confirms only a third party's statement and often rounds incident times. The bank needs the gateway's own detection timestamps, response categories, list of affected permitted routes, and automated actions. Aggregated data can be acceptable, but the interval and causal link must be testable.

If a provider changes its API, limits, or terms of use, responsibility depends on notice and the gateway's contractual duty to maintain compatibility. A sudden incompatible change may indeed sit outside its control. Ignoring an announced migration throughout the provided window belongs to gateway management. Set that boundary before the first external API update.

Service credits should rise faster than breach severity

A service credit does not compensate for the bank's loss. It creates an automatic financial consequence without requiring proof of damages. The credit therefore needs to be noticeable to the vendor and must not replace the bank's right to other remedies when the contract and law allow them.

Calculate the tiers from the monthly fee for the affected service, not from a small «support fee» line. For example, a result below the contractual target but at or above 99.90% may yield a credit of 5% of the affected service fee. A result from 99.50% up to 99.90% may yield 10%, one from 99.00% up to 99.50% may yield 20%, and a result below 99.00% may yield 30%. Boundaries must not overlap, so a value exactly on a threshold cannot cause a dispute.

These are not universal figures or a recommendation to accept them without analysis. The bank should relate the tiers to its architecture, fee amount, and cost of fallback operation. The principle is that a severe breach produces a disproportionately larger credit. A flat 5% for any outcome barely changes behavior.

«Affected service» also needs a precise definition. If one production route serves several banking applications, the vendor should not use the cost of one failed model as the base when the outage stopped the whole gateway. Conversely, a localized failure in a test profile need not create a credit against the full annual fee. The contract schedule should map invoice components to measured services and criticality profiles in advance.

It is sensible to calculate an availability credit after permitted exclusions, but the exclusions themselves need separate review. The vendor should show the original failure count, every subtraction, and the final sample. A formula that publishes only the ending percentage cannot reveal whether a disputed maintenance window or external outage reduced the credit improperly.

Currency and settlement mechanics also affect enforceability. The contract should say which invoice receives the credit, whether it can offset the next payment, and what happens if the agreement ends before that invoice. Wording that says a credit is «provided at the vendor's discretion» cancels the automatic consequence and should be removed.

Breaches of p95, RTO, and support obligations must not disappear inside availability. They need separate credits or breach points with clear conversion. The API can be available 99.99% of the time yet answer too slowly for a customer process every evening. In that case, the vendor did not deliver the promised quality. The parties can cap aggregate credits, but that cap should not erase the consequences of several independent breaches.

Automatic crediting is better than a procedure that says «submit a claim within five business days». The vendor already has telemetry and should include the calculation in its monthly report. The bank keeps the right to challenge the numbers using its own measurements. If the contract still requires a claim, the deadline should be reasonable and begin after delivery of the complete report, not after a hidden moment of failure.

Repeat breaches matter more than one credit. The agreement should give the bank a right to demand a remediation plan after repeated failure and a right to terminate without penalty after an agreed number of severe breaches in a rolling period. Otherwise, the vendor can issue small discounts for chronically poor service year after year.

The report must let the bank recalculate every number

A monthly PDF with a green indicator is not enough for SLA control. The bank should receive data that its team can use to reproduce the result: eligible request count, errors by category, latency distribution, excluded intervals, incidents, and credits applied.

A minimum aggregate record can look like this:

{"period":"2026-06","route_profile":"retail-assistant-prod","eligible":1842231,"failed":1320,"ttft_p95_ms":1420,"timeout_count":284,"excluded":91,"exclusion_reason":"bank-approved-maintenance"}

The numbers illustrate the shape, not a standard. The aggregate needs a list of identifiers for excluded events and the calculation-rule version. Request identifiers let the bank check the sample without receiving prompt text. If the vendor changes the metric algorithm, the new version should apply only after agreement and must not rewrite past periods.

Synchronize both parties' clocks, and specify the time format and permitted skew. The bank should retain its own synthetic checks and application metrics. Synthetic traffic detects a total failure when real traffic is low, while production requests show the effect under typical load. Neither source should be declared the only source of truth.

In a dispute, the parties should first compare the definition of an eligible request, time windows, and identifiers, then compare raw events. A clause saying «vendor data is final» makes independent control decorative. It is better to set a reconciliation deadline, export format, and a process for engaging an independent expert in the rare dispute that cannot be resolved.

Retention need not be unlimited, but it must be long enough for bank controls, investigations, and the contractual claim period. Risk, security, and records specialists should choose the exact period together. The SLA should guarantee report availability and prevent silent changes to a published version.

AI Router provides one OpenAI-compatible endpoint, routing across external providers and self-hosted models, plus audit logs and per-key limits. A bank procurement still needs to turn those capabilities into specific SLIs, thresholds, and evidence because a feature list does not create a commitment.

Acceptance testing should try to break the contract

Before signature, the procurement team should apply the SLA to several prepared failures. If the parties cannot calculate the same result in a test, a production outage will make the dispute longer and more expensive.

A sound acceptance set includes a slow successful response, a 5xx from an external model, a stream break after the first tokens, a prohibited fallback region, exhausted provider quota, console unavailability, and key revocation during degradation. For every event, record the expected category, timer start and end, permitted fallback, log entry, and effect on the credit in advance.

A cascade scenario is particularly useful. The primary model starts returning 429, the gateway retries too many times, the queue grows, and the fallback then receives a delayed wave of traffic. An external provider started the event, but the gateway's retry policy amplified the impact. A contract that excludes the full interval as an external outage hides the controllable part of the incident. A sound formula leaves detection delay, excessive retries, and failed switching with the gateway.

The procurement comparison should require an exact value and a reference to the relevant proposal clause, not a «yes» box. The fields are straightforward: measurement object, formula, window, thresholds by profile, RTO, support, external exclusions, evidence, credits, repeat breaches, and exit rights. An empty field means no commitment, even if the presentation promises «high availability».

Do not accept a promise that cannot be tested without internal vendor access. A good SLA gives both parties the same arithmetic and enough observable facts. If a vendor refuses to include external model errors, ask what part of the final outcome it will accept responsibility for and how many minutes of actual failure its wording permits. That calculation usually makes the advertised percentage much less impressive.

The bank should sign the percentage with the least room for dispute, not the highest one. It benefits from a contract where a slow, misrouted, or broken response counts as failure, RTO is separate from response time, and every exclusion needs evidence. Those are the lines people will read at three in the morning when the service status and customer experience no longer agree.