How banks should choose an LLM gateway or direct contracts

A bank should sign a direct contract with each provider only when it needs a particular model or cloud function as a strategic dependency and is ready to manage its risk separately. If teams need several model families, consistent access rules, local data storage, and fast switching during an outage, a gateway usually creates a more manageable setup.

This is not a choice between a "serious contract" and a convenient API. Both options require legal review, a security assessment, and a named risk owner. The difference is whether the bank repeats that control process for every provider or reviews one layer that then manages its own provider relationships. I have seen projects where a one-week pilot turned into a six-month argument between procurement, security, and engineering because the team made this decision after writing the code.

What the bank is actually choosing

The bank is choosing a control boundary, not a model. A direct contract puts that boundary between a banking application and each external API. A gateway inserts a common control layer between them: the application calls one API, while the layer applies policy and forwards the approved request to the selected provider or a locally hosted model.

People often blur three different ideas here. A "local model" means that the weights and compute run in controlled infrastructure. A "local gateway" means that the ingress point, logs, and routing rules sit in the required jurisdiction, although some requests may still go to an external API. A "model aggregator" may offer a shared catalog but does not necessarily provide bank-grade logs, key separation, or local storage. If procurement uses these terms as synonyms, suppliers will answer three different questions and the committee will compare proposals that are not comparable.

First, define four boundaries: where the bank's network ends, where the request is stripped of personal data, who chooses the model, and where the technical record of that decision remains. The architecture diagram then becomes a contractual subject. Without those boundaries, the phrase "data is not stored" is nearly useless because it does not say which data, in which component, or at which stage.

Provider review multiplies with the contract count

Direct access to three providers means three separate review cycles, not one larger review. Each provider has its own data processing terms, security questionnaire, subprocessor list, incident notice procedure, financial review, tax documents, and cross-border transfer approval. Even when the questions match, the answers and contract language will not.

A gateway reduces the bank's immediate counterparty count, but it does not remove the need to review the chain. The bank must see which downstream providers the gateway uses, which requests it may send to each one, how it reports changes to the list, and whether it can add a new provider without approval. One contract helps only when it carries obligations across the whole chain instead of hiding that chain behind a general reference to partners.

The slowest unresolved issue determines review time, not the lawyers' typing speed. Data location, audit rights, log content, or deletion procedures usually become that issue. This is why a promise of "connection in a few days" tells you nothing when comparing options. Ask both sides to complete the same evidence register: document, owner, validity period, applicable service, and exceptions. A gateway wins when that register covers several models with one control. Direct contracts win when a provider offers unique evidence or an obligation that an intermediary cannot pass through to the bank.

A sensible procurement process also defines when to review again. A new data type, a new processing region, a model with tools, or a change of subprocessor should send part of the assessment back for consideration. Otherwise, an approved text API quietly becomes a service with files, web search, and persistent state even though the original assessment covered none of them.

One contract does not create one responsibility

With a gateway, the bank gets one commercial contact, but technical responsibility remains divided. The model may produce a bad answer, an external provider may slow down, the gateway may apply the wrong route, or the banking application may send unnecessary data. The contract should assign those cases to owners, response times, and evidence.

The clause "the supplier is responsible for service availability" is too coarse. An investigation needs at least four events: the request did not reach the gateway, the gateway rejected it under policy, the chosen provider returned an error, or the response arrived on time but failed the bank's quality check. The application or network team owns the first case, the policy owner owns the second, the route operator owns the third, and the business use case owner owns the fourth. Financial compensation for availability does not answer who restores customer service in the next ten minutes.

With direct contracts, the chain is shorter for one call: the bank sees the response from a specific API and opens a ticket with that provider. But if the application chooses among three APIs itself, the bank has effectively built its own gateway. It then owns the compatibility table, error normalization, retries, limits, logs, and on-call support. Teams often leave this work out of the estimate because it is spread across repositories and departments.

A useful contract annex should answer a simple question: which fields will the bank receive for an investigation without disclosing more request content than necessary? It needs a request identifier, time, route, selected model and version, policy decision, attempt count, result code, and error source. If a supplier cannot return that set, remove the promise of "complete observability" before signing.

Centralized access reduces the error surface

A gateway gives the bank one place to issue keys, limits, and permissions, so the security team can separate the right to call the API from the right to choose any model. That distinction matters. A key that works technically does not mean its holder may send a particular data class or enable model tools.

With direct integrations, each console has its own roles, projects, key formats, and logs. The bank can connect them to its corporate identity system, but it must prove that the same rules apply in all three environments. Drift appears quietly: a test project receives a production key, an old service account survives after a pilot closes, and one provider records a model under a different name.

Set controls through policy at the banking application level, not through people's names in a provider console. The smallest useful access unit includes application, environment, data class, allowed models, budget, request rate, and expiry. The following example is an internal policy contract, not configuration for a specific product:

{
  "client": "contact-center-prod",
  "data_class": "internal-masked",
  "allowed_models": ["model-a-stable", "model-b-stable"],
  "tools": [],
  "requests_per_minute": 120,
  "monthly_budget_kzt": 900000,
  "expires_at": "2026-12-31"
}

This record prevents two familiar failures. The application cannot move on its own to a model that was not reviewed for that data class, and a leaked key does not receive unlimited, permanent access. Direct contracts can produce the same result, but the bank must maintain a translation from one policy into each provider's controls and check automatically for drift.

Storage location is not processing location

For a Kazakhstani bank, a region name in a commercial proposal cannot settle the residency question. The Republic of Kazakhstan's rules for collecting and processing personal data require personal data to be stored in a database located in the country. The law separately regulates cross-border transfer. Architects and lawyers must break the request path into operations: initial collection, storage, temporary caching, processing, logging, backup, and support access.

OpenAI's official documentation shows clearly why a region switch alone is insufficient. It separates customer content from system data, describes standard abuse monitoring logs, and notes that some features create application state. Anthropic's Zero Data Retention material also says that the arrangement applies to the API and that some storage features can follow separate rules. Google lists retention scenarios for Vertex AI and the settings needed for zero retention. I would not accept the phrase "zero retention" without an endpoint-by-endpoint table that includes every enabled feature.

A local gateway can receive a request in Kazakhstan, mask PII, write a local audit record, and send only the permitted portion outside. It cannot turn an external call into local processing. If the filtered text still contains personal data, bank secrecy information, or details that can reidentify a customer, cross-border risk remains. Masking also needs tests against real formats: names, individual identification numbers, contract numbers, free text from agents, and attachments behave differently.

A use case whose data must not leave the country needs a route to a model hosted in the country and a ban on external fallback. For less sensitive use cases, the bank may allow external models after data minimization. A good policy makes that decision before sending the request. A bad policy hopes the developer remembers the right flag in every call.

Resilience begins with response semantics

Automatic provider switching helps only for tasks where another model can produce an acceptable result. An HTTP 200 from a backup model does not mean the service has recovered. The model may differ in structured output format, context length, safety behavior, tool use, and quality on banking terminology.

A gateway makes it easier to apply one timeout, bounded retries, and a fallback route. Direct APIs give the team full control, but it must build and maintain the same mechanism inside the application. In either case, do not retry after an uncertain result without idempotency: the first attempt may have completed an action even if its response was lost. That is tolerable for drafting text. It is an operational risk for an agent that creates a case or changes an application status.

Before launch, run one reproducible failure test:

1. Fix ten reference requests, acceptable answers, and prohibited actions.
2. Force the primary provider to time out after the request has been sent.
3. Check which route the access layer chose and how many attempts it made.
4. Compare the backup response's structure and quality with the defined thresholds.
5. Confirm that the log joins both attempts under one identifier and names the source of failure.

If the team cannot run this test without manually opening three consoles, it is not ready to promise automatic fallback. For a critical process, stopping the feature and showing a controlled error is sometimes safer than silently moving to a less thoroughly reviewed model.

Token price hides most of the cost

Price lists answer only the question of variable call cost. A bank will also pay for provider review, contract work, identity integration, logs, budget controls, SDK support, testing new versions, on-call coverage, and invoice reconciliation. With several direct contracts, some of these costs repeat even though the project budget records them as staff time.

A gateway has its own economics: a possible service fee, added latency, the cost of local infrastructure, and intermediary dependency. Compare those costs with the internal cost of a control layer, not just with a markup on tokens. If a gateway charges provider rates without an API markup, that removes one question but does not make operations free. Ask how the business earns money, which services cost extra, and how the bank can reconcile the bill against route logs.

Model the cost over a year. Use the number of production applications, expected models, months of procurement work, security and legal hours, adapter development, update testing, and an incident cost expressed in units the bank is comfortable using. You do not need to invent an exact outage price. Compare the options across several understandable loss levels and see whether the decision changes.

A direct contract often makes sense for a large, stable volume with one provider, special commercial terms, or a feature the gateway cannot pass through without loss. A gateway usually wins with a changing model portfolio and many internal consumers. A hybrid setup can cost less than either extreme: a direct route for one strategic service, a shared layer for experiments and ordinary work, and a local model for restricted data.

A gateway adds its own concentration risk

One control point also becomes one dependency. A policy error can block every application, compromised administrative access can expose several routes, and a bad adapter release can distort responses from different models. That is not a reason to reject a gateway. It is a requirement to assess it as a critical banking component.

Ask for the client and key isolation design, change release process, policy version history, support engineer access procedure, and recovery plan. Check separately whether the bank can export configuration, logs, and the model mapping table. Without portable state, replacing the gateway becomes another procurement project conducted under outage pressure.

The catalog raises another uncomfortable question. Hundreds of available models expand technical choice, but a bank policy should not permit hundreds of models. The catalog and the approved register solve different problems. The first shows what is technically possible. The second contains a small number of reviewed versions with an owner, allowed data classes, evaluation results, and a review date.

I disagree with the popular advice that an intermediary always reduces reliability and a direct connection is always better. It holds for one static API and fails for a portfolio. When each application writes its own retries, stores keys, and interprets errors, intermediaries already exist. They have simply multiplied inside the bank's code and nobody operates them as one service.

The decision matrix must identify risk owners

Do not choose by voting for a brand. Use a matrix in which every criterion has evidence and an owner. The weight of a criterion depends on the use case: an employee chat, an agent suggestion, document analysis, and an agent allowed to act cannot fit in one row. The bank should describe the consequence of error first and then compare delivery models. Reversing that order usually bends requirements around a service the team already likes.

For provider review, the direct option creates a separate cycle for each counterparty. Evidence includes a completed register of questionnaires, certificates, processing terms, subprocessors, and feature exceptions. A gateway gets one primary cycle, but its register must disclose the supply chain. Vendor management or procurement owns this criterion, while security confirms the technical part. If a gateway does not report a subprocessor change, review savings become a blind spot.

For contracts and invoices, the direct option means several terms, currencies, consumption thresholds, and dispute procedures. A gateway combines them into one commercial window, but the bank should receive spending detail by application, model, and route. Finance verifies that logs can reproduce the invoice, while legal establishes which terms prevail if the gateway agreement conflicts with an upstream provider restriction. "At the model's rate" is not enough for control without rules for rounding, caching, and repeated requests.

For access, direct contracts provide roles and keys in each console. Evidence is not a screenshot but a recurring machine-readable export of users, service accounts, permissions, and last activity. A gateway should provide an equivalent export of the common policy and its change history. The bank's IAM team remains the owner even if the supplier issues keys. Outsourcing the operation does not transfer responsibility for deciding who inside the bank gets access.

For residency, the direct option depends on the region, endpoint, and enabled feature of each service. With a gateway, assess local ingress, local logs, the external route, and the local route separately. The evidence is an operation-level data map, not a picture with geographic labels. It records request and response content, metadata, cache, logs, backups, processing location, and deletion period. Legal defines applicable restrictions, the data owner approves permitted content, and the architect proves that the actual configuration matches the map.

For outages, a direct contract gives the bank a path to a specific provider and fewer intermediate components. A gateway gives common diagnostics and may switch routes, but it adds a failure point of its own. Evidence is a test report with attempt identifiers, error codes, switching time, and the backup answer's validation result. SRE or production support accepts this criterion only after a test. A service level agreement without a reproducible scenario says nothing about recovery of a particular banking function.

For exit, direct contracts require migration of each adapter and another comparison of model behavior. A gateway may keep the API stable, but it creates dependency on the policy, log, and catalog formats. Before contracting, the bank should request configuration, log, model register, and evaluation exports in a readable format. The architecture owner runs a trial migration of one application to an alternative endpoint. If that move cannot happen without the current supplier's help, the exit plan exists only on paper.

Add change speed to the matrix. Providers update model versions, retention rules, and available features outside the bank's calendar. With direct contracts, each team watches its own service or a central group builds a shared register. A gateway can normalize changes, but the bank must learn about a new version before automatic switching. Evidence includes notice, a review window, version pinning, and a clear rollback. Model governance should own this work, not the developer who happens to spot an answer changing first.

Assess quality separately because a catalog entry does not prove model fitness. The bank defines its own request set, languages, formats, prohibited actions, and thresholds. A provider or gateway may help run the evaluation, but the banking use case owner makes the admission decision. An average score cannot be a pass: a model may perform well overall and fail a rare case with severe consequences. Tie results to the exact model and routing policy version.

Give each criterion one of four states: proven, proven with a restriction, accepted as residual risk, or unproven. This is not a sheet of green checkmarks. "Proven with a restriction" must name the allowed data class or feature, while residual risk needs a review date and approval from someone authorized to accept it. An unproven criterion blocks only the affected route, not necessarily the entire pilot. This lets the bank test a safe use case without granting the experiment access to every data class.

After the matrix is complete, every disputed cell has a person who accepts the residual risk. If nobody will approve external storage, the route cannot open. If the business accepts a stop during an outage, there is no need to pay for elaborate fallback. The matrix turns a vague claim of "safer" into a verifiable decision and prevents collective discussion from dissolving accountability.

The NIST AI Risk Management Framework organizes work into govern, map, measure, and manage, and explicitly says its actions are not a universal checklist. That is a useful procurement lens: the contract belongs to govern, the data flow map to map, quality and failure tests to measure, and switching and incident handling to manage. Buying an API completes only part of the first function. A complete matrix shows who keeps working after signature because model risk changes with the use case, data, and version.

A hybrid setup is usually more honest than a binary choice

A bank rarely needs to choose one path forever for every workload. Split routes by error consequences and data sensitivity. Restricted documents and identifiers stay on a locally hosted model. Ordinary text after masking may go through a gateway to an approved external provider. A unique feature from a strategic provider gets a direct contract if an intermediary cannot preserve its behavior or contractual assurances.

AI Router offers one OpenAI-compatible endpoint that can route requests to external models and locally hosted open-weight models, apply PII masking, retain audit logs, and enforce limits at key level. For a bank, it is a candidate for the common control layer, not a reason to skip its own assessment of the supply chain, data modes, and recovery.

Start the pilot with two routes and a predefined failure, not with the model that writes the most impressive answer. One route must forbid data from leaving the country. The other may use an external provider after masking. Apply the same access policy, logging, limit, and failure test, then hand the evidence to security, legal, and the process owner.

The decision is ready for procurement when the bank can explain the fate of one request without a supplier presentation: who had the right to send it, which data was removed, where it was processed, why that model was chosen, what the log recorded, and who acts on an error. If different teams answer each question and their answers conflict, the argument about direct contracts is premature. Define the control boundary the bank can actually operate first.