A local gateway does not guarantee data sovereignty

A local API gateway gives you control over the system's entry point, but it does not preserve data sovereignty by itself. If the gateway sends a prompt to an overseas provider, the text leaves the local perimeter for at least the duration of processing. The claim that "data stays with us" is false in this design, even when the provider promises not to store anything.

The argument usually breaks down over the word "store". A team points to its local log database and considers the matter settled. Yet an overseas model receives the text, the network layer sees addresses and headers, security systems inspect the request, and backups may follow the rules of a completely different service. Accumulation, transfer, processing, temporary caching, and staff access all need separate checks.

I consider a perimeter local only when the processing location, recipient, lifetime of each copy, and proof of deletion are known for every data class. If even one row in that map is unknown, the architecture has an unverified assumption, not sovereignty.

A gateway controls the route, not all processing

A local gateway decides where a request goes, which fields it passes, and what it writes to its own logs. It does not change the model's execution location, the external provider's rules, or the geography of that provider's supporting systems.

Separate three properties that teams often conflate. Data residency answers where data is stored. Processing locality answers where the result is computed. Data sovereignty is broader: who controls the data under which jurisdiction, who can access it, what copies appear, and whether the organization can prove compliance with its own rules. A local log provides residency for one copy. It says nothing about a copy in a foreign API queue.

In a typical request, the application opens a TLS connection to a gateway inside Kazakhstan. The gateway authenticates the client, applies a limit, may mask fields, and then opens a second TLS connection to the provider. Encryption protects both channels in transit. At each endpoint, the text is decrypted for processing. The phrase "everything is encrypted" therefore does not prove that the overseas party never saw the prompt.

There is a return path too. A generated answer may contain source data that the model restored from context: a patient's name from the question, a contract number from an instruction, or part of an internal conversation. That answer passes through the provider's filters, the gateway, and application logs. The map must cover input and output, not only the outgoing prompt.

The practical test is simple. If an external operator must receive plaintext to perform the task, processing has occurred outside the local perimeter. A contract may make it permissible. A technical design may reduce the risk. But neither turns it into local processing.

Every request has four separate flows

You need to examine four flows: content, metadata, telemetry, and durable copies. They have different recipients and lifetimes, so one "not used for training" checkbox does not settle the issue.

Content includes the system prompt, user messages, RAG documents, tool call results, and the model's answer. For a multimodal request, it also includes images, audio, and text extracted from them. Teams often mask the messages field but forget that personal data has entered a file name, a tool description, or a diagnostic error excerpt.

Metadata exists even for an empty prompt. It includes a project identifier, API key or its fingerprint, IP address, region, model name, token count, request time, error code, and trace identifier. Each item can look harmless alone. Together, a department identifier, usage frequency, and a model selected for medical text may disclose the system's purpose or a specific workflow.

Telemetry lives in load balancers, WAFs, APM systems, traces, cost meters, and abuse detection systems. The gateway might not log prompts while an observability library automatically captures an HTTP request body on error. This is how a careful application policy loses to an agent setting that nobody included in the data diagram.

Durable copies appear in backups, delayed queues, prompt caches, evaluation datasets, support exports, and crash dumps. The retention period for a primary table does not apply to a disk snapshot. Deleting a record today does not mean it vanished from a backup that can only be restored as a whole.

For each flow, fill in four fields: source data class, transformation before transfer, every recipient, and maximum lifetime. An empty unknown value is more useful than an invented 0 days: it shows directly that the route cannot yet be approved.

Overseas generation always crosses the border

If a model runs abroad and receives the payload, cross-border transfer and external processing occur even when the provider claims zero data retention. Zero retention describes the state after processing. It does not cancel the fact that the provider received the data.

The Law of the Republic of Kazakhstan On Personal Data and Their Protection defines accumulation, storage, use, dissemination, depersonalization, and processing separately. That distinction matters to an architecture decision: no accumulation at the provider does not mean no processing. Lawyers must determine the legal basis and transfer conditions for the specific dataset, while engineers must give them an exact diagram instead of the promise "we have a local endpoint".

The supplier's country of incorporation does not provide the answer either. You need the execution region of the selected model, the content filtering region, the location of security logs, the support service location, and the rules for remote access. A global API endpoint may choose a region on its own. A region shown on an invoice may refer to a management resource rather than every inference stage.

Do not rely on general wording from a privacy page. Check the terms for the specific service, model, plan, and feature. Google Cloud's zero data retention documentation, for example, separately describes prompt logging for abuse monitoring, in-memory caching, and grounding features with their own retention rules. Microsoft's Azure Direct Models documentation separately explains automated review, possible storage of selected prompts for human review, and modified monitoring for approved customers. I agree with that separation: a brand-level promise is too coarse for an architecture control.

Permitted transfer and local processing remain different statuses. Write exactly that in the decision register: external_processing: approved, not data_stays_local: true. Signing a contract does not make the second statement true.

Masking reduces transfer volume but does not change geography

Masking helps only when the external provider receives data that cannot reasonably be linked to a person or protected object. Replacing a few obvious fields while preserving context often leaves a route back to identification.

Consider this request: "Draft a response to the chief physician of the district hospital about a complication after a rare operation on 12 May; the patient is 14". Removing the name and national identification number does not make this text anonymous. The recipient's role, institution, date, age, and rare event form quasi-identifiers. The model is still processing information connected to a very small group of people.

A workable design uses tokenization before the gateway or inside the trusted zone. The application replaces specific values with random tokens, stores the mapping locally, and restores values after generation only where permitted. Tokens must not encode the source value, and the external request must not contain the recovery key. Free text needs an entity detector plus domain rules because a regular expression will find a national identification number but miss a diagnosis and the description of a rare case.

Test masking quality on your own corpus. Build a set of examples with names, addresses, contract numbers, diagnoses, internal codes, and quasi-identifiers. Measure missed detections and unnecessary removals for every example. Average accuracy offers little comfort if the one missed account number already violates the transfer rule.

Some tasks lose their meaning after masking. A model cannot check a contract accurately if the relationships between parties and amounts have been destroyed. There are two honest options in that case: run a suitable model inside the local perimeter, or do not apply an LLM to that data class. "Just anonymize everything" is popular advice because it promises to preserve one route. It is wrong when context is needed for the result or reidentification remains possible.

Provider policy must become a machine rule

A contract and documentation help only after the team turns their restrictions into an enforceable routing policy. Otherwise, a new developer will select a prohibited model by its short name, and a fallback route will quietly bypass the approved region.

The fragment below is a minimal policy. The field names are illustrative, but the behavior must be real: the gateway rejects a request when its classification and route are incompatible.

data_classes:
  public:
    allowed_routes: [external, local]
  internal:
    allowed_routes: [external_zdr, local]
    require_redaction: true
  restricted:
    allowed_routes: [local]

routes:
  external_zdr:
    region_allowlist: [approved-region]
    prompt_retention: none
    training_use: disabled
    human_review: disabled
    fallback: deny
  local:
    processing_country: KZ
    fallback: deny

Do not fill in prompt_retention: none based on a sales manager's word. Attach the terms version, review date, list of exceptions, and proof that the setting is active. If zero retention requires separate approval or disabling a feature, the status must remain pending until the team sees confirmation in the control panel or management API.

Check training, abuse monitoring, human review, support, caching, and conversation state separately. A training ban does not prohibit a security log. A zero retention term for basic inference may not apply to batch, assistants, grounding, files, or fine-tuning. The same model name through two intermediaries does not imply the same terms.

The policy must fail closed. If the classifier is unavailable, the region is unknown, or an approved route has exhausted its limit, the request must not go to "any model that works". Availability matters, but it does not authorize the system to move its processing boundary.

Fallback most often breaks the promise

Failover keeps the service running, but it is also the mechanism most likely to move data outside the approved perimeter. The primary route passes review, while someone adds the backup at night after the first outage and leaves it broadly configured.

Walk through the failure over time. The application sends a request to a local model. It responds slowly, and the client library decides that a timeout occurred while the server keeps computing. The gateway repeats the same prompt with an external provider. Two models have now processed the data, and the late local answer may enter a log as an "unknown error". If retries do not share one identifier, the audit sees two unrelated events.

This failure has four unpleasant consequences. The team loses an unambiguous recipient list. One request produces several copies. Billing and metrics stop revealing the actual cause. The user does not know that the processing mode changed. All of this happens without a client code change.

Set fallback: deny for restricted data. For less sensitive classes, allow a backup route only inside the same group of conditions: the same permitted geography, retention, review mode, and feature set. Compare routes by policy, not model quality.

Retries need an idempotent identifier at the gateway and a record of every attempt. The log should include the original request ID, attempt number, selected route, switching reason, data class, and policy result. It does not need the prompt itself. This lets the team prove where the request tried to go without creating another content copy.

A cache is also a route. A semantic cache may return an answer without calling a model, but its vector representation and stored response remain data. Check the index location, tenant isolation, TTL, backup, and deletion. A cache in the local gateway improves control only if its storage is local and its lifetime is clear.

Audits need events, not screenshots

Evidence of sovereignty should be collected for every request and checked automatically. A configuration screenshot shows the interface at one moment, but it does not prove the route of yesterday's call.

Record a decision event without prompt content. This JSON can go to a local immutable store and be joined to the application log:

{
  "request_id": "01J...",
  "data_class": "restricted",
  "policy_version": "2026-07-27.3",
  "route": "local-qwen",
  "processing_country": "KZ",
  "content_logged": false,
  "fallback": "deny",
  "decision": "allow",
  "reason": "class_route_match"
}

A rejection should have the same shape with decision: "deny" and a specific reason. The audit test then becomes a query over events, not a conversation with the platform owner. For example, find every event in the restricted class where the country is not KZ, the route begins with external, or the policy version is missing. The expected result for the claim "restricted data does not leave the country" is zero rows.

The gateway log is not enough. Reconcile it with outbound network flows, provider invoices, and model metrics. If the provider bills tokens but the local log knows nothing about the request, you have a bypass or an observability gap. If the gateway logged an external call but network controls did not see it, inspect telemetry export and clock trust.

Protect the log itself with separate write and read permissions, time synchronization, a signature or immutable storage, a limited retention period, and identifier masking. An audit log containing full prompts becomes the system's most complete and therefore most dangerous database. Keep the decision and minimum metadata, not the content.

Run negative tests with every release. Send a synthetic restricted request, disable the local model, exhaust the approved provider's quota, and replace the region in configuration. The system should reject each attempt, and the log should explain every rejection. A successful happy path does not test the boundary.

Permissibility depends on data class and purpose

Overseas generation may be permissible when an organization knows the data composition, has a legal basis, approves the recipients, and technically keeps the route within those conditions. There is no universal answer for every prompt.

Public content can usually go to an external model after contract terms have been checked. It includes published material, public instructions, and synthetic tests without hidden identifiers. Even then, do not transfer internal keys, traces, or private system prompts: a public document does not make the entire request public.

Internal data needs purpose assessment and minimization. A marketing draft and a financial forecast cause different harm when disclosed, although both may carry the internal label. Add domain subclasses: trade secrets, personal data, banking secrecy, medical information, and government data. The label should come from the source system, not from a model guessing after it has received the text.

Send restricted data only through a route that policy explicitly allows. If a rule requires processing in Kazakhstan, the model must run in Kazakhstan, and its logs, caches, backups, and administrative access must satisfy the same decision. A local gateway in front of external inference does not meet that requirement.

Record approval in a short decision entry: processing purpose, field classes, recipients and subprocessors, countries, lifetime of every copy, training and monitoring modes, transfer basis, risk owner, and review date. Do not bury exceptions in a contract appendix. Put them into route restrictions and tests.

A lawyer determines permissibility, the data owner accepts residual risk, and an engineer proves the actual path. When an engineer declares a transfer lawful or a lawyer infers the technical route from a presentation, the organization gets a confident answer without verifiable grounds.

Subprocessors extend the access boundary

Every subprocessor that receives content or a recoverable derivative belongs inside the actual processing boundary. The model supplier's name on an architecture diagram does not describe the whole chain.

Behind one API, there may be a separate cloud operator, content filtering service, abuse detection system, observability supplier, and support service. One party sees the full prompt, another sees selected excerpts, and a third sees metadata. An assessment needs every party and its role. "Data is transferred to the model provider" is too narrow if the provider may engage other companies under the contract.

Ask for the current subprocessor list and connect it to the specific service. A large cloud's general list may contain dozens of companies that never touch your route, so a link to the register alone is not enough. The supplier should explain which recipient categories handle inference, security, support, and disaster recovery, where they work, and what access they receive. Record the list version and the notification mechanism for changes.

Draw the chain as a sequence of trust: application, local gateway, routing intermediary, provider endpoint, filter, model, security log, backup system, and support. For each transition, mark plaintext, pseudonymized text, metadata, or an aggregate. This label matters more than the legal name of a role. A company that receives a request ID is not equivalent to one whose engineer can open the stored prompt using that ID.

Support creates a separate path that does not appear in the normal network flow. A developer receives an error, copies the prompt into a ticket, attaches a trace, and sends the data to a system with a different region and retention period. Prohibit content examples in support tickets for restricted classes. Send the request ID, error code, route version, and a synthetic example for diagnosis. If the provider needs the real payload, treat it as a separate transfer with an owner, deadline, and confirmed deletion.

Subprocessor lists change. Put change review into the route lifecycle: a new name, country, or purpose moves the external configuration into review. The route may keep working for public data, while regulated classes are better denied until assessment. Automatic list import does not make the decision, but it stops a change from disappearing in an inbox.

Deletion also travels down the chain. A provider may delete an active record immediately, keep it in an isolated backup until the cycle expires, and restrict restoration to disaster cases. That is not the same as immediate physical destruction. Record the actual mechanism, maximum period, behavior during restoration, and the way deletion is reapplied after restoration. If a supplier cannot answer for every layer, do not assign it data that requires provable destruction.

A network allowlist proves which endpoint the gateway contacted. It does not reveal internal transitions after the request was accepted. Evidence therefore has two parts: your events and network flows prove the actual transfer, while supplier contracts, settings, and reports describe subsequent processing. Neither part replaces the other.

Test sovereignty by changing the model

A sound architecture preserves its rules when the model, provider, and failure mode change. If sovereignty depends on one plan name in a slide deck, the next catalog update will remove it.

Create a route register that ties every model version to its inference location, retention terms, features, subprocessors, and next review date. The gateway allows a route only through a stable policy identifier. A best-model alias without fixed conditions is unacceptable for regulated data: it points to a local model today, while an optimizer may select an overseas model tomorrow.

Run a substitution test. Take one synthetic request from every class, then change the primary model, region, cache mode, and fallback in turn. After each change, check the policy result, outbound network flow, and audit event. If changing the model requires somebody to remember a legal restriction manually, the control still exists only in people's heads.

AI Router can separate these routes behind one OpenAI-compatible endpoint: use external models for permitted classes and assign open-weight models on its own GPU infrastructure in Kazakhstan to data that needs local processing. PII masking, audit logs, and per-key limits help enforce the policy, but each organization still has to define its classifications and permitted routes.

Do not issue one green status saying "the gateway is local". Publish a matrix by class: local storage, local processing, external processing approved, external processing prohibited. That status survives a supplier change and does not mislead the data owner.

The perimeter boundary does not follow the address used by the client application. It runs through the last system that receives plaintext, derived data, or a recoverable copy. When the team can name that system, show the routing rule, and get zero results from a negative audit, the sovereignty claim is verifiable. In every other case, it remains a promise.