How to Keep LLM Stack Backups in the Country?

A backup of an LLM stack is usable only when you can bring the application up in the permitted territory, decrypt the data through the standard process, and get the same behavior wherever that behavior is expected to remain unchanged. An archive with several folders in object storage is not enough. It may contain terabytes of data and still fail to restore search, access rights, a fine-tuned model, or audit records.

Keeping data in the country makes the task stricter. You cannot assume that only the primary cluster is local and that backups will somehow be handled by a cloud provider's settings. The territorial requirement applies to copies, temporary objects, catalogs, keys, job logs, and the disaster-recovery site. If even one layer moves to an unapproved jurisdiction, the recovery plan no longer meets the original restriction.

Back up the state the response depends on, not just the services

An LLM application rarely consists of one model and one database. A user response comes from a chain: the request passes through authentication, filtering rules, knowledge search, context assembly, a model call, moderation, and logging. Restoring Kubernetes manifests without the data behind this chain restores an empty shell.

Group artifacts by how they can be brought back:

• primary data: source documents, structured records, media, access settings, and consents;
• derived data: chunks, embeddings, vector indexes, caches, extraction results, and retrained dictionaries;
• executable state: checkpoints, tokenizers, LoRA adapters, inference configuration, images, and prompt templates;
• evidentiary state: audit logs, access events, policy versions, pipeline logs, and evaluation results;
• cryptographic state: keys, key versions, access policies, certificate chains, and KMS or HSM parameters.

These groups have different RPOs and RTOs. Losing six hours of cache is usually unpleasant but tolerable. Losing a consent record, the latest ACL version, or the key for an active archive can stop the service completely. Do not assign one retention period “for the entire LLM.” It hides the dependencies that later break recovery.

A useful minimum registry looks like this:

artifact: knowledge-search-prod
owner: ml-platform
classification: confidential
territory: KZ
source_of_truth: postgres-documents
rebuildable: true
rpo: 30m
rto: 4h
backup:
  primary_copy: kz-dc-a
  recovery_copy: kz-dc-b
  encryption_key: kms://kz-hsm/keys/rag-prod-backup-v7
restore_dependencies:
  - document-store
  - embedding-model-qwen3-embedding-8b@sha256:...
  - chunker-config@2026-07-01
  - acl-schema@v4
validation:
  - checksum
  - filtered-search-smoke-test
  - access-denial-test

This is not bureaucracy for the sake of a spreadsheet. The rebuildable: true line forces the owner to answer an uncomfortable question: exactly which data and which embedding version will be used to rebuild the index? If there is no precise answer, the index must be saved as an independent artifact.

A vector database without its corpus and embedding version cannot restore search

A vector index can be copied byte for byte and still produce the wrong system. The cause is usually not the database engine itself but missing context: document splitting changed, a filter field disappeared, the embedding model was updated, or the application began normalizing text differently.

Distinguish between two scenarios. In the first, you restore an exact collection snapshot with its segments, payload, index files, and engine metadata. In the second, you rebuild the collection from the source corpus. The first is faster. The second is cleaner when the schema changes, but it requires the complete input set and a repeatable pipeline.

For every production index, record at least this in the manifest:

{
  "collection": "support-kz-ru",
  "snapshot_time": "2026-07-23T02:30:00Z",
  "embedding_model": "approved-embedding-model",
  "embedding_revision": "sha256:9d4...",
  "dimension": 3072,
  "distance": "cosine",
  "chunking": {
    "parser": "[email protected]",
    "max_tokens": 700,
    "overlap_tokens": 100
  },
  "payload_schema": "acl-payload@v4",
  "source_cursor": "documents:842771",
  "checksum": "sha256:..."
}

Do not replace a checksum with the test “the collection opened.” An index may open while losing segments, filters, or metadata. After recovery, run a fixed set of queries: a regular search, a search filtered by department, a request from a user without permission to the document, and a request for a deleted document. The expected result of the last test should often be empty. It shows that you did not restore a data leak along with the data.

A vector database snapshot does not replace a copy of the source corpus. The index does not retain the original document in a form suitable for legal export, reparsing, or changing the model. A corpus without a pipeline version does not replace an index snapshot when the RTO does not allow several days to recalculate billions of vectors. Keep both layers if search participates in a critical process.

A checkpoint must stay with its file family

A file weighing tens of gigabytes and named final-v3 is almost never a recoverable model. To run a checkpoint predictably, you need the exact base-model revision, tokenizer, architecture configuration, chat template, quantization parameters, and inference engine version. Adapters also require the adapter type, target layers, and compatibility with the base model.

Teams most often lose not the weights themselves. The weights remain in storage, while the configuration file, runtime image, or exact base-checkpoint identifier disappears. Six months later, someone substitutes an “almost identical” model and gets different instruction following, tool calls, context length, or response language. This failure is hard to notice: the service responds, but the business process has already changed.

Package the model as an immutable object:

models/
  legal-assistant-2026-06/
    manifest.json
    base-model-ref.txt
    adapter.safetensors
    adapter_config.json
    tokenizer/
    generation-policy.yaml
    evaluation-baseline.jsonl
    signatures.sha256

manifest.json should contain file identifiers and hashes, but no secrets. Put the parameters that affect observable behavior in generation-policy.yaml: temperature, top_p, length limits, the system prompt, message template, permitted tools, and rules for forcing tool selection.

Check a restored package by more than loading the model. Run a small immutable set of requests where response format, Kazakh and Russian language, refusal to access restricted data, tool calls, and long-context handling matter. This does not replace a full model evaluation, but it catches the most expensive mistake: “the adapter loaded, so everything is fine.”

Divide logs into audit, diagnostics, and raw text

LLM service logs quickly become the most dangerous and most underestimated archive. They may contain prompts, retrieved documents, request headers, response fragments, contract numbers, addresses, access tokens, and error messages. The simple rule “save everything for investigations” creates another copy of sensitive data without a clear retention period.

Audit and diagnostics serve different purposes. Audit records who accessed the system, which policy, model, and data set they accessed, and what the system allowed or blocked. Diagnostics explain latency, provider errors, token usage, and pipeline failures. Raw requests and responses are needed only for explicitly defined investigations, quality improvement, or a contractual obligation.

A permanent audit backup will usually include:

• a subject identifier or irreversible pseudonym;
• time, request route, policy version, and decision result;
• context-source identifiers, without automatically copying their text;
• model, call parameters, result code, and trace identifier;
• a reference to a protected investigation object when storing the full prompt separately is permitted.

Keep raw prompts in another category, with a separate retention period, separate encryption, and a narrow recovery group. Do not let an engineer restoring metrics after an outage automatically gain permission to read user conversations. During an incident, people act quickly and become tired, so access boundaries must be enforced by systems rather than by an operator's good intentions.

The schema must also be preserved. Useful JSON without field descriptions and a policy version becomes a collection of guesses. Store a JSON Schema or migrations beside it, along with a decision-code reference and the configuration hash that produced the event.

Encryption keys survive a backup only with a separate plan

An encrypted archive without an available key is a lost archive. An archive and its key in the same account, under the same role, and in the same folder create another problem: an attacker obtains both the ciphertext and the means to decrypt it. What you need is separate management, not the ceremonial presence of AES in a specification.

NIST SP 800-34 directly links the backup of encrypted data to cryptographic key management: a new or replacement system must receive both the software and the key material required to read the copy. For an LLM stack, this should be understood more broadly. Recovery requires not only a secret value but also the policy, key version, emergency-role permissions, service certificates, and procedure for accessing the HSM or KMS.

A working design often looks like this:

1. The service generates a separate DEK for each archive or archive set.
2. The service encrypts the data with the DEK and then encrypts the DEK itself with a KEK from the local KMS or HSM.
3. The archive contains the ciphertext, encrypted DEK, KEK version identifier, and integrity manifest.
4. The KMS policy, operation log, and emergency role are backed up separately within the permitted territory.
5. Decryption requires at least two controlled actions: access to the archive and permission to use the KEK.

Do not copy an exportable root key into the same archive “just in case.” If the rules and device allow export, make a protected offline copy through a separate procedure with split responsibilities. If the key is non-exportable, back up the configuration and the recovery mechanism of the HSM or KMS itself within the territory. In both cases, test the procedure with a test key. A document nobody has executed is not a recovery plan.

Rotation can also break archives. Each copy's manifest must identify the key version, and the key-deletion policy must account for the oldest archive still being retained. Deleting an old KEK version after scheduled rotation is easy. Recovering the data afterward is impossible, even when all objects remain intact.

The territory must remain unchanged along the entire copy path

A restriction on keeping data in the country cannot be confirmed by a bucket name or the address of the primary cluster. Data travels through a path: source, temporary volume, backup agent, network, object storage, catalog, key system, monitoring, second data center, and recovery-test environment. Check the entire path.

Ask each component five separate questions:

• Where are the objects and their replicas physically stored?
• Where do temporary files, multipart uploads, and queues go?
• Where do the KMS and HSM operate, and where is the key-operation log stored?
• In which territory are metadata copies, catalogs, and technical support located?
• Can the disaster-recovery policy automatically create a copy in another country?

Encryption does not cancel the territorial requirement when it applies to the data itself or to its processing. The argument “only an encrypted object left the country” is acceptable only when the applicable organizational and legal rules support it. Do not make this decision at the development-team level. It must be approved in writing by the data owner, security team, and lawyers responsible for the specific information category.

You need at least two independent failure zones inside the permitted territory. These may be two local data centers or two isolated provider sites if the contract and technical design confirm the location of every copy. Two folders in one data center protect against accidental deletion, but not against a physical incident, backbone failure, or compromise of a shared account.

AI Router can be part of such a stack for teams that need local data storage, audit logs, and PII masking, but backing up application data, indexes, and keys remains the system owner's responsibility. The gateway does not know which documents you consider the source of truth or which RTO you promised the business.

Assign RPO and RTO to separate layers

RPO answers how many recent changes you are prepared to lose. RTO answers how long the service can remain unavailable. Teams often record one value for the whole application and later discover that restoring 30 TB of index takes longer than the permitted outage.

Build a recovery table whose rows correspond to real dependencies rather than team names:

Do not copy these numbers from someone else's presentation. If a call-center operator works with a knowledge base and search is unavailable for eight hours, the business may switch to a manual process. If RAG participates in payment verification or patient-case processing, a manual mode may be impossible. Discuss RTO with the process owner rather than ending the conversation when SRE gives a disk-recovery time.

For PostgreSQL, do not rely on a nightly dump if you need a small RPO. The official PostgreSQL documentation describes PITR as a combination of a base backup and continuously archived WAL. The base backup provides a starting point, while WAL replays changes up to the required moment. The documentation also warns of a practical trap: archiving may lag or stop, while the pg_wal directory continues growing until the database stops when the disk fills.

At a minimum, monitoring must show the age of the last successfully archived WAL, the lag between segment creation and delivery, free space in pg_wal, the success of the latest base backup, and whether the restored archive can be read. A signal saying “the backup job completed” fully covers none of these risks.

Practice recovery in an isolated environment

A backup without recovery testing measures only the ability to write files. Testing must show that the file set produces a working system, not merely that storage accepted another object.

Run a recovery exercise in an isolated network within the same permitted territory. Do not send production traffic there, connect real external integrations, or give the test environment permission to send messages to users. Use temporary accounts, separate resource names, and explicit outbound-traffic restrictions.

The order of recovery is usually more important than the speed of individual operations:

1. Bring up identity management, roles, policies, and access to the KMS or HSM.
2. Restore primary data and the metadata database, then check checksums and schema migrations.
3. Restore documents and the vector index or run a verified rebuild from the fixed manifest.
4. Load the model package, enable generation policies, and run the control set of requests.
5. Restore auditing, checking event continuity and access denials, and only then open the application to users.

Include deliberately unpleasant checks in the exercise. Delete a test collection and restore it to a specified time. Try to decrypt an old archive after key rotation. Confirm that a user from another department cannot find a document through the restored index. Feed a document with a parsing error into the pipeline and make sure a retry does not create duplicates.

Record the actual time of every stage, manual actions, unexpected permissions, and the amount of data transferred. After two or three exercises, you will almost certainly reduce the RTO not by buying another storage system but by removing small obstacles: a missing permission, a slow DNS switch, manual searches for the tokenizer version, or an undocumented migration order.

Errors usually hide in dependency chains

The most common recommendation sounds simple: follow the 3-2-1 rule and the job is done. Several copies on different media are genuinely useful, but the rule does not say whether one copy may legally be kept outside the country, whether its key is available, or whether it can restore vector search with the original ACLs. In an LLM stack, the number of copies does not replace a description of the state.

The second mistake is backing up only infrastructure through Terraform, Helm, or virtual-machine snapshots. IaC restores resources, but not queue contents, the knowledge base, key history, index segments, or the record of which adapter was attached to a release. A machine snapshot often captures more than necessary, including temporary files and secrets, and transfers poorly between sites.

The third mistake is treating object versioning as immutability. Versioning helps after accidental overwrites, but a privileged user or compromised account can delete versions, shorten the retention period, or disable the rule. Critical copies need a separate account, deletion protection for the retention period, and a log that cannot be quietly changed by the same permissions.

The fourth mistake is testing recovery only with a new archive. The worst failures appear in old copies after a format change, deletion of an old key, a database-engine update, or a payload-schema change. Include archives of different ages and at least one previous data format in the test plan.

The next time the team discusses backup frequency, do not start with the schedule. Open the artifact list and ask the owner of every layer to show what it will be restored from, where its second copy is, which key reads it, and which test confirms the result. Until there is an answer, a backup remains hope rather than a recovery mechanism.