How to Rotate Encryption Keys Without Downtime

Encryption key rotation without downtime does not start with clicking Rotate in KMS. It starts by acknowledging an uncomfortable fact: an LLM service does not keep its data in a single database. Prompts and responses end up in operational tables, technical logs, traces, queues, vector indexes, attachments, exports, and backups. If one key protects all of them, you have not built a simple scheme. You have tied different retention periods, access rights, and breach consequences to one failure domain.

The basic design is simple in principle but demanding in the details: a new key handles writes, old versions continue to handle reads, a migration worker gradually rewrites existing objects, and the old version is disabled only after verification. The most expensive mistake here is not cryptographic. Teams usually lose data or break the service because they do not know exactly where ciphertext lives, who reads it, or how long its copies remain available.

Rotation and re-encryption solve different problems

Rotation creates new key material or a new key version for future operations. Re-encryption changes the cryptographic protection of objects created earlier. These are different processes, and treating them as one makes the design dangerous.

Managed KMS services often make the first process almost invisible to the application. AWS KMS documentation explicitly warns that changing KMS key material does not change data keys or re-encrypt the data protected by them. Google Cloud KMS says the same thing in a different way: the new version becomes active, but old data is not re-encrypted automatically.

This matters for LLM infrastructure. If you protect records with envelope encryption, an object usually contains the payload ciphertext and an encrypted DEK. Rotating the KEK can protect new DEKs with a new version, but the old DEK and old ciphertext do not disappear. If a specific DEK is compromised, calendar-based KEK rotation does not fix the situation.

The system must answer three separate questions without guesswork:

• Which key and version protected this object?
• Can the service read an object created before the write switch?
• How many objects, including copies, still depend on the old version?

If answering the third question requires a manual search across several teams, the old version cannot be safely destroyed. You simply do not know what you will break.

One key for every storage system makes an incident bigger

Keys for logs, application data, backups, and secrets should belong to separate cryptographic domains. This is not an auditor's preference or an attempt to create more objects in KMS. These data classes have different life cycles.

Logs are often needed for investigations, but support and observability engineers read them. The application database stores conversation state, account settings, jobs, and processing results. Backups must survive a production environment failure, so they live in another account, another project, or at least under a separate policy. Secrets such as provider tokens, database passwords, and signing keys should not be treated as ordinary application data at all.

A practical minimum looks like this:

A separate key does not provide automatic security. If one service account can decrypt all four domains, you have drawn a boundary on paper. The access policy must allow each consumer only the operations it needs and only within its own domain.

OWASP's Secrets Management Cheat Sheet specifically recommends automating rotation, applying least privilege, and keeping metadata about the purpose, owner, and life cycle of secrets. I would add one requirement from practice: metadata should explain not only who owns the key, but also where the last ciphertext under that key may appear.

Inventory must count copies, not tables

Before creating a new key, map your ciphertext. A list of database tables is not enough: one record may also go to an index, an analytics export, a dead-letter queue, and a nightly copy. Rotation fails precisely on these secondary paths.

For each data class, record these fields in a registry:

• service owner and data owner;
• physical storage location and all replication paths;
• encryption format, algorithm, and version identifier;
• readers, including batch jobs, CLI tools, and the recovery procedure;
• the maximum period for which the object or one of its copies remains available.

Do not confuse data residency with the location of the primary database. If prompts from Kazakhstan are stored locally but a debugging export or backup is sent to another environment, storage requirements already fail for the chain as a whole. This is especially uncomfortable for LLM teams: logs may retain fragments of prompts, responses, user identifiers, and request headers even when the main table stores only a minimal set of fields.

Also check which data should not survive until rotation at all. A full prompt often appears in an error log because a developer wanted to see a provider failure once. After that, it can sit in an archive for years. Masking PII before logging reduces the amount of data that needs re-encryption and makes a leak less damaging, but it does not replace encryption.

A useful inventory result does not look like a presentation diagram. It looks like a verifiable line: conversation_events -> PostgreSQL primary + CDC topic + nightly backup -> app-data-kek v4 -> API, summarizer, restore-job -> retention 30 days / 180 days backup. This line shows which systems must survive dual reads and when v4 still cannot be touched.

The ciphertext format must store the version

Dual reads work reliably when the version is included in the object or in a record inseparably linked to it. Choosing a key based on creation date, file name, or a global variable almost always breaks when a migration is restarted, an old backup is restored, or a delayed message arrives.

For application-level encryption, the metadata might look like this:

{
  "cipher": "AES-256-GCM",
  "key_domain": "app-data",
  "key_version": "2026-07",
  "wrapped_dek": "base64url(...) ",
  "nonce": "base64url(...)",
  "aad": {
    "tenant_id": "t_4821",
    "record_type": "conversation_event",
    "record_id": "ev_01J..."
  },
  "ciphertext": "base64url(...)"
}

key_version selects the key version, wrapped_dek makes it possible to decrypt the DEK, and AAD binds the ciphertext to its context. If an attacker copies the ciphertext of one tenant into another tenant's record, decryption should fail authentication. Do not put fields in AAD that the application expects to change without re-encryption, such as a job status or last-updated time.

There is one detail in this format that is often missed. The key version does not need to be secret. It is needed for decryption routing, logging, and migration checks. The secret parts are the key itself and permission to perform decryption.

For small values, direct encryption through KMS may be sufficient. For logs, files, large model responses, and backups, envelope encryption is usually more practical: the service generates a random DEK, encrypts the data with it, and stores the DEK in wrapped form. This reduces the number of expensive KMS calls and allows objects to be migrated incrementally. But a DEK must never be written to a log, cached without an expiration, or sent through a queue as an ordinary JSON field.

Dual reads keep the service running during migration

A safer transition treats the change as format compatibility rather than replacing all data in a single hour. The new release should first read every permitted version and write only the new one. Only then should the bulk migration begin.

The sequence looks like this:

1. Create a new version or a new key, but do not switch writes yet.
2. Deploy code that reads old and new versions using key_version and counts them separately in metrics.
3. Switch new writers to the new version through configuration that can be rolled back quickly.
4. Start a worker that reads old objects, verifies their integrity, and writes a new cryptographic envelope.
5. After confirming coverage, prohibit new operations under the old version while leaving decryption available for the observation period.

The point of the second step is not a fallback branch such as try old key. That kind of code hides format errors and makes the service try keys one by one. Reads should be deterministic: a v4 object opens with v4, and a v5 object opens with v5. If the version is unknown, the API returns a controlled error, raises an alert, and does not make five KMS calls hoping one will work.

Here is simplified pseudocode. It deliberately does not contain “if decryption fails, try the previous key.”

def decrypt_record(envelope):
    policy = key_registry.get(
        domain=envelope["key_domain"],
        version=envelope["key_version"]
    )
    if policy is None or policy.read_status != "enabled":
        raise UnsupportedCiphertextVersion(envelope["key_version"])

    dek = unwrap(policy.kek_ref, envelope["wrapped_dek"], envelope["aad"])
    return aes_gcm_decrypt(dek, envelope["nonce"], envelope["ciphertext"], envelope["aad"])


def encrypt_record(plaintext, context):
    policy = key_registry.current_writer(domain="app-data")
    return envelope_encrypt(policy.kek_ref, plaintext, context)

The code should log the domain, version, and operation result, but never the payload, DEK, nonce, or wrapped key. The decrypt_success_total{domain,version} metric shows live reads of the old version more accurately than relying on a completed batch.

The migration worker must be idempotent

Re-encryption becomes an incident when the worker processes an object again and loses changes made by the API in parallel. The fix is straightforward: migration needs a state version, a conditional write, and a clear retry rule.

Imagine a conversation_event record. The worker reads it with row_version=18 and key_version=2026-01, decrypts it, encrypts it with the same AAD under 2026-07, and updates the record only if row_version=18 still matches. If a user request changed the record first, the conditional update fails. The worker reads the current version and tries again. It must never overwrite newer state with an old snapshot.

Check four conditions before writing:

• the object still uses the old version;
• the AAD matches the current owner and object type;
• decryption passed authentication;
• the conditional update confirmed that the object did not change between the read and the write.

Do not run migration as one SQL script that decrypts data on the client and updates millions of rows in a single transaction. That operation will lock tables, overwhelm KMS, and leave you with a poor rollback option. Work in small batches, limit concurrency, and pause when KMS errors increase, API latency rises, or the database becomes saturated.

A good worker stores a checkpoint not as “the last ID,” but as a durable traversal rule. The last ID fails when identifiers are not ordered, records are restored from a backup, or some objects are created with the old version by a delayed queue message. Use a query such as “all objects in domain X with key_version=old, ordered by primary key” and keep returning to it until the count reaches zero. Then repeat the pass after a period that covers the maximum queue delay.

Backups cannot be treated as already migrated

A backup preserves the past by design. The old key must therefore remain capable of opening an old backup while that backup is within the approved recovery period. Destroying the key after migrating the production database and forgetting about archives means losing the only reliable recovery path after the next failure.

Separate two actions. First, new backups created after the switch should use the new backup-kek. Second, old backups must either be repackaged or kept until their retention period naturally expires, with the old version still available. The second option is often cheaper and safer when policies allow the old key to remain available only for decryption in an isolated recovery procedure.

Recovery testing should be part of rotation, not an annual exercise. Take one old backup, restore it in a separate environment with no access to production endpoints, complete the recovery, and read several encrypted objects. Then repeat the check with a new backup. Record which key domain, version, role, and procedure were required.

Managed services behave differently after a customer-managed key is rotated. Google Cloud KMS documentation describes three possibilities: a service may automatically rewrap the DEK, apply the new version only to future data, or continue using the original version. Never transfer one storage system's behavior to another by analogy. Check the specific service and confirm the result through your own recovery test.

Secrets need a separate procedure, not a data migration

Secrets are connected to external systems. A database password, model access token, webhook key, or private signing key stops working not because you failed to re-encrypt its value, but because the receiving side no longer accepts the old credential.

Secrets therefore involve two independent operations. First, change the secret with the provider or target system and allow a short overlap if the protocol supports it. Then update the secret store and roll out the consumers. Changing the KMS key that protects the secret at rest is a separate step. It protects the stored value, but it does not revoke a stolen token.

A bad plan sounds like this: “We will rotate the master key, so the tokens are safe.” No. If a token has already been issued, it continues to work until it is revoked, expires, or is replaced on the provider's side. For critical integrations, check whether the provider supports two active credentials, version identifiers, and monitoring for use of the old one.

AI Router helps keep model access behind a single compatible API endpoint, but the access key for the gateway itself is still a secret with its own life cycle. Do not include its replacement in a user-data re-encryption package without a separate overlap and rollback plan.

Destroying an old key requires evidence

Do not delete an old key based on a calendar date. First close the dependency gap with data and monitoring. Otherwise, you will turn read errors into a delayed outage that appears only during a rare request or an actual recovery.

I use four pieces of evidence before disabling decryption:

1. The inventory shows that all known storage systems have either been migrated or are intentionally kept in archival read mode.
2. A metadata scanner finds no objects with the old version in active storage.
3. Old-version read metrics remain at zero for the entire selected period, including scheduled jobs.
4. The team successfully restores both an old and a new backup using the documented procedure.

After that, prohibit the creation of new ciphertext with the old version. It usually makes sense to first disable use of the key by applications while preserving a controlled way to restore access during a short observation window. Then remove decryption access, watch for errors, and only after a period covering archives and retention obligations should you plan to destroy the key material.

NIST SP 800-57 treats key management as part of designing the entire cryptographic system, not as a choice of algorithm. That is exactly right: AES-GCM cannot save a team that cannot say which data still depends on v3 and who can read it.

If you cannot answer today which old key will open a six-month-old backup and who is authorized to perform that recovery, do not wait for an incident. Start with a domain registry, add the version to the ciphertext format, and deploy dual reads. After that, rotation stops being a nighttime operation based on hoping nobody notices.