Migrating an Agent Checkpoint Schema Without Losing Tasks

A checkpoint is not technical clutter that can be renamed without consequences whenever a field changes in the code. If an agent preserves state across attempts, worker restarts, and releases, that record determines whether the agent continues work it has already started or performs it again. A migration error here turns into more than a neat stack trace. It can mean a duplicate payment, two identical emails, a lost document, or a stuck task that nobody can open.

I have seen the same scenario far too many times: a team changes the structure of state, deploys the service, and a few days later a retry loads a record created by the old code. The new deserializer either crashes or inserts a default value. The second outcome is more dangerous. The agent continues from the wrong position and leaves behind a trail that is difficult to connect to one line in a migration.

A reliable migration does not start with an SQL script. It starts by accepting a simple fact: a checkpoint is a durable execution contract. It needs a version, an unambiguous conversion path, rules for concurrent acquisition, and tests in which one task survives several versions of the program.

A checkpoint stores permission to continue, not just data

A checkpoint should contain exactly the facts needed to decide safely which step the agent may execute next. It is not a snapshot of the process's entire working memory, and it is not a debugging log.

Imagine an agent that processes an application: it receives attachments, extracts text, calls a model, creates a record in an external system, and sends a notification. It saves its state after each step. A poor checkpoint stores only step: 3. A good one stores the application ID, IDs of external objects already created, input versions, the attempt number, the next run time, and an idempotency key for a call that has not yet been confirmed.

The difference appears after a failure. If the process crashes after creating an external record but before saving its ID, a step number alone cannot show whether the operation happened. The agent will either create a duplicate or skip the work. An additional status field cannot repair this once the data has already been lost.

Separate four kinds of information:

• facts needed for the next step;
• the result of an external action that cannot be recalculated;
• data used for idempotency and reconciliation;
• diagnostic information that helps with investigation but does not change the agent's decision.

Logs and traces do not replace a checkpoint. A log may be incomplete, retained for too short a period, or have no transactional connection to the task record. A checkpoint is not a queue either: the queue says that work should be attempted, while the checkpoint says exactly where that work should resume.

This distinction is often lost when everything is put into JSON. After a few releases, the document becomes a mixture of temporary caches, debugging flags, and business state. It is hard to migrate not because JSON is complicated, but because nobody knows what each field means. Before changing the schema, make a table with the field, owner, source of truth, whether it is needed after a restart, and whether it can be recovered from somewhere else. Fields without clear answers should not influence decisions.

Every record needs its own version

The schema version must be part of the checkpoint itself, not a constant inferred from the container version, Git branch, or creation date. At almost any moment, storage contains records from several releases. A task may be waiting for a retry, blocked by a limit, in quarantine, or left behind after a manual stop.

A minimal envelope looks like this:

{
  "task_id": "job_01J9R8...",
  "schema_version": 2,
  "revision": 17,
  "status": "running",
  "lease_until": "2026-07-23T10:17:00Z",
  "state": {
    "phase": "classify",
    "source_document_id": "doc_481",
    "classification_request_id": "req_9aa"
  }
}

schema_version describes only the format and semantics of state. revision controls concurrent updates to one record. status describes the task lifecycle. Do not mix these numbers. When a team uses one version field for all three purposes, it inevitably writes a condition nobody can read: "if the version is less than 12, does that mean old JSON, old state, or another worker's update?"

Start numbering at 1. You can temporarily treat a missing version as 0 if such data already exists. But that case should have its own converter, not checks like state.foo ?? state.bar ?? "" scattered throughout the code.

Store envelope metadata in separate columns when the database allows it. In PostgreSQL, it is useful to move task_id, status, lease_until, revision, and schema_version into typed fields while keeping domain state in jsonb. You can then index active tasks, find records using an old schema, and limit queries without parsing JSON in the application.

The PostgreSQL documentation explicitly explains that standard READ COMMITTED exposes only data committed before the start of an individual statement. Two consecutive queries in one transaction may see different results. Therefore, the pattern "first read a ready task, then mark it as claimed in a separate operation" allows a race between workers.

Converters should move only forward

A checkpoint converter does not need to turn a new format back into an old one. It must deterministically bring every supported old record to the current internal representation without changing the source until the task has successfully continued.

Suppose version 1 stored one delivery object:

{
  "schema_version": 1,
  "state": {
    "phase": "send",
    "delivery": {
      "recipient": "[email protected]",
      "body": "Ready",
      "sent": false
    }
  }
}

In version 2, you split the intention to send a message from the confirmed external result. This is a sensible change: message describes what the agent wants to do, while provider_message_id proves that the provider has accepted the request. The converter must make clear that it knows no more about the old external result than the original record does.

function upgradeToV2(v1: CheckpointV1): CheckpointV2 {
  if (v1.state.phase !== "send") {
    return {
      schema_version: 2,
      state: { ...v1.state, delivery_attempt: null }
    };
  }

  return {
    schema_version: 2,
    state: {
      phase: "send",
      message: {
        recipient: v1.state.delivery.recipient,
        body: v1.state.delivery.body
      },
      provider_message_id: null,
      delivery_attempt: v1.state.delivery.sent
        ? { outcome: "unknown", migrated_from: 1 }
        : null
    }
  };
}

Notice the uncomfortable detail: sent: true does not necessarily mean that the provider has a message ID. If the old code set the flag before the network call or saved it after the call without separate confirmation, the converter cannot honestly invent the missing fact. It must move the task to a state in which the executor checks the idempotency key, queries the external service, or sends the record to an operator for review.

This is where teams often make a dangerous choice: they assume that migration must "fix" the meaning of old data. It does not. A converter changes the representation. It does not gain the right to invent execution history.

Keep the conversion chain in a separate module:

type AnyCheckpoint = CheckpointV0 | CheckpointV1 | CheckpointV2 | CheckpointV3;

function normalize(raw: AnyCheckpoint): CheckpointV3 {
  let current = raw;

  while (current.schema_version < 3) {
    switch (current.schema_version) {
      case 0: current = upgradeV0toV1(current); break;
      case 1: current = upgradeV1toV2(current); break;
      case 2: current = upgradeV2toV3(current); break;
      default: throw new UnsupportedCheckpointVersion(current);
    }
  }

  if (current.schema_version !== 3) {
    throw new UnsupportedCheckpointVersion(current);
  }
  return validateV3(current);
}

Do not write one giant migrateToLatest that recognizes dozens of field combinations. It quickly becomes a second, informal data format. Small transitions such as V1 -> V2 and V2 -> V3 are easier to test, remove, and investigate from the logs.

The Parallel Change pattern described by Danilo Sato divides an incompatible change into expansion, a transition period, and removal of the old path. For checkpoints, this means first teaching the reader to understand both formats, then creating the new format, and only afterward removing the old one.

Reading the old format and writing the new one are different tasks

Read compatibility determines what happens to tasks that have already started. Write compatibility determines what new workers create. These changes should not be deployed as one indistinguishable operation.

A workable transition from version 1 to version 2 looks like this:

1. Add version 2 to the reader and converter, but continue creating version 1.
2. Release the code and confirm that it handles real old records without normalization errors.
3. Switch the writer to version 2 while continuing to read version 1.
4. Wait until all active version 1 records have either completed or been processed.
5. Remove version 1 creation and reading only after checking storage and preparing a fallback for old exports.

The first step seems unnecessary until a rollback is needed. If the new writer has already created version 2 records and the previous release cannot read them, rollback becomes a shutdown of the entire task pool or manual editing of rows. Sometimes that is acceptable, but the decision must be made before deployment, not after the alert.

Do not confuse backward compatibility with writing both formats. The popular advice to "write both formats to reduce risk" often makes things worse. Two representations of the same state diverge during partial failures: new code updates provider_message_id, while the old sent field remains unchanged. An old worker then reads the stale version and makes the call again.

Dual writing is justified only when you have a clearly defined source of truth, a reconciliation rule, and a short lifetime for the solution. For most agents, it is better to read the old format, normalize it in memory, and write only the current schema on the next successful save. This provides gradual materialization without rewriting the entire table.

Acquiring a task and saving state must form one protocol

A format migration will not save a task if two workers can claim it at the same time. You need an explicit lease or lock protocol, as well as protection against a stale writer that finishes after a new owner.

For a task queue in PostgreSQL, a lease and optimistic revision are often enough. First, a worker atomically claims the record. It then executes the step and saves the new state only when both the revision and lease owner match.

WITH candidate AS (
  SELECT task_id
  FROM agent_checkpoints
  WHERE status IN ('ready', 'retry')
    AND run_after <= now()
    AND (lease_until IS NULL OR lease_until < now())
  ORDER BY run_after, created_at
  FOR UPDATE SKIP LOCKED
  LIMIT 1
)
UPDATE agent_checkpoints c
SET lease_owner = $1,
    lease_until = now() + interval '60 seconds',
    status = 'running',
    revision = revision + 1
FROM candidate
WHERE c.task_id = candidate.task_id
RETURNING c.task_id, c.schema_version, c.revision, c.state;

SKIP LOCKED works well for distributing independent tasks among workers, but it does not provide business consistency by itself. After acquisition, keep the returned revision. When saving, use a conditional update:

UPDATE agent_checkpoints
SET schema_version = $2,
    state = $3::jsonb,
    status = $4,
    run_after = $5,
    lease_owner = NULL,
    lease_until = NULL,
    revision = revision + 1,
    updated_at = now()
WHERE task_id = $1
  AND revision = $6
  AND lease_owner = $7
RETURNING revision;

If the query returns no row, the worker has lost the right to save the result. It must not try again with a new revision. Otherwise, a late process could overwrite a checkpoint that another worker has already advanced.

PostgreSQL warns that SERIALIZABLE transactions can end with a serialization error and that the application must retry the entire transaction. This applies to acquiring and updating the record: you can retry a short transaction that works with the record, but you must not blindly retry an external call that has already happened.

Separate the two operations. One transaction acquires the right to execute a step. The external call uses an idempotency key. A second transaction records the confirmed result. The process may die between these operations, and that gap is what determines the quality of your checkpoint.

Idempotency cannot be recovered from attractive JSON

The most difficult point lies between sending a request externally and saving the response. If the agent dies during that window, after a restart it sees only the intention to perform the action. The external system may have accepted the request, rejected it, or accepted it without returning a response in time.

For operations that support idempotency, create the key before the call and put it in the checkpoint before contacting the provider:

{
  "phase": "create_case",
  "request": {
    "customer_id": "cust_104",
    "summary": "Document review"
  },
  "idempotency": {
    "key": "job_01J9R8:create_case:0",
    "attempt": 0
  },
  "external_case_id": null
}

After a restart, the agent repeats the request with the same key. The external side either returns the previous result or rejects the repeat as already processed. If the API does not support such a key, use your own intent log and reconcile using a stable business identifier. Do not replace this with a done: true flag.

Each phase should have one of three clear statuses: the action has not started, the action is confirmed, or the result is unknown. A bare "in progress" status is almost useless after a crash. It says that the code once entered the function, but not whether the event occurred externally.

If an external operation is irreversible and has no idempotency support, add an explicit reconciliation path. For example, a task with needs_reconciliation status should not retry forever. The operator needs the original request, attempt time, correlation ID, response if available, and a description of which fields cannot be calculated. This is cheaper than later explaining to the data owner why the agent performed the action twice.

Tests must survive the process, not just the function

A converter unit test is necessary, but it catches only the most convenient class of errors. Real failures occur at the boundary between an old record, a new binary, a lease, a network call, a process stop, and the next release.

Build fixtures from real historical checkpoints. Before anonymizing the data, preserve its shape: missing fields, empty arrays, old phase names, unfinished attempts, and records with expired leases. A manual example created after the schema change is almost always too neat.

Check at least these properties:

• every supported fixture normalizes to the current version;
• normalization is repeatable: a second run does not change the result;
• the converter does not modify the input object;
• an unknown version stops the task predictably;
• required invariants of the current schema are checked after conversion.

A process test is needed for a release chain. It does not have to start a complete production environment, but it should use a real database, real checkpoint persistence, and separate processes or isolated application instances. The scenario looks like this:

1. Release A creates a version 1 task and saves a checkpoint after an external intent.
2. The test stops process A before the external result is recorded.
3. Release B claims the same row, converts version 1 to version 2, and repeats the call with the same idempotency key.
4. Release B saves the confirmed version 2 result.
5. Release C reads the task and completes it without calling the external system again.

Do not check only schema_version = 3. Check that the external stub received one logical request, that the final ID was saved, and that restarting a completed task does nothing. If the API stub can store calls by idempotency key, this test catches duplicates well.

Add a test for a stale lease owner. Worker A claims a task and hangs. The lease expires, and worker B completes the task. A then wakes up and tries to write its old checkpoint. The conditional UPDATE should return zero rows, and A should record the lost lease without executing the effect again.

There is another test teams often skip: a transition through several schemas. Do not test only V2 -> V3 if version 1 could remain in storage. Test V1 -> V2 -> V3 through the real public normalize path. Removing an intermediate converter without this check breaks long retries exactly when they are needed most.

A bulk migration is a separate operation

Background rewriting of all checkpoints is sometimes necessary, for example when changing typed columns, building a new index, or removing a sensitive field. But it does not replace compatible reading.

If a migration job updates millions of rows, it competes with workers for the same records. Do not run one huge UPDATE without limits. It creates a long transaction, expands the change log, complicates rollback, and may hold resources needed by active tasks.

Process records in batches, select only a specific old version, and use the same revision control as the executor. If a worker changes the checkpoint between the migration's read and write, the background process should skip the row and return to it later. Its goal is not to win the race, but to avoid breaking current state.

The expand, migrate, contract approach is useful here too: first make the code understand the new shape, then let the background process change old rows, and after confirming that old data is gone, remove the transition path. In Evolutionary Database Design, this transition period is described as a distinct part of the change rather than a side effect of DDL. That is the right discipline for agent state, even when the schema itself is stored in JSON.

Before a bulk migration, agree on metrics. Count active checkpoints by schema_version, normalization errors, tasks in needs_reconciliation, the age of the oldest active record, and conditional-save failures. Without this, you cannot tell whether the transition is complete or whether you simply stopped looking at old data.

Stop an unknown or damaged record instead of guessing

The worst response to an unknown schema is to catch the error, create an empty state object, and let the agent start again from the beginning. This may be acceptable for an operation that is documented as safe to repeat. For most business processes, it is hidden context loss.

Divide errors into three categories. A record in an old supported version goes through the converter. A future or unknown version goes to quarantine because the current code does not understand its semantics. A damaged record also goes to quarantine, but with a separate reason: invalid JSON, a broken invariant, a missing required ID, or a structure that does not match the stated version.

Preserve the original payload unchanged, along with the rejection reason, reader code version, and task ID. Do not overwrite the problematic checkpoint with a "fixed" empty object. An operator may need to compare it with a backup, an external service log, or the result of a newer release.

If state is serialized in a binary format, the risk is even higher. Python documentation explicitly warns that pickle is unsafe for untrusted data and must not be used for input that may have been tampered with. Even in internal storage, a binary snapshot without an explicit schema makes auditing and long-term compatibility harder.

Choose a format that can be validated independently of the executor's code. JSON with JSON Schema, Protobuf with carefully designed evolution rules, or typed columns are better options than runtime object serialization. The format alone will not fix poor semantics, but it gives you a chance to detect incompatibility before the agent performs an action.

Removing an old converter requires proof

Converters annoy developers because transition code looks temporary. But "temporary" does not mean it can be removed after two sprints. It is needed for as long as old records can appear and remain in the system.

First stop creating the old version. Then wait for all active tasks in that format to finish, including delayed retries and quarantined records. After that, check the primary store, recovery replicas, export queues, and manual retry tools. If an engineer can take a checkpoint from an archive and attempt to run it with a new worker, that path is part of the contract too.

Record the support boundary in the repository. For example, the current schema 4 reads versions 2, 3, and 4, while version 1 is no longer supported after all its records have gone through a separate review procedure. This is better than endless if statements around fields. A converter should have a removal date, but remove it based on observed absence of data, not the calendar.

The first change worth making before the next release is simple: add schema_version to every new record and prohibit the reader from silently accepting an unknown format. This will not fix past errors, but it will stop the habit of hiding incompatibility behind default values. Then build the converter chain, verify the lease protocol, and make the test pass through a real process restart. Your unfinished task will then survive releases as a working object, not as a random leftover from old code.