Where should you store conversation state so you can switch models without losing anything?

Store conversation state yourself if your application must survive a model or provider switch, an API schema change, or a failure in the middle of a tool call. Provider-hosted memory is convenient for continuing one specific chain, but it does not replace the application's own record.

The problem often starts with the phrase, «We have a conversation ID». That ID has no universal meaning outside one API. It might point to saved history, a single response, an internal execution graph, or data retained for only a short time. When a team builds a product around this pointer, it gives the provider control over session recovery, incident investigation, and migration.

This is not a call to send the entire chat log to the model every time. It is a call to distinguish data that belongs to your application from temporary context that a particular API keeps for convenience. For a banking assistant, a clinical operator, a B2B agent, and an ordinary SaaS product, this distinction quickly becomes practical.

Conversation state consists of three different things

The transcript, working context, and execution state should exist as separate entities. If you put them all into one messages array, the system may look simple at first, then stop being able to explain its own errors.

The transcript answers the question of what happened. It contains the user's input, the assistant's response, attachments, operator corrections, moderation events, and references to data versions. This is your log of domain facts. Do not blindly overwrite it with a summary, because it is what you use to investigate a disputed answer or restore a conversation after a defect.

Working context answers a different question: what should the model see on the next turn? It almost never equals the full transcript. It may include the system instruction, a current summary of older messages, recent messages, retrieved documents, the user profile, and results from several relevant tools. Context is temporary: after sending it, you can rebuild it according to the application's rules.

Execution state is needed by the agent, not by the person it is talking to. It includes the tool call, parameters, the operation ID in an external system, the result, a request for human approval, a timeout, a retry, and a cancellation. If an agent started processing a refund, sent a request to a CRM, and failed before the model's next response, you cannot continue the conversation without this state.

This is where teams confuse «remembering the chat» with «being able to resume the work». A model may remember the previous message through a server-side ID. It does not know whether an operation in an external payment system completed until you save and pass along that fact.

A practical minimum looks like this:

conversation           - владелец, tenant, политика хранения, статус
conversation_event     - неизменяемая запись реплики или доменного события
context_snapshot       - сводка, диапазон покрытых событий, версия промпта
run                    - один запуск модели, модель, параметры, трассировка
 tool_call             - имя, аргументы, idempotency_key, статус, результат
provider_cursor        - провайдер, тип API, внешний ID, срок действия если известен

The space before tool_call in this diagram does not matter, but the boundary does. conversation_event describes what was said and done in the domain. provider_cursor stores an external pointer that may speed up the next request. Do not make someone else's pointer the primary key of your conversation.

Provider-hosted memory speeds up a chain but ties you to it

Server-side conversation continuation saves you from sending the history and sometimes gives the provider access to complete intermediate execution data. This is useful, especially for long-running agent tasks. But you pay for the convenience with portability and a less transparent storage boundary.

For example, the Gemini Interactions documentation describes continuation through previous_interaction_id: the server retrieves the history by that identifier, so the client does not need to resend the entire chat. At the same time, the system instruction, tools, and generation parameters must be provided again for every new interaction. The documentation also states that storage is enabled by default and that the retention period varies by plan.

This leads to an uncomfortable but useful conclusion: even within one API, «conversation state» does not necessarily include every condition that shaped the response. If you forget to pass a new version of the tool set or the system instruction when resuming, the conversation continues under a different behavior policy. In the interface, this looks like the model suddenly forgetting something, although the cause lies in the API contract.

There is also the opposite model. OpenRouter documentation for the Responses API explicitly describes the interface as stateless: every request is independent, and the complete history must be sent again. This is not a weakness. A stateless interface forces the application to own its context explicitly, making it easier to move to another model, reproduce in a test, and filter according to access rules.

Storing state with the provider is justified in three cases:

• you are building a short-lived internal prototype without sensitive data;
• the server-side object contains intermediate data that is difficult to reproduce in the first release;
• you still store your own transcript and can build a new context without that object.

The last point separates a reasonable compromise from a dependency. If the external ID disappears, expires, or becomes unavailable, the application should be able to continue the conversation honestly from its own log. You may lose some hidden intermediate steps, but not the customer's history or the ability to avoid a dangerous repeated action.

Model switching breaks on semantics, not on JSON format

An OpenAI-compatible message format helps you connect a new endpoint, but it does not make a conversation portable automatically. The fields may look identical while the meaning of roles, tools, structured output, images, and hidden reasoning data differs.

It is especially dangerous to copy the raw request array from the old API into a new model. It often contains service messages, provider-specific call IDs, streaming response fragments, a tool-result format, or system fields that the second provider does not accept. In the best case, the new API returns 400. In the worst case, it silently interprets the history differently.

You need a canonical event layer that does not reproduce the types of a particular vendor. For example:

{
  \"event_id\": \"evt_01JX...\",
  \"conversation_id\": \"conv_8f2...\",
  \"sequence\": 42,
  \"kind\": \"tool_result\",
  \"actor\": \"application\",
  \"occurred_at\": \"2026-07-23T10:14:08Z\",
  \"payload\": {
    \"tool_name\": \"get_invoice_status\",
    \"call_id\": \"call_73a...\",
    \"input\": {\"invoice_id\": \"inv_481\"},
    \"output_ref\": \"obj://conversation-artifacts/evt_01JX...\",
    \"status\": \"succeeded\"
  },
  \"schema_version\": 1
}

Here, kind describes an application event, not an internal role used by one model. A useful set of types usually includes user_message, assistant_message, tool_call_requested, tool_result, human_approval_requested, human_approval_resolved, context_compacted, and policy_decision. Do not add a type for every minor detail. Add one when a new event changes what can be safely restored.

During migration, you build adapters in both directions:

1. the input adapter turns a provider response into canonical events;
2. the context builder selects the relevant events and creates a request for the chosen model;
3. the output adapter validates the response, extracts tool calls, and records them in the log;
4. tests continue the same conversation on the old and new models, comparing expected actions and constraints rather than wording.

The last point is often underestimated. You do not need identical wording. You need the new model to remember the customer's confirmed language, avoid a prohibited tool, and not ask for a document it has already received. Test this with continuation scenarios based on a fixed transcript, not by comparing one polished demo request.

Privacy is determined by the entire data path

Having your own PostgreSQL database in Kazakhstan does not prove that the data stayed in Kazakhstan. A conversation may involve raw messages, files, embeddings, search results, traces, caches, backups, error logs, and objects saved by the provider. If even one layer contains personal data, include it in the threat model and retention policy.

Start not with a table, but with field classification. A user message may contain a contract number, a medical description, payment details, or a trade secret. A tool result can sometimes be more dangerous than the question itself: an operator asks «where is the order», and the CRM returns an address, phone number, and complete purchase history.

For each data type, define four things: who can read the record, where it may go in the next request, how long it lives, and how to delete it. «The data is encrypted» answers none of these questions. Encryption protects the storage medium, but it does not explain why a debug log received the full prompt or why an employee from another team can read it.

A good architecture makes redaction part of context assembly. Before the model call, a separate layer should:

• replace or remove fields the model does not need;
• expand temporary tokens only for the authorized tool;
• record the masking-rule version next to the run;
• keep the original text out of traces unless the trace has separate access controls.

Do not rely on masking after the response. By then, the data may already have entered the request, the SDK log, or the API's saved state. For sensitive processes, it is preferable to give the model a reference to an authorized fact and leave full-value retrieval to a tool with an access check.

Routing adds another variable. An aggregator's single compatible endpoint may hide the choice between models and providers. OpenRouter documentation, for example, describes routing parameters that let you specify provider order, disable fallback, require parameter support, restrict routes that retain data, and select ZDR endpoints. The point is not to copy these exact fields into every stack. The point is that the data policy must participate in route selection before the request is sent, rather than live in a separate PDF that the code never reads.

Context should be assembled by budget and purpose

A complete conversation log almost always becomes poor working context. It is expensive, contains outdated decisions, and eventually gives the model more contradictions than useful information. But an aggressive summary can also break the conversation by removing exact conditions, document references, and user confirmations.

Keep two memory layers. The first, immutable layer contains events. The second, derived layer contains context snapshots. A snapshot should have an author, template version, creation time, and boundaries, such as «covers events 1 through 180». Then, if something goes wrong, you can rebuild it from the log after fixing the prompt or extraction logic.

The context builder usually works in this order:

1. it takes the system rules for the current tenant and task;
2. it adds a compact summary of confirmed facts from the older part of the conversation;
3. it selects the latest messages without compression;
4. it attaches only the documents and tool results related to the current intent;
5. it leaves a token reserve for the response and a possible tool call.

Do not store a summary as «the truth about the user». Store it as a cache with a limited lifetime. If the user says «no, the contract is different», the latest message should override the summary. If an operator corrects the request classification, that should become an explicit event rather than an invisible replacement of the old text.

There is a popular recommendation: «just save a summary after every response». It works well for a demonstration and poorly for a controlled process. A model-generated summary can itself contain a hallucination. After five compression cycles, you may have a confident, short, and incorrect description of the conversation. Create summaries after a clear threshold, keep a reference to the original events, and check critical fields with deterministic code or domain rules.

Session recovery starts with idempotency

After a failure, you cannot simply repeat the last request to the model. If the model asked for a tool call before the failure and the service managed to execute the action, repeating the request creates a duplicate. For an agent that reads data, this is unpleasant. For an agent that creates a payment, changes a plan, or sends a document, it is an incident.

The correct order is: first record the intention to call the tool, then execute it with an idempotency key, then reliably record the result. If the process dies at any stage, the recovery routine can inspect the log and understand what to do next.

1. Записать tool_call_requested со статусом pending.
2. Сформировать idempotency_key из conversation_id и call_id.
3. Выполнить внешний запрос с этим ключом.
4. Записать tool_result со статусом succeeded или failed.
5. Только после этого отправить результат модели.

The external service must support such a key or provide a way to check the operation's result using your identifier. If it can do neither, do not give the agent an irreversible action without explicit human approval.

Streaming state requires the same discipline. The text the user saw in pieces is not always equal to the final response the model produced. Do not record streaming fragments as the final message. Keep a draft separately, save the interruption event, and create the final assistant_message only after completion. If the connection breaks, the interface can show «response interrupted», while the server needs to know whether to continue generation, retry the request, or wait for the user.

For manual approval, store more than «approved». You need the argument snapshot at request time, who approved it, when, what changed before execution, and how long the decision remains valid. Otherwise, an operator may approve a refund of 10,000 tenge, while a retry applies that approval to a different amount.

An external cursor should be a cache with a clear failure path

It is useful to save the previous response ID or a provider-side conversation ID. This reduces latency and the cost of sending a long history. But the application should treat it as a cache: use it when it is compatible and available, otherwise rebuild the request from its own log.

At a minimum, provider_cursor should include the provider, model or model family, API type, external ID, creation time, instruction version, and an indicator of whether it can be used after a policy change. Do not reuse a cursor after changing the tenant, tool set, or data-processing mode. Formally, it is the same conversation, but the conditions have changed.

A useful preflight check looks like this:

cursor можно использовать, если:
- он создан для того же tenant;
- политика данных не стала строже;
- API и модель принимают этот тип cursor;
- набор инструментов совместим с сохранённой цепочкой;
- cursor не истёк и не был отозван.

иначе:
- собрать контекст из событий;
- создать новый запуск;
- сохранить новый cursor только после успешного ответа.

Test this fallback deliberately. In staging, delete the external ID, replace the model, block the previous provider, and interrupt the process between the tool call and the model response. If the team cannot continue the conversation or say exactly what has already happened, the architecture still depends on the happy path.

The right storage model depends on the cost of failure

For an internal assistant without tools, a hybrid approach is reasonable: keep the transcript and session settings in your database, build the full context on demand, and use provider-side state only as an optimization. This gives you simple recovery without forcing you to build complex orchestration immediately.

For a customer chat with personal data, keep events, access controls, masking-rule versions, and auditing in your own system. Send the provider the minimum working context. If the API offers chain storage, enable it only after checking retention, deletion, and routing conditions for the specific data class.

For an agent system that performs actions, your own event record is mandatory. Store tool state, approvals, idempotency keys, and business-rule versions separately. In this class of system, the cost of a recovery error is higher than the cost of one more table.

For teams that need to switch between cloud-hosted and locally deployed models, it is especially important not to tie domain sessions to one provider's IDs. AI Router provides one OpenAI-compatible endpoint and supports routing to different models, but portability still comes from your canonical log, not from replacing base_url.

Start with one uncomfortable exercise. Take a real active session and answer four questions in writing: which events have already happened, which external actions may have been completed, what data can be sent to the next model, and what remains if the external conversation ID disappears. If the last answer is «we do not know», fix the storage model before adding another model.