How to Unify Chat Completions and Responses API in One Gateway?

Two API contracts cannot be unified in one LLM gateway by simply replacing messages with input. That translation may pass a demo with one user question, then break function calls, streaming, strict JSON Schema, and retries. The gateway must translate the meaning of the operation, not its JSON shape: who said what, which tools are allowed, what result is required, and which unfinished call the next data fragment belongs to.

This matters to teams that do not want to rewrite every application when changing models or providers. An old service may speak Chat Completions, a new agent may use Responses API, while internal routing should remain shared. The boundary is simple: the client owns the external contract, the gateway owns the internal contract, and the adapter owns the provider contract. Mix these three layers and every new feature becomes an exception to an exception.

What the gateway should unify

The gateway should unify request and response semantics rather than pretend that two JSON objects are isomorphic. Chat Completions builds its result around choices[] and one assistant message in each branch. Responses API returns a set of typed output items that may include text, function calls, refusals, and other execution elements. In an ordinary conversation, the difference is easy to miss. In an agent loop, it determines whether work can continue without losing state.

The gateway needs its own canonical request. Clients do not need to see it, but this is the object that should be validated, logged, and passed to the router. It is useful to separate five things:

• model instructions and conversation history;
• user and assistant content parts;
• tool definitions, selection rules, and restrictions;
• the required shape of the final response;
• operation state: call identifiers, continuation of a previous response, streaming mode, and tracing metadata.

A poor internal object usually looks like a slightly extended messages[]. Within a few months it accumulates response_format, previous_response_id, tool_outputs, reasoning, provider_payload, and several Boolean flags. Eventually nobody can say which fields are required or at what stage they have meaning.

It is better to introduce an explicit operation model. A text instruction should not pretend to be a user message merely because one external API places it in a message array. A function result should not become an «assistant message»: the model did not author it, and it belongs to a specific call_id.

{
  "model_hint": "general-reasoning",
  "instructions": [
    {"kind": "text", "text": "Отвечай по правилам кредитного продукта."}
  ],
  "turns": [
    {
      "role": "user",
      "parts": [
        {"kind": "text", "text": "Можно ли досрочно погасить заем?"}
      ]
    }
  ],
  "tools": [
    {
      "kind": "function",
      "name": "find_loan_terms",
      "description": "Возвращает условия займа по типу продукта.",
      "parameters": {
        "type": "object",
        "properties": {
          "product": {"type": "string"}
        },
        "required": ["product"],
        "additionalProperties": false
      },
      "strict": true
    }
  ],
  "tool_policy": {"mode": "auto"},
  "output_contract": {"kind": "text"},
  "stream": true
}

This is not a «third API for clients». It is the gateway's ledger. Each field represents one meaning, while adapters perform only two conversions: external request to canonical operation, and canonical operation to the model's dialect.

Messages cannot be reduced to an array of strings

Text messages only look alike on the surface. In Chat Completions, clients usually send messages with developer, system, user, assistant, and tool roles. In Responses API, the input may contain messages and separate items, while instructions may live in instructions. The API can also continue work through a previous response identifier, which changes the source of context.

OpenAI's Chat Completions documentation describes the endpoint as generating a response from a list of messages. The Responses API reference, by contrast, models the operation as creating a response with a set of input and output items. Do not treat this as cosmetic. The first representation is convenient for a familiar chat, while the second expresses an agent's execution flow more accurately.

The normalizer must preserve the order of parts within each turn. A single user message may contain text and several attachments. If the adapter joins them into one string, it loses type, order, and the ability to check allowed modalities. If the selected provider accepts text only, the gateway must reject the request before sending it or apply a predefined conversion policy. An attachment must not be silently discarded while leaving the user with the impression that the model read it.

Roles also require discipline. I usually use these rules:

• developer and system go into a separate instruction collection, preserving their original order;
• user and assistant become conversation turns;
• tool does not become an ordinary turn but is linked to a tool call;
• an unknown role produces a schema error instead of being converted to user;
• name and other auxiliary fields are kept in turn metadata if the target contract can express them.

The question of system and instructions is often handled too roughly. Moving the entire system prompt into instructions only works after choosing a continuation policy. If the client uses previous_response_id, resending an old instruction may change the context or inflate the input. If the gateway stores history itself, it must know which instructions are already part of the saved state and which ones the client wants to replace.

Do not promise complete transparency where it does not exist. Continuation through a server response identifier cannot be represented losslessly in stateless Chat Completions. There are two honest options: keep your own log of canonical turns and rebuild the history, or make this capability available only for Responses API. The first option provides portability but increases context size and responsibility for data storage. The second is easier to operate, but makes the contract broader.

A tool call consists of three linked events

A tool cannot be treated as a request field. It forms a loop: the model selects a function, the application executes it, and then returns the result to the exact call requested by the model. Losing the link between these events breaks an agent more reliably than a poor prompt.

In Chat Completions, the function definition is nested in the tools[] object as function. The model returns tool_calls[] in the assistant message, and the client continues the conversation with a message using role: \"tool\" and tool_call_id. In Responses API, a function definition is usually flat: name, description, parameter JSON Schema, and strictness setting. The call arrives as a separate function_call item, and the result is sent as a separate function_call_output input item with the same call_id.

The schema below shows the minimum conversion for a function definition. The gateway must not rewrite JSON Schema «for appearance»: it must preserve it as is and separately check which subset of the schema the target model accepts.

{
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "find_loan_terms",
        "description": "Возвращает условия займа по типу продукта.",
        "parameters": {
          "type": "object",
          "properties": {
            "product": {"type": "string"}
          },
          "required": ["product"],
          "additionalProperties": false
        },
        "strict": true
      }
    }
  ]
}

For a Responses-compatible adapter, the same canonical function should become an object with this meaning:

{
  "tools": [
    {
      "type": "function",
      "name": "find_loan_terms",
      "description": "Возвращает условия займа по типу продукта.",
      "parameters": {
        "type": "object",
        "properties": {
          "product": {"type": "string"}
        },
        "required": ["product"],
        "additionalProperties": false
      },
      "strict": true
    }
  ]
}

After generation, function arguments almost always arrive as a JSON string. Do not parse and serialize them again at the boundary without a reason. Preserve the raw arguments, validate it against the schema before execution, and add the parsed object to the executor's safe context. Re-serialization can change numbers, field order, and escaping. For most JSON this does not matter, but with signatures, precise tracing, or comparison of test fixtures, the difference becomes inconvenient.

Here is the critical fragment that must survive both directions of conversion:

{
  "call_id": "call_8f31",
  "name": "find_loan_terms",
  "arguments_raw": "{\"product\":\"consumer_loan\"}",
  "status": "requested"
}

When the application finishes, the result must not be presented as assistant text either. The canonical item should look like this:

{
  "kind": "tool_result",
  "call_id": "call_8f31",
  "output": "{\"early_repayment\":true,\"fee\":0}",
  "status": "completed"
}

The Chat Completions adapter turns it into role: \"tool\", tool_call_id: \"call_8f31\". The Responses API adapter turns it into type: \"function_call_output\", call_id: \"call_8f31\". call_id must not be replaced with a custom UUID in either direction. The client, model, and audit log must see one continuous chain.

Parallel calls need a separate check. The model may return two calls with different identifiers. The application may execute them in parallel, but results should be returned in an order explicitly supported by the selected contract. Do not associate a result with a function name: the same tool may be called twice with different arguments.

Structured responses and JSON mode solve different problems

A structured response means a contract for the result's shape, not a request to «return JSON». This distinction is often missed when a gateway tries to unify response_format with one Boolean field such as json=true.

In Chat Completions, the older JSON mode is set with response_format: {\"type\":\"json_object\"}. It aims for syntactically valid JSON but does not prove that the result matches the required schema. A schema requires json_schema, with a schema name, its description, a schema object, and a strict setting. OpenAI's documentation explicitly separates JSON mode from Structured Outputs and recommends the latter for models that support schemas.

In Responses API, the schema's meaning lives in the text format setting. The canonical model should therefore store a result object rather than the external response_format field:

{
  "output_contract": {
    "kind": "json_schema",
    "name": "loan_answer",
    "strict": true,
    "schema": {
      "type": "object",
      "properties": {
        "eligible": {"type": "boolean"},
        "reason": {"type": "string"}
      },
      "required": ["eligible", "reason"],
      "additionalProperties": false
    }
  }
}

This object can be rendered as response_format for Chat Completions and as text.format for Responses API. The gateway must take one more step: match the contract to the capabilities of the specific model. Models and providers differ in their support for strict mode, nested schemas, individual JSON Schema keywords, and combinations of schema output with function calling.

The worst policy sounds appealing: if the provider does not support strict, remove the flag and continue. Teams choose it because they want «the highest possible success rate». In practice, it changes the API promise. The client expects to parse the result without recovery heuristics, but receives an object with an extra field, a string instead of an array, or text before the JSON. In banking, healthcare, and automation, this is not a minor degradation but a different risk mode.

Divide capabilities into three classes:

• exact support: the gateway sends the request without changing its semantics;
• controlled conversion: the gateway changes the shape while preserving the declared meaning;
• unsupported capability: the gateway returns an error before calling the model.

For example, converting response_format.json_schema to text.format.json_schema may be a controlled conversion. Removing strict: true is not. If you introduce a best-effort mode anyway, it must require an explicit client flag and return a guarantee-downgrade indicator in the response or tracing metadata.

Validate structured results twice. The model or provider may claim schema support, but the application must still validate the final JSON before passing it to business logic. The first check protects the gateway's output contract; the second protects a specific action, such as creating a payment order. One validator does not replace the other.

Streaming requires two state machines

Streaming between contracts cannot be implemented by replacing an SSE event name. Both Chat Completions and Responses API send data in parts, but those parts have different structures and appear in different orders.

In Chat Completions, the client expects a sequence of chunk objects. Text often arrives in choices[0].delta.content, while function arguments may accumulate in fragments inside delta.tool_calls. In Responses API, the stream consists of typed events that may separately report response creation, output item addition, text deltas, call argument deltas, and completion.

The gateway must build an internal response state machine. For each output item, it keeps an identifier, type, index, accumulated text, accumulated arguments, and status. The external serializer then emits events in the required format. This looks more complicated than a direct proxy, but otherwise you get the classic error: text has already reached the client, the function arrives later, and the gateway has already announced finish_reason: \"stop\".

It is useful to make the minimum states explicit:

1. started: the gateway accepted the request and recorded the response contract.
2. emitting: text, refusal, or tool argument parts are arriving.
3. awaiting_tool: the model finished the current response with function calls.
4. completed, failed, or cancelled: the operation received a final status.

Do not try to reproduce another API's tokenization. If a Responses provider sends a text delta as a large fragment, the Chat adapter can emit it as one delta.content. The client usually cares about character order, not the size of each chunk. Conversely, if a Chat provider sends call arguments in fragments, the Responses adapter must preserve one call_id and pass the argument deltas as parts of one item.

A partial JSON response is another unpleasant case. The model may begin a structured response and the connection may close before the closing brace arrives. Do not return the accumulated content as a valid final object. A streaming client may display intermediate text, but the final response must receive an error status, and the audit log must preserve the reason: client cancellation, gateway timeout, provider connection loss, or model error.

Errors must preserve the cause, not only the HTTP code

A unified gateway does not have to return identical error JSON to every client, but it must classify failures consistently internally. A provider 429, a local key limit, and rejection of an unsupported schema may have similar external codes, yet they are three different causes for retries and investigation.

For an OpenAI-compatible endpoint, it is reasonable to keep a familiar external form:

{
  "error": {
    "message": "Модель не поддерживает strict JSON Schema для выбранного маршрута.",
    "type": "invalid_request_error",
    "param": "response_format",
    "code": "capability_not_supported"
  }
}

Internally, add immutable fields that clients do not necessarily need to see: gateway_request_id, client_request_id, route_id, the provider identifier, the original error code, attempt number, and retryability class. Do not put raw provider response bodies in the message field. They often contain parts of the prompt, tool data, or internal routing details.

A practical classification may look like this:

• contract validation error: do not retry, fix the request;
• model capability refusal: do not retry on the same route, and allow an explicit fallback only under the client's policy;
• gateway or provider limit: retry with controlled delay if the request is idempotent;
• temporary provider error: retry within an attempt limit and time budget;
• tool execution error: return it to the model as a tool result only if the application policy allows it.

The last point is often implemented incorrectly. If an internal search service returns 500, it must not automatically become an HTTP error for the LLM request. Sometimes the agent can tell the user that a service is temporarily unavailable and suggest another path. Sometimes the application is not allowed to reveal that such a service exists or disclose failure details. This decision belongs to tool policy, not the API adapter.

Also separate a routing error from a model error. The gateway may reject a request because of a regional rule, data residency restrictions, or the absence of a model with the required capability. The model did nothing. Quality reports must not record such cases as «model errors», or you will start fixing the prompt when the routing rule is what needs to change.

Compatibility should be a matrix, not a promise

The phrase «we support the OpenAI API» does not describe real compatibility. It hides dozens of combinations: text chat, images, tool choice, multiple calls, strict schemas, streaming, state continuation, metadata, stop reasons, and usage. One route may work well with text and functions but not support strict response formatting. Another may handle JSON Schema perfectly but not support server-side conversation continuation.

Build a capability matrix for every adapter and route. Rows should be specific: text, image input, function call, parallel function calls, strict function schema, json schema output, streaming, server-side state, usage details. Columns should show whether the gateway accepts the capability, passes it through unchanged, converts it, can validate the result, and which code it returns when rejecting it.

Do not turn the matrix into a manual document that is outdated on the day it is published. Store it as configuration data and use it before routing. The decision then becomes testable:

{
  "route": "provider-a/model-x",
  "capabilities": {
    "function_call": true,
    "parallel_function_calls": true,
    "strict_function_schema": false,
    "json_schema_output": true,
    "streaming": true,
    "server_side_state": false
  }
}

When the client sends strict: true for a tool, the router filters out that route before making a network call. If policy allows fallback, it chooses only routes with the required capability. Otherwise, it returns a clear error. This is better than receiving a vague provider 400 that does not explain which field lost its guarantee.

AI Router can keep this boundary especially clear: one OpenAI-compatible endpoint lets you preserve existing SDKs and code, but new capabilities across different models still need to be checked against the matrix rather than inferred from similar parameter names.

Test conversion on chains, not isolated requests

One request fixture does not test a gateway. You need scenarios in which the next request depends on the identifier and semantics of the previous one. The most useful tests often look boring, but they find errors before production.

First scenario: user text, one function call, function result, and final text. Check that the function name, raw JSON arguments, and call_id remain identical when routing from Chat Completions to Responses API and back.

Second: two parallel calls to the same function with different arguments. Check that results are not joined by name and that completing the first call does not mark the entire response as complete.

Third: a strict JSON Schema with additional fields forbidden. Test three branches: the model supports the schema, the model does not support the schema, and the model returns an invalid result. In the second branch, the gateway must reject before generation. In the third, it must return a contract error instead of passing broken JSON onward.

Fourth: a streaming function call in which arguments arrive in several deltas. The test must assemble them into the exact string and confirm that the final status does not appear before the last part.

Fifth: a retry after a timeout. If the gateway retries the request, it must not execute a side-effecting tool twice. Operations such as creating an application, transferring money, or sending a notification need an idempotency key on the tool side. Retrying a model request and repeating a business action are different risks.

Compare semantic invariants rather than the entire response byte for byte. Creation time, provider response identifier, technical field order, and usage may differ. Check what the conversion exists to preserve: turn order, item types, call identifiers, arguments, final status, error code, and JSON Schema compliance.

For regressions, keep two groups of fixtures. The first contains input and the expected canonical operation. The second contains a canonical operation and the expected external JSON for each adapter. This separation quickly shows where the break occurred: input contract parser, internal representation, or response serializer.

State, auditing, and data storage cannot be left to the adapter

Responses API makes server-side state convenient, but convenience does not answer questions about storage and auditing. If the gateway supports previous_response_id, it must understand where history lives: with the external provider, at the gateway itself, or in both places. These options have different costs, latency, deletion rules, and legal consequences.

OpenAI's data controls documentation describes separate retention rules for /v1/chat/completions and /v1/responses; for Responses API, application state may be stored on the platform by default. This is a good example of why two endpoints cannot be declared identical just because their results look similar. Retention policy belongs to the operation contract, not to the model field.

For systems in Kazakhstan and Central Asia, add processing labels to the canonical operation: data class, data residency requirements, allowed regions, audit period, PII masking, and whether external state is permitted. The router must read these labels before selecting a model. If this happens after request adaptation, sensitive data may already have been sent somewhere it is not allowed to go.

An audit log does not have to store the full prompt. It is often enough to preserve a hash of the normalized operation, the allowed metadata set, route, model, capabilities used, duration, usage, error classes, and tool-call relationships. Full text should be kept only for narrowly defined debugging scenarios with separate access controls and retention periods.

Do not hide this under «technical details». When a gateway unifies two contracts, it receives more context than either endpoint alone: history, tools, response schema, and route. That means it becomes the place where access rules must be clearer, not looser.

Do not build universal conversion where an explicit refusal is needed

A gateway succeeds not by accepting every JSON object, but by predictably accepting supported meaning. Chat Completions and Responses API can be served by one architecture if it has a canonical operation, strict links between tools and results, a capability matrix, and separate state machines for streaming.

Leave clients with the familiar contract where they need it. Do not turn the old Chat Completions API into an artificially limited Responses API, and do not pretend that new capabilities always fit inside the old choices[0].message. Every case that cannot be expressed should have one of three outcomes: gateway-owned state, an explicitly limited mode, or a clear error.

If you already have a proxy, start with the most difficult test rather than a simple chat: two parallel tool calls, streaming arguments, and strict JSON Schema in one chain. If this scenario passes without replacing identifiers, producing an early stop, or silently removing strict, the gateway has the right foundation.