How Unifying LLM SSE Events Keeps Streaming in Order

An LLM stream cannot be reduced to a loop like for chunk: print(chunk.text). Code like that survives a demo, but in production it loses tool arguments, confuses a refusal with a network error, and declares the response complete simply because the socket closed.

A proper shared streaming layer does not try to make every provider look alike. It defines a small sequence of its own events, preserves block order, and clearly distinguishes a valid final state from an unexpected interruption. Clients get one contract, while adapters remain free to deal with provider-specific details at the system boundary.

SSE sets the frame, not the meaning of events

SSE defines a text-based way to deliver messages over HTTP with Content-Type: text/event-stream. In the HTML Standard, each message consists of lines such as event:, data:, and an empty line that ends the record. The standard also allows several data: lines in one message and an id that the browser can send back when reconnecting.

This does not mean that SSE knows how to stream a model response. It does not say what a delta is, when a tool call is open, whether fragments can be reordered, or what counts as a successful ending. That is the provider API’s semantics.

In practice, there are at least four different models:

• An OpenAI-compatible Chat Completions stream often sends JSON in data: and ends with the [DONE] line.
• The OpenAI Responses API sends typed events such as response.created, delta-addition events, and response.completed or response.failed.
• The Claude Messages API sends named SSE events and builds the response through the content block lifecycle.
• Gemini streamGenerateContent sends a sequence of GenerateContentResponse objects, where one object may contain the next part of a candidate or final information.

The mistake starts with the name. A team says, “We have SSE,” then applies a handler designed for choices[0].delta.content to Claude or expects [DONE] from Gemini. They may all use SSE on the wire, but their response protocols are different.

The common format should describe blocks, not tokens

A single text delta is convenient while the application only prints chat. Once the model calls a function, returns a structured object, issues a refusal, or produces a hidden reasoning section, a “token” stops being the unit of integration.

Use a content block as the foundation. A block has a stable block_id, an index, and a type. Deltas arrive inside it. A text block receives strings, a tool-call block receives a name, a call identifier, and JSON fragments, while a reasoning block may receive text or a separate signature for integrity checks.

The minimum contract I use between a gateway and an application looks like this:

{
  "stream_id": "st_01J...",
  "seq": 17,
  "type": "block.delta",
  "block": {
    "id": "b_1",
    "index": 1,
    "kind": "tool_call"
  },
  "delta": {
    "kind": "json_text",
    "text": "{\"city\":\"Alma\"
  }
}

This contract needs six mandatory types:

1. stream.start says that the gateway accepted the stream and assigned it an identifier.
2. block.start opens a specific block and announces its type.
3. block.delta adds an immutable fragment to an already open block.
4. block.stop closes the block and prohibits new deltas for it.
5. stream.done confirms successful completion and carries final metadata.
6. stream.error ends the stream with an error code, category, and safe message.

The stream.start event does not mean “the provider started generating.” It only means that your public contract has started. The stream.done event does not mean “the client finished reading the HTTP body.” It means that the adapter received enough confirmation of a successful result.

You can add stream.heartbeat and stream.warning, but do not make them mandatory for the client. A ping, an SSE comment, and another text delta should not change the response state.

Delta order matters more than fragment size

A provider does not promise convenient chunk sizes. One delta may contain a word, several sentences, an empty line, half of a Unicode character at the byte-reading level, or a piece of JSON without its closing brace. Your interface should not expose assumptions about token boundaries.

The reliable rule is simple: the adapter passes deltas in the order in which it received them for one logical block and does not merge different blocks for UI convenience. If blocks have indexes, the index determines their place in the final content array, while seq determines the order in which events are delivered to the client.

Consider a response in which the model first writes text, then calls a function, and writes a continuation after the function returns. The correct sequence is:

stream.start
block.start text index=0
block.delta text="Проверяю расписание."
block.stop text index=0
block.start tool_call index=1
block.delta json_text="{\"date\":\"2026-07-23\""
block.delta json_text="}"
block.stop tool_call index=1
block.start text index=2
block.delta text="На сегодня доступно..."
block.stop text index=2
stream.done

Do not show the user the second text block before the application has executed the call, even if a particular provider can generate parts of the response in parallel. Do not merge index=0 and index=2 into one buffer and hope to reconstruct the meaning later. Auditing, retries, and tool orchestration will need the original structure.

Decide separately what to do with reasoning. If product policy prohibits showing it to the user, do not present reasoning as ordinary text and do not discard it silently. Pass the block with visibility: "internal" into a protected channel or disable its delivery at the request level. Different models apply different access and verification rules to this part of the response, so a universal UI flag is risky here.

Stream completion requires proof from the provider

A closed connection means only that the connection is closed. The server may have ended the response normally, a proxy may have interrupted a long request, the user may have lost network access, or the upstream may have failed after sending half of a JSON argument. These cases cannot all be reduced to one done state.

Every adapter should have an explicit table of final signals. For example, in OpenAI-compatible Chat Completions, this may be [DONE] after valid JSON chunks. For the OpenAI Responses API, rely on terminal response.completed, response.failed, response.incomplete, or response.cancelled events rather than only the end of the HTTP stream. OpenAI events carry a sequence_number, which is useful external material for diagnostics, but your public seq should still be assigned by the gateway.

Claude’s documentation describes the lifecycle especially clearly: message_start, followed by one or more blocks through content_block_start, content_block_delta, and content_block_stop, then message_delta and the final message_stop. Claude also explicitly warns that new event types may appear and that clients must handle unknown types calmly. This is a good rule for any adapter, not just Claude.

In streamGenerateContent mode, Gemini sends a sequence of response objects rather than a mandatory final marker like [DONE]. The adapter needs to inspect the documented candidate-completion fields and the final data received in the stream. Closing after the last object is not enough by itself if no data arrived beforehand that can establish the status.

Internally, keep three terminal states:

• completed means that the provider confirmed a normal result.
• failed means that the provider sent an error or the adapter received an unambiguous protocol error.
• interrupted means that the transport failed or the client cancelled the request before a confirmed final state.

For interrupted, you can save the text deltas already received as a draft, but you must not record that response as the agent’s completed result. This is especially important when the last open block was a tool_call or a schema-based JSON response.

Tool arguments must be collected before the block closes

Partial JSON is not JSON. It sounds obvious, but this is exactly where many orchestrators call a function with a truncated argument or start “fixing” model output with regular expressions.

In input_json_delta, Claude sends string fragments called partial_json and recommends collecting them before parsing. Gemini’s structured-output documentation likewise discusses valid partial JSON strings that must be concatenated before producing the complete object. The engineering conclusion is the same: a stream can be displayed incrementally, but the executable object exists only after the block is complete.

An adapter needs one buffer per block_id, not one buffer for the entire response. Tool calls may span multiple blocks, and some APIs allow parallel calls.

type ToolBuffer = {
  name?: string;
  callId?: string;
  rawArguments: string;
  closed: boolean;
};

function appendToolDelta(buf: ToolBuffer, part: string) {
  if (buf.closed) throw new Error("delta after block.stop");
  buf.rawArguments += part;
}

function closeToolBlock(buf: ToolBuffer) {
  buf.closed = true;
  const args = JSON.parse(buf.rawArguments);
  return { name: buf.name, callId: buf.callId, arguments: args };
}

This code deliberately does not try to parse rawArguments on every delta. If the UI wants to show how the call is being formed, it can display the raw text in a technical mode. The tool executor must wait for block.stop, then validate the JSON and the argument schema.

Do not disguise a JSON.parse failure as a tool error. It is an error in the response protocol or an incompatibility between the model and structured-output mode. Logs should retain the provider ID, model ID, original events, and block position, but not user secrets or the full prompt by default.

Errors inside SSE must reach the client as errors

An HTTP 200 status at the beginning of the response does not guarantee successful generation. Once headers have been sent, the server can no longer replace the response with HTTP 429, 500, or 529. Providers may therefore send an error as a separate event in the middle of an already open stream.

Claude explicitly describes event: error with an error object such as overloaded_error. The OpenAI Responses API has separate terminal failure events. In other APIs, an error may appear as a transport failure with diagnostics in the SDK. The common layer must turn all these variants into one stream.error.

A useful error structure should not pretend that all errors are the same:

{
  "type": "stream.error",
  "stream_id": "st_01J...",
  "seq": 24,
  "error": {
    "category": "upstream_overloaded",
    "retryable": true,
    "provider_code": "overloaded_error",
    "message": "Провайдер временно перегружен"
  }
}

After stream.error, close all open blocks only in the internal state. Do not send fake block.stop events to the client. Otherwise, the client may decide that the tool arguments are complete and that the text response ended meaningfully.

Automatic retries are safe only before an externally observable effect has been produced. For ordinary chat, that means before the first delta seen by the user. For an agent with tools, the boundary is stricter: without an idempotency key, do not repeat an execution after the tool may have charged money, sent an email, or changed a record.

Unknown events must survive API updates

Providers expand their streaming protocols. A new event about reasoning, a citation, audio, or a server-side tool should not make a switch handler fail with an exception and cut off text that is already ready for the user.

Separate processing into two layers. The first layer decodes SSE and preserves the original event in a diagnostic trace. The second maps known types to your contract. It marks an unknown type as ignored, increments a metric, and continues the stream unless the provider identified the event as a terminal error.

This is not a call to ignore changes forever. Unknown types must appear in alerts and test transcripts. But crashing a live stream because of a new optional field or a ping event is worse than controlled skipping with observability.

Heartbeat events should not extend a business timeout forever either. Separate the transport timeout, which indicates that the connection is alive, from the progress timeout, which requires a meaningful delta or a documented tool-execution event. Otherwise, an upstream can keep the connection alive with pings while your user waits indefinitely.

The adapter should be a finite-state machine

Streaming quickly produces too many conditions for a collection of if statements. A small finite-state machine makes forbidden transitions visible and gives you proper tests.

The stream state can be described like this:

idle -> open -> receiving -> terminal
                     |          |
                     v          v
                  interrupted  completed | failed

Inside receiving, keep the state of every block: new, open, closed. Allow block.delta only for open, and stream.done only when all blocks are closed. If the provider sends a terminal event while a block is open, the adapter should end the stream with a compatibility error instead of guessing the missing JSON.

Here are rules worth checking for every recorded stream:

1. The first public element is always stream.start.
2. Every block.delta has a block that was previously opened and has not yet been closed.
3. seq increases strictly within one stream_id.
4. No new public events appear after stream.done or stream.error.
5. stream.done is not emitted while a block remains open.

These rules catch errors that are invisible at a glance: a repeated delta after closure, a mixed-up index during a parallel tool call, a final event sent before usage arrives, and accidental reapplication of a chunk after reconnecting.

Test transcripts, not only live requests

A live test against a model is useful, but it reproduces rare failures poorly. Real protection appears when the adapter runs against saved transcripts of actual protocol events.

For each provider, collect at least five anonymized sequences: simple text, multiple blocks, a tool call with fragmented JSON, a terminal error after part of the text, and an interruption without a terminal signal. Add a case with an unknown event. For Claude, add ping, since it can appear anywhere. For OpenAI Responses, add an unsuccessful terminal status. For Gemini, add a stream where text arrives in several response objects.

Check more than the expected final text. Check the entire sequence of normalized events. This is what such a test looks like:

expect(normalize(transcript)).toEqual([
  { type: "stream.start", seq: 1 },
  { type: "block.start", seq: 2, block: { index: 0, kind: "text" } },
  { type: "block.delta", seq: 3, delta: { text: "Привет" } },
  { type: "block.stop", seq: 4 },
  { type: "stream.done", seq: 5, status: "completed" }
]);

Add property tests on top of the transcripts. A generator can split one text delta into random fragments, insert heartbeats, and repeat unknown optional events. The final assembled text must remain the same, and the state machine must preserve its invariants.

AI Router gives teams an OpenAI-compatible input for multiple providers, but endpoint compatibility does not eliminate differences in streaming semantics. If you are building a gateway or client on top of this input, check which events your specific route preserves and normalizes, especially for tools and final statuses.

The client needs a simple contract, while the gateway needs the complete truth

The frontend usually needs only text, status, and progress. The orchestrator needs block indexes, tool call identifiers, stop reasons, usage, and the provider’s original status. Do not make the browser carry this complexity, but do not discard it in the gateway either.

A good division of responsibilities looks like this: the adapter reads the foreign stream and builds strict internal events, the orchestrator makes decisions about blocks and tools, and the client receives a safe projection of what can be shown. At the same time, the normalized log should make it possible to explain why the response ended, where the JSON was interrupted, and what actually came from the model.

If your current contract consists of delta, [DONE], and catch, do not start by rewriting every integration. Take one stream with a tool call, save it as a transcript, add block.start and block.stop, and then prohibit done without a confirmed terminal event. After that, most hidden differences between LLM APIs will stop surfacing directly in the user interface.