Version a model's chat template together with its weights

An empty response after changing models is almost never explained by LLM "moodiness" alone. Often, the team downloaded the same weights, started the same inference server, and unknowingly sent the model a different sequence of tokens. The cause is in the chat template: a Jinja file, a tokenizer setting, or a layer that decided to "standardize" roles for its own API.

A model's chat template should be treated as part of the release, alongside the weights and tokenizer. If you do not pin its version and run regression tests, you do not have a reproducible model. You have a set of artifacts that sometimes answers, sometimes returns an empty string, and sometimes turns tool calling into decorative JSON.

Hugging Face puts it plainly, without unnecessary mystique: a chat model continues a linear sequence of tokens, while the template adds roles and message boundaries. Two models fine-tuned from the same base model may expect completely different control tokens. Errors in those tokens degrade model behavior even when the weights themselves have not changed.

The chat template is part of the model contract

A model contract is more than a name in a config. It includes the weights, tokenizer, special tokens, chat template, generation parameters, and response parsing rules. When a team updates only model_id, it often changes half the contract without realizing it.

This is especially visible with open-weight models. One format separates messages with tags such as \u003c|role|\u003e, another uses pairs of instruction markers, a third adds a service channel for reasoning, and a fourth prints tool descriptions in special XML-like markup. To a person, all these variants mean "a conversation." To a model, they are different prefixes it was trained to use when predicting the next token.

There is a useful distinction that teams regularly blur: an API message schema and a training message format are not the same thing. The API schema may look identical:

{
  "role": "user",
  "content": "Найди статус заявки 4815"
}

But after rendering, one model may see:

\u003c|user|\u003e
Найди статус заявки 4815\u003c|end|\u003e
\u003c|assistant|\u003e

And another may see:

\u003cs\u003e[INST] Найди статус заявки 4815 [/INST]

You cannot derive the correct second version from the first JSON object. You have to get it from the model artifacts or from the format used by its creator during training.

The practical rule is simple: include the template hash in the release manifest along with the weight revision. If the template is stored inside the tokenizer config, hash the entire set of tokenizer files. If the inference server receives the template through a flag or a separate path, pin the exact file that was passed in, not the template's attractive name in the wiki.

model:
  repository: org/model-instruct
  revision: 8f31c2a
  tokenizer_sha256: "..."
  chat_template_sha256: "..."
serving:
  runtime: vllm
  runtime_version: "..."
  add_generation_prompt: true
  temperature: 0

This manifest does not improve responses by itself. It makes an incident traceable. Two weeks later, you can reconstruct exactly what went into the prompt instead of arguing about whether "something in the template changed."

An empty string often starts with one extra token

Empty responses can have different causes: a server filter, an output limit that is too small, a stop sequence, or a parser error. Still, formatting should be one of the first things you check because it can lead the model directly to an EOS token or into a closed service block.

A typical failure looks like this. The adapter renders the history and then manually appends an assistant marker. The template has already added that marker because add_generation_prompt=true. The prompt now contains two consecutive assistant start markers. One model starts repeating the role, another chooses EOS, and a third produces text that the server then discards as an "invalid assistant response."

The opposite mistake is just as common. The team disables add_generation_prompt because "the last message is already from the user." For a particular model, that means the input ends inside the user block. The model correctly continues the user instead of starting the assistant's response.

Do not try to fix this with temperature. Temperature affects the choice of the next token, but it does not tell the model whose turn it is. First save the exact string before tokenization and the list of token IDs around its end.

from transformers import AutoTokenizer

model_id = "org/model-instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
messages = [
    {"role": "system", "content": "Отвечай кратко."},
    {"role": "user", "content": "Сколько будет 19 * 3?"},
]

rendered = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
ids = tokenizer.encode(rendered, add_special_tokens=False)

print(repr(rendered[-240:]))
print(ids[-40:])
print(tokenizer.convert_ids_to_tokens(ids[-40:]))

In a proper investigation, this command produces three verifiable artifacts: the end of the string, the token IDs, and their textual representation. Compare them with the baseline from the previous release. Do not compare only the model's final response: it tells you where the error occurred far too late.

A separate trap involves adding special tokens twice. The Transformers documentation recommends using apply_chat_template(..., tokenize=True) when preparing for generation, because separate rendering and tokenization steps can easily add service tokens again. If your code must work with a string, explicitly set add_special_tokens=False on the tokenizer and lock this behavior down with a test.

Roles cannot be "normalized" without model-specific rules

A universal internal message format is useful. Universal role semantics without an adapter for each model are dangerous. The most unpleasant errors appear where a platform silently "fixes" the history.

For example, a product may allow several system messages: one from the application, another from an administrator, and a third from the user through task configuration. The model may accept only one system turn at the beginning. If the adapter concatenates them without a separator, instruction boundaries disappear. If it moves the second system message into the user role, it changes its priority. If it silently drops it, the team gets a mismatch between what is recorded in the audit log and what the model actually saw.

The policy should be explicit for each model:

• accept the role sequence without transformations;
• merge permitted system messages according to a documented rule;
• reject unsupported history with a clear error;
• use a separate template when the model provides a tool-use branch;
• verify that the final turn actually prepares assistant generation.

The last point matters more than it may seem. You cannot assume that an assistant role in your JSON always means "the model should continue." Sometimes the last assistant turn is a prefill: you have already started the response and want the model to complete it. That is a different mode. Transformers provides continue_final_message for it; you cannot combine it with `add_generation_prompt because one mode opens a new response while the other continues the current one.

This test should fail before a GPU is started:

def validate_history(messages: list[dict]) -> None:
    allowed = {"system", "user", "assistant", "tool"}
    for index, message in enumerate(messages):
        if message.get("role") not in allowed:
            raise ValueError(f"messages[{index}].role is unsupported")
        if message["role"] == "tool" and not message.get("content"):
            raise ValueError(f"messages[{index}] has an empty tool result")

    if messages and messages[0]["role"] == "tool":
        raise ValueError("tool result cannot start a conversation")

This is not a universal validator for every model. That is its strength: it shows where you must document your own rules instead of relying on a "compatible API."

Thinking blocks require a separate test suite

Reasoning models have added another source of false confidence. The team sees fields such as reasoning_content, thinking, or blocks like \u003cthink\u003e, chooses one format, and starts passing it to every model. Some responses then get cut off, some hidden reasoning reaches the user, and some tool calls end up inside a closed block.

A thinking block is not ordinary assistant text. The template may open it before generation, close it before the visible response, or expect a separate field in the last message. The Transformers documentation explicitly warns that if you put a prefill in content, the template may close the reasoning block before generation starts. To continue the reasoning, you need to fill the reasoning field referenced by the template itself.

Here you need a clear boundary between two tasks:

1. The model reasons within its own format, while the application stores only the final answer.
2. The application continues reasoning that has already started or reproduces a multi-step trace.

In the first case, do not pass the internal block back into the history unless the model and product require it. You will only make the prompt longer, expose something the user does not need, and add more failure points.

In the second case, create a separate test matrix. Ordinary conversations do not replace it. At minimum, test these cases: empty reasoning, reasoning prefill with empty content, completed reasoning before the final answer, a tool call after reasoning, a returned tool result, and the next assistant turn.

A useful contract test checks the rendering structure, not the beauty of the text output:

def test_reasoning_prefill_keeps_block_open(tokenizer):
    messages = [
        {"role": "user", "content": "Объясни 1 + 1"},
        {
            "role": "assistant",
            "reasoning_content": "Нужно сложить два числа. ",
            "content": "",
        },
    ]

    prompt = tokenizer.apply_chat_template(
        messages,
        tokenize=False,
        continue_final_message="reasoning_content",
    )

    assert prompt.endswith("Нужно сложить два числа. ")
    assert "\u003cthink_end\u003e" not in prompt[-80:]

The specific tokens here are illustrative, but the principle is not: the test must know the expected state of the service block at the final character of the prompt. Substitute your model's actual boundaries. If the template does not use reasoning fields, the test should fail with a configuration error rather than "adapt" itself.

Tool calling breaks at the intersection of three protocols

Tool calling involves at least three different formats: how the application describes a tool, how the template presents that description to the model, and how the server parses the generated call into an API object. Teams often test only the last layer. The model returns JSON, so "tools work." No, that only means one example passed through the parser.

The template may turn JSON Schema into a Python-like signature, XML tags, or a custom block. It may output one call or a list. It may require the tool result to use the tool role with a function name, a call ID, or only string content. Hugging Face describes tool_calls as a list in an assistant message and separately shows the tool role for the result, while emphasizing that markup and special tokens depend on the model and must match its training format.

Here is the minimum conversation that must be part of your regression suite:

messages = [
    {"role": "user", "content": "Какая погода в Алматы?"},
    {
        "role": "assistant",
        "tool_calls": [
            {
                "type": "function",
                "function": {
                    "name": "get_weather",
                    "arguments": {"city": "Алматы"},
                },
            }
        ],
    },
    {
        "role": "tool",
        "name": "get_weather",
        "content": "{\"temperature_c\": 18, \"condition\": \"ясно\"}",
    },
]

For this scenario, check five things.

• The template outputs the get_weather description in the format the model expects when tools are present.
• The assistant tool call contains exactly the name get_weather, and the city argument is not turned into a string with unnecessary escaping.
• The tool result has the correct role and block boundaries.
• After the tool result, the template adds the start of a new assistant response.
• Tool JSON does not appear in the final text shown to the user when the model is supposed to return an ordinary answer.

A popular but poor recommendation goes like this: "Let's make every model produce strict OpenAI function calling with a system prompt." It is popular because it quickly produces a demo. In production, it fails on long histories, multiple tools, and model updates. A system prompt does not replace the tokens, format, and examples on which the model was trained to call tools.

You also need a test that many teams miss: several calls in one turn. Even if the selected model usually calls one tool, your contract must define the behavior. Either the executor supports an array and runs permitted calls in the specified order, or the validator rejects the second call. Silently taking the first call means losing an action without recording an error.

A prompt snapshot is more useful than judging whether the response "looks fine"

Behavioral evaluations are necessary, but they are poor at locating a broken template. The same prompt may accidentally produce a good answer at temperature 0.7 and a bad one at temperature 0.0. A rendering snapshot shows the cause before generation.

For every supported model family, create a fixtures directory. It should contain the input messages and expected rendering results. Do not store real client text, access tokens, or internal system responses there. Use short synthetic phrases that make boundaries visible.

fixtures/
  normal_chat.json
  normal_chat.prompt.txt
  system_and_user.json
  system_and_user.prompt.txt
  tool_roundtrip.json
  tool_roundtrip.prompt.txt
  reasoning_prefill.json
  reasoning_prefill.prompt.txt

The snapshot itself should not be the only oracle. Add invariants that report the nature of the error. For example, in an ordinary request the prompt must end with an assistant start marker. In a tool round trip, exactly one such marker should appear after the tool result. In a history with assistant prefill, a new assistant header must not be added. An input without tools must not contain function descriptions from the previous test, or you have state leaking between requests.

assert rendered.count(assistant_start) == 1
assert rendered.endswith(assistant_start)
assert tool_schema_marker not in rendered_without_tools
assert tokenizer.eos_token_id not in input_ids[:-1]

The last check depends on the model format: some templates legitimately use EOS between messages. Do not copy this assert literally. Define your invariant based on where EOS is allowed and what it means. That is the engineer's job: turn implicit knowledge about the model into a rule that can be checked.

A tokenization test catches errors that are invisible in the string

Two prompts may look identical in a log and tokenize differently. The cause may be an added BOS token, an invisible space, a difference in Unicode normalization, or the tokenizer automatically inserting special tokens on the second call.

For several short fixtures, therefore, store not only the rendered text but also the expected tail of the token IDs. You do not have to pin the entire array: it changes when the text is correctly edited. It is enough to pin role boundaries and final service tokens.

def test_generation_suffix(tokenizer):
    messages = [{"role": "user", "content": "ping"}]
    ids = tokenizer.apply_chat_template(
        messages,
        tokenize=True,
        add_generation_prompt=True,
    )

    tail = ids[-6:]
    assert tail == [151644, 8948, 198, 151645, 198, 151646]

The numbers in the example are not universal. In your repository, they must be the real IDs of the selected tokenizer. If the team is afraid to store them because "they are unreadable," it is giving up the best signal that the template and tokenizer no longer match.

Check Unicode separately. Products in Kazakhstan often accept Russian, Kazakh in Cyrillic, Kazakh in Latin script, English, and mixed text in one message. The template must not alter user content when concatenating roles. Run a fixture with І, ң, ғ, apostrophes, line breaks, and a JSON string in a tool result. This is not a translation quality check. It checks that your middleware does not corrupt the bytes before tokenization.

The test matrix should follow transitions, not "features"

A poor test suite is organized by feature names: "there is a chat test," "there is a tools test," and "there is a reasoning test." Such a suite can pass even when the assistant to tool to assistant transition is broken. Formatting depends on neighboring messages, so you need to test transitions between roles and modes.

The minimum matrix for a model without reasoning includes system to user to assistant generation, user to assistant prefill, user to assistant tool call, assistant tool call to tool result to assistant generation, and several consecutive user turns if the product allows them before calling the model.

For a reasoning model, add transitions inside and around the thinking block. For a multimodal model, add every content block type accepted by the template. For a server with automatic tool choice, add a test that the parser understands exactly the format produced by the model. vLLM and similar runtimes separate chat template selection from tool-call parser selection; compatibility between these two parts must not be assumed.

I would start with ten fixtures per model, not a hundred. Use five conversations for basic roles, three for tools, and two for reasoning or prefill. When one finds an incident, add the smallest reproducible fixture and do not close the issue without it. This way the suite grows from real failures rather than imagined coverage.

A model update requires a canary, not trust in compatibility

An OpenAI-compatible endpoint is convenient because the client changes its base URL rather than its entire codebase. But transport compatibility does not guarantee chat template compatibility. Two servers can accept the same messages payload and build different prompts for different models. That is fine as long as the platform makes the process transparent and verifiable.

Before an update, run the fixtures against the old and new artifacts. Compare the rendering first, then the token IDs, then the structured tool-calling result, and only after that the quality of the free-form response. If the template changed deliberately, the review should explain the reason for every difference in the snapshot. "The new template from the model repository" is not a reason, but a source of change.

In AI Router, teams can keep one OpenAI-compatible client, but the production route should still have a separate contract profile for each model family: role format, tool mode, thinking fields, parser, and test matrix. This is especially useful when routing changes the model based on cost, latency, or data residency requirements.

Do not release a model to general traffic until you can answer one simple question: what exact string will be sent to it for every critical transition in your application? If the team cannot show that string, it is testing the conversation with the model by ear. That is not enough for production.