How to Build a Reproducible Model Package for a GPU Node

A new GPU node does not have to respond the same way as the old one, even if you copied the weights file to it and successfully brought up the API. The discrepancy is usually not caused by the weights themselves. It comes from the tokenizer, the chat template, decoding parameters, quantization, the engine version, or an undocumented manual change in the model directory.

I have seen this failure in a familiar form: a team replaces the hardware, checks one request, celebrates the speed, and a week later the product team notices that responses have become shorter, JSON sometimes breaks, and the system instruction seems weaker. Engineers start comparing GPUs and drivers. Often the problem is in tokenizer_config.json, a different chat_template.jinja, or an EOS-token value that nobody considered part of the delivery.

A reproducible model package should be a release artifact, not a folder someone once assembled on their own machine. Its purpose is simple: on a new node, produce the same response contract for the same inputs. Byte-for-byte text matching is useful when achievable, but in production it is more important to define in advance what exactly counts as the same.

Weights do not describe the whole model

The weights file defines the network parameters, but it does not tell the runtime how to turn user messages into a token sequence or where to stop generation. That is why the phrase «we deployed the same model» guarantees nothing without an artifact list.

For a typical instruction model, the package should include at least:

• weights in the selected format and every part of the sharded set;
• config.json with the architecture, token IDs, and position settings;
• tokenizer files such as tokenizer.json, tokenizer.model, tokenizer_config.json, and special_tokens_map.json;
• the chat template, if it is stored in chat_template.jinja or in the tokenizer configuration;
• generation rules and stopping constraints;
• a description of the runtime, quantization, and launch method.

In the Transformers ecosystem, the chat template is stored with the tokenizer and applied with apply_chat_template(). Hugging Face documentation also states that when a tokenizer is saved, its template is saved with it, including as a separate chat_template.jinja file. That is the correct behavior, but it will not help if you copied only model.safetensors during the move.

Separate the four things teams constantly mix together.

The first is artifact identity. It means that the file bytes match. Hashes verify it.

The second is model-input identity. It means that the server built exactly the same sequence of token IDs from the same array of messages. The tokenizer and chat template determine this.

The third is decoding identity. It depends on temperature, top_p, top_k, seed, stop-sequence logic, retries, and length limits.

The fourth is execution identity. It depends on the inference-engine version, attention mode, CUDA, driver, GPU architecture, and weight representation.

If you confuse the first level with the others, the team will get perfect SHA256 values and still have to investigate different responses.

The message template changes the input more than it seems

A chat template is not a presentation layer. It creates the actual prompt on which the model continues the text. One template adds a system message, another ignores it. One places EOS after every message, another adds an assistant-response start marker. One serializes tools as JSON, another passes them as ordinary text.

Take one application request:

{
  "messages": [
    {"role": "system", "content": "Отвечай только JSON."},
    {"role": "user", "content": "Назови столицу Казахстана."}
  ]
}

On one node, the template might turn it into a string roughly like this:

\u003cbos\u003e\u003csystem\u003e
Отвечай только JSON.\u003ceos\u003e
\u003cuser\u003e
Назови столицу Казахстана.\u003ceos\u003e
\u003cassistant\u003e

On another node, the library may use a different version:

\u003cbos\u003e[INST] Отвечай только JSON.

Назови столицу Казахстана. [/INST]

The model does not see the original JSON API request. It sees the token IDs produced after this serialization. The two versions above have different lengths, different special tokens, and different context. Expecting the same output would be a mistake.

Check the template as program code, not as a text note. For every release, include in the test set one dialog with system, several user and assistant turns, empty content, Unicode, JSON in a message, and, if the product uses tools, a tool call. Check not only the result, but also the generated prompt or token-ID array.

A minimal script for taking such a snapshot might look like this:

from transformers import AutoTokenizer
import json

path = "./bundle/model"
tokenizer = AutoTokenizer.from_pretrained(path, local_files_only=True)
messages = [
    {"role": "system", "content": "Отвечай только JSON."},
    {"role": "user", "content": "Назови столицу Казахстана."}
]

rendered = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
ids = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True
)

print(json.dumps({
    "rendered": rendered,
    "token_count": len(ids),
    "token_ids": ids
}, ensure_ascii=False, indent=2))

Do not try to store only a visually clean text version of the prompt. It helps a person, but it does not provide a strict comparison when special characters are hidden in the text or similar-looking tokens differ. Store both the text and the token IDs. If the token IDs differ, the problem occurred before the GPU.

The manifest connects files to the launch contract

An artifact directory becomes a deliverable package only after you add a manifest. The manifest should answer questions that an on-call engineer should not have to search for in a chat: where the files came from, which revision was approved, exactly what must be checked, which engine runs the model, and which response properties are considered acceptable.

Do not use floating pointers such as main, latest, or a model name without a revision. By default, Hugging Face Hub downloads the latest repository revision, while snapshot_download() lets you pass an explicit revision parameter. The documentation shows that local snapshots can be tied to a specific revision. Use that as a foundation, but a repository identifier alone is not enough: after downloading, still verify your own file manifest.

Here is an example of manifest.json. The values are deliberately placeholders. In a real delivery, do not leave fields with ellipses and do not allow a script to silently insert default values.

{
  "bundle_format": 1,
  "model_id": "instruction_model_32b",
  "source_revision": "8f2c1a7b4d8e9f00112233445566778899aabbcc",
  "artifacts": {
    "config.json": "sha256:4c1f...",
    "generation_config.json": "sha256:9a20...",
    "tokenizer.json": "sha256:6dc8...",
    "tokenizer_config.json": "sha256:28f1...",
    "special_tokens_map.json": "sha256:73ab...",
    "chat_template.jinja": "sha256:1ef4...",
    "model_00001_of_00004.safetensors": "sha256:ad92...",
    "model_00002_of_00004.safetensors": "sha256:c030...",
    "model_00003_of_00004.safetensors": "sha256:71a6...",
    "model_00004_of_00004.safetensors": "sha256:fe25..."
  },
  "runtime": {
    "engine": "approved_engine",
    "engine_version": "0.0.0",
    "container_digest": "sha256:0e4d...",
    "quantization": "none",
    "dtype": "bfloat16",
    "tensor_parallel_size": 2
  },
  "generation": {
    "temperature": 0,
    "top_p": 1,
    "max_tokens": 256,
    "seed": 12345,
    "stop_token_ids": [2]
  },
  "acceptance_suite": "acceptance_2026_07_23.json"
}

Do not put «model version» in a single field. It is a convenient human label and a poor technical identifier. In the manifest, record the source-storage revision, the hash of every file, the package build version, the engine, and its image separately. Then an investigation will not begin with an argument about what the team meant by «release 3».

A model card is useful too, but it does not replace a manifest. Hugging Face describes a model card as a document for reproducibility and information about use, training, and evaluation. Keep the origin, license, limitations, and link to the internal approval process there. The manifest, meanwhile, should be machine-readable and block startup when artifacts do not match.

Checksums must be calculated before the first launch

A hash recorded in a wiki does not protect a delivery. Verification must be part of the installation procedure and must fail before the process loads the weights into memory.

On the build machine, create the list like this:

cd bundle/model
find . -type f ! -path './.cache/*' -print0 | sort -z | xargs -0 sha256sum \u003e ../SHA256SUMS

On the new node, verify it like this:

cd bundle
sha256sum -c SHA256SUMS

A valid output looks like this:

model/config.json: OK
model/tokenizer.json: OK
model/chat_template.jinja: OK
model/model_00001_of_00004.safetensors: OK

The sha256sum -c command catches a corrupted or replaced file, but it does not catch an extra file that the runtime might select instead of the expected one. The installer should therefore work in an empty directory and allow only files listed in the manifest. After verification, the script should traverse the tree, normalize relative paths, and compare the set of files found with artifacts in manifest.json.

A particularly dangerous situation occurs when two weight formats are present. The folder contains both .bin and .safetensors, and different runtime versions select different files. The team thinks it deployed one model, but the nodes are actually using different representations. Keep only the format approved for the specific launch method in the release directory. Store the source-file archive separately.

SHA256 protects integrity, not origin. If an attacker or an erroneous process changes both the file and the manifest, verification will pass. For packages that move between environments, sign the manifest with a release key and verify the signature on the node using a trusted public key. A hash answers «does the file match the manifest?», while a signature answers «who approved this manifest?». These are different checks, and both are needed when a package crosses trust boundaries.

Generation determinism has limits

Temperature 0 does not make the entire service mathematically invariant. This setting usually disables random sampling and leads to selecting the most likely next token, but tied or nearly tied logits, different kernels, and floating-point differences can still change the choice. After the first different token, all subsequent text will follow another path.

Do not promise users «the same answer on any hardware» until you have tested it. Instead, define one of three contracts.

The first is strict: the same token-ID array on an approved hardware configuration and exact image. It makes sense for regression tests and narrow tasks with temperature 0.

The second is application-level: the same required JSON fields, types, classification values, tool call, or routing decision. It suits most services.

The third is qualitative: the response passes a separate evaluator against a rubric. It is needed for open-ended text, but it is not suitable as the only migration safeguard because the evaluator itself may change its decision.

Record generation parameters in full. «We use temperature 0» is not a complete setting. You need max_tokens, top_p, top_k, min_p when available, seed, repetition penalty, presence penalty, frequency penalty, stop strings, stop token IDs, the logic for removing the stop marker from the response, and behavior when the limit is reached. In one API, a stop string may be checked after decoding; in another, before it. For JSON, this can determine whether you receive a closing brace or an incomplete object.

In a release package, it is preferable to specify stop token IDs when their semantics are known for the particular model. Keep string stop sequences only where the application genuinely requires them and where you have tested the result with multibyte Unicode. The string </answer> looks safe until the model emits part of the marker in a different token split or the client cuts off the stream before the server does.

The engine version and GPU are part of the verification scope

A container is useful because it fixes Python, libraries, and the server process. It does not fix the kernel driver or the physical GPU. A node with the same image but a different accelerator architecture or a driver outside the supported range may fail to start, select another kernel, or behave differently under heavy load.

NVIDIA describes CUDA compatibility as a set of limited modes, not as a universal promise. For CUDA 11 and later, compatibility within the same major-version family may be possible with a sufficiently recent driver, but the documentation separately warns about feature limitations and problems in applications that use PTX with an older driver.

Do not turn this into a manual list of dozens of versions in an article or runbook. Capture the node's actual specification during acceptance and attach it to the release record:

nvidia-smi --query-gpu=name,driver_version,memory.total --format=csv,noheader
python -c "import torch; print(torch.__version__); print(torch.version.cuda); print(torch.cuda.get_device_name(0))"

The output should go into the deployment log together with the container digest and acceptance-suite result. Comparing them will show exactly what changed when the new cluster started responding differently.

Quantization belongs here too. FP16, BF16, AWQ, GPTQ, and other formats are not interchangeable copies of the same behavior. They can change logits, memory use, the allowed batch size, and the final text. Specify the quantization method, converter version, calibration set if one was used, grouping parameters, and file format. Two models cannot be called identical simply because they have the same source checkpoint.

Acceptance tests should catch familiar failures

A single question such as «What is the capital of Kazakhstan?» checks that the server is alive and the model knows an obvious fact. It says almost nothing about delivery correctness. A good test set is small but deliberately inconvenient.

Build five classes of cases:

• a system instruction requiring a strict format;
• a multi-turn dialog in which the response depends on the assistant's previous turn;
• Unicode, Cyrillic, Kazakh characters, and mixed text;
• structured JSON with schema validation;
• a tool or RAG-context scenario, if the product has one.

Store each case as an input, expected invariants, and, where appropriate, reference token IDs or reference text. Do not put real customer requests in the package. Synthetic examples are better: they can be safely sent to any environment, included in CI, and discussed during an incident.

Example of one test:

{
  "id": "json_city_kz_01",
  "messages": [
    {"role": "system", "content": "Верни JSON с полем city. Без пояснений."},
    {"role": "user", "content": "Столица Казахстана?"}
  ],
  "generation": {
    "temperature": 0,
    "top_p": 1,
    "max_tokens": 32,
    "seed": 12345
  },
  "expect": {
    "json_schema": {
      "type": "object",
      "required": ["city"],
      "properties": {"city": {"type": "string"}},
      "additionalProperties": false
    },
    "rendered_prompt_sha256": "sha256:replace_me",
    "must_contain": ["Астана"]
  }
}

Here, rendered_prompt_sha256 checks the path to the model, namely message serialization. JSON Schema and must_contain check the output. If only the prompt hash is broken, do not spend the night analyzing CUDA. If the prompt hash matches but the JSON is invalid, look at the generation parameters, engine, and stopping conditions.

Do not make the acceptance suite too fragile. A test that requires a generative model to produce the exact same paragraph on every GPU will create false alarms. A test that accepts any nonempty text protects nothing. Each case should have one clear purpose: preserve the format, maintain context, call the correct tool, retain Unicode, or stay within an allowed limit.

Treat moving to a new node as a release

Manual directory copying over SSH almost always ends with the change history remaining in one person's head. Even if it works today, tomorrow nobody will know where the second tokenizer came from or why one node received a different configuration file.

A working procedure looks like this:

1. The build environment obtains a specific revision of the source artifacts in an empty directory and records it in the manifest.
2. A script selects the permitted files, checks for duplicate formats, calculates SHA256, and creates SHA256SUMS.
3. A separate process creates or updates the tests, calculates hashes for the generated prompts, and runs them in the reference environment.
4. The package, manifest, signature, and test set go into immutable release storage.
5. The new node receives the package, verifies the signature and checksums, starts the container with the specified parameters, and then runs the acceptance suite before being added to the load balancer.

The most unpleasant error at step four is the ability to replace files at the same path. If object storage allows overwriting models/prod/current, you have created a pointer, not a release. The package path must include an immutable version or manifest hash. You can store a pointer to the approved release separately, but it must not be the only source of truth.

When routing multiple models, this approach provides another practical benefit. AI Router can be used as an OpenAI-compatible access layer, but the model identifier in a request must still refer to a specific approved package, not to an unclear name such as assistant_prod. Otherwise, changing a node or a local model variant will look to the client like an unexplained change in behavior.

A useful package can be restored without its author

The check is simple and uncomfortable: give the package to an engineer who was not involved in building it and ask them to bring it up on a clean node. Do not provide verbal hints, forward a «small fix» in a messenger, or allow files to be changed until acceptance is complete.

This engineer should receive a clear result at every stage: which files were installed, which hashes were verified, which image was started, which template was applied, which tests passed, and where the GPU specification is stored. If they get stuck on the question «which tokenizer is the right one here?», the release is not complete.

Do not chase a beautiful folder of weights. Build a package that proves its origin, describes execution, and catches a change in behavior before traffic reaches the new node. Then moving between GPUs stops being a risky operation and becomes a routine release procedure.