How to reproduce a bad response from a trace and eval

A bad response cannot be fixed from a chat screenshot alone. A screenshot shows the symptom but hides the cause: a different system prompt, another model route, an outdated search chunk, a failed tool call, or a generation parameter that someone changed in the configuration.

Combining an eval with an exact trace turns the complaint «the model answered incorrectly» into a verifiable object. It has an input, context, execution path, evaluation, criteria version, and reproducible package. Without this connection, the team argues over plausible explanations. With it, the team compares facts.

An eval should point to a specific run

An eval is not an inherent property of a prompt, model, or dataset. It belongs to a specific application execution in which the user asked a question, the system found documents, the agent called tools, and the model generated a response.

This distinction is often blurred. A team creates a table with the columns question, answer, and score, then tries to understand why an answer received zero. The table does not contain what the model actually saw. System instructions, dialogue history, the order of retrieved chunks, tool state, and routing rules are rarely included. At best, an analyst guesses the cause. At worst, they fix the wrong component and create a new regression.

Every evaluation should have two references:

• trace_id for the root trace of the user request;
• target_span_id for the step being evaluated: the final answer, retrieval, a tool call, or a classification.

The root trace answers «what happened while processing the request». The target span answers a different question: «what exactly received this evaluation». One user session can include several generations. For example, an agent may first plan actions, then query an internal catalog, and then write a response to the customer. The quality label for the final answer must not be attached to the model call that produced the plan.

A good evaluation record looks like an event that can be opened separately from the observability interface. It contains the evaluator's name, its immutable version, evaluation type, result, explanation, and trace references. If a person provided the evaluation, save the annotator's role and the reason for the decision. If an LLM judge provided it, save the judge model, rubric version, and the input sent to the judge.

Phoenix documentation clearly states a useful distinction: traces show what happened during a run, while eval adds a repeatable quality signal. That is true, but in practice one more step is needed: the evaluation must point to the complete set of causal artifacts from the run. Otherwise, it works for a quality chart but poorly supports engineering work.

The unit of investigation is a run, not a conversation

A run is an immutable record of one attempt by an application to process one request. A conversation may last for hours and contain dozens of messages. A trace may cover only one HTTP request. Reproduction requires an object between them: the processing of one specific user message.

Assign run_id at the application boundary, before the first call to retrieval or a model. Pass it into spans, audit logs, the eval record, and background task queues. trace_id remains a technical telemetry identifier, while run_id becomes the identifier for the business operation. This makes it easier to connect retries, asynchronous steps, and several traces belonging to one request.

For example, a user asks: «Can I close a deposit early without losing the interest?» The application normalizes the request, searches for the relevant policy, calls a product-checking tool, and generates a response. If the response is wrong, the investigation needs one run with all its child operations, not «the chat messages from that day».

Do not treat a retry as the same run. If the network fails after the model request and the client sends the request again, create a new run_id, but retain parent_run_id or retry_of_run_id. Otherwise, the statistics will mix a technical failure with an independent user request, and the eval will count two attempts as one observation.

A useful minimum set of identifiers:

• run_id for the business operation;
• trace_id for the telemetry tree;
• span_id for an individual action;
• request_id for an external HTTP call;
• conversation_id for the session, if history affects the response.

These values do not replace one another. When a team puts only conversation_id everywhere, it loses the boundaries of a specific attempt. When it stores only trace_id, it becomes difficult to match a run with a product record, support request, or delayed task.

The schema should store actual inputs, not the developer's intentions

A developer's intention sounds like this: «we use the support prompt, model X, knowledge-base search, and temperature 0.2». The actual run may look different. The router selected another available model, the prompt was loaded under the current label, retrieval applied a department filter, and the SDK passed max_tokens from an environment variable.

The schema needs the actual values sent to every external or logical step. Do not reconstruct them later from the repository, configuration, and the on-call engineer's memory. Those sources may already have changed.

Below is the minimum package for the final generation. The fields can be divided between trace storage, an artifact catalog, and an eval database, but the connections between them must be direct.

{
  "run_id": "run_01JQ8K7C4V",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "started_at": "2026-07-23T14:18:09Z",
  "application": {
    "name": "deposit-assistant",
    "release": "2026.07.23.4",
    "environment": "production"
  },
  "prompt": {
    "name": "deposit-answer",
    "version_id": "prv_8d7f5a",
    "template_sha256": "2d3a...91c4",
    "rendered_messages_ref": "artifact://runs/run_01JQ8K7C4V/messages.json"
  },
  "generation": {
    "requested_model": "reasoning-model",
    "resolved_model": "provider/model-revision",
    "provider": "provider-name",
    "temperature": 0.2,
    "top_p": 1.0,
    "max_output_tokens": 700,
    "seed": 18421,
    "response_format": "text"
  },
  "retrieval_ref": "artifact://runs/run_01JQ8K7C4V/retrieval.json",
  "tool_calls_ref": "artifact://runs/run_01JQ8K7C4V/tools.json",
  "output_ref": "artifact://runs/run_01JQ8K7C4V/output.json"
}

requested_model shows the application's choice. resolved_model shows what actually processed the request. These are different fields even in a system without complex routing: a provider may return a revision that differs from the logical name in the request.

Keep the artifact reference separate from the content itself. Small, safe metadata is convenient to search directly in a trace. Full prompts, retrieved documents, and tool responses are often large and sensitive. It is more sensible to store them in protected storage with access controls, a checksum, and an expiration period. The trace should contain enough information to find the artifact and verify that it has not been replaced.

OpenTelemetry's GenAI semantic conventions describe these same categories: input and output messages, provider name, requested and returned model, token limits, retrieved documents, and tool-call arguments and results. The standard is useful as a shared vocabulary. Do not try to fit every domain detail into it. Add custom attributes for index version, access policy, template identifier, and routing reason.

The prompt version must be immutable

A prompt name does not reproduce a prompt. The production label does not reproduce a prompt. Even a Git commit does not always reproduce a prompt if the template is loaded from a separate service and variables are assembled from several configurations.

Store at least three things: the logical name, an immutable version_id, and a checksum of the exact template. For an investigation, also save the rendered messages after variables have been substituted. The version explains which template was used. The rendered messages explain exactly what the model saw.

There is an important boundary here. Do not put the template and user data into one identifier. The prompt version should change when instructions, format, examples, tool declarations, or rendering logic change. The user's question and retrieved documents belong to the run, not to the template version.

«Silent» changes are especially dangerous. A team edits the system instruction through an admin panel, keeps the same prompt name, and does not update the version. A week later, eval shows a quality drop, but good and bad responses can no longer be compared. Formally, they all belong to the same prompt.

The judge's prompt requires the same discipline. If an LLM-as-a-judge evaluates answer completeness, its rubric, model, and parameters are part of the measurement methodology. You cannot silently rewrite the rubric and then draw one before-and-after chart as if the scale had not changed. That is not a product quality improvement. It is a change of measuring instrument.

Retrieved documents must be recorded after ranking

RAG fails not only because retrieval returned an irrelevant document. It also fails when the document was relevant but outdated, when the required passage ranked sixth but only the first four entered the prompt, when an access filter excluded the current policy, or when the system truncated a chunk before an important exception.

The retrieval record should therefore reflect the final context set, not just the query sent to the vector database. Save the original search query, index name and revision, filters, ranking algorithm, candidate list in post-rerank order, and the list of chunks actually inserted into the prompt.

For every inserted chunk, record:

• stable document_id and chunk_id;
• text version or checksum;
• initial retrieval score and rerank score, if applicable;
• position in the final context;
• exclusion reason, if the candidate did not reach the model.

Do not use only a URL or document path. A knowledge-base page at the same address may change ten times. On a replay, you will receive plausible but different context and may wrongly conclude that the problem has disappeared.

Another distinction often damages eval. Document relevance and final-answer correctness measure different things. A document may be topically close but not contain the condition needed for the answer. Conversely, a set of good documents does not guarantee that the model will cite the correct rule. Phoenix documentation explicitly separates chunk-level retrieval eval from system-level Q&A evaluation. Keep these evaluations separate and connect both to the same run.

If the final answer is wrong, start with this question: «Could the model have answered correctly using only the context it received?» If not, do not spend the day rewriting the system instruction. Fix the index, filters, chunking, reranker, or knowledge-base coverage.

Tools require a causal log, not just a result

An agent's response often depends on the outside world more than on the model. A balance check, application status, rate, stock level, or access rule can change between two identical requests. If you save only the final text, a replay at another time will produce a different result, and you will not know whether the agent was wrong then or is wrong now.

For every tool call, record the tool declaration, validated arguments, call time, response, error, and the agent's decision after the response. It is especially useful to store the normalized result that was actually passed to the model. A raw HTTP response may contain fields that the application later removed or transformed.

Consider a typical failure. An agent should call get_deposit_terms with a product code. The model extracts the code from the history and passes DP-018. The tool returns terms for an old product because the service accepted an outdated alias. The agent's answer looks polished but contains the wrong rule. If the trace stores only «tool completed successfully», the investigation hits a dead end. If it stores the argument, normalized response, and directory version, the cause is immediately visible.

Do not treat secrets as «the price of debugging». Tool calls often contain tokens, account numbers, and personal data. Apply field-level masking before recording them. You can replace account_number with a stable salted hash to correlate requests without exposing the number. The authorization field should not enter the trace at all.

A tool call that the agent planned but did not make is also data. Add a span or event for the action choice: available tools, selected tool, reason for refusal, and iteration limit. This helps distinguish «the tool returned an incorrect result» from «the model decided that the tool was unnecessary».

The replay package should run outside production

Do not reproduce a bad response by directly repeating the production HTTP request. You might charge money again, send an email, change an application, or read data that has already been updated. Reproduction should run in an isolated environment and deny side effects by default.

Build a package from the trace that contains a run manifest and dependency snapshots. You do not need to duplicate the entire observability store. Include only the artifacts whose absence would change the meaning of the response.

{
  "replay_version": 1,
  "source_run_id": "run_01JQ8K7C4V",
  "mode": "offline",
  "messages": "artifacts/messages.json",
  "retrieval": "artifacts/retrieval-final.json",
  "tool_transcript": "artifacts/tool-transcript.json",
  "generation": {
    "model": "provider/model-revision",
    "temperature": 0.2,
    "top_p": 1.0,
    "max_output_tokens": 700,
    "seed": 18421
  },
  "side_effect_policy": "deny",
  "expected": {
    "evals": ["groundedness=fail", "answer_correctness=fail"],
    "output_sha256": "optional"
  }
}

In offline mode, retrieval does not access the current index. It reads retrieval-final.json. Tools do not call production services. They return the recorded responses from tool-transcript.json. This is not a production imitation for show. It isolates the decision that already happened from the current state of the data.

You need two replay modes. The first is exact: it sends the saved messages, documents, and tool responses to the model again. It answers how the model behaves on the historical context. The second is diagnostic: it runs the current system on the original user request. It answers whether the new version has fixed the problem under real conditions. Do not combine them in one report.

Set limits on full-text access. In some organizations, a package cannot be downloaded to a developer's laptop. Store it in a controlled environment, run replay next to protected storage, and grant access by role. Reproducibility does not justify uncontrolled copying of customer data.

Matching text is not the only criterion

Many teams call replay successful only when the new response is character-for-character identical to the old one. For a generative model, that requirement is too strict and often provides little value. A small wording difference may not change the cause of the failure, while a provider model update may change the text even with identical parameters.

Check reproduction in layers. First, the action tree should match: the same prompt, same document set, same tool order, same actual model, and same parameters. Then check meaning: the same incorrect policy, omitted fact, impermissible operation, or eval failure.

For structured responses, save a normalized representation. If the model returns JSON, remove timestamps, random identifiers, and key ordering before comparing required values. For text responses, compare claims instead: is the rate wrong, is the term incorrect, or does the answer cite a document that was not in the context?

A seed helps but does not promise identity. It does not cancel provider system-layer updates, different batching order, or implementation differences. The seed field should still be saved because it reduces the number of variables and makes discrepancies easier to see honestly.

If exact replay diverges, do not end the investigation with «the model is nondeterministic». First compare the manifests. The difference is often hidden in one field: a new max_output_tokens, a different response schema, a changed tool list, added history, or a provider that accepted a compatible request but routed it to another model revision.

An eval should point to a fix

An evaluation is useful when it helps assign ownership of the problem. The label bad tells the retrieval team, tool owner, and prompt developer almost nothing. Instead of one general score, create a small cause taxonomy that can be checked against the trace.

For a RAG response, five categories are enough to start:

• retrieval_missing: the context lacks material needed for the answer;
• retrieval_wrong: the context contains unsuitable or outdated material;
• tool_failure: a tool returned an error, stale data, or misinterpreted the arguments;
• generation_ungrounded: the required facts were in the context, but the model distorted or ignored them;
• policy_failure: the response violated a format, safety, or permitted-action rule.

One run may receive several labels. That is normal. An incorrect retrieved document may lead to an ungrounded generation at the same time. Do not force the annotator to choose one «main» cause when the trace shows a chain of causes.

Connect categories to actions. retrieval_missing sends the example to the knowledge-base expansion queue or to a set of queries for search improvement. tool_failure goes to the integration owner with the arguments and response snapshot. generation_ungrounded becomes a test case for prompt, model, and decoding variants. This turns eval from a metric display into an input for the engineering cycle.

Turn failed runs into a regression dataset, but do not do so blindly. Remove duplicates of the same error, preserve the distribution of query types, and attach the expected cause. Otherwise, the team will teach the system to answer a dozen similar examples while rare and costly failures remain untested.

Retention and masking determine the value of a trace

A complete LLM application trace quickly becomes an archive of sensitive content. It contains the user's question, message history, retrieved documents, function arguments, and response. Collecting all of this without access and deletion rules is dangerous. Collecting nothing leaves the team unable to investigate incidents.

The answer is not a false choice between «keep all text forever» and «keep only metrics». Divide data by sensitivity. Technical metadata, hashes, versions, durations, document IDs, and eval results can live longer. Full messages and tool results should have short retention periods, masking, access logs, and separate protected storage.

OpenTelemetry warns that input messages, search queries, and model outputs may contain sensitive data. Treat this as a design requirement, not a note in the documentation. Masking should happen before export because deleting data after it has been recorded does not guarantee cleanup of replicas, backups, and third-party systems.

For teams that need to keep data in Kazakhstan, AI Router can be part of the model-call and telemetry-control architecture, but the replay schema remains the application's responsibility. Store the actual provider, model, and route next to the run, not in a separate billing report that cannot be matched to the bad response.

Start with one real failure that the team could not explain within an hour. Take its trace, add the missing fields, build an offline replay, and attach one human evaluation and one automated eval. You will then see which data you are losing today and which fields in the next incident will save days of work.