Why do you need tail based sampling for LLM traces?

Rare LLM failures almost always lose to ordinary traffic when traces are selected at random. Thousands of short, successful requests arrive while one agent gets stuck on its third tool call, spends money on retries, and disappears from observability with the same probability as a harmless chat.

Tail based sampling fixes this exact imbalance. First, the system receives enough spans to see the outcome of a trace. It then keeps everything that is expensive, slow, or broken, while retaining a small share of normal traffic. This is not a way to «save storage space». It is a way to stop throwing away material needed for investigations.

For LLMs, this works better than it does for many ordinary HTTP services because the outcome of a request comes together at the end: you know the model and route, token count, actual cost, number of retries, the agent's final state, and the result of every tool call. Head sampling decides too early. It does not know that a request that looked harmless at the start will become the most expensive case of the day two seconds later.

Tail sampling selects the outcome, not the promise

A head sampler makes its decision in the SDK, usually when the root span is created. It is inexpensive and protects the application from unnecessary telemetry, but it sees only the beginning of the operation. Once it drops the trace_id, a later span with a tool error cannot restore the history.

A tail sampler runs in the collector. It gathers spans by trace_id, waits for a defined period, evaluates the accumulated trace, and releases either all its spans or none of them. The OpenTelemetry Collector Contrib documentation explicitly says that, for an effective decision, all spans from one trace must reach the same collector instance. It also lists rules based on delay, status code, string and numeric attributes, Boolean attributes, span count, probability, and traffic limits.

The word «all» matters here. A separate span with ERROR, without the root span, model, route, and neighboring attempts, is almost useless. You need the complete causal path: the user operation, model selection, generation, tool, retry, final response, and result record.

Tail sampling has a cost. The collector temporarily stores unfinished traces, makes decisions with a delay, and requires careful routing. For a service endpoint that needs only average latency, this may be unnecessary. For agent workflows, RAG, payment checks, medical assistants, and expensive generations, it is usually justified.

Do not confuse tail sampling with logging every prompt. Trace selection answers the question, which operation should be preserved. Data policy answers another question: which fields may be sent to the collector at all. The second question must be handled before the first.

p99 cannot be written into a rule as a magic number

The phrase «preserve p99» sounds precise, but in a configuration it often becomes a mistake. p99 is a property of a set of observations over a period, not of one trace. You cannot look at a span lasting 4.2 seconds and honestly call it p99 until the window, comparison group, and distribution have been defined.

Comparing all LLM requests against one p99 is especially harmful. Generating a short summary and performing a multistep contract check have different normal durations. A fast model and a reasoning model also belong to different distributions. If you mix them, the threshold will be too low for one stream and uselessly high for another.

First build latency metrics before sampling, using only the measurements you need:

• operation.name, for example support.reply or contract.check;
• model family or route class identifier;
• operation type: generation, embedding, rerank, tool execution;
• outcome code.

The OpenTelemetry Span Metrics Connector aggregates request, error, and duration metrics from spans. It can calculate quantiles regardless of which complete traces you later preserve.

Then turn the analysis into a working threshold. For example, once a week the team sees that, for contract.check on a particular route, the boundary of the latency tail is 8 seconds. This is not an «eternal p99». It is a current operational threshold that you review when you change the model, provider, cache, or workflow itself.

Record the measured duration on the root span. For convenience, add a flag calculated by the application or a processor:

llm.trace.duration_ms = 8427
llm.tail.latency_outlier = true
llm.operation = "contract.check"
llm.route_class = "reasoning"

Always keep the number. A Boolean flag is useful for a simple rule and for investigations, but it hides the size of the deviation. If you revise the threshold, the number lets you reclassify traces that have already been stored.

Do not put a rule such as «preserve everything above 5000 ms» on the entire cluster just because it looks neat in YAML. It will quickly turn a slow but normal route into an endless source of expensive traces.

An expensive request and a long request are not the same thing

Duration often correlates with cost, but they are different signals. A long request may be waiting for an external API or a tool queue while consuming very few tokens. A short generation may be expensive because of a large context, an expensive model, or several parallel agent branches.

Calculate cost where the application already knows the facts. After the provider or gateway responds, collect input and output tokens, apply the rate, account for retries, and record the result on the root span. If you cache prompts, record cached tokens separately. Otherwise, a rough formula will produce the wrong cost.

For example:

llm.usage.input_tokens = 18640
llm.usage.output_tokens = 2176
llm.observed_cost_usd = 0.1374
llm.retry_count = 2

The attribute name here is deliberately application-specific. Do not expect a telemetry standard to know your currency, contract rate, discount, cache, or internal GPU cost. OpenTelemetry GenAI semantic conventions standardize information about the model and tokens, and, when explicitly enabled, the contents of prompts, completions, tool calls, and tool results. Pricing logic remains the responsibility of your system.

Choose a monetary threshold separately for each operation. For a high-volume classifier, 5 cents may justify an investigation. For a rare legal workflow, it may be a normal price. Then preserve traces above the threshold in full, regardless of their duration or status.

A common bad recommendation sounds like this: «Keep only requests with a high token count». People like it because a token counter is available almost everywhere. But tokens do not explain why a request is expensive. In a trace with 40,000 input tokens, the cause may be an expected long document. In a trace with 6,000 tokens, it may be three retries caused by an incorrect tool schema. The sampler should catch both cases, but mark them with different reasons.

A successful HTTP response can hide an agent failure

In an LLM application, a transport error and a task error often diverge. The provider returned HTTP 200, the model generated JSON, and the tool received its arguments, but the CRM rejected the request because of permissions. Or the tool returned an empty result, the agent retried, exhausted its limit, and sent the user a confident but useless answer.

If you select only status.code = ERROR, you will miss a significant share of these cases. The final outcome of the application action should be treated as the error, not only an HTTP client failure.

I usually add one span for the actual tool execution and record at least:

name = "tool.execute"
llm.tool.name = "customer_lookup"
llm.tool_call.index = 2
llm.tool_call.failed = true
llm.tool_failure.kind = "permission_denied"
llm.tool.retryable = false

If the operation genuinely failed, set the span to ERROR. If the tool ran but returned an expected empty result, do not declare it an error just for sampling. Add an explicit result such as llm.tool.result = "empty" and decide separately whether it is worth keeping. Otherwise, the error metric will become noisy and engineers will stop trusting it.

The trace itself should receive its final status after the workflow ends. The root span may have ERROR if the agent did not achieve its goal, even when every network call completed successfully. It is also useful to record llm.workflow.outcome: completed, failed, abandoned, guardrail_blocked. Your team defines these values, so document them and keep the spelling consistent across services.

This is where tail sampling outperforms rules applied to individual logs. It can see that one tool failure was compensated for by a successful retry, while another caused the entire operation to fail. Preserving both indiscriminately is expensive. Dropping both means missing failures that affect users.

Rules should move from exceptions to the control group

A good strategy consists of mandatory preservation reasons and one control sample. In order, think first about workflow errors, failed tools, cost thresholds, and latency. At the end, keep a small probabilistic share of ordinary successful operations.

The control group is not for investigating an outage. It shows what a normal trace looks like after a release, helps compare selected exceptions with the background, and can reveal a new class of problems that you have not learned to mark yet. Without it, you see only the kinds of pain you already know about.

Below is an example configuration for OpenTelemetry Collector Contrib. Replace the attribute names and thresholds with your own. The syntax uses the supported status_code, boolean_attribute, numeric_attribute, latency, and probabilistic policies. The processor documentation describes them as selection rules based on status, attributes, duration, and traffic share.

processors:
  tail_sampling:
    decision_wait: 20s
    num_traces: 50000
    expected_new_traces_per_sec: 250
    decision_cache:
      sampled_cache_size: 100000
      non_sampled_cache_size: 100000
    policies:
      - name: workflow-errors
        type: status_code
        status_code:
          status_codes: [ERROR]

      - name: failed-tool-call
        type: boolean_attribute
        boolean_attribute:
          key: llm.tool_call.failed
          value: true

      - name: expensive-request
        type: numeric_attribute
        numeric_attribute:
          key: llm.observed_cost_usd
          min_value: 0.05

      - name: slow-contract-check
        type: and
        and:
          and_sub_policy:
            - name: contract-check
              type: string_attribute
              string_attribute:
                key: llm.operation
                values: [contract.check]
            - name: contract-tail
              type: latency
              latency:
                threshold_ms: 8000

      - name: ordinary-control-group
        type: probabilistic
        probabilistic:
          sampling_percentage: 1

This version deliberately does not try to force every condition into one monolithic policy. Separate rules are easier to explain during an incident review: the trace was preserved because of a workflow error, tool error, cost, latency for a specific operation, or the control sample. If your collector version can record the policy that made the decision, enable that capability and check it in the backend. The processor has a feature gate that adds the tailsampling.policy, tailsampling.composite_policy, and cache decision attributes.

Do not add always_sample at the end in the hope that it will act as a «fallback». It will cancel all your savings. If you need a sample of ordinary traffic, use a probabilistic rule with an explicitly defined share.

Mask fields first, then keep the trace in the buffer

A complete LLM trace is especially risky because it can easily contain documents, personal data, account numbers, medical information, tool arguments, and responses from internal systems. The argument «we will not store most traces anyway» does not help: before the tail sampler makes a decision, it has already received and temporarily stored them.

Divide data into three levels. The first can be used in rules and stored broadly: status, latency, tokens, cost, operation name, error type, retry count, and template hash. The second is allowed only in protected storage and for a short retention period: an edited response fragment, document identifier, and rejection reason code. The third should not enter the telemetry pipeline without a separately approved basis: original documents, raw prompts, tool argument contents, and responses containing PII.

A practical approach is to place the masking processor before the tail sampler. It must remove or replace a sensitive value rather than relying on an exporter after the decision. For diagnostics, the shape of the data is often enough:

llm.prompt.template_id = "claim-review-v7"
llm.prompt.characters = 48122
llm.prompt.sha256 = "..."
llm.tool.arguments_schema = "customer_lookup:v3"
llm.tool.arguments_bytes = 312

A hash does not automatically make data safe. Short or predictable values can be brute-forced. A hash is useful for correlating identical inputs, but it must not justify sending secrets to an external environment.

For teams in Kazakhstan, data location is often an architectural question rather than a consent checkbox in a form. AI Router can provide one OpenAI-compatible endpoint and host some open-weight models on your own GPU infrastructure, but attribute masking and trace retention rules must still run before any exporter. No API gateway can fix a trace into which the application has already written a user's passport details.

Choose decision_wait based on late spans, not intuition

A tail sampler does not know that a trace is «definitely complete» if the application does not send it a special completion signal. Child spans may arrive after the root span. That is why it uses the temporary decision_wait window. A window that is too short cuts off a late tool span and creates an incomplete history. A window that is too long increases export latency and memory usage.

The processor documentation gives a direct warning: a trace may be dropped if it is removed from the ring buffer before decision_wait. The authors recommend checking the eviction age and comparing its percentiles with decision_wait itself. Similar values indicate a risk of loss as load increases.

Do not choose 30 seconds just because it is the default. Measure it:

1. Start with a window that covers ordinary asynchronous tool calls in your workflow.
2. Build a distribution of late span ages and count traces removed because the buffer was full.
3. Increase the window if a significant share of required child spans arrives after the decision.
4. Increase capacity or the number of instances if traces disappear before the window expires.
5. Check the integrity of saved long traces manually again.

The processor provides metrics for this: trace eviction age, late span age, number of new trace_ids, total decisions, and decision timer latency. Do not ignore otelcol_processor_tail_sampling_sampling_decision_timer_latency either. If the decision pass itself takes more than a second, it adds latency and increases the risk of losing data before a decision is made.

Most teams look only at the volume of exported data. That is a late metric. First watch for loss inside the sampler itself. Otherwise, you optimize export while failing to notice that the buffer has already discarded an expensive trace.

One trace must arrive in one place

In Kubernetes and multiregion systems, the most troublesome failure is quiet. The root span reaches collector A, tool spans reach collector B, and the agent's final span reaches collector C. Each instance sees only a fragment without the deciding attribute. The backend shows broken pieces, and the team incorrectly blames instrumentation.

A trace_id-based route must come before the tail sampler. It must be deterministic: all telemetry records with one identifier must be sent to the same instance in the group. Simple round-robin distribution after the SDK is not sufficient. The Collector documentation separately emphasizes the requirement that spans from one trace reach one instance together.

Test this instead of relying on a diagram. Run one artificial agent operation with a root span, two parallel tools, and a final span with ERROR. Add the test.case_id attribute to every span. The backend should show one trace with the expected number of spans and one sampling reason. If you see two or three traces with the same test.case_id, your routing is already broken.

Also account for late spans. The processor keeps a decision cache so that spans arriving after the accumulated trace has been removed receive the same outcome. The cache size should match the traffic and the lifetime of delayed spans. Otherwise, part of the history will be preserved and part will disappear after a positive decision has already been made.

Limit volume in bytes, not with percentage-based self-deception

A percentage sample says what share of traces you are trying to preserve. It does not say how much space they will take. One ordinary trace with five spans may be hundreds of times smaller than an agent investigation with long events, several tools, and response fragments.

Measure two values separately: the number of saved traces and their byte size. The tail sampling processor has a bytes_limiting policy that uses a token bucket based on the actual protobuf size of traces. It limits the sustained data flow and allows a configured burst, which is useful when the number of complete errors suddenly grows after an incident.

A byte limiter must not silently discard all errors. First decide what budget you are willing to allocate to mandatory classes: workflow errors, failed tool calls, and expensive requests. Leave the remainder for the control sample. If the stream of mandatory errors exceeds the budget, that is an incident signal, not a reason to make the incident invisible.

A useful practical check is to take saved traces from every class every few days and count the average size, number of spans, number of events, presence of prompt fields, and share with retries. This shows what is expanding storage. Often the culprit is not the LLM span itself but a detailed event containing the tool's HTTP response body, which someone added «temporarily».

Sampling must be tested like business logic

Selection rules are code, even when they live in YAML. Test them whenever the attribute schema, model, SDK, or workflow changes. Rename llm.tool_call.failed to tool.failed, forget to update the policy, and a week later you will discover that there is nothing left to investigate.

Build a set of synthetic traces that runs in CI or on a test collector. It should include at least: an ordinary successful request, a fast expensive request, a slow normal workflow, a root operation error, a failed tool call with HTTP 200, and an error that was successfully fixed by a retry. For each case, specify the expected result and policy name in advance.

Check the consequences too. An erroneous trace should contain the complete path to the cause. An expensive trace should include cost and token usage. A successful control trace should show a normal span tree. If the backend receives only individual child spans, the test has failed even if the sampling percentage formally matches.

Once this works, do not chase an ever more complicated matrix of conditions. Add a new rule only after you can name the incident class it catches and the action an engineer will take after finding it. Good tail sampling leaves few traces, but each one gives you a reason to open it in the morning rather than being just another successful model response.