How a Confidence Interval for LLMs Changes Model Selection

Switching to a new LLM just because it scored 78% instead of 74% is risky. Those four percentage points may represent a real gain, or they may be ordinary noise from a particular test set. Until the team builds an interval for the difference, the number in the table answers only one question: who came out ahead in this run.

In model evaluation, I more often see not bad statistics but no statistics at all. A team carefully collects 100 examples, runs several candidates, picks the top row, and starts a migration. A month later, the advantage disappears on another dataset, in another language, or under real-world traffic. A confidence interval does not automate the decision, but it separates an observed difference from a decision you can defend to product, security, and finance teams.

A difference in scores does not prove superiority

A single benchmark score is an estimate, not a property of the model. If model A solved 156 of 200 tasks and model B solved 148, the difference is 4 percentage points. But 200 tasks do not represent all of the system's future work. They are a sample from a broader set of requests, documents, users, and edge cases.

On another reasonable task set, A might win by 1 point, lose by 2, or win by 7. A confidence interval shows the range of plausible values for the average difference under the chosen statistical model. It does not reduce uncertainty to zero. It makes you show that uncertainty next to the attractive number.

The most common mistake is comparing two independent percentages. The same case should pass through both models. That way, you see not only shared successes but also the structure of disagreements:

• both models succeeded;
• both failed;
• A succeeded and B did not;
• B succeeded and A did not.

Most datasets contain many cases of the first two types. They do little to help choose a winner. The last two types carry the decision: these are the tasks where the candidates actually diverged.

If A and B solve the same number of tasks, but A wins on difficult legal requests while B wins on simple classifications, the average score can also hide what matters to the business. First define the unit of decision: one request, one document, one conversation, or one customer case. Then build the comparison around that unit.

A confidence interval describes a method, not the probability of a completed range

A 95% confidence interval does not mean that the already calculated range has a 95% probability of containing the true difference. The parameter either falls within a particular interval or it does not. The 95% refers to the procedure: if you repeatedly take comparable samples and build an interval in the same way each time, about 95% of those intervals will cover the true mean.

This distinction is not an academic quibble. The statement "we have a 95% probability that A is better" creates false confidence in the decision. A more accurate statement is: "Under this protocol, the 95% interval for the difference runs from X to Y." If zero falls inside it, the data are compatible with both an advantage for A and an advantage for B.

The NIST/SEMATECH Engineering Statistics Handbook defines an interval for a mean as the mean plus or minus a critical t value multiplied by the standard error. It also directly links the width of the interval to two things: sample size and the standard deviation of the observations. For LLMs, this translates without any mystery: more independent and relevant cases narrow the interval, while inconsistent responses, debatable labels, and unstable generation widen it.

An interval does not answer the question "Do we like this model?" It answers a narrower question: which average differences on a predefined metric are consistent with what we observed in the test. A decision to switch models also needs a useful-effect threshold, migration cost, and hard constraints that cannot be offset by an average score.

Paired comparison preserves information about every task

For a binary metric, such as "the response passed the check or not," create a difference for each task:

d_i =  1, если A прошла, а B не прошла
d_i = -1, если B прошла, а A не прошла
d_i =  0, если результат одинаков

The mean of d_i equals the difference between the success rates of A and B. A paired calculation is better than an independent comparison of percentages because the shared difficulty of a task is removed from the difference. A very difficult contract and a very easy FAQ do not inflate the error simply because both appeared in the set.

Here is a minimal example you can place next to the results export. In it, A wins 34 cases, B wins 26, and the outcome is the same for 140 cases.

import numpy as np
from scipy import stats

# +1: A выиграла кейс, -1: B выиграла, 0: ничья
wins = np.array([1] * 34 + [-1] * 26 + [0] * 140)
n = len(wins)
mean_diff = wins.mean()
se = wins.std(ddof=1) / np.sqrt(n)
t_crit = stats.t.ppf(0.975, df=n - 1)
low, high = mean_diff - t_crit * se, mean_diff + t_crit * se

print(f"Разница: {mean_diff:.2%}")
print(f"95% ДИ: [{low:.2%}, {high:.2%}]")

The output looks like this:

Разница: 4.00%
95% ДИ: [-3.64%, 11.64%]

In the table, A looks better: 78% versus 74%. However, the interval includes both a loss of 3.64 points and a gain of 11.64 points. The statement "A is better" does not hold up here. The accurate wording is: on this set, A showed an estimated advantage of 4 points, but there is not enough data to reliably distinguish it from B.

Do not replace the paired calculation with two separate accuracy intervals, and do not compare whether those intervals overlap. The overlap of separate intervals is not a valid criterion for the interval of their difference. Calculate the difference at the task level from the start.

Dataset size helps more slowly than people expect

The width of an interval decreases in proportion to the square root of the number of independent cases. Doubling the dataset does not cut uncertainty in half. To get an interval that is roughly twice as narrow with the same variability, you need roughly four times as many independent tasks.

Take the same difference profile, but with 2,000 cases: A wins 340, B wins 260, and the other 1,400 have the same outcome. The estimated difference remains 4 points. The standard error falls from about 3.87 to 1.22 points, and the 95% interval becomes approximately [1.60%, 6.40%]. Zero is no longer in the range.

That is not a reason to automatically collect 2,000 rows. If they are all nearly identical variants of one template, there is little new information. A thousand paraphrased requests about the same form field do not equal a thousand independent user tasks.

Check whether the dataset contains clusters:

• several requests from one document;
• a series of messages from one conversation;
• templates from one customer or one product;
• variants of the same prompt;
• synthetic cases generated from a shared source.

If a cluster is the real unit of use, resample and calculate the interval by cluster. For example, for a contract analysis assistant, calculate the average result per contract rather than treating every clause in a contract as an independent observation. Otherwise, the table will produce a narrow interval that exists only on paper.

Response variability often matters more than the average score

Two scores of 80% may require completely different amounts of validation. If nearly all tasks produce the same outcome for both models and the differences occur in a few stable cases, the interval may narrow quickly. If the models constantly trade places on tasks of different difficulty, uncertainty will remain high even with a decent average result.

In a binary paired comparison, variability comes specifically from nonzero d_i values. When A wins 40 cases, B wins 0, and the other 160 are ties, the advantage looks more stable than when A wins 80 and B wins 40, even though both scenarios may produce a similar percentage difference. In the second case, the candidates diverge more often, and some disagreements work against A.

For a quality scale, such as 1 to 5, the situation is even more complicated. One annotator may give a 4 instead of a 3 for a complete explanation, while another gives a 2 for a dangerous factual error. The average hides the nature of the disagreement. Keep the original scores by model, task, and annotator. Then you can check whether the fluctuations are a property of the models, a feature of the task, or a disagreement between people.

Do not average incomparable types of errors into one "overall score" without weights the team is prepared to defend. An error in the friendly tone of a response and incorrect advice in a medical document should not influence the choice equally. It is better to make critical violations a separate hard criterion and compare the remaining indicators separately.

Repeated calls do not automatically increase the dataset

Temperature, sampling, provider changes, and an unstable LLM judge add another layer of variability. If you make five model calls for one question, you have five generation observations, but not five independent user tasks.

A poor approach looks like this: the team runs 100 tasks 10 times, gets 1,000 rows, and builds an interval while treating every row as independent. The interval becomes very narrow because the calculation imagines 1,000 different tasks. In reality, the test still covers 100 tasks, and the responses within each group are related.

There are two practical options. For most products, it is enough to set the number of repeats in advance, calculate each model's average score within a task, and then compare those averages in pairs across tasks. If generation variability itself is part of the decision, use a hierarchical bootstrap: first sample tasks with replacement, then sample repeats with replacement within the selected tasks.

A fixed seed helps debug discrepancies, but it should not hide instability. If the product runs at temperature 0.7, an evaluation at temperature 0 does not describe its behavior. Freeze the call parameters before the experiment: model, version, system prompt, temperature, token limit, tools, response schema, and retry policy.

Bootstrap is useful for metrics that do not like simple formulas

A paired t interval works well for an average difference, especially when there are enough tasks and the distribution of means behaves calmly. But LLM evaluation often includes F1, judge-based win rate, average cost with a heavy tail, ranking metrics, and complex aggregation rules. A paired bootstrap is more convenient for these cases.

The algorithm is simple. Repeatedly sample the same number of tasks from the original set with replacement. Each time, calculate the difference between the metrics for A and B, then take the 2.5th and 97.5th percentiles of the resulting distribution. It is important to choose task indices once for both models. If you bootstrap the candidates independently, you break the pairing and lose precision.

import numpy as np

rng = np.random.default_rng(42)
# score_a и score_b содержат метрику для одного и того же задания
score_a = np.array([...])
score_b = np.array([...])

boot = []
for _ in range(10_000):
    idx = rng.integers(0, len(score_a), len(score_a))
    boot.append(score_a[idx].mean() - score_b[idx].mean())

low, high = np.quantile(boot, [0.025, 0.975])
print(low, high)

You can use mean() only for an average metric calculated by task. For F1, first recalculate F1 on the selected indices, then add the difference to boot. For conversations, pass an entire conversation to the procedure rather than an individual message. Bootstrap does not fix a poor test set, but it honestly reflects the shape of the uncertainty produced by the chosen metric.

Statistical significance is not the same as business value

A zero outside the interval means that the data are poorly consistent with no average difference under your protocol. It does not mean that the gain justifies migration. With 20,000 requests, you can obtain a statistically convincing advantage of 0.2 points that does not pay for changing the routing, retraining the team, and repeating security acceptance.

Before launching the experiment, set a minimum practically useful effect, meaning a threshold for a useful difference. Suppose the new model justifies a switch only if it reduces the error rate by at least 2 points. Then the decision is not "the interval is above zero," but "the lower bound of the interval is above 2 points." If the range is [0.8%, 3.7%], A is probably better, but you have no basis for promising the required effect.

Sometimes the situation is reversed. The new model is cheaper or faster, and you are willing to accept a small quality decline, but no more than 1 point. In that case, set a non-inferiority boundary: the lower bound of the quality difference between A and B must be above -1 point. Then separately check whether the savings provide enough money or latency improvement. Do not hide the tradeoff behind a single rating.

For systems with mandatory requirements, average quality is not the main arbiter at all. If a model sometimes exposes personal data, violates a required format, or produces an unacceptable response in a sensitive scenario, it needs a separate stop criterion. An interval around the overall score does not override that prohibition.

The best candidate on the same dataset almost always looks better than it is

If a team runs 15 models, 30 system-prompt variants, and several judges, then declares the candidate with the highest score the winner, it has selected on noise. Even identical candidates will produce a random leader. The more attempts there are, the greater the chance of choosing a lucky deviation rather than a real effect.

Do not try to fix this with a beautiful p-value after the fact. Divide the work into three parts. Use a development set to search for the prompt, tool schema, and short list of candidates. On a validation set, lock the configuration and eliminate obvious underperformers. Keep an untouched holdout for one final comparison of a pre-limited list.

The judge also needs to be fixed. If an LLM judge evaluates the same answer differently or prefers the style of one model, add human review for a sample of disputed and critical cases. Measuring inter-annotator agreement is useful, but do not turn the agreement coefficient into a decorative number. Review specific disagreements and rewrite the rubric where people are evaluating different properties of the response.

The decision to switch models must survive data slices

The average interval across the full dataset is useful, but it is rarely enough for production. Break the results into slices that correspond to risk: Russian and Kazakh languages, document types, short and long contexts, industry terminology, requests with tools, and sensitive cases. Do not expect every slice to produce a narrow interval. Small slices will honestly show where data are missing.

Do not create dozens of slices just for a colorful dashboard. Every slice should answer a specific threat. If the medical workflow makes up a small share of traffic but errors there are costly, it needs its own dataset and its own threshold. An overall gain does not offset a regression in that workflow.

For teams that run LLM traffic through a single API gateway, it is convenient to store the model identifier, prompt version, call parameters, slice, and annotator decision in the evaluation log. AI Router can help with a unified request route and audit logs, but the statistical decision is still made by your protocol, not by the gateway.

The practical rule is simple: do not claim that a model is better until you can show the difference, the interval, the dataset composition, the aggregation rule, and the decision threshold set in advance. If one of these items is missing, you compared demos, not models.

The next run does not require a huge research program. Take the current results, pair them by task, and build an interval for the difference. If it is wide, do not argue about the winner. Collect exactly the independent, high-risk cases that are missing, then repeat the check on a dataset that nobody used to tune the prompt.