Has Test-Set Contamination Already Ruined Your Eval?

A high LLM score says nothing about model quality until you separate generalization from recognition. If test wording, answers, or close variants have already appeared on the public internet, in a fine-tuning dataset, or in your team's prompts, the model may pass the evaluation from memory. In production it will encounter a different document, a different order of conditions, and a different exception, while your leaderboard will not notice.

Test-set contamination cannot be removed with a single deduplication tool. Exact copies are uncommon and easy to catch. The more dangerous case looks different: fifty tasks follow one public template, a synthetic generator rearranges the conditions, and an editor replaces the entity names. To a person, these are different rows. To a model that has already seen the original benchmark and thousands of retellings, they are one familiar exam.

In its GPT-4 report, OpenAI described a separate check for overlap with evaluation data. That is a good engineering habit, but it does not prove that every external test is clean: with a closed model, an outside team cannot see the full pretraining corpus. The authors of "Proving Test Set Contamination in Black-Box Language Models" develop black-box methods precisely because weights and training data are often unavailable.

Contamination and leakage undermine evaluation in different ways

Contamination means that the model may have encountered evaluation material before your run. The source may be public: a GitHub repository, a competition page, a dataset on Hugging Face, a discussion containing correct answers, or a study guide that rewrote the questions. In that case, you do not have to find someone inside the company to blame, but you do have to stop calling the score a clean test of generalization.

Leakage between development and testing happens inside your own process. The team added a test example to a few-shot prompt. An analyst sent an export to a contractor for labeling. An engineer included test errors in an SFT dataset. A manager showed a model vendor ten of the most difficult cases and then measured improvement using those same lines. This is no longer a data-origin risk. It is a direct defect in the development loop.

Mixing these cases is harmful. With external contamination, you weaken the evidentiary value of the benchmark and add independent checks. With internal leakage, you invalidate the result for the affected records, trace the data path, and change access controls. Saying "we did not train the model" does not excuse a situation where test examples lived in a system prompt or in a set used for manual model selection.

There is a third case that is often hidden behind the word "leakage": a repeated template. The model has not seen this exact line, but dozens of examples use the same operation, the same answer distribution, and the same prompt style. Formally, there is no overlap. In practice, one skill receives disproportionate weight and the final score becomes fragile.

One high score does not separate memory from skill

A model that knows the answer and a model that knows how to solve the task can produce the same string. Ordinary accuracy does not distinguish between these causes. So it is pointless to ask the model, "Have you seen this question?" It does not keep a verifiable source log and may confidently give the wrong answer about its own training history.

You need an invariance check. Keep the logic of the task, but change features that should not affect the answer. Rearrange the conditions. Remove brand names. Replace the everyday scenario with a neutral one. Add an irrelevant detail. Ask for the answer in a different format. If the model solves the original perfectly but drops sharply on an equivalent version, you have a familiarity signal, not proof. That signal is already enough to inform a procurement decision.

The paper "NLP Evaluation in trouble" calls training on a test split followed by evaluation on that same split the most serious case and emphasizes that the scale of the problem is difficult to measure. The practical conclusion is simpler than the academic wording: do not publish one percentage and pretend it measures one thing.

Track four values for each task family:

• performance on the original lines;
• performance on equivalent paraphrases;
• performance on new tasks from the same workflow;
• the share of cases where not only the answer but also the model's confidence changes.

The gap between the first and second values is more useful than the average score on a public set. It shows how dependent your evaluation is on the surface form of the text. The gap between the second and third values shows something else: whether the skill transfers to real incoming documents that do not resemble training tasks.

The checklist starts with the origin of every row

Dataset validation does not start with embeddings. First find out where every record came from. If the team cannot answer that question, it does not know what it is evaluating.

For each row, create a registry with the fields item_id, source, source_url_or_ticket, created_at, author_or_owner, license_or_access, transformation, cluster_id, answer_owner, and release_status. In transformation, do not write "cleaned". Record the specific action: "translated from English", "paraphrased by an editor", "generated from template v3", or "extracted from a customer request, PII removed".

Then go through this checklist. It is short, but the team should answer every item with a document, not a memory.

1. Check whether the original question or its translation appeared in a public repository, article, educational material, competition, or public dataset.
2. Check whether the example appeared in prompts, demonstration requests, manual tests, SFT, preference data, or reports from earlier experiments.
3. Find exact and near duplicates inside test, as well as overlaps between test and train, dev, and the error set.
4. Group paraphrases and synthetic variants into clusters, then calculate how much weight each cluster has in the final metric.
5. Mark rows where the answer can be guessed from the option position, length, JSON pattern, or a repeated word in the question.

The last point is underestimated. In multiple-choice tests, the correct option is often more likely to appear in one position, contain a more detailed formulation, or repeat a term from the question. This is not pretraining contamination, but the defect produces a false score in exactly the same way: the model learns a shortcut instead of the required operation.

Do not try to solve the problem with licensing or access controls alone. A dataset may be closed to you and already be part of a provider's training data. Conversely, a public set can still be useful for regression if you openly call it public, do not use it as the sole criterion, and pair it with a private check.

Exact matches are caught by a script, not careful reading

Reading a table by eye is good at catching obvious copies and bad at catching punctuation changes, Unicode differences, translations, and rearranged phrases. Below is a minimal check worth running every time you update the eval set. It looks for exact matches after normalization and pairs with high character-level similarity.

import csv
import re
import unicodedata
from difflib import SequenceMatcher
from collections import defaultdict

def normalize(text: str) -> str:
    text = unicodedata.normalize("NFKC", text).lower()
    text = re.sub(r"\s+", " ", text).strip()
    text = re.sub(r"[^\w\s]", "", text)
    return text

def read_items(path: str):
    with open(path, newline="", encoding="utf-8") as f:
        return list(csv.DictReader(f))

train = read_items("train.csv")
test = read_items("test.csv")

train_index = defaultdict(list)
for row in train:
    train_index[normalize(row["prompt"])].append(row["id"])

for test_row in test:
    prompt = normalize(test_row["prompt"])
    if prompt in train_index:
        print({
            "type": "exact_overlap",
            "test_id": test_row["id"],
            "train_ids": train_index[prompt]
        })

    for train_row in train:
        score = SequenceMatcher(
            None, prompt, normalize(train_row["prompt"])
        ).ratio()
        if score >= 0.92:
            print({
                "type": "near_duplicate",
                "test_id": test_row["id"],
                "train_id": train_row["id"],
                "similarity": round(score, 3)
            })

The output should have the form of records, not a single aggregate number:

{'type': 'exact_overlap', 'test_id': 't-041', 'train_ids': ['tr-882']}
{'type': 'near_duplicate', 'test_id': 't-117', 'train_id': 'tr-301', 'similarity': 0.947}

Do not put the 0.92 threshold into your policy and assume it is universal. For short questions it is too crude; for long legal passages it may miss a copy with one changed phrase. Take the first 50 matches, have a person label them "duplicate", "same template", or "coincidental similarity", and choose a threshold for your type of text.

This script does not look for leakage in the model's pretraining. It does a more practical job: it keeps your team from accidentally evaluating on its own train set and prevents dozens of copies of one case from posing as a broad task set. That alone eliminates many nonsensical reports.

Semantic search helps, but it does not deliver a verdict

Embeddings find pairs that do not match word for word: translations, rearranged conditions, and rewritten scenarios. They are useful for creating a manual review queue. They do not prove that two texts measure the same skill, and they do not prove that the model saw one of them before the run.

Build one vector for each example from prompt and another from prompt + expected_answer. The second index matters: two formulations may look similar but require different decisions. Then retrieve the nearest neighbors from test against train, dev, SFT data, and public sources if you have local copies. A person should decide what to do with each pair.

A bad practice looks like this: the team takes cosine similarity above 0.85, deletes everything, and declares the set clean. You will throw away different but topically similar records, leave dangerous structural paraphrases, and lose the audit trail. Good practice stores the reviewer's decision: keep, merge_cluster, remove_from_test, rewrite, or investigate_source.

The authors of "How Contaminated Is Your Benchmark?" propose Kernel Divergence Score as a way to measure possible contamination through differences in representation behavior before and after fine-tuning. It is an interesting research method, but it requires access to the model and the fine-tuning stage. For choosing an API model, it does not replace a control set and transformed tasks.

Do not present semantic search as forensic proof. Its role is simple: find the places where human attention will be useful.

Synthetic variants often preserve someone else's exam

Teams like synthetic data because it quickly increases volume and does not require exposing a real customer request. That is reasonable as long as you do not confuse volume with independence.

Imagine a public task: "A customer asks to cancel a transfer after 18:00. What should the operator do?" The generator creates one hundred variants: it changes the bank to a marketplace, the transfer to a refund, 18:00 to 17:30, the customer's name, and the currency. Every variant still checks the same familiar rule in the same form. If the model has seen the original task, the documentation, and many similar benchmarks, the synthetic set has merely multiplied one signal.

For synthetic data, check not the uniqueness of the lines but the independence of the decision. For each template, answer three questions in writing:

• what fact or constraint must the person solving the task extract;
• what changes between variants and can change the correct answer;
• what wrong path should become plausible after the change.

If only the entity name or number changes, do not add the variant to the test. Keep it in a load-testing, format-checking, or regression set. It may be useful, but it should not increase your confidence in model quality.

Templates with a fixed answer are especially dangerous. For example, in 80 percent of cases the system should answer "transfer to an operator" because the generator builds tasks around a prohibited action. The model can learn the safe phrase and achieve a high score without analyzing a single condition. Calculate class distributions for each template, not only for the dataset as a whole.

Repeated wording gives one skill too much weight

If twenty records differ only by contract number, they are not twenty independent observations. They are one scenario repeated twenty times. With this setup, average accuracy inflates confidence and hides failures on rare scenarios.

A cluster should be built around the operation, not the topic. "Extract the national ID number from a scan", "check the amount on an invoice", and "find the date in a contract" all relate to document processing but test different actions. By contrast, "determine whether a transfer is allowed under the limit" with different names and amounts is usually one cluster if the rules and traps are the same.

Show two results in the report. The first is row-level and helps find specific errors. The second is cluster-level: first calculate the metric inside each cluster, then average clusters with equal weight. If the difference is noticeable, do not argue over which score is "real". Record the reason: one or more scenarios dominate the set.

For critical processes, one more cut is useful: worst-cluster score. It answers an unpleasant but practical question: which type of request does the model handle worst? In banking, healthcare, or government services, a high average score does not compensate for one repeated failure on a sensitive operation.

Dynamic sets such as Dynabench were built around examples on which a particular model fails, while preserving human solvability. The idea is not to catch the model with tricks forever. It is to keep the test from freezing into a form that developers and models have already learned.

Use a private set for decisions and a public set for monitoring

Public benchmarks are useful. They make it convenient to track regression, compare prompt configurations, and see broad differences between models. But they cannot, on their own, establish that a model will handle your requests, contracts, internal rules, or local language context.

A private set does not have to be huge. It has to be well selected. It should contain real operations, known error patterns, recent documents, conflicting conditions, and cases where the correct answer depends on local policy. Do not add everything. Add what the process owner is willing to take responsibility for.

Separate access. A developer who changes the prompt may see the error category and aggregate feedback, but does not always need to see the original text and reference answer. Otherwise, after several iterations, the private test becomes training material without any formal training step.

For LLM applications, it is useful to maintain three layers:

• an open set for regression and reproducible comparisons;
• a working set with de-identified but familiar cases for debugging;
• a private set for release decisions and periodic independent checks.

AI Router can simplify running the same eval through different OpenAI-compatible models when a team needs to compare candidates without rewriting client code. But a gateway will not make the set independent for you. Example provenance, access controls, and cluster-level scoring remain the team's responsibility.

A report should show uncertainty, not hide it in a footnote

A poor report says: "Model A scored 84.2%, while model B scored 81.7%." It does not show what produced the difference, which data may have been public, or how many similar records went into the average.

A good report contains four separate sections. The first lists model versions, the system prompt, generation parameters, run date, and evaluation method. The second shows the dataset's provenance: the share of public, internal, synthetic, and private records. The third presents row-level and cluster-level metrics together with the original-versus-paraphrase gap. The fourth lists excluded rows and the reason for each exclusion.

Do not hide the phrase "possible contamination" at the bottom. If it could change the model choice, it belongs next to the final table. A manager who sees this limitation before procurement will make a more useful decision than one who is shown a polished score and told about the problem after an incident.

Start with one action: take the latest 100 test records, build clusters, create one equivalent paraphrase for each important cluster, and recalculate the result. If the leader changes, you have not found an annoying margin of error. You have discovered that the old test measured familiarity with the task format more strongly than model performance.