Benchmark Saturation & Contamination — why the leaderboard is lying to you

For NAO + anyone who has ever quoted a benchmark number. This is the measurement-integrity companion to [[llm-evaluation]]. That article asks how do you score a model at all; this one asks the uglier follow-up: when a model scores 95% on a famous test, is that intelligence, or did the model see the answer key during training? The honest answer is "often you genuinely cannot tell," and that uncertainty is the single biggest threat to trusting AI progress reports today. Every number below is sourced. The frontier here is genuinely unsettled — we'll flag exactly where.

The one-sentence idea

A benchmark is only meaningful if the model has never seen the questions before; contamination is the leakage of test items (or near-copies) into training data, and saturation is what happens when every frontier model clusters near the ceiling so the benchmark stops discriminating — and the two feed each other, because a contaminated benchmark saturates fast and a saturated benchmark gets recycled into training corpora that contaminate the next one.

What it is (intuition first)

Imagine a teacher who writes one exam and uses it every year. Eventually copies of last year's exam circulate among students. The ones who memorized it score 100% — not because they learned calculus, but because they learned this exam. The teacher, looking only at scores, concludes the class is brilliant. That is exactly the position the AI field is in.

Two distinct failures hide behind a high benchmark score:

• Contamination (the exam leaked). The test questions, or paraphrases/translations of them, ended up in the model's training data. The model isn't reasoning — it's recalling. This inflates the score in a way that does not transfer to genuinely new problems.
• Saturation (the exam got too easy). Even with zero contamination, once the best models all score 88–95% on a benchmark, the remaining gap is noise — annotation errors, ambiguous items, formatting luck. The benchmark can no longer tell a better model from a worse one. It is "solved" as a measurement instrument even if the underlying task isn't solved in any deep sense.

The vicious loop that makes this a crisis rather than a nuisance: a popular benchmark gets scraped from the web into the next pretraining corpus → contamination → scores jump → the benchmark saturates → researchers build a harder benchmark → it gets popular → it gets scraped → repeat. MMLU (2020), GSM8k (2021), HumanEval (2021), GPQA (2023), and FrontierMath / Humanity's Last Exam (2025) are successive rungs of this ladder. Each was "hard" for about 18 months.

The reason this matters more than ordinary measurement error: contamination is adversarially aligned with the thing you're trying to measure. A lab racing to top a leaderboard has every incentive (sometimes accidental, sometimes not) to let the test in. So the noise isn't zero-mean — it points up.

Why it matters

• Capability claims become unfalsifiable. "Model X scored 92% on benchmark Y" is, as a standalone sentence, close to meaningless without a contamination audit and a model knowledge-cutoff date. This is the core of what people call the eval crisis.
• It corrupts the feedback loop for research. [[scaling-laws]] promised that loss falls predictably with compute — but downstream benchmark gains are what people actually celebrate, and those gains are exactly the ones contamination fakes. If you can't trust the benchmark, you can't tell whether a new technique helped or just leaked.
• Procurement and safety decisions ride on these numbers. Enterprises pick models, and regulators reason about capability/risk, using leaderboard figures that may be contaminated by double-digit percentages.
• It's measurably large. Surveys in 2025 find contamination reaching up to ~45% on some commonly used benchmarks across popular models (survey, arXiv:2502.17521↗). The 2025 Leaderboard Illusion analysis of Chatbot Arena (arXiv:2504.20879↗) showed a related distortion — private variant testing and selective disclosure let providers overfit the Arena itself, a saturation-by-gaming of a benchmark that was supposed to be contamination-proof because it's live human votes.

How it works (real mechanics)

Defining contamination formally

The 2025 static-to-dynamic survey distinguishes two regimes (arXiv:2502.17521↗):

• Exact contamination — a test item appears verbatim in training: ∃ d such that d ∈ D_train and d ∈ D_test.
• Syntactic / semantic contamination — a test item appears after transformation: punctuation/whitespace normalization, synonym substitution, paraphrase, or translation. Semantics preserved, surface form changed. This is the dangerous kind, because the cheapest detectors only catch the exact kind.

Detection method 1 — n-gram overlap (the industry default)

This is what GPT-3, GPT-4, PaLM and Llama-2 reported using. The recipe:

1. Choose n (GPT-3 used 8–13 depending on dataset; GPT-4 used a 50-character span; EleutherAI's harness fixes n = 13 for simplicity).
2. For each test item, enumerate all contiguous n-grams.
3. Mark the test item contaminated if any of its n-grams collides with any training document.

EleutherAI's lm-evaluation-harness implements it as a hash-index scan (docs↗):

# 1. build an index: every 13-gram -> {eval_ids, doc_ids that contain it}
# 2. scan training-set 13-gram files in sorted order
# 3. mark any (eval_item, training_doc) pair sharing a 13-gram as dirty
# 4. evaluator.py emits a "<metric>_decontaminate" score = score over clean items only

Decontamination is opt-in per task (should_decontaminate / doc_to_decontamination_query) — meaning most evals you see were not decontaminated unless the author bothered.

The fatal weakness: n-gram matching is exact-substring. Change a few words, or translate the question to German and back, and zero 13-grams collide. The contamination is invisible to the detector but fully present in the model. This is not hypothetical (next section).

Detection method 2 — LLM-based / semantic decontamination

Because rephrasing defeats n-grams, LMSYS proposed a two-stage LLM decontaminator (blog↗, arXiv:2311.04850↗):

1. Embedding retrieval — for each test item, retrieve the top-k most similar training items by embedding cosine similarity (see [[embeddings]]).
2. LLM adjudication — pass each (test, candidate) pair to a strong LLM (e.g. GPT-4) asking "is one a rephrase of the other?"

On 200 MMLU pairs (100 verbatim, 100 rephrased) this two-stage detector beat n-gram, BERT-embedding-only, and string-match baselines on F1, catching paraphrases the others missed.

Detection method 3 — membership inference (Min-K% Prob)

When you don't have the training corpus (closed models), you can't do overlap checks at all. Instead you ask the model itself: does this text look "too familiar"? This is membership inference. The canonical method is Min-K% Prob (Shi et al., arXiv:2310.16789↗).

Intuition: an unseen passage will contain a few low-probability "surprise" tokens; a memorized passage won't — even its rare tokens were learned. So look only at the model's least confident tokens and check how confident it still is.

# Min-K% Prob for a candidate text x = (x_1 ... x_t)
for each token x_i:
    lp_i = log p(x_i | x_1 ... x_{i-1})        # model's token log-prob
Min-K%(x) = the k% of tokens with the LOWEST p   # e.g. k=20: bottom 20%
score(x)  = (1/E) * sum( lp_i for x_i in Min-K%(x) )   # E = |Min-K%(x)|
# higher (less negative) score  ->  more likely the model was trained on x
# threshold the score to decide member / non-member

It needs no reference model and no corpus access — just token log-probs. On WikiMIA (a clever benchmark splitting Wikipedia text by edit date relative to the model's training cutoff, so membership ground truth is known), Min-K% averaged AUC ≈ 0.72, beating Perplexity (0.67), Neighbor (0.65), Zlib (0.65), Lowercase (0.61) — a ~7.4% gain over the best baseline.

Read the 0.72 honestly. AUC 0.72 is "clearly better than coin-flip," not "reliable." Membership inference at the individual-document level remains weak and easy to fool; it's most useful in aggregate over many items.

Mitigation — dynamic & private benchmarks

If you can't reliably detect leakage, the alternative is to make leakage impossible by construction. Four families (arXiv:2502.17521↗):

• Temporal cutoff — only test on material created after the model's training cutoff. LiveBench refreshes from recent sources every few months; LiveCodeBench scrapes new competitive-programming problems continuously; AntiLeak-Bench queries knowledge that didn't exist before the cutoff. Limitation: every model has a different cutoff, so "post-cutoff" is model-specific and the benchmark must be re-sliced per model.
• Template / procedural generation — never ship fixed items; ship generators. GSM-Symbolic fills numeric placeholders in solution templates; DyVal builds random DAGs whose answers are computed, not stored; NPHardEval synthesizes random graph instances. Infinite fresh items, zero memorizable surface.
• LLM rewriting — paraphrase/perturb existing benchmarks on the fly (DARG, LatestEval, C2LEVA).
• Private holdouts — keep a portion of items secret forever and only run them server-side. Humanity's Last Exam maintains a private held-out set explicitly to measure overfitting against the public set (agi.safe.ai↗, Epoch AI↗); FrontierMath keeps its hardest problems unpublished. This is the strongest defense and the hardest to run reproducibly (you must trust the holder).

The survey scores 27 dynamic benchmarks on six axes — Correctness, Scalability, Collision (do regenerated items overlap?), Stability-of-complexity, Diversity, Interpretability — and finds no single method satisfies all six. Dynamic benchmarks trade contamination-resistance for new problems: procedural items can be unrealistically uniform; temporal items drift in difficulty over time.

Key ideas & tradeoffs

• GSM1k is the cleanest natural experiment. Scale AI rebuilt GSM8k from scratch as GSM1k — same style, difficulty, step-count, answer-magnitude, human-solve-rate — but guaranteed never published, so guaranteed uncontaminated (arXiv:2405.00332↗, NeurIPS 2024). Re-running models on it revealed accuracy drops of up to ~8%, with several model families showing systematic overfitting across sizes. Crucially, the per-model gap correlated with each model's probability of generating GSM8k examples (Spearman r² ≈ 0.36) — direct evidence that the gap is memorization, not bad luck. The honest twist: frontier models showed minimal overfitting and clearly generalized. So contamination is real and measurable, but it is not the whole story — the best models really are reasoning, not just reciting. Resist both "benchmarks are all fake" and "benchmarks are all real."
• Rephrasing is the unkillable attack. Yang et al. (arXiv:2311.04850↗) trained a 13B Llama on *rephrased* MMLU/HumanEval/GSM-8K test items and matched GPT-4-level scores (e.g. MMLU 85.9%) — while passing standard n-gram decontamination. They also found 8–18% of HumanEval already overlaps with common pretraining sets (RedPajama, StarCoder), and — chillingly — found contamination inside synthetic data generated by GPT-3.5/4, i.e. the leak can be laundered through a teacher model and you'd never grep for it.
• Saturation without contamination is still fatal to a benchmark. HLE went from frontier models scoring ~10–15% in early 2025 to mid-40s by early 2026 (top reported ~46, Epoch AI tracking). At that pace the field's "hardest exam" has a measurement-useful lifespan of a couple of years before the discriminating signal collapses into ceiling noise.
• Live human-vote benchmarks aren't immune either. Chatbot Arena was the great hope for a contamination-proof eval (fresh human prompts, pairwise votes → Bradley-Terry Elo; see [[llm-evaluation]]). The Leaderboard Illusion (arXiv:2504.20879↗) showed it can be gamed: private multi-variant submission + selective publication lets a provider overfit the Arena's prompt distribution. Different mechanism, same outcome — the number no longer measures what you think.
• Detection vs. avoidance. Detectors (n-gram, LLM-judge, Min-K%) are reactive and probabilistic — they tell you probably leaked. Dynamic/private benchmarks are proactive — they make leakage structurally impossible but cost more, drift in difficulty, and (for private holdouts) require trusting an opaque scorer.

Honest caveats & open questions

• No detector is trustworthy at the single-item level. Min-K%-style membership inference sits around AUC 0.7 on the friendly WikiMIA setup and degrades on real, paraphrased, partially-memorized data. Treat any "we proved model X was trained on benchmark Y" claim with suspicion unless it's an aggregate statistical argument, not one document.
• You usually can't audit closed models at all. Overlap detection needs the training corpus. For GPT/Claude/Gemini-class models that corpus is secret, so the only tools are membership inference (weak) and clean-mirror experiments like GSM1k (expensive, one-off). Most public contamination claims about closed models are circumstantial.
• "Contamination-free" is not the same as "good." A procedurally generated benchmark can be perfectly clean and still measure the wrong thing (e.g. template-pattern-matching, not reasoning). Cleanliness is necessary, not sufficient.
• Difficulty drift in dynamic benchmarks is unsolved. If LiveCodeBench's new problems happen to be easier this quarter, scores rise for reasons unrelated to model quality. Temporal benchmarks need difficulty normalization that nobody has nailed.
• Is saturation evidence of progress or of a bad benchmark? Genuinely contested. When everyone scores 95%, you cannot tell from the number alone whether the task is solved or the test is broken — you need item-level error analysis, which almost nobody publishes.
• The frontier-most claim — that top models "really generalize" (GSM1k) — rests on a handful of clean-mirror studies. It's the most reassuring finding in this whole article and also the least replicated. Hold it loosely.

How it connects to OpenAlice

OpenAlice measures Alice's autonomous-coding capability with SWE-bench and runs an internal UI benchmark at bench.blal.pro (see org memory: reference-bench-platform-blal-pro-2026-06-09). Both are squarely exposed to everything above:

• SWE-bench contamination is a live, known issue. Many SWE-bench task instances come from public GitHub PRs that predate model cutoffs — i.e. the fix may be in pretraining. This is precisely why SWE-bench Verified and time-sliced variants exist. When Atlas/Norbert report an Alice SWE-bench delta, the contamination caveat is not optional boilerplate — a jump could be a real capability gain or a corpus-overlap artifact. Pair every score with the model's knowledge cutoff and prefer post-cutoff task slices, mirroring the temporal-cutoff mitigation above.
• bench.blal.pro is, structurally, a private benchmark — internal UI tasks, not on the public web. That's the good property (the private-holdout defense). But it inherits the private-holdout cost: only the holder (us) can score it, so results aren't externally reproducible, and our raw-model-vs-Alice-harness gap (noted in the bench memory) must be reported transparently or we reinvent the Leaderboard Illusion in miniature.
• Atlas itself should treat leaderboard numbers as contaminated-until-audited. When the index surfaces a "model X scores N% on Y" claim, the truthful annotation is the cutoff date + whether the benchmark is static/dynamic/private — exactly the metadata this article argues is load-bearing.

See also [[llm-evaluation]] (scoring mechanics, Arena Elo, LLM-as-judge), [[agent-evaluation]] (trajectory eval, live-site agent benchmarks where contamination meets environment-staleness), and [[scaling-laws]] (why loss extrapolates cleanly while benchmark gains are the contamination-prone ones).