Eval-harness design

What it is (intuition first)

A benchmark number — "82.4% on MMLU" — is the visible tip. The eval harness is the machine under it: the code that loads a dataset, turns each row into a prompt, runs a model, extracts an answer from the model's free-form text, scores it against a gold label, and aggregates the scores into that one number. Change any link in that chain — the prompt template, the few-shot order, the regex that plucks the answer, the random seed — and the number moves, with the model held fixed. So a harness is not a thin wrapper; it is the experimental apparatus, and most of the reproducibility crisis in LLM evaluation is a harness-design crisis.

The cleanest way to see it: a harness is a pipeline of stages, each a place where a bug or a choice silently changes the result. A standard pipeline (the shape EleutherAI's lm-evaluation-harness and most others share):

config load → task instantiation (dataset + split) → prompt rendering → batched model calls → filters (extract the answer) → metric (score it) → aggregation (roll up subtasks)

This article is about designing that pipeline well: what makes a good task, how to choose a grader (exact-match, programmatic, or LLM-as-judge), how to tame variance so the number is stable, how to keep contamination from turning measurement into memorization, and how to wire the whole thing into CI so a model or prompt change can't quietly regress.

This is the engineering companion to the methodology articles: [[llm-evaluation]] and [[agent-evaluation]] cover what to measure; [[benchmark-contamination]] and [[verifiers-and-reward-hacking]] cover how measurement gets gamed. Here the subject is the harness as a piece of software you have to build and trust.

Why it matters

• The harness is the result. Two harnesses scoring the same model on the same dataset routinely disagree by several points purely from prompt formatting, answer-extraction, and few-shot choices. If you can't reproduce a number, you can't compare models — and comparison is the entire point. The harness, not the benchmark, is what you're really publishing.
• Graders are where correctness leaks. A too-strict exact-match marks a right answer wrong; a too-loose regex marks a wrong answer right; an un-calibrated LLM-judge invents its own standard. The grader is the smallest, most error-prone, least-tested part of most eval setups — and it sits directly on the score.
• Variance hides and fakes progress. Run the same eval twice with a different seed and the number moves. Without controlling and reporting that noise, you can't tell a real 1-point gain from sampling jitter — and teams routinely ship "improvements" inside the error bars.
• Contamination turns a test into a lookup. If benchmark rows leaked into training, you're measuring memorization, not capability ([[benchmark-contamination]]). A harness that doesn't decontaminate is a harness that can be fooled.
• CI gating is what makes evals *bite*. An eval you run once is a blog post; an eval that fails a build is a guardrail. The design that makes regression-gating possible — deterministic, fast, high-baseline regression suites — is different from the design that makes capability measurement possible.

How it works (real mechanics)

1. Task design — the unit that everything else scores

A task is one (input, gold-answer, grader) contract plus how to render it. Anthropic's agent-evals guidance gives the sharpest acceptance test for a good task: "a good task is one where two domain experts would independently reach the same pass/fail verdict." If two experts disagree, the spec is broken, not the model. Two more rules from the same source:

• Start small, from real failures. "20–50 simple tasks drawn from real failures is a great start" — sourced from manual testing, bug trackers, support queues — beats waiting for hundreds of synthetic ones.
• Test both directions. "Test both the cases where a behavior should occur and where it shouldn't" and avoid class-imbalanced sets, or you optimize one-sidedly (a model that always says "yes" aces a yes-only suite).

In code, the task carries its rendering and scoring. In EleutherAI's harness a task is a YAML config with fields like doc_to_text (row → prompt), doc_to_target (row → gold answer), and metrics — so the same dataset can be re-templated, and the template is versioned alongside the data rather than hidden in prose.

2. Graders — exact, programmatic, LLM-judge

The grader maps (model_output, gold) to a score. Three families, with a real tradeoff (Anthropic's framing):

Anthropic's summary: code-based graders are "Fast, Cheap, Objective, Reproducible, Easy to debug" but "Brittle to valid variations"; model-based graders are "Flexible, Scalable" and "Captures nuance" but require "calibration with human graders for accuracy." The design rule that follows: prefer the cheapest grader that is faithful. Use exact-match where the answer space is closed (multiple choice, a number), programmatic where correctness is a property you can compute (does the code pass tests? did it call the right tool?), and reserve LLM-judge for genuinely open outputs (a summary, a tone, an explanation).

Two non-obvious mechanics sit between the model and the grader:

• Filters / answer extraction. The model emits free text; the harness must extract the answer before scoring (the "filters" stage — regex, extract, ensembling). A weak extractor is a silent grader bug: the model was right, the regex grabbed the wrong span. This stage deserves its own tests.
• Normalization. Lowercasing, whitespace, punctuation, "The answer is X" stripping — done before exact-match. Under- or over-normalizing both move the score; document the normalization as part of the task.

3. LLM-as-judge — design it like a measuring instrument

When you must use a judge, the failure modes are documented and consistent. The reliability literature (e.g. arXiv 2412.12509, Can You Trust LLM Judgments?) catalogs prompt sensitivity, verbosity bias (longer answers scored higher), self-preference bias (a model favoring its own family's outputs), miscalibration, and seed-dependent variance — the judge's verdict changes when only the random seed changes. Design countermeasures Anthropic and the rubric literature converge on:

• One rubric per dimension, not one rubric for everything — "vague rubrics produce inconsistent judgments." Grade correctness, completeness, and tone with separate isolated judges rather than one omnibus prompt.
• Give the judge an out — "providing an instruction to return 'Unknown'" — so it abstains instead of hallucinating a verdict.
• Calibrate against humans, repeatedly. A judge is only trustworthy to the extent its verdicts track expert verdicts on a held-out calibration set; treat judge-vs-human agreement as a metric you monitor, not a one-time check.
• Read the transcripts. Anthropic is blunt: "You won't know if your graders are working well unless you read the transcripts and grades from many trials." The judge is code with a flaky dependency; you verify it by inspection.

4. Variance & seeds — make the number stable and honest

LLM eval scores are random variables, from two sources: sampling noise (a non-zero temperature, or a non-deterministic judge) and few-shot order / example selection. The harness must control and report both.

• Seed every randomness source, separately. EleutherAI exposes --seed as a list — python,numpy,torch,fewshot, default 0,1234,1234,1234 — precisely because few-shot sampling and model sampling are different randomnesses you may want to pin or vary independently (--seed 0,None,8,52, with None skipping a component). A single global seed is the common, leaky shortcut.
• Report distributions, not point estimates. For stochastic agents Anthropic recommends pass@k (≥1 success in k tries — when one success matters) and pass^k (all k succeed — when consistency is essential); which you pick is a product decision, not a default. Either way you run k trials and characterize the spread, rather than trusting one run.
• Beware brittle formatting. Recent work ("A Single Character can Make or Break Your LLM Evals," arXiv 2510.05152) shows scores shifting on prompt-formatting trivialities — a reason to fix the template, version it, and treat formatting as a controlled variable, not a free choice. (Flagged: single-paper finding; the direction is well-corroborated, the magnitude is dataset-specific.)

5. Contamination control

If test rows are in training data, the score measures recall, not reasoning. The harness-level defense is decontamination by n-gram overlap: flag a test document as contaminated when it shares an n-gram with the training corpus. The OpenAI-derived convention used in the harness literature is N between 8 and 13, chosen per-dataset. This is necessarily approximate — paraphrased or reformatted contamination slips past n-gram matching, and you usually don't have the training corpus to check against. So decontamination is a mitigation, not a guarantee; the deeper treatment, including held-out/canary and dynamic-benchmark strategies, is [[benchmark-contamination]]. Design implication: prefer freshly authored or private eval sets for anything load-bearing, and treat public-benchmark scores as contamination-suspect by default.

6. Task versioning & reproducibility plumbing

A harness that lives changes — bugs get fixed, templates get rewritten — and that silently breaks comparability across time. EleutherAI's answer is a task VERSION field: each task carries a version, incremented on any breaking change, and reported in the results, so a number computed under the old (buggy) implementation is never silently compared against the new one. Unit tests pin task behavior so a refactor can't drift the semantics unnoticed. Round it out with: request caching (--cache_requests, so re-runs are cheap and identical), commit-hash / env capture in the output, and explicit metadata (model, params, seeds) logged beside every score. Reproducibility is not a property of intentions; it's these mechanics.

7. CI regression gating

The final design split is the most practical. Anthropic distinguishes two eval types that want opposite designs:

• Capability evals — start at a low pass rate, exist to measure improvement headroom. Noisy, hard, slow-moving. Not a build gate.
• Regression evals — sit at "nearly 100% pass rate" and exist to catch backsliding. As a capability eval saturates, you "convert" it into a regression suite.

Only the regression suite belongs in CI, and it must be engineered for it: deterministic graders (exact/programmatic, not a flaky judge), fixed seeds, fast and cheap (so it runs per-PR), and a clear pass/fail threshold that fails the build on regression. The capability suite runs on a slower cadence (nightly, per-release) where its cost and noise are acceptable. This is the same trace→dataset→experiment→promote loop described in [[agent-observability-and-tracing]], with the regression suite as the "promote only green changes" gate.

Key ideas & tradeoffs

• Cheapest faithful grader wins. Exact/programmatic where you can, LLM-judge only where the output is genuinely open — and even then, isolated per-dimension rubrics calibrated to humans, not an omnibus prompt.
• Determinism is a design axis, not an afterthought. The harness that gates CI must be reproducible; the one that explores capability can afford to be noisy. Conflating the two gives you a flaky build gate and a blind capability signal.
• Variance reporting separates real gains from jitter. A point estimate with no spread is not a measurement. Seeds + pass@k/pass^k turn "it went up" into "it went up beyond noise."
• Versioning is comparability over time. Without a task VERSION + frozen templates, your own history becomes incomparable the first time you fix a bug.
• Contamination control is a mitigation, not a proof. N-gram decontam catches verbatim leakage and misses everything subtler; private/fresh sets are the real defense for load-bearing numbers.

Honest caveats & open questions

• No single "correct" harness. Much of the inter-harness disagreement is defensible choices (log-likelihood vs. generation scoring, answer extraction, few-shot format). There is no universal right answer — only documented, versioned, reproducible ones. Be suspicious of any leaderboard that doesn't publish its harness.
• LLM-judge biases are real and only partly fixable. Verbosity, self-preference, position, and seed variance are documented (arXiv 2412.12509) and mitigations (rubrics, abstention, calibration) reduce but don't remove them. A judge is a noisy instrument you must keep re-calibrating, not a ground truth.
• The "experts agree" bar is aspirational for hard domains. Anthropic's two-experts-agree test is excellent for crisp tasks and genuinely hard for fuzzy ones (style, helpfulness, partial credit) — exactly the tasks that need judges.
• Decontamination assumes access you usually lack. N-gram overlap requires the training corpus; for closed models you can't run it at all, so contamination is suspected, not measured.
• Vendor framings are vendor framings. The capability-vs-regression split, pass@k/pass^k, and "read the transcripts" come from Anthropic's engineering blog — strong, widely-echoed practice, but practitioner guidance, not a controlled study. Flagged as such; the concepts generalize, the exact thresholds are yours to set.
• No fabricated numbers. Where this article cites figures — N=8–13 for decontam, the 0,1234,1234,1234 seed default, "20–50 tasks" — they are quoted from the cited sources. "Harnesses disagree by several points" is a well-attested qualitative claim; the exact gap is benchmark-specific and not asserted here.

How it connects to OpenAlice

OpenAlice already runs harness-shaped infrastructure, and the house disciplines map onto these design rules:

• The fixture-runner + bench platform *are* an eval harness. OpenAlice runs an internal UI-bench platform and ~24 module-specific fixture-runners — config → task → run → grade → aggregate is exactly the pipeline above. The bar to add: per-task versioning and seed control so a green run today is comparable to a green run next month.
• "Zero regressions tolerated" is CI regression-gating. The house rule that no change ships with a regression is the regression-eval-at-~100%-pass suite in policy form; making it bite means deterministic graders + fixed seeds in the gate, with the noisier capability/SWE-bench-style evals on a slower cadence ([[agentic-rl-long-horizon-coding]]).
• Lab/branch discipline is contamination + variance hygiene. Personality- and memory-sensitive changes go to a branch + lab before mvp precisely because a single prod cutover avoids the variance and self-preference traps a naive A/B over live chats would introduce ([[verifiers-and-reward-hacking]]).
• "Always verify from `usage`, read the transcripts" is grader-verification. Alice's house rule to confirm behavior from real output rather than assume is the same instinct as Anthropic's "you won't know your graders work unless you read the transcripts" — the eval is only as trustworthy as the inspection behind it ([[agent-observability-and-tracing]]).
• Personality evals need judges, carefully. Alice's soul/behavioral-envelope checks are open-ended, so they fall to LLM-judge territory — exactly where isolated per-dimension rubrics, abstention, and human calibration matter most, and where a single cosine-distance number is the wrong instrument ([[agent-evaluation]]).

The throughline: a benchmark number is only as trustworthy as the harness that produced it — and a harness is trustworthy only when its tasks are expert-agreeable, its graders are the cheapest faithful kind, its variance is reported, its data is decontaminated, and its regression suite fails the build. Everything else is a number without a measurement behind it.

See also

[[llm-evaluation]] · [[agent-evaluation]] · [[benchmark-contamination]] · [[verifiers-and-reward-hacking]] · [[agent-observability-and-tracing]] · [[agentic-rl-long-horizon-coding]]