Generative verifiers & verifier scaling (GenRM)

For NAO + anyone who has built a best-of-N loop and wondered "but how good does the *judge* have to be?" The companion article [[process-outcome-rewards]] covers what to reward (every step vs only the answer). This one is about the verifier as a first-class, scalable object: not "score = a number from a classifier head," but "verification is itself a generation problem you can reason through, vote on, and pour test-time compute into." That reframing — GenRM (Zhang et al. 2024) — plus the hard economics of when scaling the judge actually helps, is the topic. It is the difference between treating your reward model as a frozen oracle and treating it as a second LLM that thinks.

If you remember one sentence: a verifier is just another model running at inference time, so everything you know about test-time compute — chain-of-thought, majority voting, scaling curves, and the ways they break — applies to the *judge* too, not only the *generator*.

What it is (intuition first)

Go back to the exam-grading metaphor from [[process-outcome-rewards]]. A discriminative verifier is a grader who has been trained to glance at a solution and stamp a number on it — 0.81. No working shown, no reasoning, just a reflex score from a classification head bolted onto an LLM. That is how almost all reward models worked through 2023: take a pretrained LLM, throw away its ability to write, and keep only a scalar output.

A generative verifier (GenRM) is a grader who writes out the grading. You prompt it: "Here is a problem and a candidate solution. Reason step by step about whether it is correct, then answer Yes or No." It produces a chain-of-thought critique — "step 3 divides by x−2 but x=2 is in the domain, so this is invalid" — and the score is literally the probability the model assigns to the token `Yes` at the end. Verification has become next-token prediction.

That one move unlocks everything a normal LLM can do, now aimed at judging:

• It can reason before deciding (GenRM-CoT) — and catch subtle errors a reflex score misses.
• It can vote with itself — sample the verification rationale K times and take the majority Yes/No, spending test-time compute on verification exactly the way [[test-time-compute-reasoning]] spends it on generation.
• It scales — a bigger GenRM and a GenRM given more thinking tokens both get more accurate, on a curve, just like a generator.

The discriminative verifier threw away the LLM's best trick (generation) to make a judge. GenRM keeps it. That is the whole idea: verification is not a classification problem grafted onto a language model — it is a language problem.

The one-sentence version. Discriminative verifier: "score = head(solution)." Generative verifier: "score = P(Yes | think-then-answer about the solution)" — a judge that reasons, votes, and scales.

Why it matters

1. The generator–verifier gap is the free lunch of reasoning

There is a deep asymmetry the whole field leans on: checking an answer is easier than producing one. Verifying reduces (often) to a near-binary decision; generating requires sampling a correct sequence out of an enormous joint distribution, where every token can go wrong. So for many problems a model can generate a correct solution somewhere in N samples (high Pass@N) even though it can't reliably do it in one shot — and if a verifier can pick that correct one out, you convert "the answer is in there somewhere" into "the answer is the one we return."

The size of that opportunity has a name: the generation–verification gap — roughly Pass@N − (accuracy of your selection strategy). It is the headroom a better verifier can still capture. Weaver (Saad-Falcon et al. 2025) measures and attacks exactly this gap. A good verifier is the lever that turns spare inference compute into accuracy — which is why verifiers, not just generators, are now a scaling axis ([[scaling-laws]], [[test-time-compute-reasoning]]).

2. It changes where the bottleneck is

In a best-of-N or tree-search system, the generator sets the ceiling (Pass@N) and the verifier decides how much of that ceiling you actually reach. Past a point, buying a better verifier beats buying more samples. Knowing whether you are generator-bound or verifier-bound is the single most useful diagnostic for a test-time-compute system — and most teams never measure it.

3. It is where "LLM-as-a-judge" becomes rigorous

Everyone uses an LLM to grade outputs ([[agent-evaluation]], [[llm-evaluation]]). GenRM is the trained, calibrated, vote-able version of that instinct, and the verifier-dynamics literature (below) is the honest accounting of when that judge helps versus when it confidently steers you wrong.

For OpenAlice this is the theory behind safely turning Alice's spare inference into correctness — picking the best of several candidate patches or answers — without trusting a single fragile score. See the final section.

How it works (real mechanics)

GenRM: reward modeling as next-token prediction

Take a problem x and candidate solution y. A discriminative reward model computes r = σ(head(x, y)). GenRM instead builds a verification prompt and reads off a token probability:

prompt   = f"{x}\n{y}\nIs the answer correct (Yes/No)?"
r_GenRM  = P_LLM("Yes" | prompt)        # direct GenRM

The key training choice: GenRM is trained with plain next-token prediction, jointly on verification *and* solution generation — it never stops being a language model. Two flavors:

• Direct GenRM — just the P(Yes) readout above.
• GenRM-CoT — the verifier first generates a verification rationale (a critique), then emits Yes/No. Crucially, you can train this on synthetic rationales (model-written critiques of known-good/bad solutions), and Zhang et al. report that synthetic rationales alone are enough to catch subtle math errors.

Then the part that makes it a test-time-compute object — majority voting over rationales:

def genrm_cot_score(x, y, verifier, K):
    yes = 0
    for _ in range(K):
        rationale, verdict = verifier.sample(x, y)   # think, then Yes/No
        yes += (verdict == "Yes")
    return yes / K        # vote-averaged correctness probability

More votes → a better verifier, with no retraining — the same self-consistency trick ([[test-time-compute-reasoning]]) applied to judging.

What GenRM actually buys (reported best-of-N gains)

From the GenRM paper (Zhang et al. 2024, ICLR 2025). These are the authors' numbers; treat as paper-reported, not independently replicated here:

GenRM beat discriminative verifiers, DPO verifiers, and vanilla LLM-as-a-judge in their comparisons, and — the headline for this article — scaled favorably with both model size and test-time (voting) compute. The verifier has its own scaling curve.

The generator–verifier gap, formally-ish

Let the generator's best-of-N ceiling be Pass@N (probability a correct solution appears in N samples) and let the selection strategy actually return correct with probability Sel@N. Then:

gap(N) = Pass@N − Sel@N        # headroom a perfect verifier would still close

Pass@N rises toward 1 as you sample more; a perfect verifier makes Sel@N = Pass@N. Real verifiers leave a gap. Weaver (Stanford / UW-Madison / Together AI, 2025) closes it without expensive labels by combining many weak verifiers via weak supervision: binarize each verifier's output, drop uninformative ones, then estimate each verifier's accuracy from pairwise agreement statistics alone (no ground truth) and take an accuracy-weighted ensemble — needing only ~1% of the data as a dev set. Reported: with Llama-3.3-70B as generator and 33 reward-models/judges as verifiers, Weaver hit 87.7% average across MATH500 / GPQA-Diamond / MMLU-College / MMLU-Pro — within ~1% of o3-mini's selection on those sets, and +13.5–17.1% over majority voting, closing the gap ~14.5% on average for 70B models. (Vendor-adjacent benchmark framing; numbers are the authors'.)

Distilling the judge back down

Running 33 verifiers per query is absurd in production. Weaver's answer: distill the ensemble into a small cross-encoder using the ensemble's pseudolabels as targets. They report a 400M-param distilled verifier retaining ~98.7% of the accuracy gains at ~1/3000th the verification compute — a single-GPU judge that still beats majority voting handily. This is [[distillation]] applied to verification: scale the judge up to find the signal, then compress it for serving.

When scaling the verifier hurts — verification dynamics

The most important honest result for practitioners. "Variation in Verification" (Zhou et al. 2025) studies when best-of-N actually beats the generator:

• There is an accuracy threshold. A verifier must exceed a minimum accuracy for best-of-N to beat the base generator at all; below it, verification makes things worse than just taking the model's first answer. A bad judge is worse than no judge.
• Scaling the verifier is non-monotonic. A bigger/stronger verifier does not reliably help — under some conditions it hurts. What matters is calibration, not raw capacity.
• Verifier collapse / false-positive amplification. When you scale N, you draw more samples — including more plausible-looking wrong ones. A miscalibrated verifier confidently selects these false positives, and the problem gets worse with larger N. More samples can lower accuracy if the judge can't tell a convincing wrong answer from a right one.

The mental model: best-of-N is an adversarial search against your own verifier. Drawing more candidates is sampling harder for the verifier's blind spots. This is the same reward-hacking pressure from [[process-outcome-rewards]], now framed as a scaling phenomenon: the failure mode grows with the compute you throw at it.

Where GenRM sits in the verifier taxonomy

The survey "Trust but Verify!" (Venktesh et al. 2025) organizes verifiers on three orthogonal axes — and GenRM is one cell, not the whole space:

The survey's blunt warning sits on the bottom-right cells (model-based): reward hacking, miscalibration, and poor domain transfer are intrinsic to learned verifiers — exactly why the un-hackable corner (tool/rule-based outcome checks: unit tests, a symbolic solver, [[rlvr]]) is preferred whenever the domain allows it.

Key ideas & tradeoffs

• A verifier is a model at inference time — treat it like one. It has a scaling curve (size + votes), a calibration profile, and an adversarial failure mode. Stop thinking of "the reward model" as a fixed oracle.
• Generative > discriminative when subtlety matters. Reasoning-then-judging catches errors a reflex score misses; the cost is K× the verification compute for voting. Discriminative verifiers are cheaper and fine when errors are gross.
• Calibration beats capacity. The biggest judge is not the best judge. A smaller, well-calibrated verifier can outperform a larger overconfident one, because best-of-N rewards ranking correctly, not scoring confidently.
• Measure whether you're generator- or verifier-bound. Plot Pass@N (oracle selection) against Sel@N (your verifier). If Pass@N is flat, buy a better generator; if the gap is wide, buy a better verifier. Most teams guess.
• Ensemble then distill. Weaver's recipe — combine many weak verifiers without labels, then compress to one cheap judge — is the pragmatic shape of "scale the judge" that survives a production budget.
• The un-hackable verifier wins when it exists. A generative model-based judge is a tool of last resort. If the answer is machine-checkable (tests, a known numeric answer), a rule-based outcome check is a perfect, un-gameable verifier — this is the [[rlvr]] lesson and why DeepSeek-R1 chose it.

Honest caveats & open questions

This is the section the demos skip.

• The standout numbers are paper-reported and benchmark-specific. GenRM's 73% → 93.4% (GSM8K) and Weaver's 87.7% average ≈ o3-mini are the authors' results on chosen benchmarks with chosen generators; I have not re-run them. Easy→hard generalization gains and saturated benchmarks like GSM8K both invite caution. Read them as "this direction works," not as portable constants.

• More compute on the judge can lower accuracy. The Zhou et al. result is the uncomfortable one: scaling N or scaling the verifier is not monotone. False positives compound. Any "just sample more and verify" pitch that ignores the verifier's accuracy floor is selling you a footgun.

• GenRM doesn't escape reward hacking — it relocates it. A judge that scores "this looks like correct reasoning" can be fooled by fluent-but-wrong critiques and verbose padding, same as a learned PRM ([[process-outcome-rewards]]). Voting reduces variance, not systematic bias: if the verifier is wrong the same way every sample, the majority is confidently wrong.

• DeepSeek-R1 still abandoned model-based verifiers for the RL loop. Its "Unsuccessful Attempts" section (DeepSeek-AI 2025) rejected model-based process verifiers as RL signals — reward hacking + retraining cost — and used rule-based outcome rewards under GRPO. GenRM's strongest, safest use is as a frozen test-time selector, not as an RL objective you grind against. Verifier-as-RL reward and verifier-as-best-of-N selector are different risk classes; don't conflate them.

• Weak-supervision ensembling assumes verifiers' errors are roughly independent. Weaver's label-free accuracy estimation leans on pairwise disagreement carrying signal. If many verifiers share the same blind spot (all fine-tuned from one base, all length-biased), the ensemble can be confidently, correlatedly wrong, and the no-labels estimate flatters itself.

• "Verification is easier than generation" is a tendency, not a theorem. It holds strongly where checking is near-binary and tool-assisted (run the tests), weakly or not at all for open-ended generation (is this essay good?) where there is no oracle and the verifier is as confused as the generator. The gap is domain-dependent; don't assume it.

• Compute-allocation guidance is still thin. Nobody has a clean "spend X% of test-time FLOPs on verification vs generation" law. The survey flags the tradeoff but does not quantify it; in practice it's empirical per task.

How it connects to OpenAlice

• Best-of-N with a verifier is the highest-ROI use for Alice. Generate several candidate answers/patches, score them, return the best. GenRM says the scorer can be a reasoning, self-voting LLM — and the verifier-dynamics work says first measure that the scorer clears the accuracy floor, or best-of-N will quietly hurt. This is the same selection shape as our [[fusion-and-llm-councils]] and [[mixture-of-agents]] patterns: many candidates, then a judge. GenRM is the rigorous, scalable judge for that family.

• Autonomous coding: prefer the rule-based verifier. For the repo→PR / SWE-bench loop, the verifier already exists and is un-hackable: do the tests pass? That is the [[rlvr]] / DeepSeek lesson — a tool-based outcome check beats a learned generative judge wherever it applies. Reserve a GenRM-style judge for ranking candidate patches that all build green (style, risk, "is this the right fix?"), never as the sole correctness signal. This mirrors our standing rule that cargo/tests are the source of truth and self-reports are not — an outcome check, not a verifier's vibe.

• Ensemble-then-distill maps onto our cost ladder. Weaver's "combine many weak verifiers, then distill to a cheap one" is the verification analogue of the lab's cost-ladder research rig and [[model-routing]]: spend big to find the signal offline, serve a small distilled judge online.

• Atlas wiki placement. This is the verifier-centric companion to [[process-outcome-rewards]] (what to reward), [[test-time-compute-reasoning]] (where verifiers spend inference), [[rlvr]] (the un-hackable verifier), [[self-play-and-verifier-driven-training]] (verifiers as the flywheel's referee), and [[agent-evaluation]] / [[llm-evaluation]] (LLM-as-judge, rigorously). If you read one neighbor next, read [[process-outcome-rewards]] — together they are the full picture of what and who does the grading.

Sources

• Zhang, Hosseini, Bansal, Kazemi, Kumar, Agarwal — Generative Verifiers: Reward Modeling as Next-Token Prediction (GenRM, GenRM-CoT, majority-vote verification, best-of-N gains, verifier scaling; ICLR 2025): https://arxiv.org/abs/2408.15240
• Saad-Falcon, Buchanan, Chen et al. (Stanford / UW-Madison / Together AI) — Weaver: Shrinking the Generation-Verification Gap with Weak Verifiers (label-free weak-supervision ensemble of verifiers, 400M distilled judge): https://arxiv.org/abs/2506.18203
• Zhou, Xu, Zhou, Singh, Gui, Joty — Variation in Verification: Understanding Verification Dynamics in Large Language Models (verifier accuracy threshold, non-monotone scaling, verifier collapse / false-positive amplification, 2025): https://arxiv.org/abs/2509.17995
• Venktesh, Rathee, Anand — Trust but Verify! A Survey on Verification Design for Test-time Scaling (taxonomy: discriminative/generative × outcome/process × rule/model; model-based failure modes, 2025): https://arxiv.org/abs/2508.16665
• Lightman et al. (OpenAI) — Let's Verify Step by Step (the discriminative PRM/ORM baseline GenRM is measured against): https://arxiv.org/abs/2305.20050
• DeepSeek-AI — DeepSeek-R1 (why model-based verifiers were dropped from the RL loop in favor of rule-based outcome rewards): https://arxiv.org/abs/2501.12948