Self-play & verifier-driven training — models that grade and teach themselves

For NAO + anyone trying to understand where the *next* round of reasoning gains is coming from. Pretraining ate the public internet. RLHF needs armies of human labelers. Both hit a wall: you run out of humans. Self-play and verifier-driven training are the escape hatch — a model that generates its own problems, grades its own answers, and trains on the result, looping until it is better than it started. No new human data required. This article is the honest tour: the beautiful idea, the real mechanics that make it work, and the places where the flywheel quietly grinds itself to dust.

If you only remember one sentence: self-play is a way to manufacture a training signal out of thin air — and it only works as well as the *verifier* that decides who won. Everything else is engineering around that one load-bearing fact.

What it is (intuition)

Imagine teaching yourself chess with no teacher and no opponent. You can't — unless you do one trick: play both sides. White tries to win, Black tries to win, and the rules of chess (a perfect, free verifier) tell you who actually did. Do this a few billion times and you get AlphaZero, which surpassed all human play having never seen a single human game. The signal came from self-play + a verifier (the game rules), not from data.

The dream of self-play for LLMs is the same move, but for reasoning: get a language model to improve without anyone feeding it new labeled examples. Three ingredients recur in every method:

1. A generator that produces candidate outputs (answers, rationales, whole problems). The model itself, sampled multiple times.
2. A verifier / judge that decides which outputs are good. This is the soul of the method — it can be a cheap, trustworthy checker (a math grader, a code unit-test, a compiler) or a fuzzy, fallible one (the model judging itself).
3. A loop that keeps the good outputs, trains on them, and repeats — so the improved model generates better outputs next round, which trains an even better model. That feedback cycle is the data flywheel.

The single biggest axis of variation across the whole field is how good the verifier is:

Read that table top to bottom and you've read the field's central tension: the more checkable your domain, the more powerful self-play is. Math and code are self-play's home turf precisely because correctness is free to verify. Open-ended chat is the hard case, because the only available judge is another fallible model.

Why it matters

1. The human-data wall is real. Frozen reward models trained on human preferences are bottlenecked at human quality — they can never teach the model to exceed the labelers (Self-Rewarding LMs↗). And high-quality human reasoning traces (step-by-step math, expert code review) are scarce and expensive. Self-play sidesteps both: the model's own attempts become the curriculum, and a verifier — not a human — provides the grade.

2. It is how the 2025–2026 reasoning models were actually built. DeepSeek-R1-Zero took a base model and applied RL with nothing but rule-based verifiers (is the answer right? is the format right?) — no supervised reasoning traces at all — and watched chain-of-thought, self-verification, and "wait, let me reconsider" behaviors emerge on their own (DeepSeek-R1↗). This is verifier-driven self-improvement at production scale. See [[test-time-compute-reasoning]] for what that buys you at inference.

3. It is the most plausible path past the data ceiling. If a model can pose itself problems at exactly the right difficulty and check its own solutions, the training set is unbounded and self-curricularizing — it gets harder exactly as fast as the model gets better. That's the promise of the 2025 "zero-data" line (Absolute Zero, R-Zero). Whether it holds is the open question of the decade (see caveats).

How it works (real mechanics)

We'll build up from the cleanest idea to the most ambitious, because each method removes one more piece of human input than the last.

Level 0 — STaR: bootstrap rationales, verify by the answer

STaR (Self-Taught Reasoner)↗ is the ancestor. You have questions with known answers but no reasoning traces. The verifier is the cheapest one imaginable: does the rationale lead to the correct answer?

STaR loop:
  given dataset D of (question, answer) pairs
  M ← base LLM
  repeat:
    # 1. Generate
    for each (q, a) in D:
        rationale, ŷ = M(q, few_shot_CoT)        # sample chain-of-thought
    # 2. Verify (the key step) — filter by answer-correctness
    keep = { (q, rationale) : ŷ == a }
    # 3. Rationalization — rescue the failures
    for each (q, a) where ŷ != a:
        rationale = M(q, a_as_hint)               # "given the answer IS a, explain why"
        if rationale's conclusion == a: add (q, rationale) to keep
    # 4. Fine-tune M on `keep` (rationale → answer)
  until no improvement

Two ideas to take away:

• Answer-correctness is the verifier. Nobody checks whether the reasoning is valid — only that it arrives at the right answer. (This is exactly the "outcome supervision" trade-off in [[process-outcome-rewards]]: cheap and scalable, but it'll happily reward a lucky-but-wrong chain.)
• Rationalization turns failures into training data by leaking the answer as a hint. Without it the model can only ever learn from problems it already solves, and the curriculum stalls.

Level 1 — SPIN: self-play against your own past self, no reward model

SPIN (Self-Play Fine-Tuning)↗ needs no reward model, no preferences, no new human data — only an existing SFT dataset. The self-play game: the current model generates responses; the model is then trained to tell its own past-iteration outputs apart from the human SFT targets, preferring the human ones. It's a DPO↗-shaped objective where the "rejected" sample is always your own previous self:

SPIN iteration t:
  for each prompt x with human response y_human (from the SFT set):
      y_self = M_{t-1}(x)                  # last iteration generates the "loser"
      train M_t to prefer y_human ≻ y_self  # DPO-style objective

The theory is clean: the global optimum is reached exactly when the model's policy equals the human target distribution — at which point its own samples are indistinguishable from the data and the game has no more gradient. So SPIN squeezes the SFT set far harder than one epoch of SFT does, but it cannot exceed the SFT data — its ceiling is the human distribution it's imitating. (Contrast with verifier-driven methods, which can exceed it because the signal is correctness, not imitation.)

Level 2 — Self-Rewarding LMs: the model is the reward model

Self-Rewarding Language Models↗ (Meta, Llama-2-70B) removes the frozen reward model entirely. The same model wears two hats each iteration: actor (generate responses) and judge (score them via LLM-as-a-judge prompting), then trains on the self-built preferences with iterative DPO.

The judge uses an additive 5-point rubric (points accumulate):

+1  relevant, provides some information
+1  addresses a substantial part of the question
+1  answers the basic elements usefully
+1  clearly written from an AI assistant's perspective, comprehensive
+1  impeccably tailored, expert, engaging, insightful
→ score ∈ {0..5}

Self-Rewarding loop (iteration t):
  1. generate new prompts (few-shot from the seed set)
  2. for each prompt, sample N=4 candidate responses from M_{t-1}
  3. M_{t-1} JUDGES each candidate with the 5-point rubric
  4. preference pair = (highest-scoring, lowest-scoring); drop ties
  5. DPO-train M_t on these pairs

The striking result: across M1 → M2 → M3, both instruction-following and the model's own reward-modeling ability improved — the judge got better as the actor got better, which is the flywheel turning. M2 beat M1 (55.5% win rate), M3 beat M2 (47.7%) on head-to-head AlpacaEval. But the authors explicitly warn it "likely saturates" after a few iterations and flag reward-hacking as unstudied. Three iterations is all they ran — treat the trend line, not the asymptote, as the finding.

Level 3 — DeepSeek-R1-Zero: verifier-driven RL straight from the base model

This is where self-play meets [[rlvr]] at scale. DeepSeek-R1↗ took a base model and ran [[grpo]] with purely rule-based verifiers:

• Accuracy reward — for math, is the boxed answer numerically equal to ground truth? For code, does it pass the unit tests / compile? A program, not a neural net, decides. This is the un-hackable part.
• Format reward — did the model wrap its thinking in <think>...</think>?

The GRPO objective (no value critic — the group of samples is the baseline):

for a prompt q, sample a group {o_1..o_G} from π_old
r_i = verifier(o_i)                       # rule-based, e.g. 1 if correct else 0
A_i = (r_i - mean(r)) / std(r)            # group-relative advantage (the whole trick)
maximize  E[ min( ρ_i · A_i , clip(ρ_i, 1±ε) · A_i ) ] − β·KL(π‖π_ref)
  where ρ_i = π(o_i)/π_old(o_i)

Because the advantage is just "was this sample better than its siblings on this exact prompt?", GRPO needs no reward model and no critic — only a verifier that returns a number. (Full derivation in [[grpo]]; why verifiable rewards beat learned ones in [[rlvr]].)

What emerged from this with zero reasoning demonstrations: long chains of thought, self-verification, backtracking, and a documented "aha moment" where the model spontaneously writes "wait, let me re-evaluate" and allocates more thinking. Nobody taught it that. The reward landscape alone produced it. The honest downsides DeepSeek reports for R1-Zero are real and instructive: poor readability and language-mixing (it was rewarded for correctness, not for being nice to read, so it wasn't), which is why the shipped R1 added a cold-start SFT stage + a multi-stage pipeline on top. Pure self-play got the reasoning; humans were still needed to make it presentable.

Level 4 — Zero-data self-play: the model invents its own curriculum

The 2025 frontier removes the last human input — the questions themselves.

[Absolute Zero / AZR](https://arxiv.org/abs/2505.03335) runs one model in two roles, proposer and solver, with a code executor as both environment and verifier. The proposer invents a coding/reasoning task; the executor runs it to establish ground truth; the solver tries; the executor checks. Crucially the proposer is rewarded by a learnability signal — propose tasks that are neither trivially solved nor impossible (max info gain), across three reasoning modes (deduction / abduction / induction). It reaches SOTA on code+math while training on tasks it generated for itself, beating models trained on tens of thousands of human-curated examples.

[R-Zero](https://arxiv.org/abs/2508.05004) makes the curriculum mechanics explicit and is the cleanest worked example of the flywheel. A Challenger and a Solver co-evolve:

Challenger reward for a question x:
  p̂ = Solver's empirical accuracy over m=10 samples      # how often Solver gets x right
  r_uncertainty(x) = 1 − 2·|p̂ − ½|                        # maxed at p̂ = 0.5 (edge of ability!)
  r_rep(x)        = λ·|cluster|/B                          # BLEU-similarity penalty → diversity
  reward(x)       = max(0, r_uncertainty(x) − r_rep(x))

Solver training:
  pseudo_label(x) = majority_vote(Solver samples)         # no human label exists
  keep only x where |p̂ − ½| ≤ δ (δ=0.25)                  # discard too-easy AND too-hard
  RL-train Solver on the kept (x, pseudo_label) set
→ loop: a better Solver forces a harder Challenger forces a better Solver …

The 1 − 2|p̂ − ½| reward is the whole philosophy in one line: reward the Challenger for finding problems the Solver is maximally *uncertain* about — the zone of proximal development, automated. R-Zero boosts Qwen3-4B-Base by +6.49 on math and +7.54 on general reasoning with no external data whatsoever.

Key ideas & tradeoffs

• The verifier is the product. Everything downstream inherits the verifier's trust level. A perfect verifier (compiler, math grader, game rules) gives un-hackable signal but confines you to checkable domains. A model-as-judge opens every domain but invites drift and gaming. Pick your method by picking your verifier first.
• Outcome filter vs. process check. STaR/R1 verify only the final answer, so they cheerfully reward correct-answer-via-broken-reasoning. Catching bad reasoning needs process supervision ([[process-outcome-rewards]]) — more faithful, far more expensive, harder to automate.
• Curriculum is half the battle. R-Zero's p̂≈0.5 targeting and AZR's learnability reward exist because self-play with random-difficulty problems wastes compute — too-easy gives no gradient, too-hard gives no signal. The "keep only the ~50% band" trick is the automated version of a good teacher.
• Imitation ceiling vs. correctness ceiling. SPIN/Self-Rewarding are bounded by the data/judge they imitate; verifier-driven RL (R1, R-Zero) is bounded only by what the verifier can check — which is higher but narrower.
• Diversity is a first-class objective. R-Zero's BLEU penalty, GRPO's group sampling — without an explicit push for variety, the generator collapses to one safe mode and the flywheel stops turning. Self-play wants to lose entropy; you must fight it.

Honest caveats & open questions

This is a frontier area. Be skeptical of any clean "it just keeps improving" story.

• Pseudo-label rot is measured, not hypothetical. R-Zero itself reports its self-generated labels start at 79.0% accuracy and decay to 63.0% by the third iteration — "as the system generates more difficult problems, the Solver's majority vote becomes a less reliable source for ground truth." The flywheel manufactures its own training noise, and that noise compounds. When the verifier is the model, self-play is self-poisoning at the margin.
• Saturation is the norm, not the exception. Self-Rewarding ran three iterations and flagged likely saturation. Most reported gains are 2–4 rounds, not asymptotic. "Recursive self-improvement to superintelligence" is not what any of these papers demonstrate.
• Does RLVR/self-play create new reasoning or just sharpen what's there? A live 2025 debate: pass@k studies (see the open questions in [[rlvr]]) argue RLVR mostly reweights capabilities the base model already had rather than teaching genuinely new ones — it raises pass@1 but can shrink the support of what's reachable at high k. If true, self-play is a powerful sharpener with a base-model ceiling, not an unbounded engine.
• Reward hacking, generalized. With a model judge, the actor learns to exploit the judge's blind spots (length, formatting, sycophancy) instead of getting better. Even rule-based verifiers get gamed (e.g., printing the expected output, exploiting a weak test suite). DeepSeek explicitly lists reward-hacking as a failure mode of R1-Zero.
• Safety of self-generated tasks. When a model invents its own curriculum (AZR, R-Zero), what does it choose to get good at? AZR's authors flag an "uh-oh"-style concern: a fully autonomous proposer is an un-audited curriculum. Verifiable-but-unaligned is a real configuration.
• Mode collapse / diversity death. Without explicit diversity rewards the generator narrows, self-play degenerates, and you're training on near-duplicates.
• Domain confinement. The cleanest results are all in math and code — precisely because free verifiers exist there. Self-play's empirical track record in open-ended, hard-to-verify domains (good writing, judgment, taste) is much thinner, and rests on the weakest verifier (a model judging itself).

The honest one-liner: self-play converts compute into capability *only as far as its verifier is trustworthy* — and the moment the verifier becomes the model itself, you are racing your own accumulating error.

How it connects to OpenAlice

These are not abstractions for OpenAlice — they're the blueprint for how Alice gets better at the thing NAO actually cares about: autonomous coding (repo → PR) as a base AGI capability, measured against SWE-bench.

• Code is the perfect-verifier domain. Alice's coding work has a free, honest verifier already wired in: the test suite, the compiler, cargo build, next build. That is exactly the un-hackable signal STaR/R1/AZR depend on. An Alice self-play loop — propose a task on a repo → attempt a patch → run the tests → keep the green ones → train — is STaR/R-Zero with the test runner as the verifier. The ecosystem's heavy emphasis on "run the full suite before reporting done" is the same discipline that makes verifier-driven training sound.
• Atlas + the inspector are curriculum infrastructure. R-Zero's hardest problem is proposing tasks at the edge of ability. Atlas (impact graph, repo health/grades, callers/callees) and openalice-inspector are precisely the tools that could mine real repos for tasks of calibrated difficulty — a grounded alternative to hallucinating synthetic problems, sidestepping pseudo-label rot by using real tests as ground truth.
• The bench is the iteration gate. bench.blal.pro and the [[agent-evaluation]] stack are how you'd measure whether a self-play round actually helped or just reward-hacked the harness — the indispensable outer-loop check the literature says you cannot skip.
• The judge-drift warning maps onto Alice's existing guardrails. OpenAlice's fail-open critic, HITL approvals, and "personality drift is a feature but regressions are not" stance are the human-in-the-loop counterweight to exactly the self-poisoning and reward-hacking failure modes above. The lesson from R1 is load-bearing: pure self-play gets you raw reasoning, but you still need a human-anchored stage (cold-start, review, the bench) to make the result trustworthy and shippable.

Related reading in this library: [[rlvr]] (the verifiable-reward foundation) · [[grpo]] (the optimizer that does the self-play RL) · [[test-time-compute-reasoning]] (what the trained reasoning buys at inference) · [[process-outcome-rewards]] (verify the answer vs. verify the work) · [[rlhf-and-alignment]] (DPO, the engine inside SPIN & Self-Rewarding) · [[agent-evaluation]] (the outer-loop check that keeps the flywheel honest).