RLVR — Reinforcement Learning from Verifiable Rewards

For NAO + anyone trying to understand where the 2026 reasoning models came from. RLVR is the deceptively simple idea that made o1, DeepSeek-R1, and every "thinking" model after them possible: stop training a neural network to *guess* whether an answer is good, and instead *check* whether it is actually correct. If the math answer is right, reward = 1. If the code passes the tests, reward = 1. Otherwise 0. That binary, ungameable signal — fed back through a policy-gradient RL loop — is enough to grow long chains of reasoning out of a base model with no human-written reasoning examples at all.

This article goes intuition-first, then gives you the real mechanics (GRPO objective, group-relative advantage, the exact reward functions), then is honest about the big open question hanging over the whole paradigm: does RLVR teach the model to reason, or just sharpen reasoning it already had?

Pairs with [[rlhf-and-alignment]] (the parent — RLVR is RLHF with the reward model deleted) and [[test-time-compute-reasoning]] (what you do with the reasoning RLVR produces). Background: [[deepseek-architecture]], [[scaling-laws]].

The one-sentence idea

RLVR is reinforcement learning where the reward comes from a *deterministic verifier* — a program that checks correctness — instead of a learned reward model that predicts human preference.

That is the entire conceptual move. Everything else (which RL algorithm, how you shape the reward, how you stop the model gaming it) is engineering on top of that one swap. The swap matters because it removes the single most fragile, most-gamed component of the classic RLHF stack.

What it is — intuition first

To see why this is a big deal, picture the two ways you can give an LLM a "score" for an answer.

The old way (reward model, from RLHF). You collect humans ranking answers, you train a separate neural network to imitate those rankings, and then you let the policy chase that network's approval. The problem: a reward model is itself a flawed, finite neural net. The policy is an optimizer with billions of parameters whose entire job is to find the highest-reward output — so it eventually finds the reward model's blind spots and exploits them. It writes confident, long, flattering nonsense that the reward model loves and humans hate. This is reward hacking, and it is the central leak of RLHF (covered in [[rlhf-and-alignment]]).

The RLVR way (verifier). For some tasks you don't need to predict human opinion, because there's a ground truth. Did 2x + 3 = 7 get solved as x = 2? Run a string/symbolic match. Does the function pass the unit tests? Run them. Did the output obey "respond in exactly 3 bullet points"? Count the bullets. The reward is computed by code, not by a model, so there is nothing to hack — or rather, the only way to "hack" it is to actually be correct, which is exactly what you wanted.

So RLVR trades breadth (a reward model can score any response, including "write a poem about my cat") for integrity (a verifier can only score checkable tasks, but its score can't be fooled). For the domains where verification is possible — math, code, logic, formal instruction-following — that trade is overwhelmingly worth it, and those just happen to be the domains where reasoning lives.

The punchline that shocked everyone in early 2025: if you take a strong base model and do nothing but this — sample answers, reward the correct ones, push the policy toward them — the model spontaneously learns to write long chains of thought, to double-check itself, to backtrack ("wait, let me reconsider"), and to allocate more tokens to harder problems. No one demonstrated those behaviors. They emerged because they raise the probability of being verifiably correct. That's DeepSeek-R1-Zero.

Why it matters

It is the engine of the 2026 reasoning-model era. OpenAI's o1, DeepSeek-R1, and the wave of "thinking" models are, at their core, base models post-trained with large-scale RL against verifiable rewards. R1-Zero pushed AIME 2024 pass@1 from 15.6% → 71.0% (and to 86.7% with majority voting) using only rule-based rewards and no supervised reasoning data at all (DeepSeek-AI 2025). The reasoning wasn't injected by humans; it was grown by the reward signal.
It removes the most-gamed part of RLHF. Replacing the learned reward model with a verifier is what makes large-scale RL stable enough to run for thousands of steps without the policy collapsing into reward-hacked garbage. DeepSeek explicitly says they avoided neural reward models in R1-Zero training because of reward hacking at scale.
It is cheap to scale. A verifier is a few lines of Python (a math grader, a sandbox running tests, a constraint checker). Compare that to the cost of collecting human preference data and training+serving a reward model the size of the policy. RLVR data is "free" wherever ground truth exists.
It reframes alignment. RLVR shows that for a whole class of capabilities you don't need preferences at all — you need correctness. That's a different, more honest control surface than "what humans clicked thumbs-up on."
It exposes a deep question about what RL does. Because the signal is so clean, RLVR became the cleanest experiment for asking: does RL add capability, or just reweight what's already there? (See the caveats — the answer is genuinely unsettled, and it's one of the most important open problems in the field.)

How it actually works — the real mechanics

The RLVR loop is the standard policy-gradient RL loop, with the reward swapped:

for each training step:
    q       <- sample a prompt with a known checkable answer
    {o_1..o_G} <- sample G outputs from the current policy π_θ(·|q)   # "rollouts"
    for each o_i:
        r_i <- VERIFIER(q, o_i)      # ← code, not a neural net. 1 if correct else 0
    Â_i   <- advantage from {r_i}    # how much better than the group baseline?
    update π_θ to raise log-prob of high-advantage outputs (clipped + KL-regularized)

Two design choices fill in the blanks: (A) what is the reward function, and (B) which RL algorithm turns rewards into a gradient. Modern RLVR almost always pairs a rule-based reward with GRPO.

(A) The reward function — verifiable, usually binary

Tulu 3 (Lambert et al. 2024), which coined the name "RLVR," defines it as the RLHF objective with the reward model replaced by a verification function. Their reward is literally a constant for correctness:

                ⎧ α   if the verifier says the answer is correct
R_RLVR(x, y) =  ⎨
                ⎩ 0   otherwise

with α = 10 in their experiments. The full optimized objective keeps the RLHF KL leash to the reference model:

max_{π_θ}  E_{y ~ π_θ(·|x)} [ R_RLVR(x, y) ]
         = E_{y ~ π_θ(·|x)} [ v(x, y) − β · KL(π_θ(·|x) ‖ π_ref(·|x)) ]

where v(x,y) = α on a verified-correct completion and 0 otherwise. Tulu 3 runs this with PPO and reports the value model is best initialized from a general reward model. Their verifiers: for GSM8K / MATH, extract the final answer and check it against ground truth (string / numeric / symbolic match); for IFEval, run the explicit constraint checkers (e.g. "answer in JSON", "≤ 100 words").

DeepSeek-R1-Zero (DeepSeek-AI 2025) uses a slightly richer rule-based reward, the sum of two ungameable components:

Accuracy reward — for deterministic-answer tasks, force the final answer into a parseable form (e.g. inside \boxed{} for math, or run code against test cases) and check it. Correct → reward.
Format reward — require the model to wrap its reasoning in <think>…</think> before the answer. This is the lever that creates the visible chain-of-thought: the model is paid to externalize its reasoning.

Crucially R1 used no neural reward model in this phase — explicitly to dodge reward hacking over long training runs.

(B) GRPO — the RL algorithm of choice

The RL workhorse for RLVR is GRPO (Group Relative Policy Optimization), introduced in DeepSeekMath (Shao et al. 2024). GRPO is PPO with the critic (value network) deleted — which is what makes it cheap enough to scale RLVR.

The problem GRPO solves. PPO needs an advantage A_t ("was this token better than expected?"). Vanilla PPO computes that with a learned value function V(s) — a second network, roughly as large as the policy, trained alongside it via GAE. That doubles your memory and adds a notoriously finicky thing to tune.

GRPO's trick: be your own baseline. For each prompt q, sample a group of G outputs {o_1, …, o_G}, score them all with the verifier, and use the group's own mean reward as the baseline. No critic needed:

        r_i − mean(r_1, …, r_G)
Â_i  =  ─────────────────────────         (outcome supervision: same Â for every
            std(r_1, …, r_G)               token in output o_i)

The intuition is exactly the REINFORCE-with-baseline idea: "of the G answers I tried to this question, was this one above or below average?" The std-normalization keeps the scale stable across easy and hard prompts. A subtle consequence: if all G samples get the same reward (all right or all wrong), std → 0 and the advantage is zero — that prompt contributes no gradient this step. (This is both a feature — auto-curriculum toward the frontier of the model's ability — and a known pathology, advantage collapse on too-easy/too-hard prompts.)

The GRPO objective (Eq. 3 of DeepSeekMath) — PPO's clipped surrogate, summed over the group, with the KL pulled out of the reward and into the loss:

                   1   G    1   |o_i|        ⎡        ⎛  π_θ(o_i,t | q, o_i,<t)        ⎞
J_GRPO(θ) = E  [ ── Σ   ──── Σ      min ⎢ ρ_i,t Â_i , clip⎜ρ_i,t, 1−ε, 1+ε⎟ Â_i ⎥
                   G  i=1  |o_i| t=1        ⎣                            ⎠      ⎦
                                                    − β · D_KL( π_θ ‖ π_ref ) ]

with the importance ratio   ρ_i,t = π_θ(o_i,t | q, o_i,<t) / π_θ_old(o_i,t | q, o_i,<t)

and the KL estimated with Schulman's unbiased, always-positive estimator (so you don't have to compute the full distribution):

D_KL(π_θ ‖ π_ref) = π_ref(o_i,t|·)/π_θ(o_i,t|·) − log( π_ref(o_i,t|·)/π_θ(o_i,t|·) ) − 1

Note the contrast with PPO-RLHF (see [[rlhf-and-alignment]]), which usually folds the KL penalty into the per-token reward. GRPO adds KL directly to the loss, which avoids contaminating the advantage estimate.

Outcome vs process supervision. The version above is outcome supervision: one reward at the end, shared by every token. DeepSeekMath also defines process supervision, where a verifier (or process reward model) scores intermediate steps, and each token's advantage is the sum of normalized rewards from that step onward — denser signal, but you need a step-level verifier, which is much harder to build than a final-answer checker. Most production RLVR uses outcome supervision because a final-answer verifier is trivial and ungameable, whereas a process verifier reintroduces a learned, hackable model.

The R1 recipe in full (why it's not just RLVR)

R1-Zero (pure RLVR on the base model) reasoned brilliantly but had ugly outputs — language-mixing, poor readability. The shipped DeepSeek-R1 therefore wrapped RLVR in a 4-stage pipeline: (1) a small cold-start SFT on a few thousand long-CoT examples for readability, (2) large-scale RLVR for reasoning (the engine), (3) rejection sampling from the RL checkpoint → new SFT data + general RLHF for helpfulness/harmlessness, (4) a final RL pass mixing verifiable and preference rewards. RLVR is the capability engine; the SFT/RLHF stages are polish and safety. This sandwich — cold-start SFT → RLVR → preference RLHF — is now the standard reasoning-model recipe.

Key ideas & tradeoffs

Verifier integrity > reward-model breadth. The whole value proposition is that a verifier can't be gamed except by being correct. You give up the ability to score open-ended tasks (creativity, tone, "is this a good poem"). Hence RLVR is a component, not a replacement for RLHF — you still need preference learning for everything uncheckable.
Binary, sparse, delayed reward — and it works anyway. Classical RL lore says sparse 0/1 rewards are brutal to learn from. RLVR gets away with it because the base model is already a strong prior: even a weak model gets some answers right by chance, GRPO's group baseline turns "this rollout beat its siblings" into signal, and the search space (token sequences) is one the model already navigates fluently.
No critic = cheaper, more stable. Dropping PPO's value network (GRPO) roughly halves memory and removes a major source of instability; DeepSeek reports ~4.5× faster RL than their PPO baseline. This is largely why RLVR could be scaled to thousands of steps.
Reward shape is load-bearing. The <think> format reward is what externalizes reasoning; the accuracy reward is what makes it correct reasoning. Drop either and the magic degrades. Reward design, not just the algorithm, does the work.
It's an auto-curriculum. Because zero-variance groups give zero gradient, GRPO naturally concentrates learning on problems that are sometimes solvable — the frontier of current ability. That's a feature you get for free, and a failure mode when your data is all too-easy or all too-hard.
GRPO has known biases. The length/std normalization introduces subtle biases (long wrong answers under-penalized, etc.); follow-ups like DAPO, Dr.GRPO, and GSPO tweak the normalization and clipping. Treat GRPO as a strong default, not a finished algorithm.

Honest caveats & open questions

This is the part the hype skips. RLVR is real and important, but the most basic question about it is not settled.

Does RLVR add reasoning, or just *sharpen* it? The sharpest negative result: Yue et al. (2025), "Does RLVR Really Incentivize Reasoning Capacity?" measured pass@k for base vs RLVR-trained models. At small k (k=1) the RLVR model wins — it's more reliable. But at large k, the base model matches or *exceeds the RLVR model. Their conclusion: RLVR-elicited reasoning "originates from and is bounded by the base model," and the setup "does not elicit fundamentally new reasoning patterns." Read literally, RLVR is **redistributing probability mass onto solutions the base model could already produce** — making the model better at picking a correct path it already knew, not at finding genuinely new ones. (This is debated; other 2025 papers argue sufficiently long RL can* expand the boundary. The honest status: contested, methodology-sensitive, and the single most important open question in the area.)
The "invisible leash." Wang et al. (2025) formalize a related worry: because RLVR can only reward outputs the policy already samples with nonzero probability, it provably cannot put mass on solutions in the base model's support gaps — and it tends to shrink support (entropy/diversity collapse) rather than grow it. RLVR may sharpen you toward your origin, not past it.
Entropy / diversity collapse. In practice, policy entropy falls sharply during RLVR; the model becomes overconfident and stops exploring, which caps further gains and hurts pass@k. A whole 2025–2026 sub-literature (entropy regularization, clip-low/clip-high asymmetry, differential smoothing) exists just to fight this.
Verifiers can still be gamed — just differently. "Ungameable" is relative. Math graders accept wrong reasoning that lands on the right number (lucky guesses, canceling errors); code that passes weak tests isn't correct; format rewards get satisfied without real thinking. The verifier's coverage is the new attack surface — you moved the reward-hacking problem from "fool the reward model" to "exploit the verifier's gaps."
It only works where ground truth exists. Math, code, formal logic, checkable-constraint instruction-following. For most of what an assistant does (advice, writing, judgment, empathy) there is no verifier, so RLVR simply doesn't apply — you're back to preference learning and its leaks.
Process supervision reintroduces a model. Denser step-level rewards need a process reward model, which is itself learned and hackable — partially undoing the "no model to game" benefit. There's an inherent tension between signal density and verifier integrity.

The fair summary for 2026: RLVR is the most reliable known method to turn latent reasoning into consistent, sampled-at-k=1 reasoning, and the engine behind every frontier reasoning model — but whether it expands the ceiling or only raises the floor toward an existing ceiling is genuinely unresolved. Anyone who tells you it's settled is selling something.

How it connects to OpenAlice

Alice's autonomous-coding capability is a verifiable-reward task by nature. The org treats repo→PR coding as a base AGI capability measured by SWE-bench (see project-alice-agi-coding-capability). "Do the tests pass?" is a verifier — the exact RLVR signal. If/when we move from prompting frontier models to training an Alice policy on coding, the natural objective is RLVR: sample patches, run the test suite, reward green. The whole openalice-lab / benchmark / fixture-runner rig (24 module-specific fixture-runners; bench.blal.pro) is, in RLVR terms, a verifier farm — it already produces the binary correctness signals an RLVR loop would consume. That reframing is worth keeping in mind.
It explains the model lineup. OpenAlice runs on gpt-5.x reasoning models; their long-CoT "thinking" behavior is RLVR-produced. Understanding RLVR is understanding why these models pause, reason, self-check, and burn test-time compute — the consumer-side topic of [[test-time-compute-reasoning]].
The caveats are operationally relevant. The pass@k / sharpening result is why sampling-and-verifying (best-of-n against our own tests) can beat a single greedy call — the base capability is broader than any single RLVR-sharpened sample. If Alice is solving a hard coding task, generate several candidate patches and let the test suite pick the winner rather than trusting one confident attempt. That's the Yue-et-al. finding turned into an engineering tactic.
Reward-hacking literacy carries over. Everything we worry about with verifier gaps (weak tests passing, format-satisfied-but-wrong) is the same failure class as RLHF reward hacking in [[rlhf-and-alignment]] — relevant to our HITL gates, approval protocol, and safety floors.

RLVR — Reinforcement Learning from Verifiable Rewards

RLVR — Reinforcement Learning from Verifiable Rewards

The one-sentence idea

What it is — intuition first

Why it matters

How it actually works — the real mechanics

(A) The reward function — verifiable, usually binary

(B) GRPO — the RL algorithm of choice

The R1 recipe in full (why it's not just RLVR)

Key ideas & tradeoffs

Honest caveats & open questions

How it connects to OpenAlice

See also