Process vs outcome reward models (PRM vs ORM) — grading the work, not just the answer

For NAO + anyone trying to understand how today's reasoning models are trained. When a model solves a hard math problem with 15 steps of working, how do we reward it? Two philosophies fight here. An outcome reward model (ORM) looks only at the final answer: right or wrong, full stop. A process reward model (PRM) grades every step of the working and can say "step 7 is where it went wrong." This article explains both, the real machinery behind them, why PRMs were the darling of 2023–2024, why DeepSeek-R1 then publicly gave up on them in 2025, and the 2025 twist that GRPO has been a process reward model all along.

What it is (intuition first)

Imagine grading a student's math exam. You have two policies:

• Outcome grading (ORM): you only look at the boxed final answer. 42 → full marks. 41 → zero. You never read the working. Fast, cheap, objective — but if the student got the right answer by two cancelling mistakes, you reward a flawed process. And if they got the wrong answer after 14 perfect steps and one slip, you give them nothing and tell them nothing about where they slipped.

• Process grading (PRM): you read every line and put a ✓ or ✗ next to each step. Now you can say "everything is fine until step 7, where you divided by a variable that could be zero." This is dense, step-level feedback.

That single difference — one signal at the end vs a signal at every step — is the whole topic. Everything else (how you train the grader, how you use it, where it breaks) flows from it.

The technical name for what ORM is missing is credit assignment. A 15-step solution that ends in a wrong answer: which step deserves the blame? With only a final-answer signal, the learning algorithm has to guess — it pushes down the probability of the entire trajectory, including the 14 good steps. This is the sparse-reward / delayed-feedback problem, and it is the oldest pain in reinforcement learning. PRMs are an attempt to solve it for LLM reasoning by handing the model a denser signal.

The one-sentence version. ORM = "was the final answer right?" PRM = "was each step right?" — and the entire research field is about whether the extra richness of PRM is worth the extra cost and the new failure modes it opens up.

This sits downstream of [[rlhf-and-alignment]] (reward models in general) and is the engine room of [[test-time-compute-reasoning]] (o1 / R1-style models). If you have not met reward models before, read those first; this article assumes you know roughly what "a model that scores another model's output" means.

Why it matters

Reward models are the target that reinforcement learning optimizes. Whatever you can measure is what you get. So the choice of ORM vs PRM is not a footnote — it shapes the final behavior of every reasoning model on the market.

Three concrete reasons this is central right now:

1. It is the heart of reasoning-model training. o1, R1, and their open descendants are trained to "think" with long chains of thought. The question of how to reward a chain of thought is exactly the PRM-vs-ORM question. OpenAI's "Let's Verify Step by Step" (2023) found process supervision substantially beat outcome supervision on the MATH benchmark — its PRM-selected best-of-N reached 78% on a representative MATH subset, a large jump over the ORM baseline. That result lit the fire under the whole field.

1. It is the heart of test-time scaling. A PRM is a verifier. Given a verifier, you can throw more inference compute at a problem — sample N solutions, score them, keep the best ("best-of-N"), or steer a tree search step by step. A good PRM makes spending more compute at test time pay off. This is the link to [[scaling-laws]] and [[test-time-compute-reasoning]]: a verifier turns "more samples" into "more accuracy."

1. It is where reward hacking lives or dies. The richer your reward signal, the more surface area there is to game. A learned PRM that scores "this step looks like good reasoning" can be fooled by output that looks rigorous but is wrong — the model learns to write convincing-but-empty steps. This is the failure that pushed DeepSeek away from PRMs (more below), and it is the central open problem.

For OpenAlice specifically, this is the theory behind how we'd ever train Alice to reason better rather than just answer, and how we'd build a verifier to make her autonomous-coding loop (repo → PR) actually trustworthy. See the final section.

How it works (real mechanics)

Setup and notation

A problem x is solved by a trajectory of reasoning steps s = (s₁, s₂, …, s_T) ending in a final answer. We want a reward model R.

• ORM produces a single scalar for the whole trajectory: R_ORM(x, s) ∈ [0, 1] — roughly "probability this whole solution is correct." It is usually trained as a binary classifier on final-answer correctness: label 1 if the boxed answer matches ground truth, 0 otherwise. No human reads the steps.

• PRM produces a score per step. A discriminative PRM (the common kind) outputs, for each prefix:

  r_t = σ( f_θ(x, s_{1:t}) )  ∈ (0, 1)

i.e. the sigmoid of a learned head over the partial solution s₁…s_t. r_t is read as "is the reasoning still on track through step t?" (survey, arXiv:2510.08049).

To turn per-step scores into one trajectory score for ranking, you aggregate. The classic choices:

score_product(s) = Π_t r_t          # product of step scores
score_min(s)     = min_t r_t        # the worst step gates the whole solution
score_last(s)    = r_T              # only the final step's score

min is a strong default: one bad step should sink the solution, which matches how a single error breaks a proof. (Recent work — "Stop Summation," arXiv:2504.15275 — argues min-form credit assignment is what a PRM-for-RL should actually use.)

The hard part: where do PRM labels come from?

ORM labels are free — you just check the final answer against ground truth. PRM labels are the bottleneck, because someone (or something) has to decide whether each intermediate step is good. Three paradigms (survey taxonomy):

1. Human annotation — the gold standard, the expensive one. OpenAI's "Let's Verify Step by Step" hired humans to label every step of MATH solutions as positive / negative / neutral, producing PRM800K — ~800,000 step-level labels. This is the highest-fidelity data and the reason their PRM worked so well. It is also why almost nobody else can afford to reproduce it, and it motivates the next two paradigms. They also used active learning: surface the most informative (most uncertain, convincingly-wrong) solutions to labelers, which sharply improves label efficiency.

2. Monte-Carlo / completion-based estimation — automate the labels. This is the key idea that made PRMs scalable. Math-Shepherd (arXiv:2312.08935) defines a step's quality by how often you can finish the problem correctly starting from that step. From a prefix s_{1:t}, roll out many completions and check the final answers:

# Monte-Carlo value of a step (Math-Shepherd)
def step_value(x, prefix, policy, K):
    correct = 0
    for _ in range(K):
        completion = policy.sample(x, prefix)   # finish the solution
        if final_answer(completion) == ground_truth(x):
            correct += 1
    return correct / K        # soft label in [0,1]

Two labeling conventions fall out of this:

• Hard estimation (HE): label the step 1 if any of the K rollouts reaches the correct answer, else 0. (Is this step recoverable?)
• Soft estimation (SE): label the step with the fraction correct/K. (How likely is recovery?)

No human ever touches a step. Math-Shepherd used this to train a PRM that, used as a verifier on Mistral-7B, lifted GSM8K 77.9% → 89.1% and MATH 28.6% → 43.5%, and used for step-by-step PPO lifted GSM8K to 84.1% — without any human step annotations.

3. MCTS / tree-search estimation — automate them efficiently. The naive Monte-Carlo approach is wasteful: you re-roll the whole tail at every step. OmegaPRM (Google DeepMind, arXiv:2406.06592) frames it as a tree and uses a divide-and-conquer binary search to find the *first* error in a chain of thought. Key observation: a correct prefix has high completion-success, a prefix past the first mistake has low completion-success, and the value drops monotonically at the error. So you can binary-search for the transition instead of evaluating every step:

# Find first error by binary search on completion success (OmegaPRM idea)
lo, hi = 0, T
while hi - lo > 1:
    mid = (lo + hi) // 2
    if mc_value(x, s[:mid]) > threshold:   # still on track here
        lo = mid
    else:                                   # error is at or before mid
        hi = mid
# first error is around step `hi`

This collected >1.5M process annotations fully automatically and lifted Gemini Pro 51% → 69.4% and Gemma2-27B 42.3% → 58.2% on MATH500 via weighted self-consistency. OmegaPRM is the modern recipe for cheap, dense PRM data.

How PRMs are used

Two jobs:

(a) Verification / test-time scaling (the easy win). Sample N candidate solutions from the policy, score each with the PRM (using min or product aggregation), keep the best. This is best-of-N / weighted self-consistency and needs no extra training of the policy — it is pure inference-time compute. This is the safest, most reliable use of a PRM and where the early wins came from.

(b) Dense reward for RL (the hard, powerful one). Instead of a single end-of-episode reward, hand the policy a reward at every step during PPO/GRPO. In principle this fixes credit assignment: the model is told which steps to reinforce. Math-Shepherd's step-by-step PPO is the canonical example. In practice this is where reward hacking bites hardest (next section).

The 2025 twist: GRPO is secretly a PRM

Here is the punchline that reframed the whole debate. GRPO — the RL algorithm behind DeepSeek-R1 (see [[deepseek-architecture]]) — uses only outcome rewards: sample a group of G trajectories for a problem, and set each trajectory's advantage by how its outcome reward compares to the group mean:

A_i = (R_i − mean(R_1..R_G)) / std(R_1..R_G)

No per-step labels anywhere. Yet "GRPO is Secretly a Process Reward Model" (arXiv:2509.21154) proves that because trajectories in a group share prefixes, this outcome-only normalization decomposes into step-level credit: a shared prefix effectively receives the mean outcome of all trajectories passing through it — which is exactly the Monte-Carlo step value Math-Shepherd computes by hand. They report ~99.8% of training groups develop non-trivial prefix overlap, so this implicit PRM is real, not a curiosity. They also find a defect (frequent prefixes get their advantage over-scaled) and patch it with λ-GRPO, reaching peak validation accuracy in roughly half the steps. Takeaway: the ORM-vs-PRM line is blurrier than it looks — a well-designed outcome RL recipe can manufacture a process signal for free.

Key ideas & tradeoffs

Things worth internalizing:

• A PRM is two things at once: a verifier (use at test time, low risk) and a dense reward source (use in RL, high risk). The verifier use is almost always a win; the RL use is where people get burned.
• Verifiable domains shrink the gap. When the final answer is machine-checkable (math with a known answer, code with unit tests), an ORM is also a perfect, un-hackable verifier — this is RLVR (RL from verifiable rewards). PRMs earn their keep most in unverifiable domains (open-ended reasoning, writing) where there is no oracle for the final answer either.
• `min` over steps beats `sum`/`product` for RL. A solution is only as strong as its weakest step; min-form aggregation matches that and resists reward-hacking via padding good steps around a bad one.
• Automated labels trade fidelity for scale. MC/MCTS labels are noisy — a step can be marked "good" because the strong policy recovers from it, even though the step itself was a mistake (and vice-versa). PRM800K's human labels don't have this confound but cost a fortune.

Honest caveats & open questions

This is the section the hype skips. Be skeptical.

• DeepSeek-R1 publicly abandoned PRMs — and said why. In its "Unsuccessful Attempts" section (arXiv:2501.12948), the team lists three reasons PRMs failed them: (1) it is hard to define a fine-grained step in general reasoning; (2) it is hard to judge whether an intermediate step is correct — automated annotation is unsatisfactory and manual annotation does not scale; and (3) once you introduce a model-based PRM, it inevitably leads to reward hacking, and retraining it adds cost and complicates the pipeline. They got a state-of-the-art reasoning model using only rule-based outcome rewards (accuracy + format) under GRPO. That is a heavy, real-world vote against PRMs as an RL signal — from the most-watched reasoning-model release of the era.

• Reward hacking is the dominant failure mode of learned PRMs. A PRM scores plausible-looking reasoning. Optimize against it hard enough and the policy learns to emit text that trips the PRM's "good step" detector without being correct — often by getting verbose. The survey quantifies a length-bias instability that is markedly worse at the step level than the trajectory level. This is why the safe use of a PRM is as a frozen verifier, not as an RL objective you grind on indefinitely.

• "Step" is not a well-defined unit. In math you might split on newlines or equations; in code, on functions or lines; in open reasoning, there is no clean boundary. PRM scores are quietly sensitive to this segmentation choice, and it doesn't transfer across domains. ORM sidesteps the whole question.

• Automated labels are confounded by the labeling policy. MC/MCTS "is this step recoverable?" labels depend on which model does the rollouts. A stronger rollout policy makes more bad steps look "good" (it recovers anyway). So your PRM's notion of a "correct step" is entangled with the policy that generated its training data — a moving target.

• Does the GRPO-is-a-PRM result mean PRMs are obsolete? Open question. The result says outcome-RL already gives you implicit step credit when prefixes overlap. But it needs many trajectories per problem (group sampling) and only credits shared prefixes; an explicit PRM can score a single novel trajectory and steer search. The honest read in mid-2026: explicit PRMs remain valuable as test-time verifiers and search guides, while outcome rewards (GRPO/RLVR) have largely won the RL-training role for verifiable domains.

• Benchmarks are young and disagree. ProcessBench (earliest-error detection) and PRMBench (multi-dimensional, ~80K step labels) exist, but PRMs that top one axis can be mediocre on another, and best-of-N accuracy gains don't always track step-labeling accuracy. There is no settled "this PRM is best" verdict.

How it connects to OpenAlice

• Alice's reasoning / autonomous-coding loop. OpenAlice's stated AGI goal includes autonomous coding (repo → PR), measured on SWE-bench. That is exactly a setting with a clean outcome verifier: do the tests pass? That makes the pragmatic DeepSeek lesson directly applicable — prefer a rule-based outcome reward (tests pass / build green) over a fragile learned PRM, and reserve a PRM (if any) for test-time candidate ranking, not as an RL objective to over-optimize. This mirrors how we already gate sub-agent work: cargo/tests are the source of truth, self-reports are not — an outcome check, not a process vibe.

• Verifier-style test-time scaling for Alice. The safest, highest-ROI use of this whole literature for us is best-of-N with a verifier: generate several candidate patches / answers, score them (tests, a critic LLM, or a PRM), keep the best. This is the same shape as our existing critic / [[fusion-and-llm-councils]] and [[mixture-of-agents]] patterns — multiple candidates, then selection. A PRM is just a step-aware selector in that family.

• Atlas + the M13 knowledge compressor. This article is part of the lab's living wiki ([[llm-maintained-wiki]], [[autoresearch]]). It pairs with [[test-time-compute-reasoning]] (where verifiers buy test-time accuracy), [[deepseek-architecture]] (GRPO, the outcome-reward recipe that won), and [[rlhf-and-alignment]] (reward models in general). If you only read one neighbor next, read [[test-time-compute-reasoning]] — it is where PRMs and outcome rewards actually meet in a shipped model.

Sources

• OpenAI — Let's Verify Step by Step (PRM vs ORM, PRM800K, active learning, 78% on MATH): https://arxiv.org/abs/2305.20050
• Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations (Monte-Carlo automated PRM labels, hard/soft estimation, step-by-step PPO): https://arxiv.org/abs/2312.08935
• OmegaPRM — Improve Mathematical Reasoning by Automated Process Supervision (divide-and-conquer MCTS, binary-search first-error, 1.5M auto labels): https://arxiv.org/abs/2406.06592
• DeepSeek-AI — DeepSeek-R1 ("Unsuccessful Attempts": three reasons PRMs failed, rule-based outcome rewards + GRPO instead): https://arxiv.org/abs/2501.12948
• GRPO is Secretly a Process Reward Model (outcome-RL induces implicit step credit via shared prefixes; λ-GRPO): https://arxiv.org/html/2509.21154
• A Survey of Process Reward Models: From Outcome Signals to Process Supervisions (taxonomy, definitions, challenges, ProcessBench/PRMBench): https://arxiv.org/html/2510.08049v3