Atlas
kb://library/grpostable2026-06-16

GRPO & policy optimization — dropping the critic, letting the group be the baseline

libraryeducationgrpoppodapodr-grporlrlhfrlvrreasoningpolicy-optimizationdeepseekm13

GRPO & policy optimization — dropping the critic, letting the group be the baseline

For NAO + anyone trying to understand the algorithm that taught DeepSeek-R1 (and most of the 2025–2026 reasoning models) to think. The one-sentence version: GRPO is PPO with the expensive half deleted. PPO needs a second neural network — a critic roughly as big as the model itself — to estimate "how good is this state?". GRPO throws that critic away and instead samples a group of answers to the same question, then asks each answer "did you beat the group average?". That average is the baseline. Cheaper, simpler, and — on verifiable tasks like math and code — startlingly effective. Sourced from the DeepSeekMath paper that introduced GRPO (Shao et al. 2024), DeepSeek-R1 (2025), DAPO (ByteDance Seed, 2025), Dr. GRPO (Liu et al. 2025), and Cameron Wolfe's technical walkthrough.

This is the algorithm companion to [[rlhf-and-alignment]] (where GRPO sits in the post-training family), [[deepseek-architecture]] (the system that made it famous), and [[test-time-compute-reasoning]] (what it buys you: long chains of thought). Read those for the what and why of reasoning models; read this for the how of the optimizer underneath them.

What it is (intuition first)

Imagine you're teaching a student math and you only know one thing: whether the final answer was right or wrong. You can't grade the working. How do you turn that single bit of feedback into "do more of this, less of that"?

The naive move (plain REINFORCE) is: if the answer was right, push up the probability of every token the student wrote; if wrong, push it down. The problem is variance. Some questions are easy (everyone gets ~1.0 reward), some are hard (everyone gets ~0.0). Rewarding a 0.9 on an easy question the same as a 0.9 on a brutal one sends a noisy, mostly-useless signal. You need a baseline — a sense of "what score did I expect here?" — so you can reward only the surplus over expectation.

PPO solves this with a critic: a second neural network, trained alongside the policy, that learns to predict the expected reward of a partial answer. The critic is the baseline. It works, but it doubles your training footprint and the critic is itself hard to train well (a bad critic poisons the whole run).

GRPO's trick is almost embarrassingly simple. For each question, don't sample one answer — sample a group of, say, 64. Now you have 64 rewards. Their average is your baseline, for free, with no second network. An answer that scored above the group mean gets a positive advantage ("be more like this"); one below the mean gets a negative advantage ("be less like this"). That's it. The group is the critic. This is why it's called Group Relative Policy Optimization — advantage is computed relative to the cohort of answers to the same prompt, not against a learned value function.

The payoff lands hardest when the reward is verifiable: a math answer you can check by string-matching, code you can run against unit tests. No human labels, no learned reward model to game — just "right/wrong" plus the group baseline. That combination (rule-based reward + group baseline) is what DeepSeek-R1-Zero used to bootstrap reasoning from a base model with zero supervised fine-tuning, and it's the engine of the "RL from verifiable rewards" (RLVR) wave.

Why it matters

  • It deleted the critic. PPO trains two models. In half precision you budget roughly ~16 GB per 1 B params, so a 32 B policy + same-size critic is on the order of ~512 GB; GRPO cuts that roughly in half by training the policy alone. At frontier scale that's the difference between fitting on your cluster and not.
  • It made cheap reasoning RL real. GRPO + rule-based rewards is what let DeepSeek-R1-Zero learn long chain-of-thought, reflection, and self-correction without any human reasoning traces. The R1 line then made the recipe public, and within months it was the default RL algorithm for open reasoning models.
  • It's the substrate everyone forks. DAPO, Dr. GRPO, GSPO, GTPO/GRPO-S and a dozen others in 2025–2026 are all "GRPO but fix this one bias." Understanding GRPO is the price of entry for reading any modern post-training paper.
  • It generalized RLHF. The DeepSeekMath paper frames GRPO as a unified lens: PPO, DPO-style preference methods, and rejection sampling are all special cases of "estimate an advantage, then take a clipped policy step." See [[rlhf-and-alignment]] for that family tree.

How it works (real mechanics)

The PPO objective it descends from

PPO maximizes a clipped surrogate so each update can't move the policy too far from the sampling policy $\pi_{\theta_{old}}$:

$$\mathcal{J}_{PPO}(\theta)=\mathbb{E}\Big[\min\Big(\frac{\pi_{\theta}(o_t\mid q,o_{<t})}{\pi_{\theta_{old}}(o_t\mid q,o_{<t})}A_t,\ \text{clip}\Big(\frac{\pi_{\theta}(o_t\mid q,o_{<t})}{\pi_{\theta_{old}}(o_t\mid q,o_{<t})},\,1-\varepsilon,\,1+\varepsilon\Big)A_t\Big)\Big]$$

Here $A_t$ is the advantage at token $t$, and in PPO it comes from a learned value function $V_\psi$ via Generalized Advantage Estimation. That $V_\psi$ is the critic GRPO deletes.

The GRPO objective

For each question $q$, sample a group of $G$ outputs $\{o_1,\dots,o_G\}$ from $\pi_{\theta_{old}}$. The objective:

$$\mathcal{J}_{GRPO}(\theta)=\mathbb{E}\Bigg[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_i|}\sum_{t=1}^{|o_i|}\Big\{\min\Big(\frac{\pi_\theta(o_{i,t}\mid q,o_{i,<t})}{\pi_{\theta_{old}}(o_{i,t}\mid q,o_{i,<t})}\hat{A}_{i,t},\ \text{clip}\big(\tfrac{\pi_\theta(o_{i,t}\mid q,o_{i,<t})}{\pi_{\theta_{old}}(o_{i,t}\mid q,o_{i,<t})},1-\varepsilon,1+\varepsilon\big)\hat{A}_{i,t}\Big)-\beta\,\mathbb{D}_{KL}\big[\pi_\theta\,\|\,\pi_{ref}\big]\Big\}\Bigg]$$

Read it in pieces:

  • $\frac{1}{G}\sum_i$ — average over the group of $G$ answers.
  • $\frac{1}{|o_i|}\sum_t$ — average over the tokens of answer $i$ (this length normalization is exactly what Dr. GRPO later attacks — see below).
  • The min/clip block is identical to PPO's trust-region step.
  • $-\beta\,\mathbb{D}_{KL}$ — a leash to the reference policy. Note GRPO puts the KL term directly in the loss, not folded into the reward as classic RLHF-PPO does.

Where the advantage comes from (the whole point)

No critic. With outcome supervision (one reward $r_i$ per whole answer), the advantage for every token of answer $i$ is the group-normalized reward:

$$\hat{A}_{i,t}=\widetilde{r}_i=\frac{r_i-\text{mean}(\mathbf{r})}{\text{std}(\mathbf{r})}\quad\text{where }\mathbf{r}=\{r_1,\dots,r_G\}$$

Subtract the group mean (the baseline) → "did you beat your peers?". Divide by the group std → put every question on a comparable scale. Every token in a winning answer shares the same positive advantage; every token in a losing one shares the same negative advantage.

With process supervision (a reward at the end of each reasoning step), the advantage at token $t$ is the sum of normalized step-rewards from that point on:

$$\hat{A}_{i,t}=\sum_{\text{index}(j)\ge t}\widetilde{r}_i^{\,\text{index}(j)}$$

so later tokens get credit for the good steps that follow them — finer-grained, but you need a process reward model (PRM) to produce step rewards (see [[test-time-compute-reasoning]] on PRMs).

The KL estimator (a small, important detail)

GRPO uses the k3 / Schulman unbiased estimator, which is always $\ge 0$:

$$\mathbb{D}_{KL}\big[\pi_\theta\,\|\,\pi_{ref}\big]=\frac{\pi_{ref}(o_{i,t}\mid q,o_{i,<t})}{\pi_\theta(o_{i,t}\mid q,o_{i,<t})}-\log\frac{\pi_{ref}(o_{i,t}\mid q,o_{i,<t})}{\pi_\theta(o_{i,t}\mid q,o_{i,<t})}-1$$

The naive $\log(\pi_\theta/\pi_{ref})$ estimator is unbiased but can go negative on a single sample; this $r-\log r-1$ form is guaranteed non-negative, which keeps the penalty well-behaved per token.

Pseudocode

# One GRPO iteration. No value network anywhere in sight.
for q in batch_of_questions:                       # e.g. 16 prompts
    group = [pi_old.sample(q) for _ in range(G)]   # G=64 answers per prompt
    rewards = [reward_fn(q, o) for o in group]     # rule-based: 1.0 if correct else 0.0 (+ format)

    mu, sigma = mean(rewards), std(rewards)
    # group-relative advantage — the group IS the baseline (no critic V_psi)
    adv = [(r - mu) / (sigma + 1e-8) for r in rewards]   # one scalar per answer

    loss = 0.0
    for o_i, A_i in zip(group, adv):
        for t in range(len(o_i)):                   # A_i shared across all tokens of o_i
            ratio = pi_theta(o_i[t] | q, o_i[:t]) / pi_old(o_i[t] | q, o_i[:t])
            clipped = clip(ratio, 1 - eps, 1 + eps)
            tok = min(ratio * A_i, clipped * A_i)    # PPO trust-region step
            kl  = (pi_ref/pi_theta) - log(pi_ref/pi_theta) - 1   # k3 estimator
            loss += -(tok - beta * kl) / len(o_i)    # 1/|o_i| length-normalization
    loss /= G
    backprop_and_step(loss)
# DeepSeekMath defaults: G=64, eps=0.2, beta=0.1, batch=16 prompts (1024 samples).

Notice what's missing: there is no critic forward pass, no value-function loss, no GAE. The entire advantage machinery is three lines of arithmetic over the group's rewards.

Key ideas & tradeoffs

  • The group size $G$ is load-bearing. Because advantage is entirely relative, you need enough samples per prompt for the mean/std to be stable — DeepSeekMath uses $G=64$. Too small and the baseline is noisy; too large and you pay in sampling/inference cost (and groups of all-correct or all-wrong give zero gradient — more below).
  • Verifiable rewards are the sweet spot. Rule-based rewards (string-match the math answer, run the unit tests) can't be hacked the way a learned reward model can. DeepSeek-R1 uses accuracy + format rewards and no neural reward model for reasoning. On non-verifiable tasks (creative writing, helpfulness) you're back to a reward model and all of RLHF's reward-hacking problems return — see [[rlhf-and-alignment]].
  • Memory vs. variance trade. You trade the critic's memory cost for the sampling cost of $G$ rollouts per prompt and somewhat higher gradient variance than a well-trained critic would give. For long-CoT reasoning that trade has been overwhelmingly worth it.
  • Same advantage on every token. Outcome-supervised GRPO gives every token in an answer the same scalar advantage — no per-token credit assignment. A correct token buried inside a wrong answer gets penalized anyway. This crude credit assignment is the root of several criticisms (and the GTPO/GRPO-S line of fixes).

The descendant tree (2025–2026)

GRPO's simplicity made it a magnet for "fix one thing" papers:

  • DAPO (ByteDance Seed, 2025) — the most-adopted production refinement. Four changes: (1) Clip-Higher — decouple the clip bound into $\varepsilon_{low}$ and $\varepsilon_{high}$, raising the upper bound so low-probability tokens can still grow, fixing GRPO's entropy collapse; (2) Dynamic Sampling — drop prompts where all $G$ answers are right or all wrong (those give zero gradient), constraint $0<|\{o_i:\text{correct}\}|<G$; (3) Token-Level Loss — normalize over total tokens in the batch instead of GRPO's per-sequence $1/|o_i|$, so long sequences aren't down-weighted; (4) Overlong Reward Shaping — a soft length penalty to stop runaway generations. DAPO also removes the KL term entirely, arguing that for long-CoT the policy should diverge far from the base model, so the leash is counterproductive. Result: 50 on AIME-2024 with Qwen2.5-32B, beating R1-Zero-Qwen-32B's 47 in half the steps. Ablation: naive GRPO 30 → +overlong-filter 36 → +clip-higher 38 → +soft-punish 41 → +token-loss 42 → +dynamic-sampling 50.
  • Dr. GRPO ("GRPO Done Right", Liu et al. 2025) — a critique. It identifies two optimization biases baked into the GRPO objective: (1) the $1/|o_i|$ length normalization makes the policy prefer longer wrong answers (negative advantage divided by a bigger length = smaller penalty), explaining the infamous "response length keeps growing" curve as an artifact, not genuine reasoning; (2) the $\text{std}(\mathbf{r})$ normalization over-weights questions that are too easy or too hard (low variance → big effective weight). Dr. GRPO removes both: drop the $1/|o_i|$, drop the std division, leaving $\hat{A}_{i,t}=R(q,o_i)-\text{mean}(\{R(q,o_1),\dots,R(q,o_G)\})$ — which recovers a plain Monte-Carlo return with an unbiased group baseline. It keeps length from "growing wildly" with little accuracy cost.
  • GTPO / GRPO-S (2025) — attack the crude per-token credit assignment with entropy-weighted token/sequence rewards for more stable long-CoT training.
  • Many more (DCPO, adaptive-boundary clipping, GSPO's sequence-level ratios…) — all variations on the same skeleton.

A useful mental model: DAPO is the engineering fork (make it train stably at scale), Dr. GRPO is the theory fork (make the math unbiased). They disagree on some details (e.g. DAPO keeps std-normalized advantage; Dr. GRPO drops it) — there is no single settled "correct GRPO" yet.

Honest caveats & open questions

  • Entropy collapse is the #1 failure mode. GRPO policies tend to get confident too fast: groups start producing near-identical answers, diversity dies, the baseline becomes useless (all rewards equal → zero advantage → no gradient), and learning stalls. Clip-Higher (DAPO) mitigates it but doesn't fully solve it; a whole sub-literature in 2025–2026 is about restoring exploration after post-training.
  • Reward hacking didn't go away — it moved. Verifiable rewards dodge reward- model hacking, but only where you have a verifier. The moment you use GRPO on fuzzy objectives (with a learned reward model or an unconditional success signal), documented failures include mode collapse onto a single safe action and exploitation of the reward signal. GRPO is not a reward-hacking cure; the task has to be verifiable.
  • Crude credit assignment. One scalar advantage per answer, smeared over every token, is theoretically unsatisfying — correct tokens in wrong answers get punished, wrong tokens in right answers get rewarded. It works empirically far better than it "should," which is itself an open question.
  • The math isn't settled. Dr. GRPO shows the original objective is biased. DAPO ships a different objective that works great empirically but keeps the std-normalization Dr. GRPO removes. The community is still converging on which knobs are bugs and which are features — be suspicious of anyone who tells you there's one canonical GRPO.
  • Group sampling cost. Deleting the critic saved memory but $G$ rollouts per prompt is expensive inference. The bottleneck shifted from "fit the critic in memory" to "generate 64× the tokens" — your trainer is now a sampling engine.
  • It needs a decent base model. R1-Zero's pure-RL magic worked on a strong base (DeepSeek-V3-Base). On weak models, RLVR has much less to amplify — GRPO surfaces latent capability, it doesn't conjure it.

How it connects to OpenAlice

OpenAlice does not train models with GRPO — Alice is an agentic system on top of frontier LLMs (gpt-5.5 et al.), not a from-scratch trainer. So the relevance is conceptual and architectural, and worth being honest about:

  • The "group-relative baseline" is a reusable pattern beyond training. The core GRPO idea — sample many candidates, score them, keep what beats the group average — is exactly the shape of OpenAlice's council / best-of-N machinery. See [[fusion-and-llm-councils]] and [[mixture-of-agents]]: when Alice samples several candidate replies and a critic/ranker picks the best, that is inference-time group-relative selection. GRPO is the training-time version of the same instinct; the cockpit's ranker is the test-time version.
  • Verifiable rewards ↔ Alice's gates and critics. GRPO's lesson — "a cheap, rule-based, un-gameable verifier beats a fancy learned reward model" — maps onto Alice's safety/approval gates and critic-LLM checks: prefer a deterministic rule you can trust over a model you have to second-guess (cf. the critic fail-open work in core).
  • Why R1-style reasoning matters to the agent. GRPO is how the reasoning models Alice can call were made to think in long chains. Understanding it tells you why those models behave the way they do (long CoT, self-reflection) and why their cost profile is what it is — relevant when [[model-routing]] decides whether a task deserves a reasoning model's test-time compute.
  • Future: autonomous-coding self-improvement. NAO's directive frames repo→PR autonomous coding as a base AGI capability of Alice, measured on SWE-bench. If OpenAlice ever closes a self-improvement loop (run the test suite as a verifier, GRPO on the agent's own trajectories), this is the algorithm that would do it — code-passes-tests is the canonical verifiable reward. That's a research direction, not a shipped feature; stated here so the connection is honest, not oversold.

For the system that made GRPO famous and the efficiency stack around it, read [[deepseek-architecture]]. For where GRPO sits in the post-training family (RLHF / DPO / RLVR), read [[rlhf-and-alignment]]. For what the trained model then does with its compute, read [[test-time-compute-reasoning]].