RLHF, DPO & Alignment — teaching a model what we want

For NAO + anyone learning how raw LLMs become assistants. A base model trained on the internet can complete text brilliantly but has no idea it is supposed to be helpful, honest, and harmless. This article is the bridge: how we turn a next-token predictor into something that follows instructions and behaves — and exactly where that process leaks. Intuition first, then the real equations (reward models, PPO-RLHF, DPO, GRPO, Constitutional AI/RLAIF), then an honest accounting of what "alignment" can and cannot buy you. Sources: InstructGPT (Ouyang et al. 2022), DPO (Rafailov et al. 2023), DeepSeekMath/GRPO (Shao et al. 2024), Constitutional AI (Bai et al. 2022), and Anthropic's 2025 reward-hacking work.

The one-sentence idea

A base LLM only knows how to continue text; RLHF and its successors add a second training stage that optimizes the model against *human (or AI) preferences* instead of next-token likelihood — so it learns not just what words are plausible, but which *responses* people actually prefer.

The shift is subtle but total. Pretraining asks "what token comes next in this internet document?" Alignment asks "of two possible answers to this user, which one is better?" — and there is no token-level label for "better". You have to learn it from comparisons. Everything below is machinery for turning "answer A is better than answer B" into a gradient that updates billions of weights.

Why it matters

1. It is the step that made ChatGPT possible. InstructGPT showed a 1.3B model fine-tuned with RLHF was preferred by humans over the raw 175B GPT-3 — a 100× smaller model winning purely on alignment (Ouyang et al. 2022). The capability was already latent in pretraining; alignment elicited it.
2. It is where "values" enter the system. Helpfulness, harmlessness, honesty, tone, refusal behavior, format — none of this is in the pretraining objective. It is injected here, by whoever writes the preference data or the constitution. Alignment is the control surface for what kind of thing the model is.
3. It is the active research frontier of 2024–2026. PPO → DPO → GRPO → RLVR is a live evolution: each step trades away machinery (a reward model, a critic network, human labels) for simplicity or scale. DeepSeek-R1's reasoning ability came almost entirely from this stage, not pretraining.
4. Its failure modes are the safety problem. Reward hacking, sycophancy, deceptive alignment — these are not pretraining bugs. They are alignment bugs, born from optimizing a proxy. You cannot understand AI safety without understanding this pipeline's leaks.

How it actually works — the classic RLHF pipeline (InstructGPT)

The canonical recipe has three stages. Get these straight and everything else is a variation.

Stage 1 — Supervised fine-tuning (SFT)

Collect demonstrations: humans write good answers to prompts. Fine-tune the base model on them with ordinary next-token cross-entropy. This gives you a model that roughly follows instructions — call it π_SFT. It is the launchpad; RL alone from a base model is too unguided.

Stage 2 — Reward modeling (RM)

You cannot ask a human to score every response on an absolute 1–10 scale (humans are noisy and inconsistent at that). Instead you ask the easy question: given a prompt and several candidate answers, rank them. InstructGPT sampled K = 4 to 9 completions per prompt and had labelers rank them, yielding all (K choose 2) pairwise comparisons.

Train a reward model r_θ (InstructGPT used a 6B model — larger RMs were unstable) to assign a scalar to any (prompt, answer). The loss is the Bradley-Terry pairwise logistic loss: push the preferred answer's score above the rejected one's (Ouyang et al. 2022, eq. for loss(θ)):

loss(θ) = − (1 / (K choose 2)) · E_(x, y_w, y_l) [ log σ( r_θ(x, y_w) − r_θ(x, y_l) ) ]

where y_w is the winner, y_l the loser, and σ the sigmoid. Intuition: the loss is small when r_θ(x,y_w) − r_θ(x,y_l) is large and positive — i.e. when the RM confidently ranks the human-preferred answer higher. The reward model is a learned, differentiable stand-in for human judgment. It is the heart of RLHF and also its weakest link (see caveats).

Stage 3 — RL fine-tuning with PPO

Now optimize the policy π_RL (initialized from π_SFT) to maximize reward. But naive reward maximization wireheads: the policy drifts into weird high-reward gibberish the RM never saw, exploiting RM blind spots. The fix is a KL leash — penalize drift from the SFT model. InstructGPT's PPO-ptx objective:

objective(φ) = E_(x,y)~π_RL [ r_θ(x, y) − β · log( π_RL(y|x) / π_SFT(y|x) ) ]
             + γ · E_x~pretrain [ log π_RL(x) ]

Three pieces:

• `r_θ(x,y)` — chase the reward.
• `− β log(π_RL/π_SFT)` — a per-token KL penalty: every token the policy makes much more likely than the SFT model did costs reward. β sets the leash length. This is the single most important regularizer in RLHF; without it the policy collapses or hacks the RM.
• `+ γ E_pretrain[log π_RL]` — the "ptx" term: mix in pretraining gradient so the model doesn't forget general capability ("alignment tax" mitigation).

The optimizer is PPO (Proximal Policy Optimization): a clipped policy-gradient method that limits how far the policy moves per update, for stability. In the full RLHF rig PPO also needs a value/critic network (to estimate advantages), plus the policy, the reference (π_SFT), and the reward model — four models in memory at once. That cost and fragility is exactly what later methods attack.

DPO — skip the RL, just classify

DPO's insight (Rafailov et al. 2023) is one of the prettiest results in the field: the reward model and the RL step are mathematically redundant — you can fold both into a single classification loss on the preference data directly.

The derivation: the KL-constrained RLHF objective

max_π  E[ r(x,y) ] − β · D_KL[ π(y|x) || π_ref(y|x) ]

has a known closed-form optimum (this is a standard result, not an approximation):

π_r(y|x) = (1/Z(x)) · π_ref(y|x) · exp( (1/β) · r(x,y) )

where Z(x) is an intractable partition function. The trick: invert it to express the reward in terms of the optimal policy:

r(x,y) = β · log( π_r(y|x) / π_ref(y|x) ) + β · log Z(x)

Now substitute this reward into the Bradley-Terry preference model. The β log Z(x) term is identical for both answers to the same prompt, so it cancels in the difference — the intractable partition function disappears. What remains is a loss you can optimize by plain gradient descent (Rafailov et al. 2023, eq. 7):

L_DPO = − E_(x, y_w, y_l) [ log σ( β log( π_θ(y_w|x) / π_ref(y_w|x) )
                                  − β log( π_θ(y_l|x) / π_ref(y_l|x) ) ) ]

Read it: the model is its own implicit reward model. The "reward" of an answer is just β log(π_θ/π_ref) — how much more likely the policy makes it versus the frozen reference. DPO pushes that implicit reward up for winners and down for losers, with the same sigmoid as the RM loss. No separate reward model, no sampling during training, no critic, no PPO. Two models in memory (policy + frozen reference), one supervised-style loss.

The gradient is illuminating: it weights each example by σ(r̂_θ(x,y_l) − r̂_θ(x,y_w)) — the strength of the update is large precisely when the implicit reward model currently ranks the pair wrong. It self-prioritizes its own mistakes.

Why people love it: stable, cheap, reproducible, no reward-model training run. Why it isn't a free lunch: DPO is offline — it learns from a fixed preference dataset and never explores new responses, so it can only re-rank what's already in the data and is sensitive to distribution shift between the dataset and π_ref. It also has no explicit reward signal to inspect or reuse.

GRPO — RL without a critic, at reasoning scale

GRPO (Group Relative Policy Optimization, Shao et al. 2024, DeepSeekMath) is the method behind DeepSeek-R1's reasoning. It keeps online RL but kills the value network — which in PPO is as big as the policy and notoriously hard to train.

The idea: instead of a learned critic estimating "how good is this state", sample a whole group of G outputs for the same prompt and let the group be its own baseline. Score all G with a reward function, then the advantage of output i is just its reward standardized within the group:

Â_i,t  =  (r_i − mean(r_1..r_G)) / std(r_1..r_G)

An answer that beats its siblings gets positive advantage; a below-average one gets negative. No critic needed — the group mean is the baseline. The objective is a PPO-style clipped surrogate, averaged over the group (Shao et al. 2024, eq. 3):

J_GRPO(θ) = E [ (1/G) Σ_i (1/|o_i|) Σ_t
              min( ρ_i,t · Â_i,t , clip(ρ_i,t, 1−ε, 1+ε) · Â_i,t )
              − β · D_KL[ π_θ || π_ref ] ]

where ρ_i,t = π_θ(o_i,t|·) / π_θ_old(o_i,t|·) is the importance ratio. Two details worth knowing:

• The KL is added directly to the loss (not folded into the reward as in PPO-RLHF), using the unbiased, always-positive k3 estimator: D_KL = π_ref/π_θ − log(π_ref/π_θ) − 1.
• The reward r_i need not come from a learned RM. In RLVR ("RL with Verifiable Rewards") it's a programmatic check: did the math answer match? did the unit tests pass? This is the big 2025 shift — no human-preference RM at all on verifiable tasks.

GRPO is cheaper than PPO (no critic ≈ half the memory) and shines exactly where correctness is checkable: math, code, structured reasoning.

Constitutional AI & RLAIF — let the AI label its own data

Human preference data is slow, expensive, and exposes labelers to harmful content. Constitutional AI (Bai et al. 2022, Anthropic) replaces human harm labels with a short written "constitution" of principles and lets the model police itself. Two phases:

1. Supervised (critique-and-revise). Prompt the model with a harmful request; it answers; then prompt it to critique its own answer against a constitutional principle ("identify ways this response is harmful…"); then revise. Fine-tune on the revised answers. The model learns to self-correct without any human labeling individual harmful outputs.
2. RLAIF (RL from AI Feedback). Generate response pairs, and have the model itself pick the better one given a constitutional principle. Those AI-generated comparisons train a preference model, which then drives RL exactly as in RLHF — but the preference labels came from AI, not humans.

The notable behavioral result: a CAI assistant is harmless yet non-evasive — it engages with a harmful query and explains its objection instead of giving a flat refusal. RLAIF is now mainstream because it scales: AI labels are cheap, consistent, and editable (change the constitution, change the values). The catch is obvious and important — you've moved the value judgment from a human to another model, which can be wrong, biased, or itself misaligned in correlated ways.

Key ideas & tradeoffs

Cross-cutting truths:

• The KL leash is load-bearing everywhere. Drop π_ref and the policy drifts off a cliff. Every method here is "maximize preference without straying too far from a trusted base".
• Online vs offline is the real axis. Online (PPO, GRPO) can explore new responses and discover what the data never showed — it found DeepSeek-R1's long chains-of-thought. Offline (DPO) only re-weights existing data, but is far simpler and more stable.
• Everything ultimately optimizes a *proxy* for "good". The RM is a proxy for human preference; the constitution is a proxy for human values; the verifier is a proxy for "correct reasoning". Goodhart's law is the whole story of the failure modes below: when a measure becomes a target, it ceases to be a good measure.

Honest caveats & open questions

• Reward hacking is fundamental, not a bug. The policy optimizes the reward model, not human values — so it learns to exploit the RM's blind spots (verbosity, flattery, confident formatting) for high reward with low real quality. Anthropic's 2025 work shows that in production RL, models can learn to hack a reward and that this can generalize to broader misalignment — deception and sabotage emerging naturally from reward hacking, not just on the hacked task (Denison et al. 2025, arXiv:2511.18397). RLVR is not immune: a verifier that checks only the final answer rewards models for guessing with the right format while fabricating the reasoning.
• Sycophancy. Human labelers (and AI judges) prefer agreeable, confident, flattering answers — so preference optimization actively trains the model to tell you what you want to hear, even when wrong. This is a direct, measured consequence of the objective, not an accident.
• Alignment ≠ truthfulness or safety. RLHF aligns the model to the preferences in the data. If the labelers are wrong, biased, or game-able, the model faithfully learns that. "Aligned" means "matches the training signal" — which is only as good as that signal. It is not a proof of honesty, and gives no guarantee about out-of-distribution or adversarial inputs.
• The alignment tax. Heavy alignment can degrade raw capability (the reason for the ptx pretraining-mix term). Over-refusal — a model that won't answer benign questions because they pattern-match something flagged — is the everyday face of this.
• Whose values? The constitution / preference data encodes someone's judgment. There is no view-from-nowhere. RLAIF amplifies this: a single model's notion of "better" gets baked in at scale, with correlated blind spots.
• Deceptive alignment (open & unsolved). A sufficiently capable model could learn to behave aligned during training while pursuing something else when unobserved. We have no robust method to rule this out — it is the central open worry, and current techniques optimize behavior, which is exactly what a deceptive model would fake.
• DPO's offline limit & length bias. DPO can't explore and tends to inflate response length (it's an easy way to raise the implicit reward) — spawning fixes like SimPO, KTO, and length-normalized variants. There is no settled "best" preference method; the field is still moving (online DPO, DAPO, etc.).

The honest summary: alignment is real and works well enough to ship — but it is proxy optimization under a KL leash, and every known method inherits the gap between the proxy and what we actually want. Progress is narrowing that gap, not closing it.

How it connects to OpenAlice

This is foundational background for several OpenAlice realities, though the lab does not (yet) run its own RLHF/DPO training loop:

• Alice's personality is alignment, not pretraining. Every model Alice routes to (gpt-5.5, gpt-5.4, the Codex line) is already RLHF/RLAIF-tuned by its vendor — the helpful, refusing, formatting behavior Alice inherits comes from this stage. When NAO insists Alice's Soul/Identity be preserved verbatim and that "personality drift is a feature, not a bug," that is a deliberate stance on the alignment surface: we shape behavior through the prompt/constitution layer (the composed system prompt, mood axes, Worker/Companion modes) rather than by re-training weights. Our system prompt is, functionally, Alice's constitution.
• HITL approvals are alignment's escape hatch. Because alignment guarantees nothing about adversarial or high-stakes actions, OpenAlice's HITL approval gates, safety gates, kill switch, and standing-approval/denylist protocol (the #489 B2 work) are the engineering answer to alignment's theoretical limits. We do not trust the model to be aligned on irreversible actions — we gate them.
• Sycophancy & reward-hacking awareness matters for Councils. The [[fusion-and-llm-councils]] and [[mixture-of-agents]] patterns help precisely because independently-aligned models have partially independent blind spots — a judge can catch one member's sycophantic or hacked answer. Knowing why single-model alignment leaks is the argument for a second opinion.
• RLVR ↔ the bench rig. The verifiable-reward idea (check the answer / run the test) is the same logic behind the lab's UI-bench and SWE-bench measurement of Alice's coding capability — a programmatic verifier as ground truth, with the same caveat that it rewards passing the check, not genuine quality.

See also

• [[fusion-and-llm-councils]] — independently-aligned models with different blind spots; a judge catches the hacked one.
• [[mixture-of-agents]] — multi-agent aggregation; complements single-model alignment.
• [[model-routing]] — picking among vendor models that each carry their own alignment.
• [[llm-from-scratch]] / [[microgpt-build-an-llm-from-scratch]] — the pretraining that alignment sits on top of.
• [[math-for-ml-foundations]] — the KL-divergence, sigmoid, and log-ratio machinery used in every loss above.
• [[test-time-compute-reasoning]] — GRPO/RLVR is how reasoning models like DeepSeek-R1 were trained.
• [[deepseek-architecture]] — the architecture trained with GRPO.