Skip to content

Alignment

After SFT your model answers questions, but it may answer them rudely, verbosely, unsafely, or just averagely. Alignment teaches it which of two answers humans prefer. The pages in this section walk the full historical arc: Bradley-Terry reward models, PPO with all four models, the DPO derivation that killed three of them, GRPO which killed the fourth, and RLVR which kills the reward model entirely for verifiable domains.

Rapid Recall

Alignment optimizes the constrained objective E[r(x,y)] − β·KL(π‖π_ref). RM + PPO is the classical RLHF stack: Bradley-Terry RM, clipped surrogate, four models in memory (policy, reference, RM, value). DPO uses the closed-form optimal policy to rewrite the objective as supervised contrastive loss; no RM, no rollouts, two models. GRPO keeps PPO's rollouts and clipping but drops the critic by using the group mean reward as baseline; it is the post-2024 reasoning-model default. RLVR is GRPO with a programmatic verifier in place of the RM, the DeepSeek-R1 recipe. The 2026 frontier pipeline chains pretrain → SFT → RFT → RLVR via GRPO → RLHF/DPO. GRPO does not replace stages; it is a powerful new tool added to the stack.

The 2026 pipeline

PRE-TRAINING 10-15T tokens, knowledge lives here MID-TRAINING synthetic + high-quality annealing SFT instruction following + tool use + format RFT (rejection sampling) keep verifier-passing traces, 1-3 rounds RLVR via GRPO reasoning, math, code, agents RLHF / DPO / RLAIF safety, style, helpfulness, refusals knowledge capability alignment
Each stage solves a different problem; the order matters. GRPO is inserted as a powerful new stage — it does not remove any of the others.

The alignment problem

After SFT your model answers questions. But it may answer them rudely, verbosely, unsafely, or just averagely. How do you teach it to give preferred responses?

Key insight: humans find it much easier to compare two answers than to write the ideal answer from scratch. So instead of collecting more (prompt, ideal response) pairs, you collect (prompt, response A, response B, which was better) preference pairs. Now the question is how to train on preference data.

Pages in this section

  • Reward models and Bradley-Terry — the 1952 statistical model that backs both RLHF and DPO.
  • RL from zero and PPO — RL vocabulary, mapping onto LLMs, the four PPO models, clipped surrogate, KL penalty.
  • DPO — the closed-form derivation that removed the RM and the RL loop.
  • GRPO — group-relative advantages and why DeepSeek-R1 used it.
  • RLVR — verifiable rewards in math, code, and tool use.
  • The GRPO hype, decoded — what R1-Zero proved, what it did not, and the calibrated take.

For the runnable implementation of every method on a toy task, see Alignment Walkthrough.