The Three-Stage LLM Pipeline

Lock this mental model before any single technique. Every modern LLM — GPT-5, Claude, Llama, Qwen, Mistral — comes out of a three-stage funnel: pretrain → SFT → alignment. Each stage answers a different question with different data at very different cost.

Rapid Recall

Pretraining answers "what is the next word?" on raw text (~10 trillion tokens, $10M to $100M). SFT answers "how should I answer this instruction?" on (prompt, ideal response) pairs (~10K to 1M examples, $10K to \(1M). Alignment answers "of these two answers, which do humans prefer?" on (prompt, chosen, rejected) triples (\)10K to $10M). In 2026 a fourth stage, RLVR, has emerged for verifiable domains: the reward is a programmatic verifier, not a learned model or human preference. The base model after pretraining is a smart autocomplete; SFT teaches chat format and instruction-following; alignment teaches human preferences and refusal; RLVR teaches verifiable correctness in math, code, and tool use.

§1 The funnel

Stage 1: PRETRAINING       → A giant autocomplete engine (trillions of tokens, billions of $)
       ↓
Stage 2: SFT (SUPERVISED   → Teach it to follow instructions and chat format
           FINE-TUNING)      (thousands of human-written Q&A pairs)
       ↓
Stage 3: ALIGNMENT         → Teach it WHICH of two answers humans prefer
         (RLHF or DPO)       (pairs of responses with "this one is better" labels)
       ↓
Stage 4: RLVR (2025+)      → Optimize against programmatic verifiers in math/code/tools
                             (DeepSeek-R1 recipe; reasoning models in general)

Each stage answers a different question.

Stage	Question it answers	Data type	Cost
Pretraining	"What is the next word?"	Raw text (web, books, code)	$10M to $100M+
SFT	"How should I answer this instruction?"	(prompt, ideal response) pairs	$10K to $1M
Alignment	"Of these two answers, which do humans prefer?"	(prompt, response A, response B, which won) triples	$10K to $10M
RLVR	"Did the math answer come out right? Did the code pass the tests?"	Problems with verifiable answers	Compute, not labels

§2 What each stage actually does to the model

The base model after pretraining is essentially a very smart autocomplete. It does not know it is supposed to be helpful. It does not know "User:" and "Assistant:" are different speakers. It will happily continue offensive prompts.

SFT teaches it the chat format and instruction-following. Mechanically it is just next-token cross-entropy with the loss masked to assistant tokens; see Loss masking. Conceptually it is style transfer, not knowledge transfer. The model already knows everything from pretraining; SFT changes how it behaves.

Alignment teaches it human preferences. What is helpful, what is harmless, what tone humans actually want. The training data is now pairwise: for a single prompt, which of these two answers is better? See Alignment overview.

RLVR (2025-2026 plot twist): Reinforcement Learning from Verifiable Rewards skips human preferences entirely for domains where correctness is checkable: math answer matching, code unit tests, JSON validity, regex format compliance. This is what DeepSeek-R1's <think>...</think> reasoning was trained with; see RLVR.

§3 Pretraining, briefly (because you will not do this)

3.1 The simple story

Take the entire internet. Hide a random word. Ask the model to predict it. Repeat 10 trillion times. The model learns grammar, facts, reasoning patterns, code, and math as a side effect of getting good at next-token prediction.

3.2 Pretraining is compression

If you can predict the next token in Wikipedia perfectly, you have effectively compressed Wikipedia. The weights of a pretrained LLM are a lossy compression of the training data. Everything else the model can do — answer questions, write code, reason — emerges from being really good at this one compression task at sufficient scale.

3.3 The objective (the only math you need to remember)

For a sequence of tokens x₁, x₂, …, xₙ, minimize the negative log-likelihood:

\[ \mathcal{L}_{\text{pretrain}} = - \sum_i \log P(x_i \mid x_1, \dots, x_{i-1} ; \theta) \]

In English: "for every position, maximize the probability the model assigns to the actual next token that appeared in the training data." That is it. Same loss function since GPT-1 in 2018. The magic is entirely in scale and data quality.

3.4 What is new in 2026

• Data quality > data quantity. Frontier labs spend more on cleaning and filtering than on scraping.
• Synthetic pretraining data. A meaningful fraction of training tokens are generated by earlier, smaller models (textbook-style rewrites of web data).
• Longer context from the start. Pretraining at 32K to 128K context windows is now standard instead of extending via RoPE scaling post-hoc.
• Multimodal pretraining is standard. Images and text are interleaved from step 0, not bolted on later.

§4 2026 reality check on the funnel

The frontier is now post-training. Pretraining has mostly plateaued on data and compute returns; most of the quality delta between frontier models in 2026 comes from post-training quality (SFT data curation + alignment recipe + RLVR for reasoning). This is why post-training is the hot skill in AI engineering interviews right now.

Interview Questions

Q1: Walk me through the full post-training pipeline of a modern chat LLM.

Start from a pretrained base model (next-token prediction on trillions of tokens). Step 1: SFT on curated (prompt, response) pairs using chat templates, with loss masking on prompt tokens. This teaches format and basic instruction-following. Step 2: alignment, either RLHF (train reward model on preference pairs, then PPO with KL penalty) or DPO (directly optimize a contrastive loss on preference pairs, skipping the reward model). Step 3, increasingly common in 2026: RLVR for verifiable domains like math and code using deterministic reward signals. DPO or GRPO variants are the open-source default; PPO-style RL still wins at frontier scale.

Q2: Why is pretraining mostly plateauing in 2026?

Two reasons. First, data: most of the high-quality web has been crawled, and synthetic data hits diminishing returns past a certain mixing ratio. Second, compute: the cost of pretraining a new frontier base from scratch is into the hundreds of millions of dollars, and the marginal quality gain over a strong open-weight base is small relative to what good post-training can extract from that base. The action moved to SFT data curation, distillation, RLVR, and reasoning-trace training.

Q3: SFT alone vs SFT + alignment — when is each enough?

SFT alone is sufficient when you have high-quality demonstrations of the exact behavior you want, the domain is narrow (customer support for a specific product, SQL generation, etc.), and you do not need to rank between subtle response variations. SFT alone is not enough when you need to match nuanced human preferences (tone, conciseness, helpfulness tradeoffs), suppress behaviors you did not explicitly demonstrate (refuse harmful requests), or build a general chat model. That is when you stack alignment on top.