Evaluating Recommenders¶
You can't A/B test every change. Offline is the cheap filter on logged data; online A/B testing is the real verdict. This page covers the offline metrics split by stage (set metrics for retrieval, order-aware metrics for ranking), how to evaluate each stage separately, the online metrics that decide, and the offline–online gap that makes offline a filter and never a verdict.
Rapid Recall
Offline filters, online decides. Retrieval is judged by Recall@K (set membership, order-blind — its job is "don't miss"); ranking by order-aware metrics — NDCG@K (the king: order-aware via log discount + graded relevance + normalized to [0,1]), MAP (binary, all relevants), MRR (first hit only), and GAUC (per-user AUC, the RecSys refinement over plain AUC). Evaluate each stage on its own metric, but a stage can look great in isolation and hurt end-to-end. Online metrics (CTR, conversion, watch time, retention, GMV) measured via A/B testing are the real verdict. The offline–online gap is real and directional: offline data is biased by the collecting system and gives novel models zero credit for surfacing buried items, so it systematically underrates them.
Cross-reference. The core ranking-metric mechanics (NDCG's four layers, the precision@k → AP → MAP ladder, MRR) are derived in full on the evaluation reference page Ranking & Recommendation Metrics. This page is the RecSys-specific view: which metric goes with which stage, and the offline–online gap.
§1 Offline — order-unaware (set metrics) → for RETRIEVAL¶
- Recall@K — of the items the user actually liked, what fraction made it into top-K? The retrieval metric. Retrieval's job is "don't miss"; order within K doesn't matter (ranking sorts later).
- Precision@K — of your top-K, what fraction relevant? Less used for retrieval (which tolerates false positives).
- Hit Rate@K — did at least one relevant item appear? Coarse, for next-item tasks.
§2 Offline — order-aware (ranking metrics) → for RANKING¶
- MRR — rank of the first relevant item, score = 1/rank, averaged. Use when there's essentially one right answer. Ignores everything after the first hit.
- MAP — average Precision@k at each position where a relevant item appears, averaged over queries. Rewards putting all relevant items high. Binary relevance only.
NDCG@K — the king (built up in three pieces)¶
(1) Cumulative Gain = sum of top-K relevances, but order-blind. (2) Discounted CG = divide each by \(\log_2(i+1)\) — position 1 counts fully, position 10 barely. The log discount IS what makes it order-aware. (3) Normalized = divide by IDCG (the perfect ordering's DCG) → NDCG ∈ [0,1], comparable across queries. The log shape encodes rapidly decreasing user attention down the list.
Why NDCG dominates: order-aware (discount) + graded relevance (rel = 0/1/2/3, not just binary — MAP/MRR can't) + normalized/comparable. "@K" = only evaluate the top K users actually see.
| Metric | Order-aware? | Graded? | Best for |
|---|---|---|---|
| Recall@K | No | No | Retrieval (don't miss) |
| Precision@K | No | No | set quality |
| MRR | Yes | No | one right answer |
| MAP | Yes | No | all relevants, binary |
| NDCG@K | Yes | Yes | Ranking — the default |
| AUC / GAUC | Yes (pairwise) | No | pointwise CTR models |
AUC = P(model scores a random positive above a random negative) — pairwise ranking quality of a classifier; natural for pointwise P(click). GAUC (grouped: per-user AUC, impression-weighted) is the RecSys refinement — plain AUC can look good just separating heavy from light users rather than ranking within each user's list. Knowing GAUC over AUC signals depth.
§3 Evaluating stages separately¶
- Retrieval → Recall@K (+ candidate coverage/diversity): did held-out positives appear in candidates? Order ignored.
- Ranking → NDCG@K / MAP / GAUC on the candidate set's ordering.
- Re-ranking → list-level / beyond-accuracy: intra-list diversity, coverage, novelty, calibration error (+ accuracy to confirm you didn't tank relevance).
- End-to-end → online metrics.
The trap
A stage can look great in isolation and hurt end-to-end. Amazing Recall@K is useless if the added items confuse ranking. Stage metrics are necessary but not sufficient.
§4 Online metrics — the real verdict¶
- Engagement: CTR, conversion rate, watch time, session length, dwell, completion, return rate / DAU.
- Business: revenue, GMV, revenue-per-session, retention, ad revenue.
- Health / long-term: consumption diversity, catalog coverage, long-term retention, churn.
Measured via A/B testing (online controlled experiments) — split traffic, run to significance, compare. The only real verdict.
Proxy vs true objective (again)
You optimize a short-term proxy (CTR) but care about a long-term goal (retention/satisfaction), and they diverge — optimizing CTR can raise clicks while lowering satisfaction (clickbait). Mature systems run long-horizon A/B tests + guardrail metrics (next-day retention, report rate). Same proxy-vs-objective tension that justified multi-task and re-ranking.
The offline–online gap (senior-level point)
Offline NDCG can improve while the A/B test loses. Why: (1) offline data is biased by the system that collected it (position/popularity); you reward agreeing with the old model. (2) Counterfactual problem — you only have labels for items shown; a model surfacing items the old one buried gets zero credit offline (they look like negatives), yet may win online. Offline systematically underrates novel models. (3) offline is static; online, recommendations change behavior (novelty, fatigue). So: offline NDCG is a filter, not a verdict.
Interview questions¶
Q1: Why Recall@K for retrieval and not NDCG? Retrieval's job is "don't miss good items" — whether a relevant item is in the candidate set, regardless of its position (ranking sorts later). NDCG measures ordering, which isn't retrieval's concern; using it would measure the wrong thing.
Q2: Walk the three letters of NDCG. Cumulative Gain = sum of top-K relevances (order-blind). Discounted = divide each by \(\log_2(i+1)\) so lower positions count less — this adds order-awareness. Normalized = divide by IDCG (perfect ordering) → [0,1], comparable across queries.
Q3: Two things NDCG handles that MAP/MRR cannot? Graded relevance (rel = 0/1/2/3, not just binary) and normalization for cross-query comparability.
Q4: Counterfactual reason a better-offline-NDCG model loses online. Offline you only have labels for items previously shown. A new model surfacing items the old one buried gets zero credit offline (they look like negatives), yet those may be the online wins — so offline systematically underrates genuinely novel models.