Deep Ranking Architectures & Multi-Task Learning¶
The ranking stage is the cross-encoder: user + item + context features interact jointly, recovering what two-tower's separation threw away. This page covers the feature-interaction lineage — FM → Wide&Deep → DeepFM → DCN-v2, each solving the same problem in escalating ways — and multi-task learning (Shared-Bottom, MMoE, PLE), because a single objective is a lie.
Rapid Recall
Every deep ranking architecture solves one problem: automatically learn which feature interactions (conjunctions like "Hyderabad × Telugu") matter. FM gives each feature a latent vector and defines a pair's weight as their dot product (MF generalized to all features), pairwise only. Wide&Deep = memorization (hand-crossed linear) + generalization (embedding MLP), but the wide side still needs manual crosses. DeepFM swaps the wide part for an FM (shared embeddings, no hand-crafting). DCN-v2 stacks explicit cross layers for arbitrary bounded-order crosses. Multi-task predicts P(click), P(watch), P(like)… jointly because tasks share signal; MMoE gives each task its own gate over a shared expert pool to fix negative transfer; PLE adds task-specific experts. Keep training-loss weights (how it learns) separate from serving-fusion weights (business knobs, retuned via A/B without retraining).
§1 The feature-interaction lineage¶
The one problem they all solve
Ranking data is dominated by massive sparse categoricals (user_id, item_id, device, city, hour). Signal lives in their conjunctions: "user_from_Hyderabad" weak, "item_is_Telugu_film" weak, but "Hyderabad × Telugu" strong. Linear models can't capture products; hand-crafting crosses is combinatorially infeasible. So the entire lineage = different mechanisms for learning interactions automatically.
Factorization Machines (FM) — the foundation¶
Naive: a weight \(w_{ij}\) per feature pair — but sparse pairs never co-occur, so you can't learn them. FM's trick: give each feature a latent vector \(v_i\); the pair's interaction weight = the dot product \(v_i\cdot v_j\).
Linear part + pairwise interaction part where each pair's strength is a dot product of learned embeddings. Even if "Hyderabad × rare-item" never co-occurred, you learned \(v_{\text{Hyderabad}}\) from OTHER items and \(v_{\text{rare-item}}\) from OTHER users → generalize to the unseen pair. This is literally MF generalized to arbitrary features (MF = the special case of only user_id & item_id). Algebraic rearrangement makes the sum \(O(kn)\), not \(O(kn^2)\).
Limitation: only pairwise (degree-2). Can't do three-way or nonlinear. That ceiling drives everything after.
Wide & Deep (Google 2016) — two complementary halves¶
- Wide (memorization) — linear model on raw + manually crossed features. Exact frequent conjunctions. Bad at unseen combos.
- Deep (generalization) — embed categoricals → concat → MLP. High-order nonlinear interactions, generalizes via embeddings. Can over-generalize.
- Joint training — outputs summed before the sigmoid; each half covers the other's failure.
Trade-off: the wide part still needs manual feature engineering. The next two automate it.
DeepFM (Huawei 2017) — replace the wide part with an FM¶
Wide & Deep's wide side needs hand-crafted crosses; FM learns pairwise crosses automatically → swap the linear wide part for an FM. Two key points: (1) the FM side and deep side share the same embeddings (one table, two consumers); (2) no manual feature engineering at all — FM handles explicit pairwise, MLP handles high-order implicit. A very common production default.
DCN / DCN-v2 (Google 2017/2020) — explicit, arbitrary-order crosses¶
FM only does degree-2. DCN's cross network explicitly raises interaction order one degree per layer:
Each cross layer multiplies by the original input \(x_0\) again → polynomial degree grows by one per layer, explicitly and cheaply, with bounded parameters (no combinatorial blowup). Stack 3 layers → up to degree-4 explicit interactions. A deep MLP runs in parallel for implicit nonlinear interactions; combine at the end. DCN-v2 made the cross layer a matrix (more expressive) — a strong modern choice.
| Model | Learns interactions via | Manual eng? | Order | Identity |
|---|---|---|---|---|
| FM | embedding dot products | No | pairwise only | MF generalized to all features |
| Wide & Deep | hand crosses + MLP | ⚠️ Yes (wide) | high (implicit) + manual | memorization + generalization |
| DeepFM | FM + MLP, shared embeddings | No | pairwise explicit + high implicit | Wide&Deep minus hand-crafting |
| DCN-v2 | explicit cross layers + MLP | No | arbitrary explicit + high implicit | explicit bounded high-order crosses |
When to use what — and 'should we combine?'
They are already combinations (explicit-interaction module + deep MLP). The field converged on this. DeepFM / DCN-v2 = strong no-hand-crafting defaults (DCN-v2 more modern). Wide & Deep when you have valuable known crosses + huge frequent-pattern data (app stores). FM alone for latency-tight/smaller settings. This is the cross-encoder stage — user+item+context features interact jointly, recovering what two-tower's separation threw away.
§2 Multi-task learning¶
A single objective is a lie. Optimizing pure clicks gives clickbait. So predict many objectives at once: P(click), P(watch>30s), P(like), P(share), P(complete).
Why jointly, not N separate models¶
- Business: the final score combines these, so you need all per item.
- Statistical (the deeper reason): tasks share signal. Shared layers learn richer representations; data-rich tasks (clicks) help data-poor tasks (shares, purchases) via the shared bottom — transfer/regularization across tasks.
Shared-Bottom (the baseline)¶
One shared trunk → small per-task heads. Trained jointly; total loss = weighted sum of per-task losses.
The failure → why MMoE exists
One representation serves all tasks. When tasks conflict (click vs. complete-watch — clickbait clicks vs. genuine completion), the bottom must compromise and both suffer. This is negative transfer / the seesaw effect. Shared-bottom can't give different tasks different representations.
MMoE — Multi-gate Mixture-of-Experts (Google 2018)¶
A pool of expert MLPs (shared), and one gating network per task (a softmax over experts) deciding how much each task uses each expert.
The experts are a shared pool; differentiation happens at the gate level. Correlated tasks share experts; conflicting tasks route differently → fixes negative transfer. PLE (Tencent 2020) refines this with explicit shared and task-specific experts.
Two different weightings — keep them distinct
(1) Training loss weights: \(\mathcal{L}=w_1\mathcal{L}_{click}+w_2\mathcal{L}_{watch}+\dots\) — balance how the model learns each task (hyperparameters / uncertainty-weighting).
(2) Serving fusion weights: combine the predicted scores to sort by — often a weighted product (soft AND; an item that fails a critical objective gets killed):
\(\alpha,\beta,\gamma\) are business knobs, tuned by product priorities + A/B tests, NOT gradient descent. When strategy shifts ("push watch-time this quarter"), retune these — no retraining. That decoupling is exactly why you predict tasks separately and fuse at the end.
Placement: multi-task is a ranking-stage concern (precision + business objectives + affordable compute). Retrieval stays single-objective (just recall).
Interview questions¶
Q1: What's FM's trick for feature pairs that never co-occurred? Don't learn a per-pair weight; give each feature a latent vector and define the pair's interaction as the dot product of the two vectors. Each vector is learned from that feature's co-occurrence with OTHER features, so you generalize to unseen pairs.
Q2: What limitation of Wide & Deep does DeepFM remove, and how? Wide & Deep's wide side needs hand-crafted cross features. DeepFM replaces the linear wide part with an FM (automatic pairwise crosses) sharing embeddings with the deep MLP — removing all manual feature engineering.
Q3: What does each DCN cross layer do to interaction order, and why better than FM? Each layer raises the polynomial interaction degree by one (explicitly, with bounded parameters). FM is capped at degree-2; DCN reaches arbitrary explicit order by stacking layers.
Q4: What is negative transfer / the seesaw effect, and which architecture causes it? When tasks conflict, optimizing one drags another down because they share one representation. Shared-bottom causes it — a single trunk must compromise across all tasks.
Q5: What does "multi-gate" buy over a single shared gate? Each task gets its own gate (softmax over a shared expert pool), so correlated tasks learn to share experts while conflicting tasks route to different ones — giving each task an effectively different representation and avoiding negative transfer.
Q6: Why keep serving-fusion weights separate from training-loss weights? Fusion weights (α,β,γ on predicted scores) encode business priorities and can be retuned via A/B tests when strategy shifts — without retraining the model. Loss weights govern how the model learns. Decoupling lets product re-prioritize objectives instantly.