Multi-Channel Retrieval, ANN & MIPS¶
"Two-tower does retrieval" is sloppy. The model produces vectors; ANN does the retrieval — they are different layers. And no single channel has good recall alone, so production runs many channels in parallel and unions them. This page separates the vector factory from the lookup, lays out the typical retrieval channels (not all of them ANN), and settles the dot-product-vs-cosine / MIPS question.
Rapid Recall
The model is the vector factory; ANN (HNSW/IVF/ScaNN/FAISS) is the lookup. Recall is additive, so production fires many channels in parallel — two-tower, item-to-item, trending, fresh, co-visitation — and unions + dedupes them; total latency ≈ the slowest channel, not the sum. Not every channel is ANN (trending is a Redis sorted list, fresh is a filtered DB query). Match the index metric to the training objective: trained on raw dot product → MIPS (magnitude carries popularity/confidence signal you mustn't normalize away); trained on cosine → normalize + cosine index. The 2026 default for contrastive/softmax towers is normalize → cosine with a tuned temperature τ.
§1 The division of labor¶
- Offline (build representations): MF → item vectors \(Q\); two-tower → item vectors \(v\); content-based → item feature vectors. These algorithms' job ends at "here is a vector per item." They retrieve nothing.
- Online (actually retrieve): take the query vector (user vector \(u\), live), find highest dot products. That search is ANN (HNSW/IVF/ScaNN/FAISS). The model is the vector factory; ANN is the lookup.
Online flow for an embedding channel¶
- Compute user vector (user tower forward pass, ~1 ms) or look up \(p_u\).
- ANN query: "~500 items with highest dot product to \(u\)" — HNSW visits a few thousand nodes, not millions (~10 ms).
- Return ~500 candidate IDs + scores.
§2 Multi-channel retrieval¶
The premise of multi-channel
No single channel has good recall alone. A two-tower model captures one notion of relevance; it misses fresh items, "because you watched X," trending. So production runs many channels in parallel and unions the results — recall is additive; each channel patches the others' blind spots.
Typical channels — and not all are ANN¶
| Channel | Mechanism |
|---|---|
| Two-tower (personalized embedding) | ANN — query = user vector |
| Item-to-item ("because you watched X") | ANN — query = the item vector, not the user |
| Trending / popular | Key-value lookup (precomputed sorted list in Redis) |
| Fresh / recency | Filtered DB query |
| Collaborative co-visitation | Precomputed item-item co-occurrence table |
| Continue-watching / social | State lookup / graph query |
The nuance: ANN is a retrieval mechanism — the one for vectors — not the only one. Non-personalized channels are often just sorted lists.
Production mechanics¶
- Run in parallel — independent async calls. Total latency ≈ the slowest channel, not the sum. That's how 6 channels stay under 50 ms.
- Per-channel top-k budgets — tuned via A/B tests on downstream engagement.
- Union + dedupe — an item from 3 channels collapses to one; keep which channels fired it as a feature for the ranker.
- Cheap pre-filter — drop already-watched, region-locked, sold-out before the expensive ranker.
§3 Dot product vs cosine, and MIPS¶
Cosine = dot product after L2-normalizing both vectors. So cosine throws away magnitude and keeps only direction. The question is whether magnitude carries signal.
- Trained with raw dot product → magnitude is meaningful (often encodes popularity/confidence). Normalizing to cosine deletes that → you'd retrieve under a different objective than you trained on → train/serve mismatch. So you use MIPS (Maximum Inner Product Search) to search under the exact dot product the model optimized.
- You CAN normalize and use a cosine index — legitimate and common — but only correct if the model was trained with cosine (on normalized vectors), where magnitude was never meant to matter.
The rule
Match the index metric to the training objective. Trained on dot product → MIPS. Trained on cosine → normalize + cosine index. MIPS exists for the dot-product case because you can't normalize away magnitude when magnitude was part of the learned signal. (Aside: pure inner product isn't a true metric — no triangle inequality — so older ANN methods needed a transform; modern indexes like HNSW/ScaNN handle MIPS natively.)
What do modern two-towers actually do? (2026)¶
Both exist, but the common modern pattern is normalize, then use a tuned temperature:
- Contrastive / sampled-softmax towers (YouTube, Google lineage) typically L2-normalize both vectors → cosine, then divide by a temperature τ before softmax. Normalization stabilizes training; τ restores the sharpness normalization removes.
- Some keep plain dot product, deliberately retaining magnitude as popularity/confidence signal.
Rule of thumb: contrastive/softmax-trained towers → usually normalized cosine + temperature (the dominant 2026 default); older or magnitude-sensitive setups → raw dot product.
Interview questions¶
Q1: Why is total multi-channel latency the max of the channels, not the sum? Channels are fired as independent, concurrent async calls, so they run in parallel — you wait only for the slowest one to return, not for each in sequence.
Q2: Name two channels that don't use ANN and their mechanism. Trending/popular → key-value lookup of a precomputed sorted list (Redis). Fresh/recency → filtered DB query. (Also: co-visitation → precomputed item-item table; continue-watching → state lookup.)
Q3: In "because you watched X," what is the query vector vs. the main two-tower channel? The query is the last-watched item's vector (item-to-item ANN), not the user vector. The main two-tower channel queries with the user vector.
Q4: Why MIPS instead of just normalizing to cosine? Cosine = dot product after L2-normalization, which discards magnitude. If the model trained with raw dot product, magnitude carries signal (popularity/confidence); normalizing deletes it → train/serve mismatch. MIPS searches under the exact dot product trained on. (You can normalize + cosine-index, but only if the model was trained with cosine.)