Architecture Families and the Task → Architecture Map¶
Attention is just a mechanism. What matters is what you connect via attention. Encoder-only, decoder-only, and encoder-decoder are the three families; ViT, BERT, CLIP, and Whisper are the canonical examples; bi-encoder and cross-encoder are the retrieval/ranking patterns built on top.
Rapid Recall
Three families: encoder-only for understand-then-label (BERT), decoder-only for generate-anything (GPT, Claude), encoder-decoder for source-to-different-target (T5, Whisper). Self-attention has Q=K=V from one sequence; cross-attention has Q from a decoder and K, V from an encoder. Bi-encoder vs cross-encoder is a different concept: bi-encoder encodes inputs separately and compares vectors (fast retrieval, billions of docs); cross-encoder concatenates inputs and encodes jointly (slow, accurate, ~100 candidates). Production RAG = bi-encoder retrieve → cross-encoder rerank → LLM generate.
§1 Vision Transformer (ViT)¶
"What if we just apply a transformer to images?" And it worked, at scale.
The pipeline: split image into 16×16 patches (224² → 196 patches) → linearly project each flattened patch to an embedding (treat each patch as a token) → prepend a learnable [CLS] token → add learned positional embeddings → standard transformer encoder (bidirectional, no mask) → CLS final state → MLP head → class. The only conv-flavored op is the patch projection.
The data-scaling shock
On ImageNet-1K, ViT loses to ResNet. On ImageNet-21K it matches. On JFT-300M it beats ResNet. CNNs hardcode locality, translation-equivariance, and hierarchy as priors that help with small data; with enough data, ViT learns better priors from scratch. DeiT later made ViT work on ImageNet-1K alone via aggressive augmentation plus distillation.
Why ViT mattered most: it made vision interoperable with the transformer ecosystem. Two transformers (vision + language) compose trivially, which is what made modern multimodal LLMs possible. Family: DeiT, Swin (hierarchical, shifted windows), MAE (masked self-supervised), DINOv2, SAM, and CLIP/SigLIP encoders inside every VLM.
§2 BERT — the encoder-only era¶
Bidirectional Encoder Representations from Transformers (Google, Oct 2018). Just the encoder half: bidirectional attention, no causal mask. It established "pretrain once, fine-tune everywhere." Base: 12 layers / 110M; Large: 24 layers / 340M. Post-norm, GELU, WordPiece (~30K vocab), learned positions, plus a segment embedding (sentence A vs B).
Two pretraining objectives¶
Masked LM (MLM): Mask 15% of tokens, predict them from bidirectional context. The famous 80/10/10 split: of the chosen 15%, 80% → [MASK], 10% → random token, 10% → unchanged. Prevents train/inference mismatch ([MASK] never appears at inference) by forcing useful representations for every position.
Next Sentence Prediction (NSP): Predict whether sentence B follows A. Turned out to be a bad objective; RoBERTa removed it and improved. Too easy (mostly topic detection), taught little.
How it is used¶
- Classification:
[CLS]rep → linear → softmax. Loss: cross-entropy. - Token classification (NER): per-token rep → tag (BIO scheme). Loss: per-token CE.
- Extractive QA (SQuAD): two heads predict start and end span positions. Loss: CE on start + CE on end.
- Sentence pairs:
[CLS] A [SEP] B [SEP]→ relation (cross-encoder), or Sentence-BERT bi-encoder for similarity at scale.
Inference: single forward pass, no autoregression, no KV-cache. Fast and cheap. Still used heavily in 2026 for high-volume classification (spam, moderation, ranking, embeddings) where LLMs are overkill. Family: RoBERTa (BERT done right), ALBERT, DistilBERT, ELECTRA, DeBERTa, Sentence-BERT, mBERT.
§3 CLIP — contrastive vision-language¶
Two independent encoders (image: ViT/ResNet; text: GPT-2-style) projecting into one shared vector space. Trained on 400M web (image, caption) pairs. After L2-normalization, similarity is just a dot product. CLIP is a bi-encoder.
Training: for a batch of N pairs, build the N×N cosine-similarity matrix. Symmetric cross-entropy: each image should pick its true caption (row-wise softmax) and each caption its true image (column-wise). A learnable temperature sharpens it.
The full math (InfoNCE / symmetric contrastive):
sim is cosine similarity; τ is a learnable temperature.
Why huge batches matter
Contrastive signal strength scales with negatives per positive. Batch of 32K → each image is contrasted against 32,767 wrong captions. Small batches = weak signal. This is why CLIP-scale training needs massive multi-GPU infra. Zero-shot classification: encode class prompts ("a photo of a {class}") once, encode the image, take the nearest by cosine. No task training. CLIP matched supervised ResNet-50 on ImageNet zero-shot.
Successors: SigLIP (sigmoid loss, no huge-batch need), EVA-CLIP, CoCa, InternVL.
§4 Whisper — speech as seq2seq¶
An encoder-decoder transformer (OpenAI, Sep 2022). Speech recognition is fundamentally seq2seq: audio → text. Trained on 680K hours (V3: 5M+ hours) of weakly supervised web audio.
Preprocessing: raw 16 kHz audio is too long (480K samples / 30s) for O(N²) attention. Convert to a log-Mel spectrogram (80/128 bins, 25 ms windows, 10 ms stride) → (80, 3000). Mel scale matches human pitch perception; log compresses dynamic range. Like patches for an image.
Encoder: two strided 1D convs (downsample to 1500 frames) + sinusoidal positions + transformer layers (bidirectional). Decoder: causal self-attention over generated text + cross-attention to the 1500 acoustic frames + FFN, autoregressive. Loss: cross-entropy on transcript tokens.
Task-controlling special tokens
One model, 99 languages, multiple tasks, via the decoder prompt: <|startoftranscript|> <|en|> <|transcribe|> <|notimestamps|>. Swap <|en|> → <|fr|> for French; <|transcribe|> → <|translate|> to translate to English. Same idea as instruction tuning, but predates it. Skip the language token and the model predicts it (language ID).
Limits: 30-second chunks (boundary issues), hallucination on silence/noise, no speaker diarization, slow autoregressive decode, not streaming. Landscape: Whisper Large V3 / Turbo, Distil-Whisper, NVIDIA Parakeet (low-latency non-AR), SeamlessM4T (speech-to-speech), and audio LLMs (Qwen-Audio, GPT-4o) that make audio a native LLM modality.
§5 Self vs cross attention, bi vs cross encoders¶
Self vs cross attention — same op, different inputs¶
| Self-attention | Cross-attention |
|---|---|
Q, K, V all from the same sequence. The sequence searches itself. Q = K = V = X. Maintains coherence within a sequence. |
Q from sequence A, K and V from sequence B. One sequence searches another. Grounds a decoder in an encoder's output. |
An encoder-decoder decoder block has THREE sub-blocks: (1) masked self-attention over its own past tokens (autoregressive coherence), (2) cross-attention to the encoder output (grounding, no mask), (3) FFN. Decoder-only models (GPT) use only self-attention: the prompt and output are one sequence, no separate encoder to cross-attend to. Cross-attention also powers Stable Diffusion (image queries text), Perceiver, and Flamingo.
Bi-encoder vs cross-encoder — a retrieval pattern¶
These are retrieval/ranking patterns, not attention types. (Note: a cross-encoder internally uses self-attention over the concatenated pair. The shared word "cross" is an unfortunate coincidence.)
| Bi-encoder | Cross-encoder | |
|---|---|---|
| How | Encode the two inputs separately, compare vectors by cosine | Concatenate both inputs, encode jointly with full attention → one score |
| Speed | Very fast (precompute one side) | Slow (one forward pass per pair) |
| Accuracy | Lower | Higher (fine-grained interaction) |
| Precompute? | Yes — store doc vectors | No |
| Scale | Billions of docs | ~100 candidates |
| Use for | Retrieval / candidate generation | Reranking / final scoring |
The production pattern: retrieve → rerank
Bi-encoder retrieves top ~100 from millions in milliseconds (precomputed vectors). Cross-encoder reranks those 100 accurately. You cannot cross-encode millions (a forward pass per pair does not scale). This is Google Search, every RAG system, every modern search stack. CLIP is a bi-encoder; the SQuAD-style [CLS] A [SEP] B setup is a cross-encoder.
§6 Task → architecture decision map¶
Three families, three I/O shapes. The selection rule to burn into memory.
| You have | Use | Trained how |
|---|---|---|
| Text → label | Encoder + classifier head | MLM pretrain → CE on head |
| Text → per-token tag (NER) | Encoder + token head | Per-token CE |
| Text → span (QA) | Encoder + start/end heads | CE on start + CE on end |
| Text → fixed vector | Encoder as bi-encoder | Contrastive (InfoNCE) |
| Pair → similarity score | Encoder as cross-encoder | CE on relation |
| Audio → text | Enc-Dec (Whisper) | CE on target tokens |
| Lang A → Lang B | Enc-Dec | CE, teacher forcing |
| Doc → summary | Enc-Dec or Decoder LLM | CE on summary |
| Prompt → generation | Decoder (GPT) | Next-token pretrain |
| Instruction → answer | Decoder + instruction tuning | Pretrain → SFT → RLHF/DPO |
| Chat with tools | Decoder + function calling | SFT on tool-use traces |
| Tabular → prediction | XGBoost first; transformer only if conditions met | — |
Transformers for tabular data¶
The 2026 verdict
Default to XGBoost. Tree models handle tabular inductive biases better (robust to uninformative features, no normalization needed) and they fit the naturally jagged decision boundaries that NNs oversmooth. Tabular datasets are usually small, where transformers' data hunger hurts.
Transformers (FT-Transformer, TabPFN, SAINT) treat each feature as a token; self-attention learns feature interactions; CLS → prediction. Consider them only when: >1M rows, mixed-modality tabular (free text/images alongside columns), need for row embeddings for downstream retrieval, or temporal/sequential structure. Otherwise XGBoost / CatBoost / LightGBM win on speed, robustness, and deployment simplicity.
§7 Contrastive learning losses (quick reference)¶
Learn representations by pulling similar things together and pushing dissimilar things apart in vector space, instead of predicting a label. Each step needs an anchor, a positive (similar), and negatives (dissimilar, usually the rest of the batch).
| Triplet loss | InfoNCE (modern default) |
|---|---|
L = max(0, d(a,p) − d(a,n) + margin) |
L = −log( exp(sim(a,p)/τ) / Σᵢ exp(sim(a,xᵢ)/τ) ) |
"Anchor should be closer to positive than to negative, by at least margin." If already true → loss 0. Margin prevents collapse to zero distance. Uses one negative per step. Older (FaceNet). |
Cross-entropy in disguise: "pick the positive out of N candidates." Softmax over similarities with the positive as the label. Uses N−1 negatives per step → richer signal. CLIP, SimCLR, embedding models. |
Core mental model
Triplet = "make this pair closer than that pair, by a margin" (pairwise). InfoNCE = "classify the positive out of a lineup" (softmax framing). Same goal, different framing. InfoNCE won because scaling negatives is trivial (bigger batch = more in-batch negatives), giving a much richer gradient. Whenever you see "contrastive loss" today, assume InfoNCE.
Interview Questions¶
Q1 (Trap): ViT is permutation-invariant over patches — does shuffling change the output?
Attention is invariant, but ViT adds positional embeddings. Shuffle patches without their positional embeddings → output changes (each patch now carries a different position). Shuffle both together → invariant, but positions are baked to specific locations during training, so it never happens in practice.
Q2 (Trap): If BERT is bidirectional, can it generate text?
Technically yes (iteratively unmask), but practically no. It was not trained for coherent left-to-right generation. That is why generation became GPT's domain and modern LLMs descend from GPT, not BERT.
Q3 (Trap): Why not one shared encoder for image and text in CLIP?
Images and text have totally different structure and need different architectures. CLIP shares the output space, not parameters. Separate encoders, jointly trained so their vectors are comparable. The shared latent space, not shared weights, is what enables contrastive learning and zero-shot transfer.
Q4 (Trap): Why does Whisper hallucinate on silence?
It is a generative AR language model; it must emit some token. On silent or noisy audio, cross-attention gives an ambiguous signal and the decoder's LM side fills in whatever is most likely from prior text context. Fix with voice-activity detection (VAD) preprocessing and confidence thresholding.
Q5 (Trap): If decoder-only does everything via prompting, why do encoder-only models still exist?
Three reasons: cost/latency (a fine-tuned BERT is 10 to 100× cheaper than prompting an LLM at high volume), embeddings (you need a bi-encoder for fixed-size retrieval vectors), and reliability (extractive QA cannot hallucinate; it must return a span). Encoder-only is the production workhorse; decoder-only is the flexible frontier.