Skip to content

PEFT: LoRA and QLoRA

Parameter-Efficient Fine-Tuning is the standard 2026 stack for adapting large LLMs to a specific domain with limited data and compute. LoRA gives you ~0.1% of trainable parameters; QLoRA combines LoRA with 4-bit NF4 quantization to put a 70B model on a single 48GB GPU; paged optimizers, gradient accumulation, and gradient checkpointing handle the remaining memory pressure.

Rapid Recall

Fine-tuning a huge model fully is wasteful because the useful weight change ΔW is empirically low-rank — it lives in a tiny subspace. LoRA exploits this by learning ΔW = BA with thin matrices, training ~0.1% of params and merging at the end for zero inference cost. QLoRA stacks 4-bit NF4 quantization of the frozen base on top, and paged optimizers + gradient accumulation + gradient checkpointing handle the remaining memory pressure so a 70B model fine-tunes on a single 48GB GPU.

How the pieces connect

PEFT goal adapt big model, tiny cost ΔW is low-rank the assumption that makes it valid LoRA = BA init · scaling · target layers QLoRA + 4-bit NF4 frozen base Paged optimizer optimizer state ↔ CPU Grad accumulation big batch in pieces Grad checkpoint recompute activations
The dependency graph: low-rank assumption underpins LoRA; QLoRA adds 4-bit base; three memory tricks make a 70B model fit on one GPU.

Pages in this section

  • LoRA mechanics — the ΔW = BA decomposition, the forward pass, the parameter saving, and the merge trick.
  • Why low-rank — the two papers that prove ΔW is low-rank, plus the chef analogy that makes it intuitive.
  • Initialization and gradient flow — why B = 0 and A = random, and why everyone gets the "no gradient flows" reasoning wrong.
  • Target modules — the six attachable matrices (Q, K, V, O, W_up, W_gate, W_down) and the decision dial for which to use.
  • Quantization and NF4 — number formats, why uniform INT4 fails for NN weights, how NF4 wins, double quantization, BF16 vs FP16.
  • QLoRA assembled — the full memory budget for 70B on 48GB and the production code.
  • Memory tricks — paged optimizers, gradient accumulation, gradient checkpointing.

For the runnable LoRA SFT walkthrough on TinyLlama-1.1B, see SFT Walkthrough Part B.