Law of Large Numbers & CLT¶
The two halves of the heart of statistics. The law of large numbers says averaging gets you to the truth, fixing the destination. The central limit theorem says the error around that destination is always normal, fixing the shape and scale of the approach. Together they justify nearly every estimate and error bar in practice.
Rapid Recall
For i.i.d. data the sample mean is unbiased, \(E(\bar X_n)=\mu\), with shrinking variance \(\mathrm{Var}(\bar X_n)=\sigma^2/n\), so it locks onto \(\mu\): the law of large numbers. The strong version holds almost surely, the weak version in probability per \(n\), and Chebyshev proves the weak law in one line. The central limit theorem says \(\sqrt{n}(\bar X_n-\mu)/\sigma \to \mathcal{N}(0,1)\) regardless of the source distribution, because \(\sqrt{n}\) rescaling annihilates every moment past the mean and variance. Error shrinks like \(1/\sqrt{n}\), so four times the data halves it, and finite variance is required (Cauchy breaks it).
§8 Sample Statistics & the Law of Large Numbers¶
For i.i.d. \(X_1,\dots,X_n\) with true mean \(\mu\) and variance \(\sigma^2\), the sample mean is:
The deep question of all statistics: \(\mu\) is an unobservable fact about the world; \(\bar X_n\) is computable from data. How are they related as data grows?
Two facts that power everything¶
| Fact | Statement | Meaning |
|---|---|---|
| Unbiased | \(E(\bar X_n) = \mu\) | aimed at the right target |
| Shrinking variance | \(\text{Var}(\bar X_n) = \dfrac{\sigma^2}{n}\) | spread goes to 0 as \(n\to\infty\) |
The variance fact comes from independence (covariances zero): summing \(n\) variances gives \(n\sigma^2\), and the \(1/n\) squares to \(1/n^2\), leaving \(\sigma^2/n\). As \(n\) grows the sample mean stops wobbling and locks onto \(\mu\).
The Law of Large Numbers¶
| Version | Statement | Why this strength |
|---|---|---|
| Strong | \(\bar X_n\to\mu\) with probability 1 (almost surely) | the whole path settles at \(\mu\) and never escapes |
| Weak | $P( | \bar X_n-\mu |
Why weak is weaker: the weak law promises closeness at each large \(n\) separately; the strong law promises the entire tail of the sequence converges. Strong implies weak, not the reverse. The weak law's proof is one line, Chebyshev on \(\bar X_n\):
Trap, no balancing force (gambler's fallacy)
10 heads in a row does not make tails "due." The streak is not corrected, it gets swamped by the volume of later flips. The average drifts to \(0.5\) because the extra heads become negligible, not because the future compensates.
Where LLN lives. Life: casinos and insurance (tiny per-bet edge gives guaranteed profit over millions of bets), polling. ML: Monte Carlo estimation is the LLN; mini-batch SGD (batch gradient is a sample-mean estimate of the true gradient, variance \(\sigma^2/n\)); empirical risk minimization (minimizing average loss approximates expected loss, the reason finite data generalizes); test-set evaluation. LLN says averaging gets you to the truth.
§9 The Central Limit Theorem¶
The LLN says \(\bar X_n\to\mu\) but nothing about the shape of the error. The CLT fills that in:
Take the sample mean, subtract the truth, magnify by \(\sqrt{n}\), and the distribution of that error is always a standard normal, regardless of the source distribution.
The magic, two astonishing claims. 1. Universality. The \(X\)'s can be anything, uniform, exponential, coin flips, lopsided custom. Average enough and the shape is always the same bell. The source's identity is erased. The bell is an attractor: sums flow toward it. Why? Adding many independent things averages out their idiosyncrasies (skew, lumps), leaving only the most generic symmetric shape consistent with a fixed mean and variance, the max-entropy normal. 2. The \(\sqrt{n}\) rate. Error shrinks like \(1/\sqrt{n}\): to halve error you need 4 times the data. This is the standard error \(\sigma/\sqrt{n}\) behind every confidence interval and A/B sample-size calculation.
LLN versus CLT, complementary zoom levels¶
| What it tells you | |
|---|---|
| LLN | the destination: \(\bar X_n\to\mu\) (where it lands) |
| CLT | the shape of the approach: error times \(\sqrt{n}\) is normal, width \(\sigma\) |
LLN sees \(\text{Var}(\bar X_n)=\sigma^2/n\to 0\) (convergence); CLT rescales by \(\sqrt{n}\) to hold the variance at \(\sigma^2\) and reveal the normal underneath.
Why it works, the MGF proof¶
Standardize (\(\mu=0,\sigma=1\)), \(S_n=\sum X_j\). Independent sums multiply MGFs:
Take logs (the form is \(1^\infty\)), substitute \(y=1/\sqrt n\), apply L'Hopital twice; the two derivatives pull down \(M'(0)=0\) and \(M''(0)=1\) (the first two moments):
What the math reveals. The proof used only the first two moments. Skew, kurtosis, and all higher moments got divided away by powers of \(\sqrt{n}\) and vanished. That is the mathematical reason the source's identity disappears, only mean and variance survive, and those define a unique normal.
Caveat, finite variance required
The Cauchy distribution (no mean or variance) breaks the CLT: averaging Cauchys returns a Cauchy, never a normal. "Just average it, it will be normal" hides the finite-variance assumption; heavy-tailed data can violate it.
Normal approximation in practice¶
A \(\text{Bin}(n,p)\) is a sum of \(n\) Bernoullis, so for large \(n\) it is \(\approx\mathcal{N}(np,\,np(1-p))\):
| Approximation | When |
|---|---|
| Poisson | \(n\) large, \(p\) small, \(\lambda=np\) moderate (rare events) |
| Normal | \(n\) large, \(p\) near \(\tfrac12\) (symmetric) |
Continuity correction: approximating discrete with continuous, use \(P(X=a) = P(a-\tfrac12 < X < a+\tfrac12)\), smear the spike over a unit interval. Worked example, with \(X\sim\text{Bin}(100,0.1)\): \(np=10\), \(np(1-p)=9\), SD \(=3\), and \(P(X\le 15)\to P(X<15.5)\):
A Chebyshev sample-size example, with \(\mu=0.5,\ \sigma^2=0.25\), \(\text{Var}(\bar X_n)=0.25/n\):
Where CLT lives. Every confidence interval and error bar (estimate \(\pm 1.96\,\sigma/\sqrt n\)), all of hypothesis testing and A/B testing (the test statistic is normal because of the CLT, why z-scores and p-values work without knowing the data distribution), sample-size planning, and the justification for Gaussian noise assumptions.
Interview compression. LLN: sample mean goes to truth because \(\text{Var}(\bar X_n)=\sigma^2/n\to 0\). Weak is in probability per \(n\); Strong is almost-sure whole path. No balancing force, streaks get swamped. CLT: sums of many finite-variance independents are normal regardless of source, because \(\sqrt n\) rescaling annihilates every moment past mean and variance. Error is about \(1/\sqrt n\) (4 times the data to halve it). LLN gives the destination; CLT gives the shape and scale of the approach.
Interview Questions¶
Q1: What does the law of large numbers guarantee, and what does it not? It guarantees that the sample mean of i.i.d. data converges to the true mean as \(n\) grows, because the sample mean is unbiased with variance \(\sigma^2/n\) shrinking to zero. It does not say anything about the shape of the error at finite \(n\), and it provides no balancing force: a run of heads is not corrected, it is simply swamped by later data.
Q2: State the central limit theorem and the two things that make it remarkable. The CLT says \(\sqrt{n}(\bar X_n-\mu)/\sigma\) converges to a standard normal as \(n\to\infty\). It is remarkable first for universality, the limiting shape is normal regardless of the source distribution, and second for the \(1/\sqrt{n}\) rate, which means you need four times the data to halve the standard error.
Q3: Why does the CLT depend only on the mean and variance of the source? In the MGF proof, standardizing and raising the per-term MGF to the \(n\)-th power, then taking logs, leaves a limit determined by \(M'(0)=0\) and \(M''(0)=1\). The \(\sqrt{n}\) rescaling divides away every higher moment, so skew and kurtosis vanish and only the first two moments survive, which is why the source's identity disappears.
Q4: When does the central limit theorem fail? When the source distribution does not have finite variance. The Cauchy distribution has no mean or variance, so averaging Cauchy variables returns another Cauchy rather than converging to a normal. Heavy-tailed real data can approach this regime, so "just average it" silently assumes finite moments.