CDF, Expectation & Variance¶
The cumulative distribution function accumulates probability, expectation summarizes a distribution's center, and variance measures its spread. These three tools work identically across discrete and continuous worlds, and together they turn a distribution into the numbers that decisions and loss functions ride on.
Rapid Recall
The CDF \(F_X(x)=P(X\le x)\) is a staircase for discrete variables (jump size equals the PMF) and a smooth integral of the density for continuous ones; it is nondecreasing, right-continuous, and limits to 0 and 1. Expectation \(E(X)=\sum x\,P(X=x)\) is a probability-weighted average, and its most important property is linearity, \(E(X+Y)=E(X)+E(Y)\), which holds even under dependence because the joint structure never enters. Variance \(\text{Var}(X)=E[(X-E[X])^2]=E[X^2]-(E[X])^2\) is the average squared distance from the mean, squared so deviations do not cancel, and standard deviation is its square root in the original units.
§7 Cumulative Distribution Function (CDF)¶
Discrete: a staircase. Flat between values, jumps at each value, jump size equals the PMF there:
Continuous: a smooth rising curve. No jumps; mass is smeared, so the CDF integrates the density:
The three properties of a valid CDF.
- Monotonically nondecreasing, a running total only adds mass, never removes it.
- Right-continuous, the "\(\le\)" includes the point \(x\), so a jump's mass is counted at \(x\) (the step is drawn filled at the top).
- Limits to 0 and 1:
Independence, now for random variables.
Discrete equivalent: \(P(X=x,\,Y=y)=P(X=x)\,P(Y=y)\). Knowing \(Y\) tells you nothing about \(X\); the joint factors into the product.
A worked staircase, valid because it is nondecreasing \(0\to0.2\to0.7\to1.0\) and ends at 1:
Cross-check: \(P(X=1)+P(X=2)=0.5+0.3=0.8\). (Subtracting \(F_X(0)\) removes the \(X=0\) mass the strict "\(0<\)" excludes.)
§8 Expected Value & Its Properties¶
A probability-weighted average: each value times its probability, summed. \(\text{Im}(X)\) is the set of values \(X\) can take.
The weighted-average intuition. Average of \(1,1,1,1,1,3,3,5\) regrouped by value:
That is the expectation formula, the average with probabilities as weights.
Bernoulli and the indicator bridge.
If \(X\) is the indicator of event \(A\), then \(X\sim\text{Bern}(P(A))\) so \(E(X)=P(A)\).
Linearity of expectation, the most important property¶
Why it is true:
Splitting a sum into two needs no assumption about how \(X,Y\) relate, the joint structure never enters.
Payoff: \(E[\text{Bin}(n,p)]=np\) in one line.
And under dependence, the expected aces in a 5-card hand:
- The one-number summary of a distribution's center; decisions ride on averages (expected revenue, loss, latency).
- Every ML objective: training minimizes expected loss (risk). Empirical risk equals average loss equals empirical expectation.
loss.mean()is an expectation. - Variance, covariance (PCA), entropy and cross-entropy are all expectations.
- RL: the value of a state is expected cumulative reward.
§15 Variance & Standard Deviation¶
Why we need it. "Average return 8%" describes both a steady bond and a wild bet that is usually -50% or +66%. The mean hides risk; variance exposes it. In ML: the bias-variance tradeoff, gradient noise, prediction uncertainty, metric volatility across runs.
The average squared distance from the mean.
- Why not \(E[X-E[X]]\)? It is always 0, since positive and negative deviations cancel:
Squaring kills the cancellation.
- Why not \(E|X-E[X]|\)? It is valid (mean absolute deviation) but absolute values are not differentiable at 0 and are algebraically ugly. Squaring is smooth and plays beautifully with calculus and linearity.
The computational form (the one you use).
The mean of the squares minus the square of the mean. Derivation via linearity (treat \(E[X]\) as the constant it is):
Standard deviation.
Variance is in squared units (dollars squared); the square root returns interpretable original units, the "typical distance from the mean." Variance for math, standard deviation for human-readable spread.
Interview Questions¶
Q1: State the three properties that define a valid CDF. A CDF is monotonically nondecreasing because accumulating probability never removes mass; it is right-continuous because \(P(X\le x)\) includes the point \(x\), so a discrete jump's mass is counted at \(x\); and it limits to 0 as \(x\to-\infty\) and to 1 as \(x\to+\infty\). For a discrete variable it is a staircase whose jumps equal the PMF, and for a continuous variable it is the integral of the density.
Q2: Why does linearity of expectation hold even when variables are dependent? Because \(E(X+Y)=\sum_s (X(s)+Y(s))P(\{s\})\) splits into two separate sums by simple rearrangement, and that split never references the joint distribution of \(X\) and \(Y\). Independence is required for products like \(E(XY)\) but not for sums, which is why decomposing a dependent count into indicators and summing their means always works.
Q3: Derive the computational form of variance and explain why we square the deviations. Starting from \(E[(X-E[X])^2]\) and expanding with linearity gives \(E[X^2]-2(E[X])^2+(E[X])^2=E[X^2]-(E[X])^2\). We square rather than take the raw deviation because \(E[X-E[X]]\) is identically zero, and we prefer squaring over absolute value because the square is smooth and differentiable, which suits calculus and optimization.
Q4: What does standard deviation give you that variance does not? Standard deviation is the square root of variance, returning the spread to the original units of the variable rather than squared units. That makes it directly interpretable as the typical distance of an observation from the mean, whereas variance is the more convenient quantity for algebra and proofs.