Eigenvalues, Pivots & Rank¶
Eigenvectors are the special directions a matrix only stretches, pivots are the anchors of row reduction, and rank is the count of genuinely independent directions a matrix carries. Together they describe what a matrix really does underneath its apparent complexity.
Rapid Recall
An eigenvector satisfies \(Av=\lambda v\): it keeps its direction and is only scaled, and repeated application gives \(A^k v=\lambda^k v\), which governs Markov chains, PageRank, and stability. Eigenvalues come from the characteristic equation \(\det(A-\lambda I)=0\). Pivots are the leading nonzero entries in echelon form, and their count is the rank. Rank is the matrix's true working dimension, equal across independent columns, independent rows, pivots, and nonzero singular values. A sum of \(r\) rank-1 outer products has rank at most \(r\), so low rank means buildable from a few underlying factors.
§3 Eigenvalues & eigenvectors¶
Most vectors get rotated and stretched by a matrix. Eigenvectors are the special directions that only get stretched, no turning. Picture stretching a rubber sheet diagonally. Most painted arrows swing to point somewhere new. The arrows lying along the pure stretch and squeeze axes just get longer or shorter without turning. Those are the eigenvectors; the stretch factors are the eigenvalues.
An eigenvector is really a direction: if \(v\) works, so does any multiple of it, with the same \(\lambda\). That is why we find a line (and often normalize it).
Why we care¶
Eigenvectors are the matrix's natural axes, the coordinate system in which the machine stops mixing and is pure stretching. Concretely:
- Repeated application is trivial: \(A^k v = \lambda^k v\). Governs Markov chains, PageRank, dynamics, the largest \(|\lambda|\) dominates (\(>1\) blow-up, \(<1\) decay, \(=1\) steady state).
- Stability and conditioning: the Hessian's eigenvalues are curvatures; the ratio largest/smallest is the condition number that makes gradient descent zig-zag.
- PCA is the eigendecomposition of the covariance matrix, eigenvectors are the directions of maximum variance, eigenvalues are how much.
How to find them, step by step¶
We need a nonzero \(v\) crushed to zero by \((A - \lambda I)\). A matrix that crushes a nonzero vector is singular, which happens exactly when its determinant is zero (the determinant is the volume-scaling factor; zero means a dimension collapsed):
- Form \(A - \lambda I\).
- Set its determinant to zero, giving a polynomial in \(\lambda\).
- Solve for \(\lambda\) (the eigenvalues).
- For each \(\lambda\), solve \((A - \lambda I)v = 0\) for the eigenvector(s).
§4 Pivots¶
A completely separate idea from eigenvalues. When you row-reduce a matrix into staircase (echelon) form, the pivot is the leading nonzero entry of each row, the anchor you eliminate with. Columns with a pivot are pivot columns; columns without one are free columns.
Where pivots are used.
- Number of pivots equals rank, the true working dimension.
- Pivot columns form a basis for the column space; free columns count the null space.
- Solvability: unique, infinite, or no solution depending on pivots and free columns.
- Invertibility: an \(n\times n\) matrix is invertible if and only if it has \(n\) pivots.
- Determinant equals the product of pivots (with sign for row swaps); pivots are the diagonal of U in LU.
§5 Rank & low rank¶
Rank is the number of genuinely independent directions a matrix carries, the dimension of its column space. A \(3\times3\) matrix whose three columns all lie in a plane has rank 2: it is a 2D machine wearing a 3D costume. Rank is the matrix's true working dimension. All of these are the same number:
- independent columns,
- independent rows,
- pivots,
- nonzero singular values,
- dimension of the column space.
Full rank is the maximum \(\min(m, n)\). For a square matrix that means invertible, nonzero determinant, no zero eigenvalues. Low rank (rank deficient) means some columns are redundant; the matrix collapses dimensions and loses information.
How a matrix becomes low rank, the outer product¶
The key fact: a sum of r rank-1 outer products has rank at most r. So "low rank" literally means "buildable from a few underlying factors."
Interview Questions¶
Q1: What is an eigenvector, and why does it make repeated application easy? An eigenvector \(v\) satisfies \(Av=\lambda v\), so the matrix only scales it without turning, and any multiple of \(v\) is also an eigenvector with the same \(\lambda\). Repeated application gives \(A^k v=\lambda^k v\), turning a hard matrix power into simple scalar powers, which is why eigenvalues govern Markov chains, PageRank, and dynamical stability where the largest magnitude eigenvalue dominates.
Q2: How do you compute eigenvalues from scratch? Rewrite \(Av=\lambda v\) as \((A-\lambda I)v=0\) and require a nonzero solution, which forces \(A-\lambda I\) to be singular. That means its determinant is zero, giving the characteristic equation \(\det(A-\lambda I)=0\), a polynomial whose roots are the eigenvalues. For each eigenvalue you then solve \((A-\lambda I)v=0\) for the eigenvectors.
Q3: List several quantities that all equal the rank of a matrix. The rank equals the number of independent columns, the number of independent rows, the number of pivots in echelon form, the number of nonzero singular values, and the dimension of the column space. They are five views of the same underlying count of genuinely independent directions.
Q4: What does it mean for a matrix to be low rank, and why is that useful? A low-rank matrix has fewer independent directions than its size suggests, so it can be written as a sum of a few rank-1 outer products. This means it is buildable from a small number of underlying factors, which is the basis of compression, PCA, and low-rank adaptation, since you store and compute with far fewer numbers.