Computational Linear Algebra
This course develops linear algebra the way a working numerical analyst or machine-learning practitioner actually uses it: not as an abstract theory of vector spaces encountered once and filed away, but as a small set of concrete tools — the singular value decomposition above all — that answer a specific recurring question about every matrix one meets.
The governing idea is that a matrix is a representation of a linear operator, and the right coordinate system makes that operator transparent. The course reaches for that coordinate system as early as possible and then derives the rest of the subject from it.
Why the SVD comes early
Most textbooks build to the singular value decomposition near the end, after determinants, inverses, and the eigenvalue theory. This course inverts that order deliberately. The SVD of a matrix \mathbf{A} — the factorization
\mathbf{A} = \mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^{\mathsf T}, \qquad \mathbf{U}^{\mathsf T}\mathbf{U} = \mathbf{I},\quad \mathbf{V}^{\mathsf T}\mathbf{V} = \mathbf{I},\quad \boldsymbol{\Sigma} \text{ diagonal}
with nonnegative entries \sigma_1 \ge \sigma_2 \ge \dots \ge \sigma_r > 0 — captures, in one statement, the image, the nullspace, the rank, the operator norm, the condition number, the best low-rank approximation, and the pseudoinverse. Nothing else in the subject packages this much into so little.
The spectral theorem and the SVD are equivalent, and they share one proof. The course proves the symmetric case first — not because the SVD logically requires it, but because it is the simplest instance: one square matrix, real eigenvalues, a single orthonormal basis. The same variational argument (maximize a quadratic form over the unit sphere, restrict to an invariant orthogonal complement, and induct) yields the spectral theorem when run on a symmetric matrix \mathbf{M}, and the singular value decomposition when run on \mathbf{M} = \mathbf{A}^{\mathsf T}\mathbf{A} for an arbitrary \mathbf{A}.
Once the SVD is in hand, it explains the rank, the four fundamental subspaces, the operator norm, the condition number, the best low-rank approximation, the pseudoinverse, and least squares — but not everything. Two classical topics are genuinely independent of it: the general eigenvalue theory of nonsymmetric matrices, whose eigenvalues the SVD cannot see, and the orientation carried by the sign of the determinant. Each gets its own chapter in its own right.
Fronting the SVD is the ordering popularized by Trefethen and Bau and, independently, by Strang. The eigenvalue-first ordering, which fronts determinants and Jordan-esque pathologies before any useful factorization appears, is treated here as a pedagogical mistake worth correcting.
Prerequisites
- Comfort with calculus and basic probability; some prior exposure to matrices at the level of Gaussian elimination.
- Working Python with NumPy, SciPy, and matplotlib. Every idea is demonstrated in code as it is introduced.
- No prior exposure to the SVD, spectral theory, or numerical analysis is assumed — the course is self-contained from the definition of a vector space forward.
The mathematical voice is impersonal and terse: definitions, then results with complete proofs, then a code demonstration. Proofs are included not as decoration but because they are where the geometry lives.
The fourteen chapters
| Ch | Chapter | Center of gravity |
|---|---|---|
| 01 | Foundations | vectors, matrices, inner products, norms, orthogonal matrices |
| 02 | The spectral theorem | symmetric matrices diagonalize over an orthonormal basis |
| 03 | The singular value decomposition | geometry, existence, low-rank approximation, Eckart–Young |
| 04 | Rank & the four fundamental subspaces | rank as information, numerical rank, the subspace quartet |
| 05 | Determinant | signed volume, invertibility |
| 06 | Inverse & pseudoinverse | left/right inverses, the Moore–Penrose inverse |
| 07 | Projection & orthogonalization | orthogonal projectors, Gram–Schmidt, QR, Sherman–Morrison |
| 08 | Least squares | the normal equations, the SVD solution, gradient descent |
| 09 | Eigendecomposition | diagonalization, Gershgorin disks, multiplicities, power iteration |
| 10 | Quadratic forms | definiteness, principal axes, the Rayleigh quotient |
| 11 | Topic modeling with SVD and NMF | term–document matrices, topics as singular vectors, NMF, randomized SVD |
| 12 | PageRank | the web as a stochastic matrix, power iteration, damping |
| 13 | Robust PCA | low-rank plus sparse decomposition, nuclear norm, soft-thresholding |
| 14 | Compressed sensing | underdetermined systems, \ell_1 vs \ell_2, the restricted isometry property |
Chapters 01–03 form the spine; everything after is an application of that spine. The chapters build on one another and reward reading in order, though the table also works as a lookup index. Chapters 11–14 close the course with four applications — topic modeling, PageRank, robust PCA, and compressed sensing — each putting an idea from the first ten chapters to work outside the synthetic demos of the spine.
Conventions
Notation is the interface, so the course keeps it rigid. The central distinction is between an operator and its representation:
- Operators are linear maps between vector spaces, written in regular font: T, S. They are basis-free.
- Vectors and matrices are representations — a concrete vector or the matrix of an operator in a chosen basis — written in bold: vectors \mathbf{x}, \mathbf{y} \in \mathbb{R}^n; matrices \mathbf{A} \in \mathbb{R}^{m \times n}.
- Scalars and components are regular font: \alpha, \sigma_i, \lambda_i, and the coordinates x_i of a vector \mathbf{x}.
A vector is a column \mathbf{x} = (x_1, \dots, x_n)^{\mathsf T}; row vectors appear only as transposes. The standard inner product is \langle \mathbf{x}, \mathbf{y}\rangle = \mathbf{x}^{\mathsf T}\mathbf{y}. The identity and zero matrices are \mathbf{I} and \mathbf{0}; the identity and zero operators are I and 0. Orthogonal always means \mathbf{Q}^{\mathsf T}\mathbf{Q} = \mathbf{I}. The singular values of \mathbf{A} are \sigma_1 \ge \sigma_2 \ge \dots \ge \sigma_r > 0; the eigenvalues are \lambda_1, \lambda_2, \dots.
The source of the course is a set of eleven notebooks first written as an undergraduate study aid; this rewrite reconstructs the same material with complete proofs, consistent notation, and the numerical-analysis depth the original sketch left out.