Singular Value Decomposition

SVD article.

Motivation

Many problems in data science reduce to understanding the structure of a matrix — dimensionality reduction via PCA, solving least-squares problems, compressing images, or computing pseudoinverses. SVD provides the most general and numerically stable way to answer all of these questions.

import numpy as np

A = np.random.randn(5, 3)
U, s, Vt = np.linalg.svd(A, full_matrices=False)
Sigma = np.diag(s)

# Reconstruct
A_recon = U @ Sigma @ Vt
assert np.allclose(A, A_recon)
print(f"Shape: {A.shape} -> U: {U.shape}, s: {s.shape}, Vt: {Vt.shape}")

The Four Fundamental Subspaces

The SVD gives an orthonormal basis for each of the four fundamental subspaces of A:

• Column space \mathcal{R}(A): first r columns of U
• Row space \mathcal{R}(A^T): first r rows of V^T
• Nullspace \mathcal{N}(A): last n-r rows of V^T
• Left nullspace \mathcal{N}(A^T): last m-r columns of U

where r = \text{rank}(A).