Definition
The singular value decomposition writes any real matrix as , where () and () are orthogonal and () is zero except for the singular values on its diagonal. The columns of and are the right and left singular vectors.
At a glance
- Notation
- ,
- Input → output
- any matrix (not necessarily square) → orthonormal bases , and non-negative
- Key property
- ; ; ; rank = number of nonzero
- Cost
- for the full SVD; truncated/randomized SVD for the top is much cheaper
- Prerequisites
- eigenvectors & eigenvalues, projections & orthogonality
- Used in
- PCA, recommender systems, pseudo-inverse & least squares, model compression
The intuition
Every linear map, however messy its entries, does three simple things in sequence: rotate (possibly reflect) the input space, stretch along the coordinate axes, then rotate the output space. A unit circle therefore always lands on an ellipse; the singular values are its semi-axis lengths and the left singular vectors point along its axes.
The decomposition also reads as a ranked sum of rank-1 layers , each one a pattern (column profile times row profile) weighted by its importance. Real data matrices usually have a few large and a long tail of small ones, so a handful of layers reproduces most of the matrix.
The math
Existence via eigenvectors. is symmetric and positive semidefinite, so it has orthonormal eigenvectors with eigenvalues . Set and for . These are orthonormal, because , which is 1 when and 0 otherwise. Stacking gives , i.e. :
- right singular vectors — orthonormal input directions (eigenvectors of )
- left singular vectors — orthonormal output directions (eigenvectors of )
- singular values: how much stretches along
A x in three moves
- 1: rotate so the right singular vectors line up with the coordinate axes.
- 2: stretch axis by (and drop or pad dimensions when ).
- 3: rotate the axes onto the left singular vectors — the ellipse's principal axes.
Best low-rank approximation (Eckart–Young). Keep the first terms, . No matrix of rank at most is closer to , in either the spectral or the Frobenius norm:
- Frobenius norm — square root of the sum of squared entries
- spectral norm — the largest stretch of any unit vector
The fraction of "energy" kept is , and storing costs numbers instead of .
Pseudo-inverse and conditioning. (invert the nonzero , transpose the shape) gives the minimum-norm least-squares solution . The condition number measures how much errors in can be amplified; tiny are where solutions become unstable.
Link to PCA. For a centred data matrix ( samples), : the right singular vectors are the principal directions and are the variances along them — no covariance matrix needs to be formed.
See it
- 1x on unit circle; mark v₁, v₂
- 2y ← Vᵀx # rotate by −θ
- 3y ← Σy # stretch σ₁, σ₂
- 4y ← Uy # rotate by φ
- 5y = Ax # same ellipse
The unit circle and the right singular vectors v₁, v₂ — orthonormal, at θ = 32.2° and 122.2°.
Worked example
. Then , with eigenvalues and and eigenvectors , . So and .
Left vectors: , divided by gives ; , divided by gives . Check: , and .
Rank-1 approximation: . The error has , exactly as Eckart–Young predicts; keeps of the energy. Condition number .
In code
1import numpy as np23A = np.array([[3.0, 0.0], [4.0, 5.0]])4U, s, Vt = np.linalg.svd(A) # s ≈ [6.7082, 2.2361]; signs of u_i, v_i may flip5print(np.allclose(U @ np.diag(s) @ Vt, A)) # True6A1 = s[0] * np.outer(U[:, 0], Vt[0]) # ≈ [[1.5, 1.5], [4.5, 4.5]]7print(np.linalg.norm(A - A1)) # ≈ 2.2361 = s[1] (Eckart–Young)89X = np.random.default_rng(0).normal(size=(500, 40))10U, s, Vt = np.linalg.svd(X, full_matrices=False) # economy SVD: U is 500×4011k = 1012Xk = (U[:, :k] * s[:k]) @ Vt[:k] # best rank-10 approximation13kept = (s[:k] ** 2).sum() / (s ** 2).sum() # fraction of energy kept14x_ls = np.linalg.pinv(A) @ np.array([3.0, 9.0]) # pseudo-inverse solve: [1., 1.]Where it's used
PCA
scikit-learn computes PCA as the SVD of the centred data matrix: components are the rows of , explained variance is .
Recommenders & topic models
A truncated SVD of a user–item or term–document matrix gives low-dimensional user, item and word embeddings (latent semantic analysis).
Least squares & pseudo-inverse
np.linalg.lstsq and pinv solve via the SVD, discarding singular values below a tolerance to stay stable on rank-deficient designs.
Model compression
Replacing an weight matrix by rank- factors cuts parameters from to ; LoRA trains exactly such a low-rank update.
Check yourself
A matrix has singular values , and . What is the Frobenius-norm error of its best rank-1 approximation?
Check yourself
A matrix has singular values , . Which is true?
Remember
- ✓Every matrix is : rotate, stretch along orthogonal axes by , rotate.
- ✓ are the eigenvalues of ; maps the unit circle's axes to the ellipse's axes.
- ✓Truncating to the top terms is the best rank- approximation; the error is set by the discarded .
- ✓SVD powers PCA, pseudo-inverses, recommenders and low-rank compression; is the condition number.