Singular Value Decomposition

Every matrix factors as UΣVᵀ: rotate, scale along orthogonal axes, rotate again; keeping the top singular values gives the best low-rank approximation.

7 min read591 wordsadvanced7 viewsupdated 2026-10-03

Definition

The singular value decomposition writes any real m×nm\times n matrix as A=UΣV⊤A=U\Sigma V^\top, where UU (m×mm\times m) and VV (n×nn\times n) are orthogonal and Σ\Sigma (m×nm\times n) is zero except for the singular values σ1≥σ2≥⋯≥0\sigma_1\ge\sigma_2\ge\dots\ge0 on its diagonal. The columns of VV and UU are the right and left singular vectors.

At a glance

Notation
A=UΣV⊤=∑i=1rσiuivi⊤A=U\Sigma V^\top=\sum_{i=1}^r\sigma_iu_iv_i^\top, r=rank⁡Ar=\operatorname{rank}A
Input → output
any m×nm\times n matrix (not necessarily square) → orthonormal bases UU, VV and non-negative σi\sigma_i
Key property
Avi=σiuiAv_i=\sigma_iu_i; ∥A∥2=σ1\lVert A\rVert_2=\sigma_1; ∥A∥F2=∑iσi2\lVert A\rVert_F^2=\sum_i\sigma_i^2; rank = number of nonzero σi\sigma_i
Cost
O(mnmin⁡(m,n))O(mn\min(m,n)) for the full SVD; truncated/randomized SVD for the top kk is much cheaper
Prerequisites
eigenvectors & eigenvalues, projections & orthogonality
Used in
PCA, recommender systems, pseudo-inverse & least squares, model compression

The intuition

Every linear map, however messy its entries, does three simple things in sequence: rotate (possibly reflect) the input space, stretch along the coordinate axes, then rotate the output space. A unit circle therefore always lands on an ellipse; the singular values are its semi-axis lengths and the left singular vectors point along its axes.

The decomposition also reads as a ranked sum of rank-1 layers σiuivi⊤\sigma_iu_iv_i^\top, each one a pattern (column profile times row profile) weighted by its importance. Real data matrices usually have a few large σi\sigma_i and a long tail of small ones, so a handful of layers reproduces most of the matrix.

The math

Existence via eigenvectors. A⊤AA^\top A is symmetric and positive semidefinite, so it has orthonormal eigenvectors viv_i with eigenvalues λi≥0\lambda_i\ge0. Set σi=λi\sigma_i=\sqrt{\lambda_i} and ui=Avi/σiu_i=Av_i/\sigma_i for σi>0\sigma_i>0. These uiu_i are orthonormal, because ui⊤uj=vi⊤A⊤Avj/(σiσj)=λjvi⊤vj/(σiσj)u_i^\top u_j=v_i^\top A^\top Av_j/(\sigma_i\sigma_j)=\lambda_j v_i^\top v_j/(\sigma_i\sigma_j), which is 1 when i=ji=j and 0 otherwise. Stacking gives AV=UΣAV=U\Sigma, i.e. A=UΣV⊤A=U\Sigma V^\top:

A⊤A=V Σ⊤Σ V⊤,AA⊤=U ΣΣ⊤ U⊤,Avi=σiui,A⊤ui=σiviA^\top A=V\,\Sigma^\top\Sigma\,V^\top,\qquad AA^\top=U\,\Sigma\Sigma^\top\,U^\top,\qquad Av_i=\sigma_iu_i,\quad A^\top u_i=\sigma_iv_i
viv_i
right singular vectors — orthonormal input directions (eigenvectors of A⊤AA^\top A)
uiu_i
left singular vectors — orthonormal output directions (eigenvectors of AA⊤AA^\top)
σi\sigma_i
singular values: how much AA stretches along viv_i

A x in three moves

  1. 1V⊤xV^\top x: rotate so the right singular vectors viv_i line up with the coordinate axes.
  2. 2Σ\Sigma: stretch axis ii by σi\sigma_i (and drop or pad dimensions when m≠nm\ne n).
  3. 3UU: rotate the axes onto the left singular vectors uiu_i — the ellipse's principal axes.

Best low-rank approximation (Eckart–Young). Keep the first kk terms, Ak=∑i≤kσiuivi⊤A_k=\sum_{i\le k}\sigma_iu_iv_i^\top. No matrix of rank at most kk is closer to AA, in either the spectral or the Frobenius norm:

min⁡rank⁡B≤k∥A−B∥F=∥A−Ak∥F=∑i>kσi2,∥A−Ak∥2=σk+1\min_{\operatorname{rank}B\le k}\lVert A-B\rVert_F=\lVert A-A_k\rVert_F=\sqrt{\textstyle\sum_{i>k}\sigma_i^2},\qquad \lVert A-A_k\rVert_2=\sigma_{k+1}
∥⋅∥F\lVert\cdot\rVert_F
Frobenius norm — square root of the sum of squared entries
∥⋅∥2\lVert\cdot\rVert_2
spectral norm — the largest stretch of any unit vector

The fraction of "energy" kept is ∑i≤kσi2/∑iσi2\sum_{i\le k}\sigma_i^2/\sum_i\sigma_i^2, and storing AkA_k costs k(m+n+1)k(m+n+1) numbers instead of mnmn.

Pseudo-inverse and conditioning. A+=VΣ+U⊤A^+=V\Sigma^+U^\top (invert the nonzero σi\sigma_i, transpose the shape) gives the minimum-norm least-squares solution x=A+bx=A^+b. The condition number κ=σ1/σr\kappa=\sigma_1/\sigma_r measures how much errors in bb can be amplified; tiny σi\sigma_i are where solutions become unstable.

Link to PCA. For a centred data matrix XX (nn samples), X⊤X/(n−1)=V Σ2n−1 V⊤X^\top X/(n-1)=V\,\tfrac{\Sigma^2}{n-1}\,V^\top: the right singular vectors are the principal directions and σi2/(n−1)\sigma_i^2/(n-1) are the variances along them — no covariance matrix needs to be formed.

See it

A =[[1.5, 0.8], [-0.4, 1]]
▸ SVD · rotate, stretch, rotatestep 1/5
v₁v₂
σ₁ = 1.707σ₂ = 1.066σ₁/σ₂ = 1.6θ = 32.2°, φ = 6.5°
  1. 1x on unit circle; mark v₁, v₂
  2. 2y ← Vᵀx # rotate by −θ
  3. 3y ← Σy # stretch σ₁, σ₂
  4. 4y ← Uy # rotate by φ
  5. 5y = Ax # same ellipse

The unit circle and the right singular vectors v₁, v₂ — orthonormal, at θ = 32.2° and 122.2°.

fig.Step through A = [[1.5, 0.8], [−0.4, 1]]: Vᵀ rotates the circle by −32.2° (v₁, v₂ land on the axes), Σ stretches by σ₁ = 1.707 and σ₂ = 1.066, U rotates by 6.5°, and the result coincides with A applied directly — Av₁ and Av₂ are the ellipse's semi-axes. The near-singular preset has σ₂ = 0.022: the ellipse collapses to almost a segment.
original · rank 48, 2304 numbers
rank 5 · 485 numbers
error ×4 · ‖A − Aₖ‖/‖A‖ = 11.4%
singular values σᵢ (log scale) — kept in colour
energy kept 98.71%rel. error 11.4%storage 485 / 2304σ₁ = 21.25, σ5 = 2.033
fig.A 48×48 image rebuilt from its top k singular triplets (σ₁ = 21.25). k = 1 keeps 82.96% of the energy (41.3% relative error); k = 5 keeps 98.71% with 485 numbers instead of 2,304; k = 10 keeps 99.77% (4.8% error). The vertical stripes are nearly rank-1 and appear early; the disc's curved edge needs many terms. Past k = 23, k(m + n + 1) exceeds m·n and nothing is saved.

Worked example

A=(3045)A=\begin{pmatrix}3&0\\4&5\end{pmatrix}. Then A⊤A=(25202025)A^\top A=\begin{pmatrix}25&20\\20&25\end{pmatrix}, with eigenvalues 4545 and 55 and eigenvectors v1=12(1,1)v_1=\tfrac1{\sqrt2}(1,1), v2=12(1,−1)v_2=\tfrac1{\sqrt2}(1,-1). So σ1=45=35≈6.708\sigma_1=\sqrt{45}=3\sqrt5\approx6.708 and σ2=5≈2.236\sigma_2=\sqrt5\approx2.236.

Left vectors: Av1=12(3,9)Av_1=\tfrac1{\sqrt2}(3,9), divided by σ1\sigma_1 gives u1=110(1,3)u_1=\tfrac1{\sqrt{10}}(1,3); Av2=12(3,−1)Av_2=\tfrac1{\sqrt2}(3,-1), divided by σ2\sigma_2 gives u2=110(3,−1)u_2=\tfrac1{\sqrt{10}}(3,-1). Check: u1⊤u2=3−3=0u_1^\top u_2=3-3=0, and σ1σ2=15=∣det⁡A∣\sigma_1\sigma_2=15=|\det A|.

Rank-1 approximation: A1=σ1u1v1⊤=3520(1133)=(1.51.54.54.5)A_1=\sigma_1u_1v_1^\top=\tfrac{3\sqrt5}{\sqrt{20}}\begin{pmatrix}1&1\\3&3\end{pmatrix}=\begin{pmatrix}1.5&1.5\\4.5&4.5\end{pmatrix}. The error A−A1=(1.5−1.5−0.50.5)A-A_1=\begin{pmatrix}1.5&-1.5\\-0.5&0.5\end{pmatrix} has ∥⋅∥F2=2.25+2.25+0.25+0.25=5=σ22\lVert\cdot\rVert_F^2=2.25+2.25+0.25+0.25=5=\sigma_2^2, exactly as Eckart–Young predicts; A1A_1 keeps 45/50=90%45/50=90\% of the energy. Condition number κ=6.708/2.236=3\kappa=6.708/2.236=3.

In code

python
1import numpy as np
2
3A = np.array([[3.0, 0.0], [4.0, 5.0]])
4U, s, Vt = np.linalg.svd(A) # s ≈ [6.7082, 2.2361]; signs of u_i, v_i may flip
5print(np.allclose(U @ np.diag(s) @ Vt, A)) # True
6A1 = s[0] * np.outer(U[:, 0], Vt[0]) # ≈ [[1.5, 1.5], [4.5, 4.5]]
7print(np.linalg.norm(A - A1)) # ≈ 2.2361 = s[1] (Eckart–Young)
8
9X = np.random.default_rng(0).normal(size=(500, 40))
10U, s, Vt = np.linalg.svd(X, full_matrices=False) # economy SVD: U is 500×40
11k = 10
12Xk = (U[:, :k] * s[:k]) @ Vt[:k] # best rank-10 approximation
13kept = (s[:k] ** 2).sum() / (s ** 2).sum() # fraction of energy kept
14x_ls = np.linalg.pinv(A) @ np.array([3.0, 9.0]) # pseudo-inverse solve: [1., 1.]
Full SVD, the rank-1 approximation from the worked example, and truncated reconstruction of a data matrix.

Where it's used

PCA

scikit-learn computes PCA as the SVD of the centred data matrix: components are the rows of V⊤V^\top, explained variance is σi2/(n−1)\sigma_i^2/(n-1).

Recommenders & topic models

A truncated SVD of a user–item or term–document matrix gives low-dimensional user, item and word embeddings (latent semantic analysis).

Least squares & pseudo-inverse

np.linalg.lstsq and pinv solve via the SVD, discarding singular values below a tolerance to stay stable on rank-deficient designs.

Model compression

Replacing an m×nm\times n weight matrix by rank-kk factors cuts parameters from mnmn to k(m+n)k(m+n); LoRA trains exactly such a low-rank update.

Check yourself

A matrix has singular values 55, 33 and 11. What is the Frobenius-norm error of its best rank-1 approximation?

Check yourself

A 3×23\times2 matrix has singular values σ1=4\sigma_1=4, σ2=0\sigma_2=0. Which is true?

Remember

  • ✓Every matrix is UΣV⊤U\Sigma V^\top: rotate, stretch along orthogonal axes by σi≥0\sigma_i\ge0, rotate.
  • ✓σi2\sigma_i^2 are the eigenvalues of A⊤AA^\top A; Avi=σiuiAv_i=\sigma_iu_i maps the unit circle's axes to the ellipse's axes.
  • ✓Truncating to the top kk terms is the best rank-kk approximation; the error is set by the discarded σi\sigma_i.
  • ✓SVD powers PCA, pseudo-inverses, recommenders and low-rank compression; σ1/σr\sigma_1/\sigma_r is the condition number.
← back to Mathematics roadmap