# Singular Value Decomposition

> Every matrix factors as UΣVᵀ: rotate, scale along orthogonal axes, rotate again; keeping the top singular values gives the best low-rank approximation.

Source: https://www.mlfoundry.in/learn/singular-value-decomposition  
Section: Mathematics › Linear Algebra  
Level: advanced · 7 min read · updated 2026-10-03

**The singular value decomposition:** writes any real $m\times n$ matrix as $A=U\Sigma V^\top$, where $U$ ($m\times m$) and $V$ ($n\times n$) are orthogonal and $\Sigma$ ($m\times n$) is zero except for the **singular values** $\sigma_1\ge\sigma_2\ge\dots\ge0$ on its diagonal. The columns of $V$ and $U$ are the right and left singular vectors.

- **Notation:** $A=U\Sigma V^\top=\sum_{i=1}^r\sigma_iu_iv_i^\top$, $r=\operatorname{rank}A$
- **Input → output:** any $m\times n$ matrix (not necessarily square) → orthonormal bases $U$, $V$ and non-negative $\sigma_i$
- **Key property:** $Av_i=\sigma_iu_i$; $\lVert A\rVert_2=\sigma_1$; $\lVert A\rVert_F^2=\sum_i\sigma_i^2$; rank = number of nonzero $\sigma_i$
- **Cost:** $O(mn\min(m,n))$ for the full SVD; truncated/randomized SVD for the top $k$ is much cheaper
- **Prerequisites:** eigenvectors & eigenvalues, projections & orthogonality
- **Used in:** PCA, recommender systems, pseudo-inverse & least squares, model compression

## The intuition

Every linear map, however messy its entries, does three simple things in sequence: rotate (possibly reflect) the input space, stretch along the coordinate axes, then rotate the output space. A unit circle therefore always lands on an ellipse; the singular values are its semi-axis lengths and the left singular vectors point along its axes.

The decomposition also reads as a ranked sum of rank-1 layers $\sigma_iu_iv_i^\top$, each one a pattern (column profile times row profile) weighted by its importance. Real data matrices usually have a few large $\sigma_i$ and a long tail of small ones, so a handful of layers reproduces most of the matrix.

## The math

**Existence via eigenvectors.** $A^\top A$ is symmetric and positive semidefinite, so it has orthonormal eigenvectors $v_i$ with eigenvalues $\lambda_i\ge0$. Set $\sigma_i=\sqrt{\lambda_i}$ and $u_i=Av_i/\sigma_i$ for $\sigma_i>0$. These $u_i$ are orthonormal, because $u_i^\top u_j=v_i^\top A^\top Av_j/(\sigma_i\sigma_j)=\lambda_j v_i^\top v_j/(\sigma_i\sigma_j)$, which is 1 when $i=j$ and 0 otherwise. Stacking gives $AV=U\Sigma$, i.e. $A=U\Sigma V^\top$:

$$
A^\top A=V\,\Sigma^\top\Sigma\,V^\top,\qquad AA^\top=U\,\Sigma\Sigma^\top\,U^\top,\qquad Av_i=\sigma_iu_i,\quad A^\top u_i=\sigma_iv_i
$$

- $v_i$: right singular vectors — orthonormal input directions (eigenvectors of $A^\top A$)
- $u_i$: left singular vectors — orthonormal output directions (eigenvectors of $AA^\top$)
- $\sigma_i$: singular values: how much $A$ stretches along $v_i$

**A x in three moves**

1. $V^\top x$: rotate so the right singular vectors $v_i$ line up with the coordinate axes.
2. $\Sigma$: stretch axis $i$ by $\sigma_i$ (and drop or pad dimensions when $m\ne n$).
3. $U$: rotate the axes onto the left singular vectors $u_i$ — the ellipse's principal axes.

**Best low-rank approximation (Eckart–Young).** Keep the first $k$ terms, $A_k=\sum_{i\le k}\sigma_iu_iv_i^\top$. No matrix of rank at most $k$ is closer to $A$, in either the spectral or the Frobenius norm:

$$
\min_{\operatorname{rank}B\le k}\lVert A-B\rVert_F=\lVert A-A_k\rVert_F=\sqrt{\textstyle\sum_{i>k}\sigma_i^2},\qquad \lVert A-A_k\rVert_2=\sigma_{k+1}
$$

- $\lVert\cdot\rVert_F$: Frobenius norm — square root of the sum of squared entries
- $\lVert\cdot\rVert_2$: spectral norm — the largest stretch of any unit vector

The fraction of "energy" kept is $\sum_{i\le k}\sigma_i^2/\sum_i\sigma_i^2$, and storing $A_k$ costs $k(m+n+1)$ numbers instead of $mn$.

**Pseudo-inverse and conditioning.** $A^+=V\Sigma^+U^\top$ (invert the nonzero $\sigma_i$, transpose the shape) gives the minimum-norm least-squares solution $x=A^+b$. The condition number $\kappa=\sigma_1/\sigma_r$ measures how much errors in $b$ can be amplified; tiny $\sigma_i$ are where solutions become unstable.

**Link to PCA.** For a centred data matrix $X$ ($n$ samples), $X^\top X/(n-1)=V\,\tfrac{\Sigma^2}{n-1}\,V^\top$: the right singular vectors are the principal directions and $\sigma_i^2/(n-1)$ are the variances along them — no covariance matrix needs to be formed.

## See it

> Figure: Step through A = [[1.5, 0.8], [−0.4, 1]]: Vᵀ rotates the circle by −32.2° (v₁, v₂ land on the axes), Σ stretches by σ₁ = 1.707 and σ₂ = 1.066, U rotates by 6.5°, and the result coincides with A applied directly — Av₁ and Av₂ are the ellipse's semi-axes. The near-singular preset has σ₂ = 0.022: the ellipse collapses to almost a segment.

> Figure: A 48×48 image rebuilt from its top k singular triplets (σ₁ = 21.25). k = 1 keeps 82.96% of the energy (41.3% relative error); k = 5 keeps 98.71% with 485 numbers instead of 2,304; k = 10 keeps 99.77% (4.8% error). The vertical stripes are nearly rank-1 and appear early; the disc's curved edge needs many terms. Past k = 23, k(m + n + 1) exceeds m·n and nothing is saved.

## Worked example

$A=\begin{pmatrix}3&0\\4&5\end{pmatrix}$. Then $A^\top A=\begin{pmatrix}25&20\\20&25\end{pmatrix}$, with eigenvalues $45$ and $5$ and eigenvectors $v_1=\tfrac1{\sqrt2}(1,1)$, $v_2=\tfrac1{\sqrt2}(1,-1)$. So $\sigma_1=\sqrt{45}=3\sqrt5\approx6.708$ and $\sigma_2=\sqrt5\approx2.236$.

Left vectors: $Av_1=\tfrac1{\sqrt2}(3,9)$, divided by $\sigma_1$ gives $u_1=\tfrac1{\sqrt{10}}(1,3)$; $Av_2=\tfrac1{\sqrt2}(3,-1)$, divided by $\sigma_2$ gives $u_2=\tfrac1{\sqrt{10}}(3,-1)$. Check: $u_1^\top u_2=3-3=0$, and $\sigma_1\sigma_2=15=|\det A|$.

Rank-1 approximation: $A_1=\sigma_1u_1v_1^\top=\tfrac{3\sqrt5}{\sqrt{20}}\begin{pmatrix}1&1\\3&3\end{pmatrix}=\begin{pmatrix}1.5&1.5\\4.5&4.5\end{pmatrix}$. The error $A-A_1=\begin{pmatrix}1.5&-1.5\\-0.5&0.5\end{pmatrix}$ has $\lVert\cdot\rVert_F^2=2.25+2.25+0.25+0.25=5=\sigma_2^2$, exactly as Eckart–Young predicts; $A_1$ keeps $45/50=90\%$ of the energy. Condition number $\kappa=6.708/2.236=3$.

## In code

```python
import numpy as np

A = np.array([[3.0, 0.0], [4.0, 5.0]])
U, s, Vt = np.linalg.svd(A)            # s ≈ [6.7082, 2.2361]; signs of u_i, v_i may flip
print(np.allclose(U @ np.diag(s) @ Vt, A))   # True
A1 = s[0] * np.outer(U[:, 0], Vt[0])   # ≈ [[1.5, 1.5], [4.5, 4.5]]
print(np.linalg.norm(A - A1))          # ≈ 2.2361 = s[1]  (Eckart–Young)

X = np.random.default_rng(0).normal(size=(500, 40))
U, s, Vt = np.linalg.svd(X, full_matrices=False)   # economy SVD: U is 500×40
k = 10
Xk = (U[:, :k] * s[:k]) @ Vt[:k]       # best rank-10 approximation
kept = (s[:k] ** 2).sum() / (s ** 2).sum()         # fraction of energy kept
x_ls = np.linalg.pinv(A) @ np.array([3.0, 9.0])    # pseudo-inverse solve: [1., 1.]
```

Full SVD, the rank-1 approximation from the worked example, and truncated reconstruction of a data matrix.

- **PCA:** scikit-learn computes PCA as the SVD of the centred data matrix: components are the rows of $V^\top$, explained variance is $\sigma_i^2/(n-1)$.
- **Recommenders & topic models:** A truncated SVD of a user–item or term–document matrix gives low-dimensional user, item and word embeddings (latent semantic analysis).
- **Least squares & pseudo-inverse:** `np.linalg.lstsq` and `pinv` solve via the SVD, discarding singular values below a tolerance to stay stable on rank-deficient designs.
- **Model compression:** Replacing an $m\times n$ weight matrix by rank-$k$ factors cuts parameters from $mn$ to $k(m+n)$; LoRA trains exactly such a low-rank update.

> **Singular values are not eigenvalues:** They coincide only for symmetric positive semidefinite matrices. The shear $\begin{pmatrix}1&1\\0&1\end{pmatrix}$ has eigenvalues $1,1$ but singular values $1.618$ and $0.618$: eigenvalues describe directions that are preserved, singular values describe how much the map stretches, whatever the direction.

> **Center before using the SVD as PCA:** On uncentred data the first singular vector mostly points at the mean, and $\sigma_1^2$ measures $\lVert\bar x\rVert^2$ rather than variance. Subtract column means first (sklearn's PCA does; TruncatedSVD deliberately does not, to preserve sparsity).

> **You rarely need the full SVD:** For the top $k$ components of a large matrix, randomized SVD (`sklearn.utils.extmath.randomized_svd`) or Lanczos (`scipy.sparse.linalg.svds`) costs roughly $O(mnk)$ instead of $O(mn\min(m,n))$ and works on sparse inputs.

**Question:** A matrix has singular values $5$, $3$ and $1$. What is the Frobenius-norm error of its best rank-1 approximation?

- A. 1
- B. 3
- C. √10 ≈ 3.16
- D. 4

**Answer:** C. √10 ≈ 3.16 — Eckart–Young: the error is $\sqrt{\sigma_2^2+\sigma_3^2}=\sqrt{9+1}=\sqrt{10}\approx3.16$. (The spectral-norm error would be $\sigma_2=3$.)

**Question:** A $3\times2$ matrix has singular values $\sigma_1=4$, $\sigma_2=0$. Which is true?

- A. rank 2, and no unit vector is stretched beyond 4
- B. rank 1, and the largest ‖Ax‖ over unit x is 4
- C. rank 1, and the largest ‖Ax‖ over unit x is 16
- D. rank 0, since a zero singular value means A = 0

**Answer:** B. rank 1, and the largest ‖Ax‖ over unit x is 4 — The rank is the number of nonzero singular values, here 1. The largest stretch is $\max_{\lVert x\rVert=1}\lVert Ax\rVert=\lVert A\rVert_2=\sigma_1=4$, attained at $x=v_1$; $v_2$ is mapped to 0.

**Key points**

- Every matrix is $U\Sigma V^\top$: rotate, stretch along orthogonal axes by $\sigma_i\ge0$, rotate.
- $\sigma_i^2$ are the eigenvalues of $A^\top A$; $Av_i=\sigma_iu_i$ maps the unit circle's axes to the ellipse's axes.
- Truncating to the top $k$ terms is the best rank-$k$ approximation; the error is set by the discarded $\sigma_i$.
- SVD powers PCA, pseudo-inverses, recommenders and low-rank compression; $\sigma_1/\sigma_r$ is the condition number.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
