# Truncated SVD

> Compute only the top k singular vectors without centering the data, so it scales to huge sparse matrices such as text (LSA).

Source: https://www.mlfoundry.in/learn/truncated-svd  
Section: ML Algorithms › Unsupervised · Dimensionality Reduction  
Level: core · 16 min read · updated 2026-10-07

**Truncated SVD:** computes only the $k$ largest singular values and their singular vectors of a data matrix $X$ ($n$ rows × $d$ features), **without subtracting the column means first**. It maps every row to $k$ numbers, $z = xV_k$, and gives the best rank-$k$ approximation $X_k=U_k\Sigma_kV_k^\top$ of $X$ itself. Run on a document–term matrix, this is **Latent Semantic Analysis (LSA)**.

- **Task:** Unsupervised linear dimensionality reduction; topic / concept extraction from text (LSA)
- **Input → output:** a (usually sparse) $n\times d$ matrix → dense $n\times k$ coordinates $Z = XV_k = U_k\Sigma_k$
- **Learns:** $k$ right singular vectors (the rows of $V_k^\top$, scikit-learn's `components_`) and singular values $\sigma_1\ge\dots\ge\sigma_k$
- **Training:** randomized: about $O\big(\text{nnz}(X)\,(k+p)\,(q+1) + (n+d)(k+p)^2\big)$; never forms a dense $n\times d$ matrix
- **Prediction:** $O(\text{nnz}(x)\cdot k)$ per new row: one sparse vector–matrix product $xV_k$
- **Hyperparameters:** `n_components` $k$, `algorithm` (randomized / arpack), `n_iter` $q$, `n_oversamples` $p$; plus the input weighting (counts, tf-idf, normalisation)
- **Assumes:** the signal is approximately low-rank and linear; distances and angles in the input space are meaningful (so the weighting of features matters)

## The problem

Text, ratings and click logs come as enormous, mostly-empty matrices. A corpus of a million documents over a 100,000-word vocabulary with ten non-zero counts per row has $10^7$ non-zeros: about **0.12 GB** in sparse CSR format. The obvious way to compress it is **PCA**, but PCA first subtracts each column's mean. The mean of a word column is not zero, so after centring *every* one of the $10^{11}$ entries is non-zero, and the dense matrix needs **800 GB**. A full SVD would then cost $O(nd\min(n,d))$ on top. Neither fits in memory.

Text has a second problem. Search by matching words fails on **synonyms**: a query for *auto* has cosine similarity exactly 0 with a document that only says *car*, because the two share no word. We want a representation in which words that appear in the same contexts end up close together.

## The idea

Skip the centring and factorise $X$ as it is. Every matrix can be written as a sum of rank-1 pieces $\sigma_i u_i v_i^\top$, sorted by size $\sigma_i$ (the full theory is in the **SVD** article). Keep only the $k$ largest pieces and drop the rest. Each kept right singular vector $v_i$ is a weighted mix of features, which in text reads as a **latent concept** (*vehicles*, *fruit*). Each document is described by how much of each concept it contains. Words that co-occur, or that share neighbours like *car* and *auto* both sharing *engine*, load on the same concepts, so documents that use different words for the same thing land close together.

Because only products like $Xv$ and $X^\top u$ are ever needed, the algorithm can work directly on the sparse matrix, touching only its non-zeros. That is why it scales: the cost grows with $\text{nnz}(X)$, not with $n\times d$.

**Vocabulary**

- **Singular value σᵢ:** the length of $X$ along direction $v_i$: $\sigma_i=\lVert Xv_i\rVert$. Sorted $\sigma_1\ge\sigma_2\ge\dots\ge0$
- **Right singular vector vᵢ:** a unit vector in feature space (one weight per term), the $i$-th **component**. Stored as row $i$ of `components_`
- **Left singular vector uᵢ:** a unit vector over the rows: $u_i = Xv_i/\sigma_i$. Document coordinates are $U_k\Sigma_k$
- **Rank-k approximation:** $X_k=\sum_{i\le k}\sigma_iu_iv_i^\top$, the closest rank-$k$ matrix to $X$
- **Energy:** the fraction of $\lVert X\rVert_F^2=\sum_i\sigma_i^2$ kept by the first $k$ components
- **Latent space / concept:** the $k$-dimensional space of coordinates $z=xV_k$; each axis is a concept (a weighted mix of words)
- **Fold-in:** projecting a new row, e.g. a search query, into the latent space with the *learned* $V_k$: $z_q=qV_k$
- **LSA / LSI:** Latent Semantic Analysis / Indexing: truncated SVD of a (tf-idf) document–term matrix, used for retrieval and similarity
- **Sketch, oversampling, power iteration:** randomized SVD ideas: multiply $X$ by a random $d\times(k+p)$ matrix to capture its top range ($p$ = extra **oversamples**), and repeat $q$ times with $XX^\top$ (**power iterations**) to sharpen it

## The math

Start from the thin SVD of the uncentred matrix and cut it after $k$ terms:

$$
X = U\Sigma V^\top=\sum_{i=1}^{r}\sigma_i\,u_iv_i^\top \quad\Longrightarrow\quad X_k=U_k\Sigma_kV_k^\top=\sum_{i=1}^{k}\sigma_i\,u_iv_i^\top
$$

- $X$: $n\times d$ data matrix, rows = samples (documents), columns = features (terms); **not** centred
- $r$: rank of $X$, at most $\min(n,d)$
- $U_k$: $n\times k$, orthonormal columns $u_1..u_k$ (left singular vectors)
- $\Sigma_k$: $k\times k$ diagonal matrix of the largest singular values $\sigma_1\ge\dots\ge\sigma_k$
- $V_k$: $d\times k$, orthonormal columns $v_1..v_k$ (right singular vectors = components)

**Why truncation is the right thing to do (Eckart–Young).** Among *all* matrices of rank at most $k$, $X_k$ is the closest to $X$, and the leftover error is exactly the energy in the dropped singular values:

$$
X_k=\arg\min_{\operatorname{rank}(B)\le k}\lVert X-B\rVert_F,\qquad \lVert X-X_k\rVert_F^2=\sum_{i>k}\sigma_i^2,\qquad \text{energy}(k)=\frac{\sum_{i\le k}\sigma_i^2}{\sum_i\sigma_i^2}
$$

- $\lVert A\rVert_F$: Frobenius norm, $\sqrt{\sum_{ij}A_{ij}^2}$
- $\sum_i\sigma_i^2$: equals $\lVert X\rVert_F^2$, the total sum of squared entries

This is how $k$ acts mathematically: raising $k$ by one removes $\sigma_{k+1}^2$ from the squared error. The singular-value spectrum tells you directly how much each extra component buys.

**The transform.** The reduced coordinates are the projection of each row onto the components. Since $V^\top V_k$ is the identity stacked on zeros, the projection collapses to the first $k$ columns of $U\Sigma$:

$$
Z = XV_k = U\Sigma V^\top V_k = U\Sigma\begin{bmatrix}I_k\\0\end{bmatrix}=U_k\Sigma_k,\qquad z_q = q\,V_k
$$

- $Z$: $n\times k$ document coordinates (what `fit_transform` returns)
- $q$: a new row in the same feature space, e.g. a query's term counts
- $z_q$: its latent coordinates; compare to rows of $Z$ by cosine similarity

> **Two fold-in conventions:** Classical LSA papers fold a query in as $\hat q=qV_k\Sigma_k^{-1}$, which puts it on the scale of $U_k$. scikit-learn's `transform` returns $qV_k$, on the scale of $U_k\Sigma_k$. The two differ only by a per-axis scale. Be consistent: compare queries to documents produced by the same formula.

**Where the components come from.** Substitute the SVD into the $d\times d$ Gram matrix: $X^\top X=V\Sigma U^\top U\Sigma V^\top=V\Sigma^2V^\top$. So the $v_i$ are the eigenvectors of $X^\top X$ and $\sigma_i^2$ are its eigenvalues; $v_1$ maximises $\lVert Xv\rVert^2$ over unit vectors. Compare PCA, which diagonalises the **covariance** $\tfrac1{n-1}(X-\mathbf 1\mu^\top)^\top(X-\mathbf 1\mu^\top)$. Without centring, $X^\top X$ is a matrix of *second moments about the origin*, so for non-negative data $v_1$ points roughly along the mean vector $\mu$. In the worked example below, $\cos(v_1,\mu)=0.998$. Centre $X$ first and truncated SVD returns exactly the PCA components.

**The randomized algorithm (Halko, Martinsson & Tropp).** Finding the top-$k$ eigenvectors does not need the whole spectrum. Draw a Gaussian $\Omega\in\mathbb R^{d\times(k+p)}$. Then $Y=X\Omega$ is a set of random combinations of $X$'s columns, which with high probability spans the top-$k$ left singular directions. Power iterations multiply by $XX^\top$ before orthonormalising:

$$
Y=(XX^\top)^{q}X\Omega=U\Sigma^{2q+1}V^\top\Omega,\qquad Q=\operatorname{qr}(Y),\qquad B=Q^\top X=\tilde U\Sigma\tilde V^\top,\qquad U_k\approx Q\tilde U_{:,1..k}
$$

- $\Omega$: $d\times(k+p)$ random Gaussian test matrix
- $p$: oversamples (default 10): extra random directions that absorb error from the tail
- $q$: power iterations (`n_iter`); each one raises every singular value to a higher odd power
- $Q$: $n\times(k+p)$ orthonormal basis for the range of $Y$
- $B$: small $(k+p)\times d$ matrix; its exact SVD is cheap

The $\Sigma^{2q+1}$ factor is the key to tuning $q$. The tail directions pollute the sketch in proportion to $(\sigma_{k+1}/\sigma_k)^{2q+1}$, so each iteration shrinks that ratio geometrically. Text spectra decay slowly ($\sigma_{k+1}\approx\sigma_k$), which is why scikit-learn defaults to $q=5$ rather than 0. Oversampling $p$ works alongside it: with $k+p$ random directions instead of $k$, the chance that the sketch misses part of the top-$k$ subspace falls very quickly. Every step needs only the products $X\cdot(\text{thin})$ and $X^\top\cdot(\text{thin})$, which is $O(\text{nnz}(X)(k+p))$ on a sparse matrix.

## Worked example: LSA on seven tiny documents

Six terms, seven documents. Three are about vehicles, three about fruit, and D7, *apple car*, mixes both. The matrix holds raw counts; 16 of its 42 entries are non-zero. We keep $k=2$ components and then search for the one-word query **auto**.

Document–term count matrix X (7 × 6). D1 never contains *auto*, so a word-matching search for “auto” gives it a score of 0.

| Doc | Text | car | auto | engine | fruit | apple | juice |
| --- | --- | --- | --- | --- | --- | --- | --- |
| D1 | car car engine | 2 | 0 | 1 | 0 | 0 | 0 |
| D2 | auto auto engine | 0 | 2 | 1 | 0 | 0 | 0 |
| D3 | car auto engine engine | 1 | 1 | 2 | 0 | 0 | 0 |
| D4 | fruit fruit apple | 0 | 0 | 0 | 2 | 1 | 0 |
| D5 | apple apple juice | 0 | 0 | 0 | 0 | 2 | 1 |
| D6 | fruit apple juice juice | 0 | 0 | 0 | 1 | 1 | 2 |
| D7 | apple car | 1 | 0 | 0 | 0 | 1 | 0 |

**Rank-2 truncated SVD by hand**

1. **Gram matrix.** $G=X^\top X$ (6 × 6), e.g. $G_{\text{car},\text{car}}=2^2+1^2+1^2=6$ and $G_{\text{car},\text{engine}}=2\cdot1+1\cdot2=4$. The full matrix is in the table below. Note that $G$ has no centring and is block-structured: vehicle terms only touch fruit terms through D7's car–apple entry $G_{\text{car},\text{apple}}=1$.
2. **Top eigenpairs of G.** $\lambda_1=12.371$, $\lambda_2=11.594$, so $\sigma_1=\sqrt{12.371}=3.517$ and $\sigma_2=3.405$, with $v_1=(0.410,\,0.297,\,0.444,\,0.325,\,0.543,\,0.383)$ and $v_2=(0.363,\,0.375,\,0.527,\,-0.320,\,-0.455,\,-0.373)$ over (car, auto, engine, fruit, apple, juice). Check one entry of $Gv_1=\lambda_1v_1$ for *car*: $6(0.410)+1(0.297)+4(0.444)+1(0.543)=5.077$ and $12.371\times0.410=5.077$ (using 4-decimal $v_1$).
3. **Read the components.** $v_1$ is all-positive: it is the "how many words, of any kind" axis, almost parallel to the column-mean vector ($\cos=0.998$). $v_2$ is the **topic contrast**: positive on car / auto / engine, negative on fruit / apple / juice.
4. **Document coordinates** $z=xV_2$. D1 $=2\,v_{\text{car}}+1\,v_{\text{engine}}=(2\cdot0.410+0.444,\;2\cdot0.363+0.527)=(1.265,\,1.253)$. D7 $=v_{\text{car}}+v_{\text{apple}}=(0.953,\,-0.093)$, which sits between the topics. All seven are in the next table.
5. **How much is kept.** $\lVert X\rVert_F^2=34$ (sum of squared counts). Energy $=(12.371+11.594)/34=70.5\%$, and $\lVert X-X_2\rVert_F=\sqrt{34-23.965}=3.168$. The rank-2 reconstruction gives D1 an *auto* entry of $0.845$, even though D1 never contains the word.
6. **Fold in the query** $q=$ “auto” $=(0,1,0,0,0,0)$: $z_q=qV_2=$ the *auto* row of $V_2$ $=(0.297,\,0.375)$.
7. **Cosine to D1.** $z_q\cdot z_{D1}=0.297(1.265)+0.375(1.253)=0.845$; $\lVert z_q\rVert=0.478$, $\lVert z_{D1}\rVert=1.780$, so $\cos=0.845/(0.478\cdot1.780)=0.993$. In raw count space it was $0$.
8. **Rank the documents** (table below). Latent space: D2 (1.000), D3 (0.998), D1 (0.993), then D7 (0.542). The fruit documents score about $-0.05$ to $-0.08$. The search found D1 through the shared context word *engine*.

Gram matrix $G=X^\top X$. Its eigenvectors are the components; its eigenvalues are $\sigma_i^2$.

| G | car | auto | engine | fruit | apple | juice |
| --- | --- | --- | --- | --- | --- | --- |
| **car** | 6 | 1 | 4 | 0 | 1 | 0 |
| **auto** | 1 | 5 | 4 | 0 | 0 | 0 |
| **engine** | 4 | 4 | 6 | 0 | 0 | 0 |
| **fruit** | 0 | 0 | 0 | 5 | 3 | 2 |
| **apple** | 1 | 0 | 0 | 3 | 7 | 4 |
| **juice** | 0 | 0 | 0 | 2 | 4 | 5 |

Documents in the 2-D latent space and their cosine similarity to the query “auto”, in raw count space vs latent space.

| Doc | z₁ | z₂ | raw cosine | latent cosine (k = 2) |
| --- | --- | --- | --- | --- |
| D1 | 1.265 | 1.253 | 0.000 | **0.993** |
| D2 | 1.037 | 1.277 | 0.894 | 1.000 |
| D3 | 1.595 | 1.793 | 0.408 | 0.998 |
| D4 | 1.192 | −1.096 | 0.000 | −0.074 |
| D5 | 1.468 | −1.284 | 0.000 | −0.049 |
| D6 | 1.633 | −1.523 | 0.000 | −0.081 |
| D7 | 0.953 | −0.093 | 0.000 | 0.542 |

The same mechanism works on terms. Their latent vectors are the rows of $V_k\Sigma_k$, and the cosine between *car* and *auto* rises from $0.183$ (raw document columns) to $0.984$. The rank-2 model has learned that they are used interchangeably.

> Figure: The worked example, computed live. Top: the count matrix X (16/42 non-zero), its rank-k reconstruction X_k, and the six singular values (3.517, 3.405, 2.141, 1.781, 1.341, 0.694). Drag k to see the Frobenius error fall from 4.651 (k = 1) through 3.168 (k = 2, 70.5% of the energy) to 0 (k = 6). Bottom: documents on components 1 and 2; the dashed ray is the folded-in query. With the query “auto” and k = 2, D1 goes from raw cosine 0.000 to latent 0.993. Toggle query terms to search, and push k to 6 to watch the latent cosines turn back into the raw ones. “Centre first” switches to PCA: the matrix becomes 42/42 dense, the mean-direction component disappears, and the raw-count query no longer lines up with D1 (cosine −0.308).

> **Low rank is what creates the synonymy:** At $k=6$ (full rank) $V$ is an orthogonal rotation, so every latent cosine equals the raw cosine and D1 is back to 0. Throwing away the small singular directions forces documents that use different words in the same context onto the same few axes. **Truncation is the model, not just compression.** Too small a $k$ over-merges: at $k=1$ every document collapses onto one axis and all cosines become 1.

## The algorithm, step by step

**Randomized truncated SVD (scikit-learn's default)**

1. **Weight the input** (optional but standard for text): counts → tf-idf, often with sublinear tf. Keep it sparse.
2. **Sketch.** Draw $\Omega\in\mathbb R^{d\times(k+p)}$ with $\mathcal N(0,1)$ entries; $Y\leftarrow X\Omega$ ($n\times(k+p)$, dense but thin).
3. **Power iterations** ($q$ times): $Q\leftarrow\operatorname{qr}(Y)$; $Q\leftarrow\operatorname{qr}(X^\top Q)$; $Y\leftarrow XQ$. Re-orthonormalising (QR or LU) each time stops all columns from collapsing onto $u_1$ in floating point.
4. **Basis.** $Q\leftarrow\operatorname{qr}(Y)$, with orthonormal columns spanning the approximate top range of $X$.
5. **Project and solve small.** $B\leftarrow Q^\top X$ ($(k+p)\times d$); compute its exact SVD $B=\tilde U\Sigma\tilde V^\top$.
6. **Lift and cut.** $U\leftarrow Q\tilde U$; keep the first $k$ columns of $U$, values of $\Sigma$, rows of $\tilde V^\top$. Fix signs so the largest-magnitude entry of each $v_i$ is positive (singular vectors are only defined up to sign).
7. **Transform.** $Z=XV_k$ for training rows and $z=xV_k$ for new ones.

**Complexity**

- **Randomized:** $O\big(\text{nnz}(X)(k+p)(2q+2)\big)$ for the sparse products + $O\big((n+d)(k+p)^2\big)$ for QR and the small SVD. Memory: $X$ plus $O((n+d)(k+p))$
- **ARPACK (Lanczos):** iterative eigen-solver on $X^\top X$ via repeated $Xv$, $X^\top u$ products. More accurate for small $k$, often slower for large $k$, not parallel-friendly
- **Full SVD, for contrast:** $O(nd\min(n,d))$ time and a dense $n\times d$ copy, impossible at the 1M × 100k scale above
- **Transform:** $O(\text{nnz}(x)\,k)$ per row

## Variants and tuning

Effect of power iterations $q$ on a 2000 × 1000 sparse count matrix (1% non-zero), $k=10$, $p=10$: worst relative error among the top-10 singular values, and the 10th singular value (exact 16.653).

| n_iter q | max relative error | σ₁₀ estimate |
| --- | --- | --- |
| 0 | 0.439 | 12.137 |
| 1 | 0.142 | 14.286 |
| 2 | 0.090 | 15.159 |
| 4 | 0.047 | 15.865 |
| 7 | 0.025 | 16.240 |

A random sketch always *underestimates* the singular values: it sees the right directions only partially. Each power iteration pulls the estimates up towards the truth. Flat spectra like this one, typical of sparse data, need several iterations.

**Knobs that matter**

- **n_components k:** for LSA, 100–300 is the classic range for retrieval and clustering; for visualisation, 2–3. Pick by the energy curve, or better, by downstream validation score (retrieval precision, classifier accuracy). The energy curve of text has no clear elbow
- **algorithm:** `randomized` (default, fast, large $k$ fine) vs `arpack` (Lanczos, exact to tolerance `tol`, needs $k<\min(n,d)$)
- **n_iter q:** default 5. Raise it for slowly decaying spectra or when components look noisy across seeds; each one costs two more passes over $X$
- **n_oversamples p:** default 10. Raise it (e.g. 20–50) when k is small relative to a flat spectrum
- **Input weighting:** tf-idf instead of counts stops frequent words dominating $v_1$. L2-normalise rows (or use `Normalizer` after the SVD) so cosine compares topics, not document lengths
- **Centring:** dense data that fits in memory: just use **PCA**, which centres. Sparse data: truncated SVD, or scikit-learn's `PCA` with `svd_solver="arpack"` / `"covariance_eigh"`, which centre *implicitly* without densifying (recent versions)

Related methods: **NMF** factorises the same non-negative matrix but forces non-negative, parts-based components that read more like topics. **LDA (latent Dirichlet allocation)** is the probabilistic topic-model successor of LSA. **Incremental / streaming SVD** (e.g. gensim's `LsiModel`) updates the factorisation in chunks when the corpus doesn't fit on one machine.

## In code

```python
import numpy as np

def truncated_svd(X, k, n_oversamples=10, n_iter=4, seed=0):
    """Randomized truncated SVD (Halko et al., 2011). X may be a NumPy or scipy.sparse matrix."""
    rng = np.random.default_rng(seed)
    n, d = X.shape
    l = min(k + n_oversamples, d)
    Q = X @ rng.standard_normal((d, l))          # n x l sketch of the column space
    for _ in range(n_iter):                        # power iterations sharpen the spectrum
        Q, _ = np.linalg.qr(Q)
        Q, _ = np.linalg.qr(X.T @ Q)
        Q = X @ Q
    Q, _ = np.linalg.qr(Q)                         # orthonormal basis for range(X)
    B = (X.T @ Q).T                                # small l x d matrix  B = Q^T X
    Ub, s, Vt = np.linalg.svd(B, full_matrices=False)
    U = Q @ Ub
    U, s, Vt = U[:, :k], s[:k], Vt[:k]
    flip = np.sign(Vt[np.arange(k), np.abs(Vt).argmax(axis=1)])   # largest |entry| of each v positive
    return U * flip, s, Vt * flip[:, None]

terms = ["car", "auto", "engine", "fruit", "apple", "juice"]
X = np.array([[2, 0, 1, 0, 0, 0], [0, 2, 1, 0, 0, 0], [1, 1, 2, 0, 0, 0],
              [0, 0, 0, 2, 1, 0], [0, 0, 0, 0, 2, 1], [0, 0, 0, 1, 1, 2],
              [1, 0, 0, 0, 1, 0]], dtype=float)

U, s, Vt = truncated_svd(X, k=2)
Z = U * s                                          # document coordinates = X @ V_k
print("singular values:", s.round(4))
print("V_k^T:\n", Vt.round(4))
print("docs in LSA space:\n", Z.round(4))
print("||X - X_k||_F =", round(np.linalg.norm(X - Z @ Vt), 4))

q = np.array([0, 1, 0, 0, 0, 0.])                  # query: "auto"
zq = q @ Vt.T                                      # fold in: q V_k
cos = lambda a, B: B @ a / np.linalg.norm(B, axis=1) / np.linalg.norm(a)
print("query:", zq.round(4))
print("raw cosine:", cos(q, X).round(3))
print("LSA cosine:", cos(zq, Z).round(3))
```

From scratch: randomized truncated SVD with power iterations and scikit-learn's sign convention, run on the worked example. Prints σ = [3.5172 3.405], the document coordinates from the table, ‖X − X₂‖ = 3.1677, and the raw vs latent cosines (D1: 0.0 → 0.993).

```python
from sklearn.decomposition import TruncatedSVD
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.metrics.pairwise import cosine_similarity

docs = ["car car engine", "auto auto engine", "car auto engine engine",
        "fruit fruit apple", "apple apple juice", "fruit apple juice juice",
        "apple car"]
vec = CountVectorizer(vocabulary=["car", "auto", "engine", "fruit", "apple", "juice"])
X = vec.fit_transform(docs)                        # scipy.sparse CSR, 16 non-zeros of 42
print(type(X).__name__, X.nnz, "non-zeros")

svd = TruncatedSVD(n_components=2, algorithm="randomized", n_iter=5, random_state=0)
Z = svd.fit_transform(X)                           # = U_k Sigma_k = X V_k
print("singular values:", svd.singular_values_.round(4))
print("components:\n", svd.components_.round(4))
print("explained variance ratio:", svd.explained_variance_ratio_.round(4))
print("docs:\n", Z.round(4))

zq = svd.transform(vec.transform(["auto"]))
print("query 'auto':", zq.round(4))
print("raw cosine:", cosine_similarity(vec.transform(["auto"]), X).round(3))
print("LSA cosine:", cosine_similarity(zq, Z).round(3))

# Real LSA pipeline:
# make_pipeline(TfidfVectorizer(sublinear_tf=True), TruncatedSVD(200), Normalizer(copy=False))
```

scikit-learn on a sparse matrix built from the raw text. Same components and coordinates; note explained_variance_ratio_ = [0.0195 0.5263].

> **explained_variance_ratio_ is not sorted:** scikit-learn reports the **variance** of each output column $Z_{:,i}$ divided by the total variance of $X$. Variance is measured around the mean, but the components were chosen by uncentred energy $\sigma_i^2$. In the worked example the first component carries the most energy ($\sigma_1^2=12.371$) yet only **1.95%** of the variance, while the second carries **52.6%**. That's because $v_1$ is the mean direction, and every document has a similar amount of it. Don't use this ratio to choose $k$ or to rank components; use `singular_values_` or downstream performance.

> **Never call .toarray() to centre or scale:** `StandardScaler()` on a sparse matrix raises an error unless `with_mean=False`, and densifying to centre turns a 0.12 GB matrix into 800 GB. If you want PCA semantics on sparse data, use an implicit-centring solver; otherwise accept that component 1 mostly measures document length / overall frequency, and normalise rows afterwards.

> **Signs and seeds:** Each $(u_i,v_i)$ pair is defined only up to a sign flip, and with randomized solvers a small change in data or seed can flip or (for near-equal $\sigma$'s, like 3.517 vs 3.405 here) even rotate components. Never hard-code "component 2 = vehicles". Fix `random_state`, and compare subspaces or downstream results rather than individual vectors.

**Strengths**

- The data is large and sparse (tf-idf text, one-hot features, user–item matrices) and must stay sparse.
- You need dense, low-dimensional features for retrieval, clustering or a linear classifier on text (LSA).
- You want a fast, deterministic-given-seed, linear reduction whose transform is a single matrix product.
- You need only the top few singular directions of a huge matrix (spectral methods, low-rank approximation).

**Limitations**

- The data is dense and fits in memory: use PCA, which centres, so components and variance ratios mean what you expect.
- You need interpretable topics: components mix positive and negative weights; NMF or probabilistic topic models read better.
- The structure is nonlinear (manifolds, clusters in curved shapes): see kernel PCA, t-SNE or UMAP.
- Word order or polysemy matters: a bag-of-words SVD gives one vector per word regardless of sense; use contextual embeddings.

- **Latent semantic indexing:** Search engines and document-similarity systems map tf-idf vectors to ~200 latent dimensions, so queries match documents that share no exact words.
- **Text classification features:** TF-IDF → TruncatedSVD → logistic regression / SVM: a compact, fast pipeline for spam, sentiment and topic classification.
- **Recommender systems:** Low-rank factorisation of sparse user–item rating or click matrices (the SVD family behind the Netflix Prize models).
- **Pre-step for clustering and visualisation:** Reduce sparse text to 50–100 dims before k-means, or before t-SNE / UMAP, which struggle on 100k sparse dimensions.

**Question:** Why does scikit-learn's TruncatedSVD not centre the data, unlike PCA?

- A. Centring would change the singular values only slightly
- B. Subtracting column means makes a sparse matrix dense, destroying the memory and speed advantage
- C. SVD is undefined for centred matrices
- D. Centring would make the components non-orthogonal

**Answer:** B. Subtracting column means makes a sparse matrix dense, destroying the memory and speed advantage — Column means are non-zero, so after subtracting them every entry is non-zero. A 1M × 100k matrix with 10M non-zeros goes from about 0.12 GB (CSR) to 800 GB dense.

**Question:** A matrix has singular values $(4, 3, 2, 1)$. You keep $k=2$. What are the energy kept and $\lVert X-X_2\rVert_F$?

- A. 50% and 3
- B. 83.3% and 2.236
- C. 70% and 1.732
- D. 83.3% and 5

**Answer:** B. 83.3% and 2.236 — Energy $=(16+9)/(16+9+4+1)=25/30=83.3\%$. By Eckart–Young the error is $\sqrt{2^2+1^2}=\sqrt5=2.236$.

**Question:** In the worked example, the query “auto” has raw cosine 0 with D1 but latent cosine 0.993 at k = 2. What happens to that latent cosine at k = 6 (full rank)?

- A. It stays 0.993
- B. It becomes 1
- C. It returns to 0, because a full-rank V is just a rotation
- D. It becomes negative

**Answer:** C. It returns to 0, because a full-rank V is just a rotation — With all 6 components, $V$ is a $6\times6$ orthogonal matrix, and rotations preserve angles: $\cos(qV, xV)=\cos(q,x)=0$. The similarity comes from discarding components.

**Question:** On non-negative count data, the first truncated-SVD component typically…

- A. separates the two biggest topics
- B. points roughly along the mean document vector, so it measures overall word volume
- C. has the largest explained_variance_ratio_
- D. is identical to PCA's first component

**Answer:** B. points roughly along the mean document vector, so it measures overall word volume — Without centring, $v_1$ maximises $\lVert Xv\rVert^2$, which for non-negative rows is dominated by the shared mean direction. Here $\cos(v_1,\mu)=0.998$ and its variance ratio is only 0.0195. The topic contrast appears in component 2, which matches PCA's first component (variance ratio 0.526 vs PCA's 0.527).

**Key points**

- Truncated SVD keeps the top $k$ terms $\sigma_iu_iv_i^\top$ of the **uncentred** matrix: $Z=XV_k=U_k\Sigma_k$. On centred data it is exactly PCA.
- Eckart–Young: $X_k$ is the best rank-$k$ approximation and $\lVert X-X_k\rVert_F^2=\sum_{i>k}\sigma_i^2$, so the spectrum tells you what each extra component buys.
- Randomized SVD needs only products with $X$ and $X^\top$, so it costs $O(\text{nnz}\cdot k)$ and keeps sparse data sparse. Power iterations ($\Sigma^{2q+1}$) fix flat spectra.
- On text (LSA), low rank merges words used in the same contexts: “auto” finds the *car* document (cosine 0 → 0.993). Truncation is what creates the similarity.
- Without centring, component 1 ≈ the mean direction; `explained_variance_ratio_` is then unsorted and misleading. Use tf-idf and row normalisation.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
