Truncated SVD

Compute only the top k singular vectors without centering the data, so it scales to huge sparse matrices such as text (LSA).

16 min read1641 wordscore7 viewsupdated 2026-10-07

Definition

Truncated SVD computes only the kk largest singular values and their singular vectors of a data matrix XX (nn rows × dd features), without subtracting the column means first. It maps every row to kk numbers, z=xVkz = xV_k, and gives the best rank-kk approximation Xk=UkΣkVk⊤X_k=U_k\Sigma_kV_k^\top of XX itself. Run on a document–term matrix, this is Latent Semantic Analysis (LSA).

At a glance

Task
Unsupervised linear dimensionality reduction; topic / concept extraction from text (LSA)
Input → output
a (usually sparse) n×dn\times d matrix → dense n×kn\times k coordinates Z=XVk=UkΣkZ = XV_k = U_k\Sigma_k
Learns
kk right singular vectors (the rows of Vk⊤V_k^\top, scikit-learn's components_) and singular values σ1≥⋯≥σk\sigma_1\ge\dots\ge\sigma_k
Training
randomized: about O(nnz(X) (k+p) (q+1)+(n+d)(k+p)2)O\big(\text{nnz}(X)\,(k+p)\,(q+1) + (n+d)(k+p)^2\big); never forms a dense n×dn\times d matrix
Prediction
O(nnz(x)⋅k)O(\text{nnz}(x)\cdot k) per new row: one sparse vector–matrix product xVkxV_k
Hyperparameters
n_components kk, algorithm (randomized / arpack), n_iter qq, n_oversamples pp; plus the input weighting (counts, tf-idf, normalisation)
Assumes
the signal is approximately low-rank and linear; distances and angles in the input space are meaningful (so the weighting of features matters)

The problem

Text, ratings and click logs come as enormous, mostly-empty matrices. A corpus of a million documents over a 100,000-word vocabulary with ten non-zero counts per row has 10710^7 non-zeros: about 0.12 GB in sparse CSR format. The obvious way to compress it is PCA, but PCA first subtracts each column's mean. The mean of a word column is not zero, so after centring every one of the 101110^{11} entries is non-zero, and the dense matrix needs 800 GB. A full SVD would then cost O(ndmin⁡(n,d))O(nd\min(n,d)) on top. Neither fits in memory.

Text has a second problem. Search by matching words fails on synonyms: a query for auto has cosine similarity exactly 0 with a document that only says car, because the two share no word. We want a representation in which words that appear in the same contexts end up close together.

The idea

Skip the centring and factorise XX as it is. Every matrix can be written as a sum of rank-1 pieces σiuivi⊤\sigma_i u_i v_i^\top, sorted by size σi\sigma_i (the full theory is in the SVD article). Keep only the kk largest pieces and drop the rest. Each kept right singular vector viv_i is a weighted mix of features, which in text reads as a latent concept (vehicles, fruit). Each document is described by how much of each concept it contains. Words that co-occur, or that share neighbours like car and auto both sharing engine, load on the same concepts, so documents that use different words for the same thing land close together.

Because only products like XvXv and X⊤uX^\top u are ever needed, the algorithm can work directly on the sparse matrix, touching only its non-zeros. That is why it scales: the cost grows with nnz(X)\text{nnz}(X), not with n×dn\times d.

Vocabulary

Singular value σᵢ
the length of XX along direction viv_i: σi=∥Xvi∥\sigma_i=\lVert Xv_i\rVert. Sorted σ1≥σ2≥⋯≥0\sigma_1\ge\sigma_2\ge\dots\ge0
Right singular vector vᵢ
a unit vector in feature space (one weight per term), the ii-th component. Stored as row ii of components_
Left singular vector uᵢ
a unit vector over the rows: ui=Xvi/σiu_i = Xv_i/\sigma_i. Document coordinates are UkΣkU_k\Sigma_k
Rank-k approximation
Xk=∑i≤kσiuivi⊤X_k=\sum_{i\le k}\sigma_iu_iv_i^\top, the closest rank-kk matrix to XX
Energy
the fraction of ∥X∥F2=∑iσi2\lVert X\rVert_F^2=\sum_i\sigma_i^2 kept by the first kk components
Latent space / concept
the kk-dimensional space of coordinates z=xVkz=xV_k; each axis is a concept (a weighted mix of words)
Fold-in
projecting a new row, e.g. a search query, into the latent space with the learned VkV_k: zq=qVkz_q=qV_k
LSA / LSI
Latent Semantic Analysis / Indexing: truncated SVD of a (tf-idf) document–term matrix, used for retrieval and similarity
Sketch, oversampling, power iteration
randomized SVD ideas: multiply XX by a random d×(k+p)d\times(k+p) matrix to capture its top range (pp = extra oversamples), and repeat qq times with XX⊤XX^\top (power iterations) to sharpen it

The math

Start from the thin SVD of the uncentred matrix and cut it after kk terms:

X=UΣV⊤=∑i=1rσi uivi⊤⟹Xk=UkΣkVk⊤=∑i=1kσi uivi⊤X = U\Sigma V^\top=\sum_{i=1}^{r}\sigma_i\,u_iv_i^\top \quad\Longrightarrow\quad X_k=U_k\Sigma_kV_k^\top=\sum_{i=1}^{k}\sigma_i\,u_iv_i^\top
XX
n×dn\times d data matrix, rows = samples (documents), columns = features (terms); not centred
rr
rank of XX, at most min⁡(n,d)\min(n,d)
UkU_k
n×kn\times k, orthonormal columns u1..uku_1..u_k (left singular vectors)
Σk\Sigma_k
k×kk\times k diagonal matrix of the largest singular values σ1≥⋯≥σk\sigma_1\ge\dots\ge\sigma_k
VkV_k
d×kd\times k, orthonormal columns v1..vkv_1..v_k (right singular vectors = components)

Why truncation is the right thing to do (Eckart–Young). Among all matrices of rank at most kk, XkX_k is the closest to XX, and the leftover error is exactly the energy in the dropped singular values:

Xk=arg⁡min⁡rank⁡(B)≤k∥X−B∥F,∥X−Xk∥F2=∑i>kσi2,energy(k)=∑i≤kσi2∑iσi2X_k=\arg\min_{\operatorname{rank}(B)\le k}\lVert X-B\rVert_F,\qquad \lVert X-X_k\rVert_F^2=\sum_{i>k}\sigma_i^2,\qquad \text{energy}(k)=\frac{\sum_{i\le k}\sigma_i^2}{\sum_i\sigma_i^2}
∥A∥F\lVert A\rVert_F
Frobenius norm, ∑ijAij2\sqrt{\sum_{ij}A_{ij}^2}
∑iσi2\sum_i\sigma_i^2
equals ∥X∥F2\lVert X\rVert_F^2, the total sum of squared entries

This is how kk acts mathematically: raising kk by one removes σk+12\sigma_{k+1}^2 from the squared error. The singular-value spectrum tells you directly how much each extra component buys.

The transform. The reduced coordinates are the projection of each row onto the components. Since V⊤VkV^\top V_k is the identity stacked on zeros, the projection collapses to the first kk columns of UΣU\Sigma:

Z=XVk=UΣV⊤Vk=UΣ[Ik0]=UkΣk,zq=q VkZ = XV_k = U\Sigma V^\top V_k = U\Sigma\begin{bmatrix}I_k\\0\end{bmatrix}=U_k\Sigma_k,\qquad z_q = q\,V_k
ZZ
n×kn\times k document coordinates (what fit_transform returns)
qq
a new row in the same feature space, e.g. a query's term counts
zqz_q
its latent coordinates; compare to rows of ZZ by cosine similarity

Where the components come from. Substitute the SVD into the d×dd\times d Gram matrix: X⊤X=VΣU⊤UΣV⊤=VΣ2V⊤X^\top X=V\Sigma U^\top U\Sigma V^\top=V\Sigma^2V^\top. So the viv_i are the eigenvectors of X⊤XX^\top X and σi2\sigma_i^2 are its eigenvalues; v1v_1 maximises ∥Xv∥2\lVert Xv\rVert^2 over unit vectors. Compare PCA, which diagonalises the covariance 1n−1(X−1μ⊤)⊤(X−1μ⊤)\tfrac1{n-1}(X-\mathbf 1\mu^\top)^\top(X-\mathbf 1\mu^\top). Without centring, X⊤XX^\top X is a matrix of second moments about the origin, so for non-negative data v1v_1 points roughly along the mean vector μ\mu. In the worked example below, cos⁡(v1,μ)=0.998\cos(v_1,\mu)=0.998. Centre XX first and truncated SVD returns exactly the PCA components.

The randomized algorithm (Halko, Martinsson & Tropp). Finding the top-kk eigenvectors does not need the whole spectrum. Draw a Gaussian Ω∈Rd×(k+p)\Omega\in\mathbb R^{d\times(k+p)}. Then Y=XΩY=X\Omega is a set of random combinations of XX's columns, which with high probability spans the top-kk left singular directions. Power iterations multiply by XX⊤XX^\top before orthonormalising:

Y=(XX⊤)qXΩ=UΣ2q+1V⊤Ω,Q=qr⁡(Y),B=Q⊤X=U~ΣV~⊤,Uk≈QU~:,1..kY=(XX^\top)^{q}X\Omega=U\Sigma^{2q+1}V^\top\Omega,\qquad Q=\operatorname{qr}(Y),\qquad B=Q^\top X=\tilde U\Sigma\tilde V^\top,\qquad U_k\approx Q\tilde U_{:,1..k}
Ω\Omega
d×(k+p)d\times(k+p) random Gaussian test matrix
pp
oversamples (default 10): extra random directions that absorb error from the tail
qq
power iterations (n_iter); each one raises every singular value to a higher odd power
QQ
n×(k+p)n\times(k+p) orthonormal basis for the range of YY
BB
small (k+p)×d(k+p)\times d matrix; its exact SVD is cheap

The Σ2q+1\Sigma^{2q+1} factor is the key to tuning qq. The tail directions pollute the sketch in proportion to (σk+1/σk)2q+1(\sigma_{k+1}/\sigma_k)^{2q+1}, so each iteration shrinks that ratio geometrically. Text spectra decay slowly (σk+1≈σk\sigma_{k+1}\approx\sigma_k), which is why scikit-learn defaults to q=5q=5 rather than 0. Oversampling pp works alongside it: with k+pk+p random directions instead of kk, the chance that the sketch misses part of the top-kk subspace falls very quickly. Every step needs only the products X⋅(thin)X\cdot(\text{thin}) and X⊤⋅(thin)X^\top\cdot(\text{thin}), which is O(nnz(X)(k+p))O(\text{nnz}(X)(k+p)) on a sparse matrix.

Worked example: LSA on seven tiny documents

Six terms, seven documents. Three are about vehicles, three about fruit, and D7, apple car, mixes both. The matrix holds raw counts; 16 of its 42 entries are non-zero. We keep k=2k=2 components and then search for the one-word query auto.

DocTextcarautoenginefruitapplejuice
D1car car engine201000
D2auto auto engine021000
D3car auto engine engine112000
D4fruit fruit apple000210
D5apple apple juice000021
D6fruit apple juice juice000112
D7apple car100010
Document–term count matrix X (7 × 6). D1 never contains auto, so a word-matching search for “auto” gives it a score of 0.

Rank-2 truncated SVD by hand

  1. 1Gram matrix. G=X⊤XG=X^\top X (6 × 6), e.g. Gcar,car=22+12+12=6G_{\text{car},\text{car}}=2^2+1^2+1^2=6 and Gcar,engine=2⋅1+1⋅2=4G_{\text{car},\text{engine}}=2\cdot1+1\cdot2=4. The full matrix is in the table below. Note that GG has no centring and is block-structured: vehicle terms only touch fruit terms through D7's car–apple entry Gcar,apple=1G_{\text{car},\text{apple}}=1.
  2. 2Top eigenpairs of G. λ1=12.371\lambda_1=12.371, λ2=11.594\lambda_2=11.594, so σ1=12.371=3.517\sigma_1=\sqrt{12.371}=3.517 and σ2=3.405\sigma_2=3.405, with v1=(0.410, 0.297, 0.444, 0.325, 0.543, 0.383)v_1=(0.410,\,0.297,\,0.444,\,0.325,\,0.543,\,0.383) and v2=(0.363, 0.375, 0.527, −0.320, −0.455, −0.373)v_2=(0.363,\,0.375,\,0.527,\,-0.320,\,-0.455,\,-0.373) over (car, auto, engine, fruit, apple, juice). Check one entry of Gv1=λ1v1Gv_1=\lambda_1v_1 for car: 6(0.410)+1(0.297)+4(0.444)+1(0.543)=5.0776(0.410)+1(0.297)+4(0.444)+1(0.543)=5.077 and 12.371×0.410=5.07712.371\times0.410=5.077 (using 4-decimal v1v_1).
  3. 3Read the components. v1v_1 is all-positive: it is the "how many words, of any kind" axis, almost parallel to the column-mean vector (cos⁡=0.998\cos=0.998). v2v_2 is the topic contrast: positive on car / auto / engine, negative on fruit / apple / juice.
  4. 4Document coordinates z=xV2z=xV_2. D1 =2 vcar+1 vengine=(2⋅0.410+0.444,  2⋅0.363+0.527)=(1.265, 1.253)=2\,v_{\text{car}}+1\,v_{\text{engine}}=(2\cdot0.410+0.444,\;2\cdot0.363+0.527)=(1.265,\,1.253). D7 =vcar+vapple=(0.953, −0.093)=v_{\text{car}}+v_{\text{apple}}=(0.953,\,-0.093), which sits between the topics. All seven are in the next table.
  5. 5How much is kept. ∥X∥F2=34\lVert X\rVert_F^2=34 (sum of squared counts). Energy =(12.371+11.594)/34=70.5%=(12.371+11.594)/34=70.5\%, and ∥X−X2∥F=34−23.965=3.168\lVert X-X_2\rVert_F=\sqrt{34-23.965}=3.168. The rank-2 reconstruction gives D1 an auto entry of 0.8450.845, even though D1 never contains the word.
  6. 6Fold in the query q=q= “auto” =(0,1,0,0,0,0)=(0,1,0,0,0,0): zq=qV2=z_q=qV_2= the auto row of V2V_2 =(0.297, 0.375)=(0.297,\,0.375).
  7. 7Cosine to D1. zq⋅zD1=0.297(1.265)+0.375(1.253)=0.845z_q\cdot z_{D1}=0.297(1.265)+0.375(1.253)=0.845; ∥zq∥=0.478\lVert z_q\rVert=0.478, ∥zD1∥=1.780\lVert z_{D1}\rVert=1.780, so cos⁡=0.845/(0.478⋅1.780)=0.993\cos=0.845/(0.478\cdot1.780)=0.993. In raw count space it was 00.
  8. 8Rank the documents (table below). Latent space: D2 (1.000), D3 (0.998), D1 (0.993), then D7 (0.542). The fruit documents score about −0.05-0.05 to −0.08-0.08. The search found D1 through the shared context word engine.
Gcarautoenginefruitapplejuice
car614010
auto154000
engine446000
fruit000532
apple100374
juice000245
Gram matrix G=X⊤XG=X^\top X. Its eigenvectors are the components; its eigenvalues are σi2\sigma_i^2.
Docz₁z₂raw cosinelatent cosine (k = 2)
D11.2651.2530.0000.993
D21.0371.2770.8941.000
D31.5951.7930.4080.998
D41.192−1.0960.000−0.074
D51.468−1.2840.000−0.049
D61.633−1.5230.000−0.081
D70.953−0.0930.0000.542
Documents in the 2-D latent space and their cosine similarity to the query “auto”, in raw count space vs latent space.

The same mechanism works on terms. Their latent vectors are the rows of VkΣkV_k\Sigma_k, and the cosine between car and auto rises from 0.1830.183 (raw document columns) to 0.9840.984. The rank-2 model has learned that they are used interchangeably.

truncated SVD: factorise X as it is · keep k = 2 of 6 componentsD1D2D3D4D5D6D7carautoenginefruitapplejuice201000021000112000000210000021000112100010X counts · 16/42 non-zerocarautoenginefruitapplejuice1.00.81.20.00.10.00.90.81.1-0.10.0-0.11.31.11.7-0.10.0-0.10.1-0.10.00.71.10.90.10.00.00.91.41.00.1-0.1-0.11.01.61.20.40.20.40.30.60.4X_2 · ‖X − X_2‖_F = 3.168singular values σᵢ123456energy kept 70.5%documents on components 1 (→) and 2 (↑) · query “auto” folded in as q·Vcomp 1comp 2D1D2D3D4D5D6D7cos to query: ▬ raw ▬ latent k=2D1 car car engine0.0000.993D2 auto auto engine0.8941.000D3 car auto engine engine0.4080.998D4 fruit fruit apple0.000-0.074D5 apple apple juice0.000-0.049D6 fruit apple juice juice0.000-0.081D7 apple car0.0000.542
query terms:
σ = 3.517 · 3.405 · 2.141 · 1.781 · 1.341 · 0.694 · v₁ = (0.410, 0.297, 0.444, 0.325, 0.543, 0.383)
query in latent space: (0.297, 0.375) · best match D2 · raw cosine of D1 = 0.000, latent = 0.993
fig.The worked example, computed live. Top: the count matrix X (16/42 non-zero), its rank-k reconstruction X_k, and the six singular values (3.517, 3.405, 2.141, 1.781, 1.341, 0.694). Drag k to see the Frobenius error fall from 4.651 (k = 1) through 3.168 (k = 2, 70.5% of the energy) to 0 (k = 6). Bottom: documents on components 1 and 2; the dashed ray is the folded-in query. With the query “auto” and k = 2, D1 goes from raw cosine 0.000 to latent 0.993. Toggle query terms to search, and push k to 6 to watch the latent cosines turn back into the raw ones. “Centre first” switches to PCA: the matrix becomes 42/42 dense, the mean-direction component disappears, and the raw-count query no longer lines up with D1 (cosine −0.308).

The algorithm, step by step

Randomized truncated SVD (scikit-learn's default)

  1. 1Weight the input (optional but standard for text): counts → tf-idf, often with sublinear tf. Keep it sparse.
  2. 2Sketch. Draw Ω∈Rd×(k+p)\Omega\in\mathbb R^{d\times(k+p)} with N(0,1)\mathcal N(0,1) entries; Y←XΩY\leftarrow X\Omega (n×(k+p)n\times(k+p), dense but thin).
  3. 3Power iterations (qq times): Q←qr⁡(Y)Q\leftarrow\operatorname{qr}(Y); Q←qr⁡(X⊤Q)Q\leftarrow\operatorname{qr}(X^\top Q); Y←XQY\leftarrow XQ. Re-orthonormalising (QR or LU) each time stops all columns from collapsing onto u1u_1 in floating point.
  4. 4Basis. Q←qr⁡(Y)Q\leftarrow\operatorname{qr}(Y), with orthonormal columns spanning the approximate top range of XX.
  5. 5Project and solve small. B←Q⊤XB\leftarrow Q^\top X ((k+p)×d(k+p)\times d); compute its exact SVD B=U~ΣV~⊤B=\tilde U\Sigma\tilde V^\top.
  6. 6Lift and cut. U←QU~U\leftarrow Q\tilde U; keep the first kk columns of UU, values of Σ\Sigma, rows of V~⊤\tilde V^\top. Fix signs so the largest-magnitude entry of each viv_i is positive (singular vectors are only defined up to sign).
  7. 7Transform. Z=XVkZ=XV_k for training rows and z=xVkz=xV_k for new ones.

Complexity

Randomized
O(nnz(X)(k+p)(2q+2))O\big(\text{nnz}(X)(k+p)(2q+2)\big) for the sparse products + O((n+d)(k+p)2)O\big((n+d)(k+p)^2\big) for QR and the small SVD. Memory: XX plus O((n+d)(k+p))O((n+d)(k+p))
ARPACK (Lanczos)
iterative eigen-solver on X⊤XX^\top X via repeated XvXv, X⊤uX^\top u products. More accurate for small kk, often slower for large kk, not parallel-friendly
Full SVD, for contrast
O(ndmin⁡(n,d))O(nd\min(n,d)) time and a dense n×dn\times d copy, impossible at the 1M × 100k scale above
Transform
O(nnz(x) k)O(\text{nnz}(x)\,k) per row

Variants and tuning

n_iter qmax relative errorσ₁₀ estimate
00.43912.137
10.14214.286
20.09015.159
40.04715.865
70.02516.240
Effect of power iterations qq on a 2000 × 1000 sparse count matrix (1% non-zero), k=10k=10, p=10p=10: worst relative error among the top-10 singular values, and the 10th singular value (exact 16.653).

A random sketch always underestimates the singular values: it sees the right directions only partially. Each power iteration pulls the estimates up towards the truth. Flat spectra like this one, typical of sparse data, need several iterations.

Knobs that matter

n_components k
for LSA, 100–300 is the classic range for retrieval and clustering; for visualisation, 2–3. Pick by the energy curve, or better, by downstream validation score (retrieval precision, classifier accuracy). The energy curve of text has no clear elbow
algorithm
randomized (default, fast, large kk fine) vs arpack (Lanczos, exact to tolerance tol, needs k<min⁡(n,d)k<\min(n,d))
n_iter q
default 5. Raise it for slowly decaying spectra or when components look noisy across seeds; each one costs two more passes over XX
n_oversamples p
default 10. Raise it (e.g. 20–50) when k is small relative to a flat spectrum
Input weighting
tf-idf instead of counts stops frequent words dominating v1v_1. L2-normalise rows (or use Normalizer after the SVD) so cosine compares topics, not document lengths
Centring
dense data that fits in memory: just use PCA, which centres. Sparse data: truncated SVD, or scikit-learn's PCA with svd_solver="arpack" / "covariance_eigh", which centre implicitly without densifying (recent versions)

Related methods: NMF factorises the same non-negative matrix but forces non-negative, parts-based components that read more like topics. LDA (latent Dirichlet allocation) is the probabilistic topic-model successor of LSA. Incremental / streaming SVD (e.g. gensim's LsiModel) updates the factorisation in chunks when the corpus doesn't fit on one machine.

In code

python
1import numpy as np
2
3def truncated_svd(X, k, n_oversamples=10, n_iter=4, seed=0):
4 """Randomized truncated SVD (Halko et al., 2011). X may be a NumPy or scipy.sparse matrix."""
5 rng = np.random.default_rng(seed)
6 n, d = X.shape
7 l = min(k + n_oversamples, d)
8 Q = X @ rng.standard_normal((d, l)) # n x l sketch of the column space
9 for _ in range(n_iter): # power iterations sharpen the spectrum
10 Q, _ = np.linalg.qr(Q)
11 Q, _ = np.linalg.qr(X.T @ Q)
12 Q = X @ Q
13 Q, _ = np.linalg.qr(Q) # orthonormal basis for range(X)
14 B = (X.T @ Q).T # small l x d matrix B = Q^T X
15 Ub, s, Vt = np.linalg.svd(B, full_matrices=False)
16 U = Q @ Ub
17 U, s, Vt = U[:, :k], s[:k], Vt[:k]
18 flip = np.sign(Vt[np.arange(k), np.abs(Vt).argmax(axis=1)]) # largest |entry| of each v positive
19 return U * flip, s, Vt * flip[:, None]
20
21terms = ["car", "auto", "engine", "fruit", "apple", "juice"]
22X = np.array([[2, 0, 1, 0, 0, 0], [0, 2, 1, 0, 0, 0], [1, 1, 2, 0, 0, 0],
23 [0, 0, 0, 2, 1, 0], [0, 0, 0, 0, 2, 1], [0, 0, 0, 1, 1, 2],
24 [1, 0, 0, 0, 1, 0]], dtype=float)
25
26U, s, Vt = truncated_svd(X, k=2)
27Z = U * s # document coordinates = X @ V_k
28print("singular values:", s.round(4))
29print("V_k^T:\n", Vt.round(4))
30print("docs in LSA space:\n", Z.round(4))
31print("||X - X_k||_F =", round(np.linalg.norm(X - Z @ Vt), 4))
32
33q = np.array([0, 1, 0, 0, 0, 0.]) # query: "auto"
34zq = q @ Vt.T # fold in: q V_k
35cos = lambda a, B: B @ a / np.linalg.norm(B, axis=1) / np.linalg.norm(a)
36print("query:", zq.round(4))
37print("raw cosine:", cos(q, X).round(3))
38print("LSA cosine:", cos(zq, Z).round(3))
From scratch: randomized truncated SVD with power iterations and scikit-learn's sign convention, run on the worked example. Prints σ = [3.5172 3.405], the document coordinates from the table, ‖X − X₂‖ = 3.1677, and the raw vs latent cosines (D1: 0.0 → 0.993).
python
1from sklearn.decomposition import TruncatedSVD
2from sklearn.feature_extraction.text import CountVectorizer
3from sklearn.metrics.pairwise import cosine_similarity
4
5docs = ["car car engine", "auto auto engine", "car auto engine engine",
6 "fruit fruit apple", "apple apple juice", "fruit apple juice juice",
7 "apple car"]
8vec = CountVectorizer(vocabulary=["car", "auto", "engine", "fruit", "apple", "juice"])
9X = vec.fit_transform(docs) # scipy.sparse CSR, 16 non-zeros of 42
10print(type(X).__name__, X.nnz, "non-zeros")
11
12svd = TruncatedSVD(n_components=2, algorithm="randomized", n_iter=5, random_state=0)
13Z = svd.fit_transform(X) # = U_k Sigma_k = X V_k
14print("singular values:", svd.singular_values_.round(4))
15print("components:\n", svd.components_.round(4))
16print("explained variance ratio:", svd.explained_variance_ratio_.round(4))
17print("docs:\n", Z.round(4))
18
19zq = svd.transform(vec.transform(["auto"]))
20print("query 'auto':", zq.round(4))
21print("raw cosine:", cosine_similarity(vec.transform(["auto"]), X).round(3))
22print("LSA cosine:", cosine_similarity(zq, Z).round(3))
23
24# Real LSA pipeline:
25# make_pipeline(TfidfVectorizer(sublinear_tf=True), TruncatedSVD(200), Normalizer(copy=False))
scikit-learn on a sparse matrix built from the raw text. Same components and coordinates; note explained_variance_ratio_ = [0.0195 0.5263].

Use when

  • +The data is large and sparse (tf-idf text, one-hot features, user–item matrices) and must stay sparse.
  • +You need dense, low-dimensional features for retrieval, clustering or a linear classifier on text (LSA).
  • +You want a fast, deterministic-given-seed, linear reduction whose transform is a single matrix product.
  • +You need only the top few singular directions of a huge matrix (spectral methods, low-rank approximation).

Avoid when

  • −The data is dense and fits in memory: use PCA, which centres, so components and variance ratios mean what you expect.
  • −You need interpretable topics: components mix positive and negative weights; NMF or probabilistic topic models read better.
  • −The structure is nonlinear (manifolds, clusters in curved shapes): see kernel PCA, t-SNE or UMAP.
  • −Word order or polysemy matters: a bag-of-words SVD gives one vector per word regardless of sense; use contextual embeddings.

Where it's used

Latent semantic indexing

Search engines and document-similarity systems map tf-idf vectors to ~200 latent dimensions, so queries match documents that share no exact words.

Text classification features

TF-IDF → TruncatedSVD → logistic regression / SVM: a compact, fast pipeline for spam, sentiment and topic classification.

Recommender systems

Low-rank factorisation of sparse user–item rating or click matrices (the SVD family behind the Netflix Prize models).

Pre-step for clustering and visualisation

Reduce sparse text to 50–100 dims before k-means, or before t-SNE / UMAP, which struggle on 100k sparse dimensions.

Check yourself

Why does scikit-learn's TruncatedSVD not centre the data, unlike PCA?

Check yourself

A matrix has singular values (4,3,2,1)(4, 3, 2, 1). You keep k=2k=2. What are the energy kept and ∥X−X2∥F\lVert X-X_2\rVert_F?

Check yourself

In the worked example, the query “auto” has raw cosine 0 with D1 but latent cosine 0.993 at k = 2. What happens to that latent cosine at k = 6 (full rank)?

Check yourself

On non-negative count data, the first truncated-SVD component typically…

Remember

  • ✓Truncated SVD keeps the top kk terms σiuivi⊤\sigma_iu_iv_i^\top of the uncentred matrix: Z=XVk=UkΣkZ=XV_k=U_k\Sigma_k. On centred data it is exactly PCA.
  • ✓Eckart–Young: XkX_k is the best rank-kk approximation and ∥X−Xk∥F2=∑i>kσi2\lVert X-X_k\rVert_F^2=\sum_{i>k}\sigma_i^2, so the spectrum tells you what each extra component buys.
  • ✓Randomized SVD needs only products with XX and X⊤X^\top, so it costs O(nnz⋅k)O(\text{nnz}\cdot k) and keeps sparse data sparse. Power iterations (Σ2q+1\Sigma^{2q+1}) fix flat spectra.
  • ✓On text (LSA), low rank merges words used in the same contexts: “auto” finds the car document (cosine 0 → 0.993). Truncation is what creates the similarity.
  • ✓Without centring, component 1 ≈ the mean direction; explained_variance_ratio_ is then unsorted and misleading. Use tf-idf and row normalisation.
← back to ML Algorithms roadmap