Principal Component Analysis

Project data onto the directions of maximum variance — the top eigenvectors of its covariance matrix.

17 min read1516 wordscore8 viewsupdated 2026-10-07

Definition

Principal component analysis (PCA) finds the orthogonal directions along which centred data varies most, and represents each point by its coordinates along the top kk of them. It is the linear projection to kk dimensions that keeps the most variance — and, equivalently, the one whose reconstruction has the smallest squared error.

At a glance

Task
Unsupervised dimensionality reduction (also compression, denoising, decorrelation)
Input → output
X∈Rn×dX\in\mathbb{R}^{n\times d} → scores Z∈Rn×kZ\in\mathbb{R}^{n\times k}, with k<dk<d
Learns
the mean xˉ\bar x, kk orthonormal directions u1,…,uku_1,\dots,u_k (the principal components) and their variances λ1≥⋯≥λk\lambda_1\ge\dots\ge\lambda_k
Training
one eigendecomposition of the d×dd\times d covariance, O(nd2+d3)O(nd^2+d^3), or one thin SVD of XX, O(ndmin⁡(n,d))O(nd\min(n,d)); randomized SVD costs about O(ndk)O(ndk)
Prediction
project a new point: z=Uk⊤(x−xˉ)z=U_k^\top(x-\bar x), O(dk)O(dk)
Hyperparameters
number of components kk (or a variance fraction such as 95%); whether to standardize; whitening; solver
Assumes
the structure worth keeping is linear and shows up as large variance; features are on comparable scales

The problem

Real feature sets are redundant. The 64 pixels of a handwritten digit move together; height and weight rise together; thirteen chemical measurements of a wine are driven by a handful of underlying factors. Redundancy costs us: models train slower, distances become less meaningful, and we cannot look at data with more than three dimensions.

The obvious fix — drop some features — throws information away blindly. In the 10-point example below, x1x_1 and x2x_2 have correlation 0.926. Keeping only x1x_1 keeps 46.2% of the total variance; keeping only x2x_2 keeps 53.8%. Yet a new axis, a blend of both, keeps 96.3%. PCA finds that blend: instead of choosing among the original axes, it rotates the coordinate system to the axes the data actually uses.

The idea

Picture the data as a cloud. Move the origin to the centre of the cloud, then turn a stick through it until the shadows of the points on the stick are as spread out as possible — that stick is the first principal component. The second component is the best stick perpendicular to the first, and so on. Each point is then described by where its shadow falls on the first few sticks; the remaining, low-spread directions are dropped because points barely differ along them.

Why spread? Because a direction along which every point has the same coordinate tells you nothing about how points differ. And there is a second reading of the same picture: the leftover distance from each point to its shadow is the error you make by keeping only the shadow. Since the total spread is fixed, maximizing the spread of the shadows is the same as minimizing the leftover error.

Vocabulary

Centring
subtracting the feature means so the cloud sits on the origin; PCA directions pass through xˉ\bar x
Principal component (PC)
a unit direction uj∈Rdu_j\in\mathbb{R}^d — an eigenvector of the covariance matrix. Also called a loading vector; its entries are the loadings (how much each original feature contributes)
Score
the coordinate of a point along a component, zij=uj⊤(xi−xˉ)z_{ij}=u_j^\top(x_i-\bar x) — the length of its shadow
Explained variance
λj\lambda_j, the variance of the scores on component jj — the eigenvalue
Explained variance ratio
λj/∑lλl\lambda_j/\sum_l\lambda_l, the share of the total variance component jj keeps
Scree plot
the eigenvalues in decreasing order; one looks for the "elbow" where they flatten out
Reconstruction
mapping scores back: x^=xˉ+∑j≤kzjuj\hat x=\bar x+\sum_{j\le k}z_j u_j; its error is the part of xx in the dropped directions
Whitening
dividing each score by λj\sqrt{\lambda_j} so every kept component has unit variance

The math

Four short derivations: the variance along a direction, its maximization (which produces the eigenvalue equation), the equivalent reconstruction-error view, and the link to the SVD that libraries actually compute.

Variance along a direction

Centre the data: x~i=xi−xˉ\tilde x_i=x_i-\bar x, stacked into the rows of X~\tilde X. The score of point ii on a unit vector uu is zi=u⊤x~iz_i=u^\top\tilde x_i, and because the scores have mean zero their variance is a quadratic form in the covariance matrix:

Var⁡(z)=1n−1∑i=1n(u⊤x~i)2=u⊤(X~⊤X~n−1)u=u⊤C u\operatorname{Var}(z)=\frac{1}{n-1}\sum_{i=1}^n (u^\top\tilde x_i)^2=u^\top\Big(\frac{\tilde X^\top\tilde X}{n-1}\Big)u=u^\top C\,u
X~\tilde X
centred data, n×dn\times d, one row per example
uu
a candidate direction with ∥u∥=1\lVert u\rVert=1
CC
sample covariance matrix, d×dd\times d, symmetric and positive semi-definite; CabC_{ab} is the covariance of features aa and bb
n−1n-1
Bessel's correction (scikit-learn uses it too); it rescales every eigenvalue equally and never changes the directions

Maximizing it: the eigenvalue equation

Without the constraint ∥u∥=1\lVert u\rVert=1 we could inflate the variance just by scaling uu, so maximize u⊤Cuu^\top Cu subject to u⊤u=1u^\top u=1 with a Lagrange multiplier λ\lambda:

L(u,λ)=u⊤Cu−λ (u⊤u−1),∇uL=2Cu−2λu=0  ⟹  C u=λ u\mathcal{L}(u,\lambda)=u^\top Cu-\lambda\,(u^\top u-1),\qquad \nabla_u\mathcal{L}=2Cu-2\lambda u=0\;\Longrightarrow\; C\,u=\lambda\,u
λ\lambda
Lagrange multiplier — it turns out to be an eigenvalue of C
∇u\nabla_u
gradient with respect to uu (using ∇u u⊤Cu=2Cu\nabla_u\, u^\top Cu=2Cu for symmetric CC)

Every stationary point is an eigenvector of CC. Multiply Cu=λuCu=\lambda u on the left by u⊤u^\top: the variance captured is u⊤Cu=λu⊤u=λu^\top Cu=\lambda u^\top u=\lambda. So among the eigenvectors, the best direction is the one with the largest eigenvalue. Because CC is symmetric, its eigenvectors can be chosen orthonormal and its eigenvalues are real and ≥0\ge 0.

For the second component add the constraint u2⊥u1u_2\perp u_1. The same Lagrangian argument (with a second multiplier for u2⊤u1=0u_2^\top u_1=0) gives Cu2=λu2Cu_2=\lambda u_2 restricted to the subspace orthogonal to u1u_1 — the eigenvector with the second-largest eigenvalue. Repeating, the first kk components are the top-kk eigenvectors, and their scores are uncorrelated: Cov⁡(za,zb)=ua⊤Cub=λbua⊤ub=0\operatorname{Cov}(z_a,z_b)=u_a^\top Cu_b=\lambda_b u_a^\top u_b=0 for a≠ba\ne b.

C=UΛU⊤=∑j=1dλjujuj⊤,tr⁡C=∑a=1dCaa=∑j=1dλjC=U\Lambda U^\top=\sum_{j=1}^d\lambda_j u_ju_j^\top,\qquad \operatorname{tr}C=\sum_{a=1}^d C_{aa}=\sum_{j=1}^d\lambda_j
UU
orthogonal d×dd\times d matrix whose columns are the components u1,…,udu_1,\dots,u_d
Λ\Lambda
diagonal matrix of eigenvalues λ1≥λ2≥⋯≥λd≥0\lambda_1\ge\lambda_2\ge\dots\ge\lambda_d\ge0
tr⁡C\operatorname{tr}C
total variance: the sum of the feature variances, which a rotation cannot change

The trace identity is what makes "share of variance" meaningful: rotating to the eigenbasis just redistributes the fixed total tr⁡C\operatorname{tr}C so that as much as possible lands on the first axes.

The same answer from reconstruction error

Keep only the score on uu and rebuild x~^i=ziu\hat{\tilde x}_i=z_iu. The residual x~i−ziu\tilde x_i-z_iu is perpendicular to uu, so Pythagoras splits each point's squared length into a kept part and a lost part:

∥x~i∥2=zi2+∥x~i−ziu∥2⟹1n−1∑i∥x~i−ziu∥2=tr⁡C−u⊤Cu\lVert\tilde x_i\rVert^2=z_i^2+\lVert\tilde x_i-z_iu\rVert^2\quad\Longrightarrow\quad \frac{1}{n-1}\sum_i\lVert\tilde x_i-z_iu\rVert^2=\operatorname{tr}C-u^\top Cu
ziuz_i u
the reconstruction (shadow) of point ii
x~i−ziu\tilde x_i-z_iu
residual: the part of the point in the dropped directions

The left side is fixed, so minimizing the reconstruction error is exactly maximizing the projected variance. With kk components the error is the sum of the dropped eigenvalues, ∑j>kλj\sum_{j>k}\lambda_j, and no other rank-kk linear projection does better (the Eckart–Young theorem — see the SVD article for the matrix version).

Computing it with the SVD

Forming CC squares the condition number of the data. Libraries instead take the thin SVD of the centred data, X~=VΣU⊤\tilde X=V\Sigma U^\top (here VV holds the left singular vectors): the right singular vectors are the components, λj=σj2/(n−1)\lambda_j=\sigma_j^2/(n-1), and the scores are Z=X~Uk=VkΣkZ=\tilde XU_k=V_k\Sigma_k. The SVD article derives this factorization; Truncated SVD covers the variant that skips centring so it can work on sparse matrices.

What the hyperparameters do

At a glance

k
keeps a fraction ∑j≤kλj/∑jλj\sum_{j\le k}\lambda_j/\sum_j\lambda_j of the variance; the mean squared reconstruction error is exactly ∑j>kλj\sum_{j>k}\lambda_j. Each extra component lowers the error by its eigenvalue
Standardize
dividing each feature by its standard deviation replaces CC with the correlation matrix R=D−1/2CD−1/2R=D^{-1/2}CD^{-1/2}. Eigenvectors of CC and RR differ: unscaled PCA is dominated by whichever feature has the largest numeric variance
Whiten
zj→zj/λjz_j\to z_j/\sqrt{\lambda_j} makes the score covariance the identity. Distances stop favouring the first components — useful before k-means or ICA, harmful if small components are mostly noise (it amplifies them)

Worked example

Ten points in two dimensions (the classic tutorial set from Lindsay Smith). The features are strongly correlated, so one direction should carry almost everything. We reduce to k=1k=1 and then project a new point x∗=(2.0, 2.5)x^*=(2.0,\,2.5).

ix₁x₂x₁ − 1.81x₂ − 1.91
12.52.40.690.49
20.50.7−1.31−1.21
32.22.90.390.99
41.92.20.090.29
53.13.01.291.09
62.32.70.490.79
72.01.60.19−0.31
81.01.1−0.81−0.81
91.51.6−0.31−0.31
101.10.9−0.71−1.01
The data and the centred data. Means: xˉ1=1.81\bar x_1=1.81, xˉ2=1.91\bar x_2=1.91.

PCA by hand

  1. 1Centre. Mean xˉ=(18.1/10, 19.1/10)=(1.81, 1.91)\bar x=(18.1/10,\ 19.1/10)=(1.81,\ 1.91); subtract it from every row (right two columns above).
  2. 2Covariance. From the centred columns: ∑x~12=5.549\sum\tilde x_1^2=5.549, ∑x~22=6.449\sum\tilde x_2^2=6.449, ∑x~1x~2=5.539\sum\tilde x_1\tilde x_2=5.539. Dividing by n−1=9n-1=9: C=(0.61660.61540.61540.7166)C=\begin{pmatrix}0.6166&0.6154\\0.6154&0.7166\end{pmatrix}. Total variance tr⁡C=1.3331\operatorname{tr}C=1.3331.
  3. 3Eigenvalues. Solve det⁡(C−λI)=0\det(C-\lambda I)=0: λ2−(tr⁡C)λ+det⁡C=λ2−1.3331λ+0.0630=0\lambda^2-(\operatorname{tr}C)\lambda+\det C=\lambda^2-1.3331\lambda+0.0630=0, so λ=1.33312±1.333124−0.0630\lambda=\tfrac{1.3331}{2}\pm\sqrt{\tfrac{1.3331^2}{4}-0.0630}, giving λ1=1.2840\lambda_1=1.2840 and λ2=0.0491\lambda_2=0.0491. Check: 1.2840+0.0491=1.3331=tr⁡C1.2840+0.0491=1.3331=\operatorname{tr}C.
  4. 4Eigenvector. The first row of (C−λ1I)u=0(C-\lambda_1I)u=0 reads (0.6166−1.2840)u1+0.6154 u2=0(0.6166-1.2840)u_1+0.6154\,u_2=0, so u2/u1=0.6674/0.6154=1.0845u_2/u_1=0.6674/0.6154=1.0845. Normalizing (1, 1.0845)(1,\,1.0845) by its length 1.47521.4752: u1=(0.6779, 0.7352)u_1=(0.6779,\ 0.7352), an angle of 47.3°47.3° — just steeper than the diagonal because x2x_2 varies a little more. The second component is the perpendicular u2=(−0.7352, 0.6779)u_2=(-0.7352,\ 0.6779).
  5. 5Explained variance. PC1 keeps λ1/tr⁡C=1.2840/1.3331=96.3%\lambda_1/\operatorname{tr}C=1.2840/1.3331=96.3\%; PC2 only 3.7%3.7\%. Compare: the original x1x_1 axis keeps 0.6166/1.3331=46.2%0.6166/1.3331=46.2\%.
  6. 6Scores. zi=0.6779 x~i1+0.7352 x~i2z_{i}=0.6779\,\tilde x_{i1}+0.7352\,\tilde x_{i2}. Point 1: 0.6779(0.69)+0.7352(0.49)=0.82800.6779(0.69)+0.7352(0.49)=0.8280. All ten are in the next table; their variance is 1.2840=λ11.2840=\lambda_1, as promised.
  7. 7Reconstruction error. Rebuilding from one score, x^i=xˉ+ziu1\hat x_i=\bar x+z_iu_1, point 1 becomes (2.371, 2.519)(2.371,\ 2.519) with squared error 0.03070.0307. The ten errors sum to 0.44180.4418; divided by n−1n-1 that is 0.0491=λ20.0491=\lambda_2 — exactly the dropped eigenvalue.
  8. 8New point. x∗−xˉ=(0.19, 0.59)x^*-\bar x=(0.19,\ 0.59), so z∗=0.6779(0.19)+0.7352(0.59)=0.5626z^*=0.6779(0.19)+0.7352(0.59)=0.5626. One number now stands for x∗x^*; mapping it back gives x^∗=xˉ+0.5626 u1=(2.191, 2.324)\hat x^*=\bar x+0.5626\,u_1=(2.191,\ 2.324). Its PC2 score, the part thrown away, is 0.26030.2603.
iz₁ (PC1)z₂ (PC2)x̂ from z₁squared error
10.8280−0.1751(2.371, 2.519)0.0307
2−1.77760.1429(0.605, 0.603)0.0204
30.99220.3844(2.483, 2.639)0.1477
40.27420.1304(1.996, 2.112)0.0170
51.6758−0.2095(2.946, 3.142)0.0439
60.91290.1753(2.429, 2.581)0.0307
7−0.0991−0.3498(1.743, 1.837)0.1224
8−1.14460.0464(1.034, 1.069)0.0022
9−0.43800.0178(1.513, 1.588)0.0003
10−1.2238−0.1627(0.980, 1.010)0.0265
variance / sum1.28400.04910.4418
Scores on both components, the 1-D reconstruction and its squared error. Note that each squared error equals the PC2 score squared — the residual lives entirely along u2u_2.
00112233x₁x*udrag ● u, any ● or ○ x*total variance 1.333= along u + residualresidual0.717 (53.8%)along u0.617 (46.2%)var along u vs θ0°180°peak λ₁ at 47.3°
θ = 0.0° · u = (1.000, 0.000) · uᵀCu = 0.617 · residual = 0.717
mean = (1.810, 1.910) · C = [[0.617, 0.615], [0.615, 0.717]]
λ₁ = 1.284 at 47.3° (96.3%) · λ₂ = 0.049 · λ₁ + λ₂ = trace C = 1.333
x* = (2.00, 2.50) → z* = (x* − mean)·u = 0.190 → x̂* = mean + z*·u = (2.000, 1.910)
fig.The worked example, live. The orange line is a candidate direction u through the mean (×); rose segments are each point's residual to its shadow (orange dots). It starts on the x₁ axis: variance along u 0.617 (46.2%), residual 0.717. Drag the orange handle — the bar always sums to the total 1.333, and the curve on the right peaks at 47.3°, where u is PC1: 1.284 (96.3%) kept, residual 0.049 = λ₂. The violet ring is the new point x* = (2.0, 2.5); on PC1 its score is 0.563 and its reconstruction (2.191, 2.324). Drag any blue point to watch the covariance, eigenvalues and components move.

The algorithm, step by step

PCA(X, k)

  1. 1Centre (and, if features have different units, standardize): compute xˉ\bar x (and per-feature ss) on the training data and store them.
  2. 2Decompose. Either form C=X~⊤X~/(n−1)C=\tilde X^\top\tilde X/(n-1) and run a symmetric eigensolver (O(nd2+d3)O(nd^2+d^3)), or take the thin SVD of X~\tilde X directly (O(ndmin⁡(n,d))O(nd\min(n,d)), more stable).
  3. 3Sort components by eigenvalue, largest first. Fix each sign (e.g. make the largest loading positive) — eigenvectors are only defined up to ±\pm.
  4. 4Choose k from the cumulative explained-variance ratio, a scree-plot elbow, or downstream validation score.
  5. 5Transform: Z=(X−xˉ)UkZ=(X-\bar x)U_k, cost O(ndk)O(ndk). Inverse transform: X^=ZUk⊤+xˉ\hat X=ZU_k^\top+\bar x.
▸ PCA · maximum-variance directionstep 1/18
θ = —var along u = —explained = —
  1. 1X ← X − mean(X)
  2. 2C ← XᵀX / (n − 1)
  3. 3u ← argmax uᵀCu, ‖u‖ = 1
  4. 4 = top eigenvector of C
  5. 5Z ← X·u

Raw data: 40 points whose mean (×) is at (0.36, 0.23), not at the origin.

fig.The same computation on a 40-point cloud: centre, form the covariance, sweep a direction through 0°–180° and watch its projected variance — the sweep's peak is the top eigenvector, found in closed form. One dimension keeps 96.1% of this cloud's variance.

For large nn and dd with small kk, randomized SVD (Halko et al.) multiplies X~\tilde X by a random d×(k+p)d\times(k+p) matrix, orthonormalizes, and does an exact SVD of the small result — about O(ndk)O(ndk) with a few power iterations. scikit-learn's svd_solver="auto" forms the covariance when the data is tall and skinny (d≤1000d\le1000 and n≥10dn\ge10d), switches to randomized SVD when the matrix exceeds 500 in some dimension and an integer kk is below 80% of min⁡(n,d)\min(n,d), and otherwise runs a full SVD.

Choosing k and other practical choices

There is no single right kk; it depends on what the components are for. Three common rules, shown on the standardized Iris data (4 features, so the eigenvalues of the correlation matrix sum to 4):

ComponentEigenvalue λRatioCumulative
PC12.93872.96%72.96%
PC20.92022.85%95.81%
PC30.1483.67%99.48%
PC40.0210.52%100%
Standardized Iris. Two components keep 95.8% of the variance — enough for a faithful 2-D plot. The Kaiser rule (keep λ > 1) would keep only one.

Ways to pick k

Variance threshold
keep the smallest k reaching 90–99%. On the 64-pixel digits data that is 21 components for 90%, 29 for 95%, 41 for 99%. In scikit-learn pass a float: PCA(n_components=0.95).
Scree elbow
plot the eigenvalues and stop where the curve flattens; subjective but robust for visualization.
Kaiser rule
on standardized data keep components with λ>1\lambda>1 (each worth more than one original feature). Often too aggressive — 1 for Iris, 3 for Wine.
Downstream validation
when PCA feeds a model, treat k as a hyperparameter and cross-validate the whole pipeline. This is the only rule that measures what you care about.
Probabilistic (MLE)
n_components='mle' uses Minka's Bayesian model selection under probabilistic PCA.
VariantWhat changesUse when
Full SVD / eigendecompositionexact; all componentsd up to a few thousand
Randomized PCAapproximate top-k via random projectionslarge n and d, small k
Incremental PCAupdates the decomposition one mini-batch at a timedata does not fit in memory, or arrives as a stream
Sparse PCAL1 penalty on loadings, so each component uses few featuresyou need interpretable components
Probabilistic PCAa latent Gaussian model with isotropic noise, fit by EM or closed formmissing values, likelihoods, model selection
Kernel PCAPCA in an implicit feature space via a kernelcurved structure — see Kernel PCA
Truncated SVDno centringsparse text matrices (LSA) — see Truncated SVD
LDAmaximizes between-class over within-class scatter, using labelssupervised reduction — see LDA for dimensionality reduction
Variants of PCA and when to reach for them.

In code

python
1import numpy as np
2
3def pca_fit(X, k):
4 mean = X.mean(axis=0)
5 Xc = X - mean # 1. centre
6 C = Xc.T @ Xc / (len(X) - 1) # 2. covariance (d x d)
7 eigvals, eigvecs = np.linalg.eigh(C) # 3. symmetric eigensolver (ascending)
8 order = np.argsort(eigvals)[::-1] # sort descending
9 eigvals, eigvecs = eigvals[order], eigvecs[:, order]
10 W = eigvecs[:, :k] # 4. top-k components as columns (d x k)
11 W = W * np.sign(W[np.abs(W).argmax(axis=0), range(k)]) # fix sign: largest loading > 0
12 return mean, W, eigvals
13
14def transform(X, mean, W):
15 return (X - mean) @ W # scores (n x k)
16
17def inverse_transform(Z, mean, W):
18 return Z @ W.T + mean # back to d dimensions
19
20X = np.array([[2.5, 2.4], [0.5, 0.7], [2.2, 2.9], [1.9, 2.2], [3.1, 3.0],
21 [2.3, 2.7], [2.0, 1.6], [1.0, 1.1], [1.5, 1.6], [1.1, 0.9]])
22mean, W, lam = pca_fit(X, k=1)
23Z = transform(X, mean, W)
24X_hat = inverse_transform(Z, mean, W)
25print("mean ", mean.round(4))
26print("eigenvalues", lam.round(4), "explained", (lam / lam.sum()).round(4))
27print("u1 ", W[:, 0].round(4))
28print("scores ", Z[:, 0].round(4))
29print("recon MSE (sum/(n-1))", (((X - X_hat) ** 2).sum() / (len(X) - 1)).round(4))
30x_new = np.array([[2.0, 2.5]])
31z_new = transform(x_new, mean, W)
32print("new point ", z_new.round(4), inverse_transform(z_new, mean, W).round(4))
From scratch with NumPy, on the worked example. Prints λ = [1.284, 0.0491], u₁ = [0.6779, 0.7352], reconstruction MSE 0.0491 and the new point's score 0.5626 → (2.1913, 2.3236).
python
1import numpy as np
2from sklearn.decomposition import PCA
3
4X = np.array([[2.5, 2.4], [0.5, 0.7], [2.2, 2.9], [1.9, 2.2], [3.1, 3.0],
5 [2.3, 2.7], [2.0, 1.6], [1.0, 1.1], [1.5, 1.6], [1.1, 0.9]])
6
7pca = PCA(n_components=1).fit(X)
8print("mean ", pca.mean_.round(4))
9print("u1 ", pca.components_[0].round(4))
10print("lambda1 ", pca.explained_variance_.round(4))
11print("ratio ", pca.explained_variance_ratio_.round(4))
12print("scores ", pca.transform(X)[:, 0].round(4))
13
14z_new = pca.transform([[2.0, 2.5]])
15print("new point ", z_new.round(4), pca.inverse_transform(z_new).round(4))
16
17# On real data: standardize, then keep enough components for 95% of the variance
18from sklearn.datasets import load_wine
19from sklearn.pipeline import make_pipeline
20from sklearn.preprocessing import StandardScaler
21Xw = load_wine().data
22pipe = make_pipeline(StandardScaler(), PCA(n_components=0.95)).fit(Xw)
23print("wine: k for 95% =", pipe[-1].n_components_)
scikit-learn on the same data (same mean, component, λ₁ = 1.284, ratio 0.9632, scores and new-point result), then the standard scale-then-reduce pipeline on Wine, which needs 10 components for 95%.

Use when

  • +Many correlated numeric features and you want fewer, decorrelated inputs for a downstream model.
  • +A first 2-D or 3-D look at high-dimensional data, or a quick check of how many directions really matter.
  • +Compression and denoising, when the noise is spread over many small-variance directions.
  • +You need a deterministic, fast, invertible transform with no tuning beyond k.

Avoid when

  • −The structure is nonlinear (curved manifolds, clusters on rings) — try Kernel PCA, UMAP or an autoencoder.
  • −The signal you care about has low variance: PCA ignores labels and may discard the one direction that separates classes (use LDA).
  • −You need interpretable features: every component mixes every original feature (consider sparse PCA or feature selection).
  • −Heavy outliers or wildly different feature scales you cannot or will not fix.

Where it's used

Population genetics

The first two components of genome-wide genotype data reproduce a map of ancestry — a standard correction for population structure in association studies.

Finance

A few components (market, sector, rates) explain most co-movement of stock returns or of a yield curve; risk models are built on them.

Faces and images

Eigenfaces: each face as a combination of a few dozen principal images; also ZCA/PCA whitening before training vision models.

Preprocessing and embeddings

Shrink thousands of features, or 768-dimensional text embeddings, before clustering, nearest-neighbour search or regression.

Check yourself

Covariance eigenvalues are 6, 3 and 1. What fraction of the variance do the first two components keep, and what is the mean squared reconstruction error with k = 2?

Check yourself

In the worked example, what is the mean squared reconstruction error (sum divided by n − 1) when keeping only PC1?

Check yourself

Unscaled PCA on the Wine data puts 99.81% of the variance on PC1. What is the most likely explanation?

Check yourself

Which statement about principal component scores is true?

Remember

  • ✓Centre the data; the principal components are the eigenvectors of the covariance CC, sorted by eigenvalue. Component jj captures variance λj\lambda_j.
  • ✓Maximizing u⊤Cuu^\top Cu subject to ∥u∥=1\lVert u\rVert=1 gives Cu=λuCu=\lambda u via a Lagrange multiplier; later components are the next eigenvectors, orthogonal to earlier ones, and their scores are uncorrelated.
  • ✓Total variance tr⁡C=∑λj\operatorname{tr}C=\sum\lambda_j is fixed, so maximum kept variance equals minimum reconstruction error, which is ∑j>kλj\sum_{j>k}\lambda_j.
  • ✓Compute with an SVD of the centred data (randomized for large data); choose k by a variance threshold or by validating the downstream model.
  • ✓Standardize mixed units, watch for outliers, fit on training data only, and remember PCA is linear and label-blind.
← back to ML Algorithms roadmap