Definition
Principal component analysis (PCA) finds the orthogonal directions along which centred data varies most, and represents each point by its coordinates along the top of them. It is the linear projection to dimensions that keeps the most variance — and, equivalently, the one whose reconstruction has the smallest squared error.
At a glance
- Task
- Unsupervised dimensionality reduction (also compression, denoising, decorrelation)
- Input → output
- → scores , with
- Learns
- the mean , orthonormal directions (the principal components) and their variances
- Training
- one eigendecomposition of the covariance, , or one thin SVD of , ; randomized SVD costs about
- Prediction
- project a new point: ,
- Hyperparameters
- number of components (or a variance fraction such as 95%); whether to standardize; whitening; solver
- Assumes
- the structure worth keeping is linear and shows up as large variance; features are on comparable scales
The problem
Real feature sets are redundant. The 64 pixels of a handwritten digit move together; height and weight rise together; thirteen chemical measurements of a wine are driven by a handful of underlying factors. Redundancy costs us: models train slower, distances become less meaningful, and we cannot look at data with more than three dimensions.
The obvious fix — drop some features — throws information away blindly. In the 10-point example below, and have correlation 0.926. Keeping only keeps 46.2% of the total variance; keeping only keeps 53.8%. Yet a new axis, a blend of both, keeps 96.3%. PCA finds that blend: instead of choosing among the original axes, it rotates the coordinate system to the axes the data actually uses.
The idea
Picture the data as a cloud. Move the origin to the centre of the cloud, then turn a stick through it until the shadows of the points on the stick are as spread out as possible — that stick is the first principal component. The second component is the best stick perpendicular to the first, and so on. Each point is then described by where its shadow falls on the first few sticks; the remaining, low-spread directions are dropped because points barely differ along them.
Why spread? Because a direction along which every point has the same coordinate tells you nothing about how points differ. And there is a second reading of the same picture: the leftover distance from each point to its shadow is the error you make by keeping only the shadow. Since the total spread is fixed, maximizing the spread of the shadows is the same as minimizing the leftover error.
Vocabulary
- Centring
- subtracting the feature means so the cloud sits on the origin; PCA directions pass through
- Principal component (PC)
- a unit direction — an eigenvector of the covariance matrix. Also called a loading vector; its entries are the loadings (how much each original feature contributes)
- Score
- the coordinate of a point along a component, — the length of its shadow
- Explained variance
- , the variance of the scores on component — the eigenvalue
- Explained variance ratio
- , the share of the total variance component keeps
- Scree plot
- the eigenvalues in decreasing order; one looks for the "elbow" where they flatten out
- Reconstruction
- mapping scores back: ; its error is the part of in the dropped directions
- Whitening
- dividing each score by so every kept component has unit variance
The math
Four short derivations: the variance along a direction, its maximization (which produces the eigenvalue equation), the equivalent reconstruction-error view, and the link to the SVD that libraries actually compute.
Variance along a direction
Centre the data: , stacked into the rows of . The score of point on a unit vector is , and because the scores have mean zero their variance is a quadratic form in the covariance matrix:
- centred data, , one row per example
- a candidate direction with
- sample covariance matrix, , symmetric and positive semi-definite; is the covariance of features and
- Bessel's correction (scikit-learn uses it too); it rescales every eigenvalue equally and never changes the directions
Maximizing it: the eigenvalue equation
Without the constraint we could inflate the variance just by scaling , so maximize subject to with a Lagrange multiplier :
- Lagrange multiplier — it turns out to be an eigenvalue of C
- gradient with respect to (using for symmetric )
Every stationary point is an eigenvector of . Multiply on the left by : the variance captured is . So among the eigenvectors, the best direction is the one with the largest eigenvalue. Because is symmetric, its eigenvectors can be chosen orthonormal and its eigenvalues are real and .
For the second component add the constraint . The same Lagrangian argument (with a second multiplier for ) gives restricted to the subspace orthogonal to — the eigenvector with the second-largest eigenvalue. Repeating, the first components are the top- eigenvectors, and their scores are uncorrelated: for .
- orthogonal matrix whose columns are the components
- diagonal matrix of eigenvalues
- total variance: the sum of the feature variances, which a rotation cannot change
The trace identity is what makes "share of variance" meaningful: rotating to the eigenbasis just redistributes the fixed total so that as much as possible lands on the first axes.
The same answer from reconstruction error
Keep only the score on and rebuild . The residual is perpendicular to , so Pythagoras splits each point's squared length into a kept part and a lost part:
- the reconstruction (shadow) of point
- residual: the part of the point in the dropped directions
The left side is fixed, so minimizing the reconstruction error is exactly maximizing the projected variance. With components the error is the sum of the dropped eigenvalues, , and no other rank- linear projection does better (the Eckart–Young theorem — see the SVD article for the matrix version).
Computing it with the SVD
Forming squares the condition number of the data. Libraries instead take the thin SVD of the centred data, (here holds the left singular vectors): the right singular vectors are the components, , and the scores are . The SVD article derives this factorization; Truncated SVD covers the variant that skips centring so it can work on sparse matrices.
What the hyperparameters do
At a glance
- k
- keeps a fraction of the variance; the mean squared reconstruction error is exactly . Each extra component lowers the error by its eigenvalue
- Standardize
- dividing each feature by its standard deviation replaces with the correlation matrix . Eigenvectors of and differ: unscaled PCA is dominated by whichever feature has the largest numeric variance
- Whiten
- makes the score covariance the identity. Distances stop favouring the first components — useful before k-means or ICA, harmful if small components are mostly noise (it amplifies them)
Worked example
Ten points in two dimensions (the classic tutorial set from Lindsay Smith). The features are strongly correlated, so one direction should carry almost everything. We reduce to and then project a new point .
| i | x₁ | x₂ | x₁ − 1.81 | x₂ − 1.91 |
|---|---|---|---|---|
| 1 | 2.5 | 2.4 | 0.69 | 0.49 |
| 2 | 0.5 | 0.7 | −1.31 | −1.21 |
| 3 | 2.2 | 2.9 | 0.39 | 0.99 |
| 4 | 1.9 | 2.2 | 0.09 | 0.29 |
| 5 | 3.1 | 3.0 | 1.29 | 1.09 |
| 6 | 2.3 | 2.7 | 0.49 | 0.79 |
| 7 | 2.0 | 1.6 | 0.19 | −0.31 |
| 8 | 1.0 | 1.1 | −0.81 | −0.81 |
| 9 | 1.5 | 1.6 | −0.31 | −0.31 |
| 10 | 1.1 | 0.9 | −0.71 | −1.01 |
PCA by hand
- 1Centre. Mean ; subtract it from every row (right two columns above).
- 2Covariance. From the centred columns: , , . Dividing by : . Total variance .
- 3Eigenvalues. Solve : , so , giving and . Check: .
- 4Eigenvector. The first row of reads , so . Normalizing by its length : , an angle of — just steeper than the diagonal because varies a little more. The second component is the perpendicular .
- 5Explained variance. PC1 keeps ; PC2 only . Compare: the original axis keeps .
- 6Scores. . Point 1: . All ten are in the next table; their variance is , as promised.
- 7Reconstruction error. Rebuilding from one score, , point 1 becomes with squared error . The ten errors sum to ; divided by that is — exactly the dropped eigenvalue.
- 8New point. , so . One number now stands for ; mapping it back gives . Its PC2 score, the part thrown away, is .
| i | z₁ (PC1) | z₂ (PC2) | x̂ from z₁ | squared error |
|---|---|---|---|---|
| 1 | 0.8280 | −0.1751 | (2.371, 2.519) | 0.0307 |
| 2 | −1.7776 | 0.1429 | (0.605, 0.603) | 0.0204 |
| 3 | 0.9922 | 0.3844 | (2.483, 2.639) | 0.1477 |
| 4 | 0.2742 | 0.1304 | (1.996, 2.112) | 0.0170 |
| 5 | 1.6758 | −0.2095 | (2.946, 3.142) | 0.0439 |
| 6 | 0.9129 | 0.1753 | (2.429, 2.581) | 0.0307 |
| 7 | −0.0991 | −0.3498 | (1.743, 1.837) | 0.1224 |
| 8 | −1.1446 | 0.0464 | (1.034, 1.069) | 0.0022 |
| 9 | −0.4380 | 0.0178 | (1.513, 1.588) | 0.0003 |
| 10 | −1.2238 | −0.1627 | (0.980, 1.010) | 0.0265 |
| variance / sum | 1.2840 | 0.0491 | 0.4418 |
The algorithm, step by step
PCA(X, k)
- 1Centre (and, if features have different units, standardize): compute (and per-feature ) on the training data and store them.
- 2Decompose. Either form and run a symmetric eigensolver (), or take the thin SVD of directly (, more stable).
- 3Sort components by eigenvalue, largest first. Fix each sign (e.g. make the largest loading positive) — eigenvectors are only defined up to .
- 4Choose k from the cumulative explained-variance ratio, a scree-plot elbow, or downstream validation score.
- 5Transform: , cost . Inverse transform: .
- 1X ← X − mean(X)
- 2C ← XᵀX / (n − 1)
- 3u ← argmax uᵀCu, ‖u‖ = 1
- 4 = top eigenvector of C
- 5Z ← X·u
Raw data: 40 points whose mean (×) is at (0.36, 0.23), not at the origin.
For large and with small , randomized SVD (Halko et al.) multiplies by a random matrix, orthonormalizes, and does an exact SVD of the small result — about with a few power iterations. scikit-learn's svd_solver="auto" forms the covariance when the data is tall and skinny ( and ), switches to randomized SVD when the matrix exceeds 500 in some dimension and an integer is below 80% of , and otherwise runs a full SVD.
Choosing k and other practical choices
There is no single right ; it depends on what the components are for. Three common rules, shown on the standardized Iris data (4 features, so the eigenvalues of the correlation matrix sum to 4):
| Component | Eigenvalue λ | Ratio | Cumulative |
|---|---|---|---|
| PC1 | 2.938 | 72.96% | 72.96% |
| PC2 | 0.920 | 22.85% | 95.81% |
| PC3 | 0.148 | 3.67% | 99.48% |
| PC4 | 0.021 | 0.52% | 100% |
Ways to pick k
- Variance threshold
- keep the smallest k reaching 90–99%. On the 64-pixel digits data that is 21 components for 90%, 29 for 95%, 41 for 99%. In scikit-learn pass a float:
PCA(n_components=0.95). - Scree elbow
- plot the eigenvalues and stop where the curve flattens; subjective but robust for visualization.
- Kaiser rule
- on standardized data keep components with (each worth more than one original feature). Often too aggressive — 1 for Iris, 3 for Wine.
- Downstream validation
- when PCA feeds a model, treat k as a hyperparameter and cross-validate the whole pipeline. This is the only rule that measures what you care about.
- Probabilistic (MLE)
n_components='mle'uses Minka's Bayesian model selection under probabilistic PCA.
| Variant | What changes | Use when |
|---|---|---|
| Full SVD / eigendecomposition | exact; all components | d up to a few thousand |
| Randomized PCA | approximate top-k via random projections | large n and d, small k |
| Incremental PCA | updates the decomposition one mini-batch at a time | data does not fit in memory, or arrives as a stream |
| Sparse PCA | L1 penalty on loadings, so each component uses few features | you need interpretable components |
| Probabilistic PCA | a latent Gaussian model with isotropic noise, fit by EM or closed form | missing values, likelihoods, model selection |
| Kernel PCA | PCA in an implicit feature space via a kernel | curved structure — see Kernel PCA |
| Truncated SVD | no centring | sparse text matrices (LSA) — see Truncated SVD |
| LDA | maximizes between-class over within-class scatter, using labels | supervised reduction — see LDA for dimensionality reduction |
In code
1import numpy as np23def pca_fit(X, k):4 mean = X.mean(axis=0)5 Xc = X - mean # 1. centre6 C = Xc.T @ Xc / (len(X) - 1) # 2. covariance (d x d)7 eigvals, eigvecs = np.linalg.eigh(C) # 3. symmetric eigensolver (ascending)8 order = np.argsort(eigvals)[::-1] # sort descending9 eigvals, eigvecs = eigvals[order], eigvecs[:, order]10 W = eigvecs[:, :k] # 4. top-k components as columns (d x k)11 W = W * np.sign(W[np.abs(W).argmax(axis=0), range(k)]) # fix sign: largest loading > 012 return mean, W, eigvals1314def transform(X, mean, W):15 return (X - mean) @ W # scores (n x k)1617def inverse_transform(Z, mean, W):18 return Z @ W.T + mean # back to d dimensions1920X = np.array([[2.5, 2.4], [0.5, 0.7], [2.2, 2.9], [1.9, 2.2], [3.1, 3.0],21 [2.3, 2.7], [2.0, 1.6], [1.0, 1.1], [1.5, 1.6], [1.1, 0.9]])22mean, W, lam = pca_fit(X, k=1)23Z = transform(X, mean, W)24X_hat = inverse_transform(Z, mean, W)25print("mean ", mean.round(4))26print("eigenvalues", lam.round(4), "explained", (lam / lam.sum()).round(4))27print("u1 ", W[:, 0].round(4))28print("scores ", Z[:, 0].round(4))29print("recon MSE (sum/(n-1))", (((X - X_hat) ** 2).sum() / (len(X) - 1)).round(4))30x_new = np.array([[2.0, 2.5]])31z_new = transform(x_new, mean, W)32print("new point ", z_new.round(4), inverse_transform(z_new, mean, W).round(4))1import numpy as np2from sklearn.decomposition import PCA34X = np.array([[2.5, 2.4], [0.5, 0.7], [2.2, 2.9], [1.9, 2.2], [3.1, 3.0],5 [2.3, 2.7], [2.0, 1.6], [1.0, 1.1], [1.5, 1.6], [1.1, 0.9]])67pca = PCA(n_components=1).fit(X)8print("mean ", pca.mean_.round(4))9print("u1 ", pca.components_[0].round(4))10print("lambda1 ", pca.explained_variance_.round(4))11print("ratio ", pca.explained_variance_ratio_.round(4))12print("scores ", pca.transform(X)[:, 0].round(4))1314z_new = pca.transform([[2.0, 2.5]])15print("new point ", z_new.round(4), pca.inverse_transform(z_new).round(4))1617# On real data: standardize, then keep enough components for 95% of the variance18from sklearn.datasets import load_wine19from sklearn.pipeline import make_pipeline20from sklearn.preprocessing import StandardScaler21Xw = load_wine().data22pipe = make_pipeline(StandardScaler(), PCA(n_components=0.95)).fit(Xw)23print("wine: k for 95% =", pipe[-1].n_components_)Use when
- +Many correlated numeric features and you want fewer, decorrelated inputs for a downstream model.
- +A first 2-D or 3-D look at high-dimensional data, or a quick check of how many directions really matter.
- +Compression and denoising, when the noise is spread over many small-variance directions.
- +You need a deterministic, fast, invertible transform with no tuning beyond k.
Avoid when
- −The structure is nonlinear (curved manifolds, clusters on rings) — try Kernel PCA, UMAP or an autoencoder.
- −The signal you care about has low variance: PCA ignores labels and may discard the one direction that separates classes (use LDA).
- −You need interpretable features: every component mixes every original feature (consider sparse PCA or feature selection).
- −Heavy outliers or wildly different feature scales you cannot or will not fix.
Where it's used
Population genetics
The first two components of genome-wide genotype data reproduce a map of ancestry — a standard correction for population structure in association studies.
Finance
A few components (market, sector, rates) explain most co-movement of stock returns or of a yield curve; risk models are built on them.
Faces and images
Eigenfaces: each face as a combination of a few dozen principal images; also ZCA/PCA whitening before training vision models.
Preprocessing and embeddings
Shrink thousands of features, or 768-dimensional text embeddings, before clustering, nearest-neighbour search or regression.
Check yourself
Covariance eigenvalues are 6, 3 and 1. What fraction of the variance do the first two components keep, and what is the mean squared reconstruction error with k = 2?
Check yourself
In the worked example, what is the mean squared reconstruction error (sum divided by n − 1) when keeping only PC1?
Check yourself
Unscaled PCA on the Wine data puts 99.81% of the variance on PC1. What is the most likely explanation?
Check yourself
Which statement about principal component scores is true?
Remember
- ✓Centre the data; the principal components are the eigenvectors of the covariance , sorted by eigenvalue. Component captures variance .
- ✓Maximizing subject to gives via a Lagrange multiplier; later components are the next eigenvectors, orthogonal to earlier ones, and their scores are uncorrelated.
- ✓Total variance is fixed, so maximum kept variance equals minimum reconstruction error, which is .
- ✓Compute with an SVD of the centred data (randomized for large data); choose k by a variance threshold or by validating the downstream model.
- ✓Standardize mixed units, watch for outliers, fit on training data only, and remember PCA is linear and label-blind.