# Principal Component Analysis

> Project data onto the directions of maximum variance — the top eigenvectors of its covariance matrix.

Source: https://www.mlfoundry.in/learn/pca  
Section: ML Algorithms › Unsupervised · Dimensionality Reduction  
Level: core · 17 min read · updated 2026-10-07

**Principal component analysis (PCA):** finds the orthogonal directions along which centred data varies most, and represents each point by its coordinates along the top $k$ of them. It is the linear projection to $k$ dimensions that keeps the most variance — and, equivalently, the one whose reconstruction has the smallest squared error.

- **Task:** Unsupervised dimensionality reduction (also compression, denoising, decorrelation)
- **Input → output:** $X\in\mathbb{R}^{n\times d}$ → scores $Z\in\mathbb{R}^{n\times k}$, with $k<d$
- **Learns:** the mean $\bar x$, $k$ orthonormal directions $u_1,\dots,u_k$ (the principal components) and their variances $\lambda_1\ge\dots\ge\lambda_k$
- **Training:** one eigendecomposition of the $d\times d$ covariance, $O(nd^2+d^3)$, or one thin SVD of $X$, $O(nd\min(n,d))$; randomized SVD costs about $O(ndk)$
- **Prediction:** project a new point: $z=U_k^\top(x-\bar x)$, $O(dk)$
- **Hyperparameters:** number of components $k$ (or a variance fraction such as 95%); whether to standardize; whitening; solver
- **Assumes:** the structure worth keeping is linear and shows up as large variance; features are on comparable scales

## The problem

Real feature sets are redundant. The 64 pixels of a handwritten digit move together; height and weight rise together; thirteen chemical measurements of a wine are driven by a handful of underlying factors. Redundancy costs us: models train slower, distances become less meaningful, and we cannot *look* at data with more than three dimensions.

The obvious fix — **drop some features** — throws information away blindly. In the 10-point example below, $x_1$ and $x_2$ have correlation 0.926. Keeping only $x_1$ keeps 46.2% of the total variance; keeping only $x_2$ keeps 53.8%. Yet a *new* axis, a blend of both, keeps **96.3%**. PCA finds that blend: instead of choosing among the original axes, it rotates the coordinate system to the axes the data actually uses.

## The idea

Picture the data as a cloud. Move the origin to the centre of the cloud, then turn a stick through it until the shadows of the points on the stick are as **spread out** as possible — that stick is the first principal component. The second component is the best stick *perpendicular* to the first, and so on. Each point is then described by where its shadow falls on the first few sticks; the remaining, low-spread directions are dropped because points barely differ along them.

Why spread? Because a direction along which every point has the same coordinate tells you nothing about how points differ. And there is a second reading of the same picture: the leftover distance from each point to its shadow is the error you make by keeping only the shadow. Since the total spread is fixed, **maximizing the spread of the shadows is the same as minimizing the leftover error**.

**Vocabulary**

- **Centring:** subtracting the feature means so the cloud sits on the origin; PCA directions pass through $\bar x$
- **Principal component (PC):** a unit direction $u_j\in\mathbb{R}^d$ — an eigenvector of the covariance matrix. Also called a *loading vector*; its entries are the **loadings** (how much each original feature contributes)
- **Score:** the coordinate of a point along a component, $z_{ij}=u_j^\top(x_i-\bar x)$ — the length of its shadow
- **Explained variance:** $\lambda_j$, the variance of the scores on component $j$ — the eigenvalue
- **Explained variance ratio:** $\lambda_j/\sum_l\lambda_l$, the share of the total variance component $j$ keeps
- **Scree plot:** the eigenvalues in decreasing order; one looks for the "elbow" where they flatten out
- **Reconstruction:** mapping scores back: $\hat x=\bar x+\sum_{j\le k}z_j u_j$; its error is the part of $x$ in the dropped directions
- **Whitening:** dividing each score by $\sqrt{\lambda_j}$ so every kept component has unit variance

## The math

Four short derivations: the variance along a direction, its maximization (which produces the eigenvalue equation), the equivalent reconstruction-error view, and the link to the SVD that libraries actually compute.

## Variance along a direction

Centre the data: $\tilde x_i=x_i-\bar x$, stacked into the rows of $\tilde X$. The score of point $i$ on a unit vector $u$ is $z_i=u^\top\tilde x_i$, and because the scores have mean zero their variance is a quadratic form in the covariance matrix:

$$
\operatorname{Var}(z)=\frac{1}{n-1}\sum_{i=1}^n (u^\top\tilde x_i)^2=u^\top\Big(\frac{\tilde X^\top\tilde X}{n-1}\Big)u=u^\top C\,u
$$

- $\tilde X$: centred data, $n\times d$, one row per example
- $u$: a candidate direction with $\lVert u\rVert=1$
- $C$: sample covariance matrix, $d\times d$, symmetric and positive semi-definite; $C_{ab}$ is the covariance of features $a$ and $b$
- $n-1$: Bessel's correction (scikit-learn uses it too); it rescales every eigenvalue equally and never changes the directions

## Maximizing it: the eigenvalue equation

Without the constraint $\lVert u\rVert=1$ we could inflate the variance just by scaling $u$, so maximize $u^\top Cu$ subject to $u^\top u=1$ with a Lagrange multiplier $\lambda$:

$$
\mathcal{L}(u,\lambda)=u^\top Cu-\lambda\,(u^\top u-1),\qquad \nabla_u\mathcal{L}=2Cu-2\lambda u=0\;\Longrightarrow\; C\,u=\lambda\,u
$$

- $\lambda$: Lagrange multiplier — it turns out to be an eigenvalue of C
- $\nabla_u$: gradient with respect to $u$ (using $\nabla_u\, u^\top Cu=2Cu$ for symmetric $C$)

Every stationary point is an **eigenvector** of $C$. Multiply $Cu=\lambda u$ on the left by $u^\top$: the variance captured is $u^\top Cu=\lambda u^\top u=\lambda$. So among the eigenvectors, the best direction is the one with the **largest eigenvalue**. Because $C$ is symmetric, its eigenvectors can be chosen orthonormal and its eigenvalues are real and $\ge 0$.

For the second component add the constraint $u_2\perp u_1$. The same Lagrangian argument (with a second multiplier for $u_2^\top u_1=0$) gives $Cu_2=\lambda u_2$ restricted to the subspace orthogonal to $u_1$ — the eigenvector with the second-largest eigenvalue. Repeating, the first $k$ components are the top-$k$ eigenvectors, and their scores are **uncorrelated**: $\operatorname{Cov}(z_a,z_b)=u_a^\top Cu_b=\lambda_b u_a^\top u_b=0$ for $a\ne b$.

$$
C=U\Lambda U^\top=\sum_{j=1}^d\lambda_j u_ju_j^\top,\qquad \operatorname{tr}C=\sum_{a=1}^d C_{aa}=\sum_{j=1}^d\lambda_j
$$

- $U$: orthogonal $d\times d$ matrix whose columns are the components $u_1,\dots,u_d$
- $\Lambda$: diagonal matrix of eigenvalues $\lambda_1\ge\lambda_2\ge\dots\ge\lambda_d\ge0$
- $\operatorname{tr}C$: total variance: the sum of the feature variances, which a rotation cannot change

The trace identity is what makes "share of variance" meaningful: rotating to the eigenbasis just redistributes the fixed total $\operatorname{tr}C$ so that as much as possible lands on the first axes.

## The same answer from reconstruction error

Keep only the score on $u$ and rebuild $\hat{\tilde x}_i=z_iu$. The residual $\tilde x_i-z_iu$ is perpendicular to $u$, so Pythagoras splits each point's squared length into a kept part and a lost part:

$$
\lVert\tilde x_i\rVert^2=z_i^2+\lVert\tilde x_i-z_iu\rVert^2\quad\Longrightarrow\quad \frac{1}{n-1}\sum_i\lVert\tilde x_i-z_iu\rVert^2=\operatorname{tr}C-u^\top Cu
$$

- $z_i u$: the reconstruction (shadow) of point $i$
- $\tilde x_i-z_iu$: residual: the part of the point in the dropped directions

The left side is fixed, so **minimizing the reconstruction error is exactly maximizing the projected variance**. With $k$ components the error is the sum of the dropped eigenvalues, $\sum_{j>k}\lambda_j$, and no other rank-$k$ linear projection does better (the Eckart–Young theorem — see the **SVD** article for the matrix version).

## Computing it with the SVD

Forming $C$ squares the condition number of the data. Libraries instead take the thin SVD of the centred data, $\tilde X=V\Sigma U^\top$ (here $V$ holds the left singular vectors): the right singular vectors are the components, $\lambda_j=\sigma_j^2/(n-1)$, and the scores are $Z=\tilde XU_k=V_k\Sigma_k$. The **SVD** article derives this factorization; **Truncated SVD** covers the variant that skips centring so it can work on sparse matrices.

## What the hyperparameters do

- **k:** keeps a fraction $\sum_{j\le k}\lambda_j/\sum_j\lambda_j$ of the variance; the mean squared reconstruction error is exactly $\sum_{j>k}\lambda_j$. Each extra component lowers the error by its eigenvalue
- **Standardize:** dividing each feature by its standard deviation replaces $C$ with the **correlation matrix** $R=D^{-1/2}CD^{-1/2}$. Eigenvectors of $C$ and $R$ differ: unscaled PCA is dominated by whichever feature has the largest numeric variance
- **Whiten:** $z_j\to z_j/\sqrt{\lambda_j}$ makes the score covariance the identity. Distances stop favouring the first components — useful before k-means or ICA, harmful if small components are mostly noise (it amplifies them)

## Worked example

Ten points in two dimensions (the classic tutorial set from Lindsay Smith). The features are strongly correlated, so one direction should carry almost everything. We reduce to $k=1$ and then project a new point $x^*=(2.0,\,2.5)$.

The data and the centred data. Means: $\bar x_1=1.81$, $\bar x_2=1.91$.

| i | x₁ | x₂ | x₁ − 1.81 | x₂ − 1.91 |
| --- | --- | --- | --- | --- |
| 1 | 2.5 | 2.4 | 0.69 | 0.49 |
| 2 | 0.5 | 0.7 | −1.31 | −1.21 |
| 3 | 2.2 | 2.9 | 0.39 | 0.99 |
| 4 | 1.9 | 2.2 | 0.09 | 0.29 |
| 5 | 3.1 | 3.0 | 1.29 | 1.09 |
| 6 | 2.3 | 2.7 | 0.49 | 0.79 |
| 7 | 2.0 | 1.6 | 0.19 | −0.31 |
| 8 | 1.0 | 1.1 | −0.81 | −0.81 |
| 9 | 1.5 | 1.6 | −0.31 | −0.31 |
| 10 | 1.1 | 0.9 | −0.71 | −1.01 |

**PCA by hand**

1. **Centre.** Mean $\bar x=(18.1/10,\ 19.1/10)=(1.81,\ 1.91)$; subtract it from every row (right two columns above).
2. **Covariance.** From the centred columns: $\sum\tilde x_1^2=5.549$, $\sum\tilde x_2^2=6.449$, $\sum\tilde x_1\tilde x_2=5.539$. Dividing by $n-1=9$: $C=\begin{pmatrix}0.6166&0.6154\\0.6154&0.7166\end{pmatrix}$. Total variance $\operatorname{tr}C=1.3331$.
3. **Eigenvalues.** Solve $\det(C-\lambda I)=0$: $\lambda^2-(\operatorname{tr}C)\lambda+\det C=\lambda^2-1.3331\lambda+0.0630=0$, so $\lambda=\tfrac{1.3331}{2}\pm\sqrt{\tfrac{1.3331^2}{4}-0.0630}$, giving $\lambda_1=1.2840$ and $\lambda_2=0.0491$. Check: $1.2840+0.0491=1.3331=\operatorname{tr}C$.
4. **Eigenvector.** The first row of $(C-\lambda_1I)u=0$ reads $(0.6166-1.2840)u_1+0.6154\,u_2=0$, so $u_2/u_1=0.6674/0.6154=1.0845$. Normalizing $(1,\,1.0845)$ by its length $1.4752$: $u_1=(0.6779,\ 0.7352)$, an angle of $47.3°$ — just steeper than the diagonal because $x_2$ varies a little more. The second component is the perpendicular $u_2=(-0.7352,\ 0.6779)$.
5. **Explained variance.** PC1 keeps $\lambda_1/\operatorname{tr}C=1.2840/1.3331=96.3\%$; PC2 only $3.7\%$. Compare: the original $x_1$ axis keeps $0.6166/1.3331=46.2\%$.
6. **Scores.** $z_{i}=0.6779\,\tilde x_{i1}+0.7352\,\tilde x_{i2}$. Point 1: $0.6779(0.69)+0.7352(0.49)=0.8280$. All ten are in the next table; their variance is $1.2840=\lambda_1$, as promised.
7. **Reconstruction error.** Rebuilding from one score, $\hat x_i=\bar x+z_iu_1$, point 1 becomes $(2.371,\ 2.519)$ with squared error $0.0307$. The ten errors sum to $0.4418$; divided by $n-1$ that is $0.0491=\lambda_2$ — exactly the dropped eigenvalue.
8. **New point.** $x^*-\bar x=(0.19,\ 0.59)$, so $z^*=0.6779(0.19)+0.7352(0.59)=0.5626$. One number now stands for $x^*$; mapping it back gives $\hat x^*=\bar x+0.5626\,u_1=(2.191,\ 2.324)$. Its PC2 score, the part thrown away, is $0.2603$.

Scores on both components, the 1-D reconstruction and its squared error. Note that each squared error equals the PC2 score squared — the residual lives entirely along $u_2$.

| i | z₁ (PC1) | z₂ (PC2) | x̂ from z₁ | squared error |
| --- | --- | --- | --- | --- |
| 1 | 0.8280 | −0.1751 | (2.371, 2.519) | 0.0307 |
| 2 | −1.7776 | 0.1429 | (0.605, 0.603) | 0.0204 |
| 3 | 0.9922 | 0.3844 | (2.483, 2.639) | 0.1477 |
| 4 | 0.2742 | 0.1304 | (1.996, 2.112) | 0.0170 |
| 5 | 1.6758 | −0.2095 | (2.946, 3.142) | 0.0439 |
| 6 | 0.9129 | 0.1753 | (2.429, 2.581) | 0.0307 |
| 7 | −0.0991 | −0.3498 | (1.743, 1.837) | 0.1224 |
| 8 | −1.1446 | 0.0464 | (1.034, 1.069) | 0.0022 |
| 9 | −0.4380 | 0.0178 | (1.513, 1.588) | 0.0003 |
| 10 | −1.2238 | −0.1627 | (0.980, 1.010) | 0.0265 |
| **variance / sum** | **1.2840** | **0.0491** |  | **0.4418** |

> Figure: The worked example, live. The orange line is a candidate direction u through the mean (×); rose segments are each point's residual to its shadow (orange dots). It starts on the x₁ axis: variance along u 0.617 (46.2%), residual 0.717. Drag the orange handle — the bar always sums to the total 1.333, and the curve on the right peaks at 47.3°, where u is PC1: 1.284 (96.3%) kept, residual 0.049 = λ₂. The violet ring is the new point x* = (2.0, 2.5); on PC1 its score is 0.563 and its reconstruction (2.191, 2.324). Drag any blue point to watch the covariance, eigenvalues and components move.

> **Two objectives, one answer:** Watch the bar in the figure while you turn $u$: the orange part (variance kept) and the rose part (squared residual) always add to the same total, $\operatorname{tr}C$. That is the Pythagoras identity from the math section. "Keep the most variance" and "lose the least information under squared error" are not two goals that happen to agree — they are the same goal written twice.

## The algorithm, step by step

**PCA(X, k)**

1. **Centre** (and, if features have different units, **standardize**): compute $\bar x$ (and per-feature $s$) on the *training* data and store them.
2. **Decompose.** Either form $C=\tilde X^\top\tilde X/(n-1)$ and run a symmetric eigensolver ($O(nd^2+d^3)$), or take the thin SVD of $\tilde X$ directly ($O(nd\min(n,d))$, more stable).
3. **Sort** components by eigenvalue, largest first. Fix each sign (e.g. make the largest loading positive) — eigenvectors are only defined up to $\pm$.
4. **Choose k** from the cumulative explained-variance ratio, a scree-plot elbow, or downstream validation score.
5. **Transform:** $Z=(X-\bar x)U_k$, cost $O(ndk)$. **Inverse transform:** $\hat X=ZU_k^\top+\bar x$.

> Figure: The same computation on a 40-point cloud: centre, form the covariance, sweep a direction through 0°–180° and watch its projected variance — the sweep's peak is the top eigenvector, found in closed form. One dimension keeps 96.1% of this cloud's variance.

For large $n$ and $d$ with small $k$, **randomized SVD** (Halko et al.) multiplies $\tilde X$ by a random $d\times(k+p)$ matrix, orthonormalizes, and does an exact SVD of the small result — about $O(ndk)$ with a few power iterations. scikit-learn's `svd_solver="auto"` forms the covariance when the data is tall and skinny ($d\le1000$ and $n\ge10d$), switches to randomized SVD when the matrix exceeds 500 in some dimension and an integer $k$ is below 80% of $\min(n,d)$, and otherwise runs a full SVD.

## Choosing k and other practical choices

There is no single right $k$; it depends on what the components are for. Three common rules, shown on the standardized Iris data (4 features, so the eigenvalues of the correlation matrix sum to 4):

Standardized Iris. Two components keep 95.8% of the variance — enough for a faithful 2-D plot. The Kaiser rule (keep λ > 1) would keep only one.

| Component | Eigenvalue λ | Ratio | Cumulative |
| --- | --- | --- | --- |
| PC1 | 2.938 | 72.96% | 72.96% |
| PC2 | 0.920 | 22.85% | 95.81% |
| PC3 | 0.148 | 3.67% | 99.48% |
| PC4 | 0.021 | 0.52% | 100% |

**Ways to pick k**

- **Variance threshold:** keep the smallest k reaching 90–99%. On the 64-pixel digits data that is 21 components for 90%, 29 for 95%, 41 for 99%. In scikit-learn pass a float: `PCA(n_components=0.95)`.
- **Scree elbow:** plot the eigenvalues and stop where the curve flattens; subjective but robust for visualization.
- **Kaiser rule:** on standardized data keep components with $\lambda>1$ (each worth more than one original feature). Often too aggressive — 1 for Iris, 3 for Wine.
- **Downstream validation:** when PCA feeds a model, treat k as a hyperparameter and cross-validate the whole pipeline. This is the only rule that measures what you care about.
- **Probabilistic (MLE):** `n_components='mle'` uses Minka's Bayesian model selection under probabilistic PCA.

Variants of PCA and when to reach for them.

| Variant | What changes | Use when |
| --- | --- | --- |
| Full SVD / eigendecomposition | exact; all components | d up to a few thousand |
| Randomized PCA | approximate top-k via random projections | large n and d, small k |
| Incremental PCA | updates the decomposition one mini-batch at a time | data does not fit in memory, or arrives as a stream |
| Sparse PCA | L1 penalty on loadings, so each component uses few features | you need interpretable components |
| Probabilistic PCA | a latent Gaussian model with isotropic noise, fit by EM or closed form | missing values, likelihoods, model selection |
| Kernel PCA | PCA in an implicit feature space via a kernel | curved structure — see **Kernel PCA** |
| Truncated SVD | no centring | sparse text matrices (LSA) — see **Truncated SVD** |
| LDA | maximizes between-class over within-class scatter, using labels | supervised reduction — see **LDA for dimensionality reduction** |

> **Standardize features with different units:** PCA chases raw variance, so units decide the answer. On the UCI Wine data (13 features) unscaled PCA puts **99.81%** of the variance on PC1, which is almost exactly the *proline* column (loading 1.000) — simply because proline is measured in the hundreds while the others are in single digits. After standardizing, PC1 keeps 36.2% and mixes many features, and 10 components are needed for 95%. Use a scaler unless the features share a unit and their variance is meaningful (pixels, spectra).

> **Outliers grab components:** Variance is a sum of squares, so one far point can bend or claim a component. Adding the single point $(3.5,\,0.5)$ to the worked example drops PC1's share from 96.3% to 70.6% — the new PC2 exists mostly to describe that one outlier. Inspect scores for extreme values, or use robust PCA variants.

## In code

```python
import numpy as np

def pca_fit(X, k):
    mean = X.mean(axis=0)
    Xc = X - mean                                   # 1. centre
    C = Xc.T @ Xc / (len(X) - 1)                    # 2. covariance (d x d)
    eigvals, eigvecs = np.linalg.eigh(C)            # 3. symmetric eigensolver (ascending)
    order = np.argsort(eigvals)[::-1]               #    sort descending
    eigvals, eigvecs = eigvals[order], eigvecs[:, order]
    W = eigvecs[:, :k]                              # 4. top-k components as columns (d x k)
    W = W * np.sign(W[np.abs(W).argmax(axis=0), range(k)])  # fix sign: largest loading > 0
    return mean, W, eigvals

def transform(X, mean, W):
    return (X - mean) @ W                           # scores (n x k)

def inverse_transform(Z, mean, W):
    return Z @ W.T + mean                           # back to d dimensions

X = np.array([[2.5, 2.4], [0.5, 0.7], [2.2, 2.9], [1.9, 2.2], [3.1, 3.0],
              [2.3, 2.7], [2.0, 1.6], [1.0, 1.1], [1.5, 1.6], [1.1, 0.9]])
mean, W, lam = pca_fit(X, k=1)
Z = transform(X, mean, W)
X_hat = inverse_transform(Z, mean, W)
print("mean      ", mean.round(4))
print("eigenvalues", lam.round(4), "explained", (lam / lam.sum()).round(4))
print("u1        ", W[:, 0].round(4))
print("scores    ", Z[:, 0].round(4))
print("recon MSE (sum/(n-1))", (((X - X_hat) ** 2).sum() / (len(X) - 1)).round(4))
x_new = np.array([[2.0, 2.5]])
z_new = transform(x_new, mean, W)
print("new point ", z_new.round(4), inverse_transform(z_new, mean, W).round(4))
```

From scratch with NumPy, on the worked example. Prints λ = [1.284, 0.0491], u₁ = [0.6779, 0.7352], reconstruction MSE 0.0491 and the new point's score 0.5626 → (2.1913, 2.3236).

```python
import numpy as np
from sklearn.decomposition import PCA

X = np.array([[2.5, 2.4], [0.5, 0.7], [2.2, 2.9], [1.9, 2.2], [3.1, 3.0],
              [2.3, 2.7], [2.0, 1.6], [1.0, 1.1], [1.5, 1.6], [1.1, 0.9]])

pca = PCA(n_components=1).fit(X)
print("mean      ", pca.mean_.round(4))
print("u1        ", pca.components_[0].round(4))
print("lambda1   ", pca.explained_variance_.round(4))
print("ratio     ", pca.explained_variance_ratio_.round(4))
print("scores    ", pca.transform(X)[:, 0].round(4))

z_new = pca.transform([[2.0, 2.5]])
print("new point ", z_new.round(4), pca.inverse_transform(z_new).round(4))

# On real data: standardize, then keep enough components for 95% of the variance
from sklearn.datasets import load_wine
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
Xw = load_wine().data
pipe = make_pipeline(StandardScaler(), PCA(n_components=0.95)).fit(Xw)
print("wine: k for 95% =", pipe[-1].n_components_)
```

scikit-learn on the same data (same mean, component, λ₁ = 1.284, ratio 0.9632, scores and new-point result), then the standard scale-then-reduce pipeline on Wine, which needs 10 components for 95%.

> **Signs are arbitrary:** If $u$ is an eigenvector, so is $-u$. Two libraries (or two versions of one) may return components with flipped signs and scores with flipped signs; both are correct. scikit-learn fixes signs deterministically, which is why it matches the from-scratch code here. Never compare loadings across fits without aligning signs first.

**Strengths**

- Many correlated numeric features and you want fewer, decorrelated inputs for a downstream model.
- A first 2-D or 3-D look at high-dimensional data, or a quick check of how many directions really matter.
- Compression and denoising, when the noise is spread over many small-variance directions.
- You need a deterministic, fast, invertible transform with no tuning beyond k.

**Limitations**

- The structure is nonlinear (curved manifolds, clusters on rings) — try Kernel PCA, UMAP or an autoencoder.
- The signal you care about has low variance: PCA ignores labels and may discard the one direction that separates classes (use LDA).
- You need interpretable features: every component mixes every original feature (consider sparse PCA or feature selection).
- Heavy outliers or wildly different feature scales you cannot or will not fix.

- **Population genetics:** The first two components of genome-wide genotype data reproduce a map of ancestry — a standard correction for population structure in association studies.
- **Finance:** A few components (market, sector, rates) explain most co-movement of stock returns or of a yield curve; risk models are built on them.
- **Faces and images:** Eigenfaces: each face as a combination of a few dozen principal images; also ZCA/PCA whitening before training vision models.
- **Preprocessing and embeddings:** Shrink thousands of features, or 768-dimensional text embeddings, before clustering, nearest-neighbour search or regression.

**Question:** Covariance eigenvalues are 6, 3 and 1. What fraction of the variance do the first two components keep, and what is the mean squared reconstruction error with k = 2?

- A. 60%, error 4
- B. 90%, error 1
- C. 90%, error 3
- D. 66%, error 1

**Answer:** B. 90%, error 1 — Kept: $(6+3)/(6+3+1)=0.9$. The error equals the sum of the dropped eigenvalues, here just $\lambda_3=1$.

**Question:** In the worked example, what is the mean squared reconstruction error (sum divided by n − 1) when keeping only PC1?

- A. 0.0491 — the second eigenvalue
- B. 1.2840 — the first eigenvalue
- C. 0.4418 — the sum of squared PC1 scores
- D. 0 — the reconstruction is exact

**Answer:** A. 0.0491 — the second eigenvalue — The ten squared errors sum to 0.4418, and $0.4418/9=0.0491=\lambda_2$. Everything lost lives in the dropped direction $u_2$.

**Question:** Unscaled PCA on the Wine data puts 99.81% of the variance on PC1. What is the most likely explanation?

- A. The wines really vary along one chemical axis
- B. One feature (proline) has a far larger numeric variance than the rest, so it dominates
- C. PCA overfits small datasets
- D. The data was not centred

**Answer:** B. One feature (proline) has a far larger numeric variance than the rest, so it dominates — PC1's loading on proline is 1.000. After standardizing, PC1 keeps only 36.2% — the spread was an artefact of units.

**Question:** Which statement about principal component scores is true?

- A. They are statistically independent
- B. They are uncorrelated, with variances equal to the eigenvalues
- C. They all have unit variance
- D. They separate the classes as well as possible

**Answer:** B. They are uncorrelated, with variances equal to the eigenvalues — $\operatorname{Cov}(z_a,z_b)=u_a^\top Cu_b=0$ and $\operatorname{Var}(z_j)=\lambda_j$. Uncorrelated is weaker than independent (that is ICA's goal); unit variance requires whitening; PCA never sees labels.

**Key points**

- Centre the data; the principal components are the eigenvectors of the covariance $C$, sorted by eigenvalue. Component $j$ captures variance $\lambda_j$.
- Maximizing $u^\top Cu$ subject to $\lVert u\rVert=1$ gives $Cu=\lambda u$ via a Lagrange multiplier; later components are the next eigenvectors, orthogonal to earlier ones, and their scores are uncorrelated.
- Total variance $\operatorname{tr}C=\sum\lambda_j$ is fixed, so maximum kept variance equals minimum reconstruction error, which is $\sum_{j>k}\lambda_j$.
- Compute with an SVD of the centred data (randomized for large data); choose k by a variance threshold or by validating the downstream model.
- Standardize mixed units, watch for outliers, fit on training data only, and remember PCA is linear and label-blind.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
