# Kernel PCA

> Run PCA in a kernel-induced feature space using only the kernel matrix, capturing nonlinear structure.

Source: https://www.mlfoundry.in/learn/kernel-pca  
Section: ML Algorithms › Unsupervised · Dimensionality Reduction  
Level: advanced · 17 min read · updated 2026-10-07

**Kernel PCA:** is principal component analysis carried out in a high- (often infinite-) dimensional feature space $\phi(x)$ that is never built explicitly. Everything is computed from the $n\times n$ **kernel matrix** $K_{ij}=k(x_i,x_j)=\phi(x_i)^\top\phi(x_j)$: centre it, take its top eigenvectors, and each eigenvector becomes a *nonlinear* projection $z(x)=\sum_i \alpha_i\,k(x_i,x)$ of the input.

- **Task:** Unsupervised nonlinear dimensionality reduction / feature extraction
- **Input → output:** $n$ points $x_i\in\mathbb R^d$ → $m$ scores $(z_1,\dots,z_m)$ per point, plus a map for new points
- **Learns:** dual coefficients $\alpha^{(j)}\in\mathbb R^n$ and eigenvalues $\mu_j$ of the centred kernel matrix; it must also keep the training set
- **Training cost:** $O(n^2 d)$ to build $K$, $O(n^3)$ for a full eigen-decomposition ($O(n^2 m)$ with iterative solvers); $O(n^2)$ memory
- **Prediction cost:** $O(n\,d + n\,m)$ per new point — one kernel evaluation against every training point
- **Hyperparameters:** kernel (RBF, polynomial, sigmoid…), its parameters ($\gamma$, degree, $c_0$), number of components $m$
- **Assumes:** the structure you care about is *linear* in the chosen feature space — i.e. the kernel encodes the right notion of similarity; features on comparable scales

## The problem

Ordinary **PCA** (see the PCA article) looks for straight directions of maximum variance. That is exactly right when the data lie near a flat subspace and exactly wrong when they lie on a curve. Take two concentric rings: an inner ring of radius 1 and an outer ring of radius 3. The one number that describes the structure — the radius — is a *nonlinear* function of $(x_1,x_2)$. Worse, because both rings are symmetric, their covariance matrix is a multiple of the identity: every direction has the same variance, PCA has no preferred axis, and any 1-D projection mixes the rings.

The classical fix is to add features by hand — $x_1^2$, $x_2^2$, $x_1x_2$ — and run PCA on those. For two inputs and degree 2 that is 6 features, but a degree-3 polynomial map of a 784-pixel image has $\binom{787}{3}\approx 81$ million, and the Gaussian (RBF) kernel corresponds to infinitely many. Kernel PCA gets the result of PCA in that space *without ever writing a feature vector down*.

## The idea

Two observations make it work. First, the principal axes in feature space always lie in the span of the mapped training points, so each axis can be written as $v=\sum_i\alpha_i\phi(x_i)$ — $n$ numbers instead of a vector of unknown length. Second, everything PCA needs (variances, projections, normalisation) then reduces to inner products $\phi(x_i)^\top\phi(x_j)$, which the kernel function computes directly from the raw inputs. That is the **kernel trick** applied to PCA instead of to a classifier (compare the SVM article).

The payoff is that a "projection onto an axis" becomes a curved function on the input space. With an RBF kernel the first component of the rings data is, in effect, a radius detector: its level sets are circles, so all inner-ring points get one score and all outer-ring points another.

**Vocabulary**

- **Feature map φ:** a (possibly infinite-dimensional) embedding $\phi:\mathbb R^d\to\mathcal H$; never computed explicitly
- **Kernel k:** a function with $k(x,x')=\phi(x)^\top\phi(x')$ — e.g. RBF $e^{-\gamma\lVert x-x'\rVert^2}$, polynomial $(x^\top x'+c_0)^p$, linear $x^\top x'$
- **Gram / kernel matrix K:** the $n\times n$ table of all pairwise kernel values $K_{ij}=k(x_i,x_j)$; symmetric and positive semidefinite
- **Centring:** subtracting the feature-space mean $\bar\phi$ from every $\phi(x_i)$; done implicitly as $\tilde K=HKH$
- **Dual coefficients α:** the weights expressing a principal axis as $v=\sum_i\alpha_i\phi(x_i)$; one vector $\alpha^{(j)}$ per component
- **Eigenvalue μ:** eigenvalue of $\tilde K$; the variance captured along that axis is $\mu/n$
- **Score z:** the coordinate of a point along a kernel principal axis, $z=v^\top(\phi(x)-\bar\phi)$
- **Out-of-sample projection:** scoring a point that was not in the training set, using its kernel values against the training points
- **Pre-image:** a point $x$ in input space whose $\phi(x)$ is close to a given feature-space vector — needed for denoising/reconstruction, and generally does not exist exactly

## The math

Assume for now that the mapped points are centred, $\sum_i\phi(x_i)=0$. PCA in feature space diagonalises the covariance operator

$$
C=\frac1n\sum_{i=1}^{n}\phi(x_i)\,\phi(x_i)^\top,\qquad C\,v=\lambda\,v
$$

- $\phi(x_i)$: the i-th training point mapped into feature space
- $C$: feature-space covariance (an operator if the space is infinite-dimensional)
- $v,\ \lambda$: a principal axis and the variance along it
- $n$: number of training points

**Step 1 — the axis lives in the span of the data.** Expand $Cv$: $\;Cv=\frac1n\sum_i\phi(x_i)\,\big(\phi(x_i)^\top v\big)$. Each bracket is a scalar, so for $\lambda>0$, $v=\frac{1}{n\lambda}\sum_i\big(\phi(x_i)^\top v\big)\phi(x_i)$ — a linear combination of the mapped points. Name the coefficients $\alpha_i$:

$$
v=\sum_{i=1}^{n}\alpha_i\,\phi(x_i)
$$

- $\alpha_i$: dual coefficient of training point i for this axis

**Step 2 — turn the eigen-problem into one about K.** Substitute $v$ into $Cv=\lambda v$ and take the inner product of both sides with $\phi(x_k)$ for every $k$. Every $\phi^\top\phi$ becomes a kernel value, and stacking the $n$ equations gives $\frac1n K^2\alpha=\lambda K\alpha$. Any solution of the smaller problem below solves this one (the remaining solutions differ only by null-space directions of $K$, which do not change $v$):

$$
\tilde K\,\alpha=\mu\,\alpha,\qquad \mu=n\lambda
$$

- $\tilde K$: the centred n×n kernel matrix
- $\mu$: eigenvalue of $\tilde K$; the feature-space variance is $\lambda=\mu/n$

**Step 3 — normalise.** PCA axes have unit length. $\lVert v\rVert^2=\sum_{i,j}\alpha_i\alpha_j\phi(x_i)^\top\phi(x_j)=\alpha^\top\tilde K\alpha=\mu\,\alpha^\top\alpha$, so a unit eigenvector $u$ of $\tilde K$ must be rescaled:

$$
\alpha=\frac{u}{\sqrt{\mu}}\quad\Longrightarrow\quad \lVert v\rVert=1
$$

- $u$: unit-length eigenvector of $\tilde K$ (what numpy's eigh returns)

**Step 4 — project.** The score of any point is its inner product with the axis, which again needs only kernel values. For the training points it simplifies to $z=\tilde K\alpha=\mu\alpha=\sqrt\mu\,u$.

$$
z(x)=v^\top\big(\phi(x)-\bar\phi\big)=\sum_{i=1}^{n}\alpha_i\,\tilde k(x_i,x)
$$

- $z(x)$: score of a (possibly new) point x on this component
- $\tilde k(x_i,x)$: the centred kernel value between training point i and x (below)

**Centring without coordinates.** We cannot subtract $\bar\phi=\frac1n\sum_j\phi(x_j)$ directly, but expanding $(\phi(x_i)-\bar\phi)^\top(\phi(x_j)-\bar\phi)$ gives four kernel terms. In matrix form, with $\mathbf 1_n$ the $n\times n$ matrix whose entries are all $1/n$:

$$
\tilde K = K-\mathbf 1_nK-K\mathbf 1_n+\mathbf 1_nK\mathbf 1_n = HKH,\qquad H=I-\mathbf 1_n
$$

- $H$: the centring matrix; it subtracts row and column means
- $\mathbf 1_n$: $n\times n$ matrix with every entry $1/n$

$$
\tilde k(x_i,x)=k(x_i,x)-\frac1n\sum_{j}k(x_j,x)-\frac1n\sum_{j}K_{ij}+\frac1{n^2}\sum_{j,l}K_{jl}
$$

- $\tfrac1n\sum_j K_{ij}$: mean of row i of the training kernel matrix (store it at fit time)

Because each $\alpha^{(j)}$ is an eigenvector of $\tilde K$ with $\mu>0$, its entries sum to zero; the two terms that do not depend on $i$ therefore cancel in $z(x)$, leaving the shortcut $z(x)=\sum_i\alpha_i\big(k(x_i,x)-\bar K_{i\cdot}\big)$, where $\bar K_{i\cdot}$ is the training row mean.

> **Linear kernel = ordinary PCA:** With $k(x,x')=x^\top x'$, $\tilde K=\tilde X\tilde X^\top$ is the Gram matrix of the centred data. Its nonzero eigenvalues equal those of $\tilde X^\top\tilde X=nC$, and its scores are exactly the PCA scores — this is the "dual" or Gram form of PCA, the same identity behind the SVD article. Kernel PCA is PCA with the dot product swapped out.

**What γ does (RBF).** For small $\gamma$, $e^{-\gamma\lVert x-x'\rVert^2}\approx 1-\gamma\lVert x-x'\rVert^2$; after centring the constant drops out and $\tilde K\approx 2\gamma\,\tilde X\tilde X^\top$ — kernel PCA collapses back towards linear PCA. For large $\gamma$, every off-diagonal entry tends to 0, so $K\to I$ and $\tilde K\to H$, whose eigenvalues are $1$ (with multiplicity $n-1$) and $0$: every direction carries the same variance, each point is its own "component", and nothing has been learned. Useful values sit between: $1/\gamma$ should be comparable to the squared distance between points that you want to count as similar.

**What the eigenvalues tell you.** $\mu_j/\sum_l\mu_l$ is the share of feature-space variance on component $j$ — but that variance is measured in a space the kernel defined. Changing $\gamma$ changes the space, so explained-variance shares cannot be compared across kernels or $\gamma$ values, and a high share does not mean the component is *useful*.

## Worked example: two rings, eight points

Four points on a ring of radius 1 (A1–A4) and four on a ring of radius 3 (B1–B4), all on the axes. We use an RBF kernel with $\gamma=0.5$, so $k(x,x')=e^{-\lVert x-x'\rVert^2/2}$.

The data, and the first kernel-PCA score each point receives (computed below). Linear PCA cannot separate the rings: the covariance is 2.5·I, so every direction has variance 2.5.

| Point | x₁ | x₂ | radius | ring | PC1 score z₁ |
| --- | --- | --- | --- | --- | --- |
| A1 | 1 | 0 | 1 | inner | −0.4010 |
| A2 | 0 | 1 | 1 | inner | −0.4010 |
| A3 | −1 | 0 | 1 | inner | −0.4010 |
| A4 | 0 | −1 | 1 | inner | −0.4010 |
| B1 | 3 | 0 | 3 | outer | +0.4010 |
| B2 | 0 | 3 | 3 | outer | +0.4010 |
| B3 | −3 | 0 | 3 | outer | +0.4010 |
| B4 | 0 | −3 | 3 | outer | +0.4010 |

**Building and centring the kernel matrix**

1. **Squared distances.** Inner–inner: 2 (neighbours) or 4 (opposite). Outer–outer: 18 or 36. Inner–outer: 4 (same ray, e.g. A1–B1), 10 (a right angle apart) or 16 (opposite).
2. **Kernel values** $e^{-d^2/2}$: $e^{-1}=0.3679$, $e^{-2}=0.1353$, $e^{-5}=0.0067$, $e^{-8}=0.0003$, $e^{-9}=0.0001$, $e^{-18}\approx 0$. The matrix is shown in the table below.
3. **Row sums.** An inner row: within its ring $S_{AA}=1+2(0.3679)+0.1353=1.8711$, towards the outer ring $S_{AB}=0.1353+2(0.0067)+0.0003=0.1491$. An outer row: $S_{BB}=1+2(0.0001)+0\approx1.0002$ and $S_{BA}=0.1491$.
4. **Means for centring.** Row (= column) means: $(1.8711+0.1491)/8=0.2525$ for inner points and $(0.1491+1.0002)/8=0.1437$ for outer points; grand mean $0.1981$.
5. **Centre.** $\tilde K_{ij}=K_{ij}-\bar K_{i\cdot}-\bar K_{\cdot j}+\bar K$. For example $\tilde K_{A1,A1}=1-2(0.2525)+0.1981\approx0.6930$ and $\tilde K_{B1,B1}=1-2(0.1437)+0.1981\approx0.9108$ (unrounded means).

The RBF kernel matrix $K$ ($\gamma=0.5$). Each inner point is strongly similar to its ring neighbours (0.3679) and barely to the outer ring; outer points are almost orthogonal to everything — their ring is too spread out for this $\gamma$.

| k(·,·) | A1 | A2 | A3 | A4 | B1 | B2 | B3 | B4 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| **A1** | 1.0000 | 0.3679 | 0.1353 | 0.3679 | 0.1353 | 0.0067 | 0.0003 | 0.0067 |
| **A2** | 0.3679 | 1.0000 | 0.3679 | 0.1353 | 0.0067 | 0.1353 | 0.0067 | 0.0003 |
| **A3** | 0.1353 | 0.3679 | 1.0000 | 0.3679 | 0.0003 | 0.0067 | 0.1353 | 0.0067 |
| **A4** | 0.3679 | 0.1353 | 0.3679 | 1.0000 | 0.0067 | 0.0003 | 0.0067 | 0.1353 |
| **B1** | 0.1353 | 0.0067 | 0.0003 | 0.0067 | 1.0000 | 0.0001 | 0.0000 | 0.0001 |
| **B2** | 0.0067 | 0.1353 | 0.0067 | 0.0003 | 0.0001 | 1.0000 | 0.0001 | 0.0000 |
| **B3** | 0.0003 | 0.0067 | 0.1353 | 0.0067 | 0.0000 | 0.0001 | 1.0000 | 0.0001 |
| **B4** | 0.0067 | 0.0003 | 0.0067 | 0.1353 | 0.0001 | 0.0000 | 0.0001 | 1.0000 |

**Eigen-decomposition, scores and a new point**

1. **Guess the eigenvector by symmetry.** Rotating the picture by 90° permutes the points within each ring, so try $u=\tfrac{1}{\sqrt8}(-1,-1,-1,-1,+1,+1,+1,+1)$ — "inner versus outer". Its entries sum to 0, so $\tilde Ku=HKHu=HKu$.
2. **Check it.** $(Ku)_{\text{inner}}=\tfrac{1}{\sqrt8}(-S_{AA}+S_{AB})=\tfrac{-1.7220}{\sqrt8}$ and $(Ku)_{\text{outer}}=\tfrac{1}{\sqrt8}(-S_{BA}+S_{BB})=\tfrac{0.8511}{\sqrt8}$. $H$ subtracts their mean $-0.4355/\sqrt8$, leaving $\mp1.2865/\sqrt8$ — exactly $1.2865\,u$. So $u$ is an eigenvector with $\mu_1=\tfrac{S_{AA}+S_{BB}}{2}-S_{AB}=1.2865$.
3. **Is it the top one?** A full decomposition (numpy) gives the spectrum $1.2865,\ 1.0833,\ 1.0833,\ 1.0237,\ 0.7813,\ 0.7813,\ 0.3757,\ 0$. Yes: the radial component is first, with $1.2865/6.4152=20.1\%$ of the feature-space variance. The tied pair $1.0833$ are "angular" modes.
4. **Rescale.** $\alpha=u/\sqrt{\mu_1}=\pm\tfrac{1}{\sqrt{8\times1.2865}}=\pm0.3117$ (negative for A's, positive for B's).
5. **Training scores.** $z=\mu_1\alpha=\pm\sqrt{1.2865/8}=\pm0.4010$: the whole inner ring collapses to $-0.4010$ and the outer ring to $+0.4010$. One number now separates the rings perfectly — something no linear projection can do.
6. **New point** $x^*=(0,\,2.5)$. Squared distances to A1–A4: $7.25,\ 2.25,\ 7.25,\ 12.25$; to B1–B4: $15.25,\ 0.25,\ 15.25,\ 30.25$. Kernel values: $0.0266,\ 0.3247,\ 0.0266,\ 0.0022$ (sum 0.3801) and $0.0005,\ 0.8825,\ 0.0005,\ 0.0000$ (sum 0.8835).
7. **Project** with the shortcut $z^*=\sum_i\alpha_i\,(k(x_i,x^*)-\bar K_{i\cdot})$: $\;0.3117\big[(0.8835-4\times0.1437)-(0.3801-4\times0.2525)\big]=0.3117\times0.9386=0.2926$.
8. **Read it.** $z^*=0.2926$ is much closer to the outer ring's $+0.4010$ than to $-0.4010$, so in this 1-D representation the new point belongs with the outer ring. (The origin, by contrast, projects to $-0.6067$: beyond the inner ring.)

## Explore it

> Figure: Kernel PCA live. Left: input space shaded by the score z₁(x) (warm = positive, blue = negative) — with the RBF kernel at γ = 0.5 on the worked example the shading is a radial bowl, the eight points score ±0.4010, and the star at (0, 2.5) scores 0.2926. Right: every point at its (z₁, z₂) scores; they glide when you change the kernel or γ. Bottom: the centred kernel matrix K̃ that is actually decomposed, and its eigenvalue shares (μ₁ = 20.1%). Switch to the linear kernel and the two eigenvalues tie at 20 with no separating direction; push γ to 20 and all seven nonzero eigenvalues become 1. On the 48-point two rings, γ = 0.5 lets one threshold on z₁ separate the rings 100% versus 66.7% for linear PCA.

Things to try: drag the star across the gap between the rings and watch $z_1$ change sign; on the rings, sweep $\gamma$ from 0.02 to 20 and watch the separation appear then degrade as each point becomes its own island; on the moons, notice that the top component is *not* the one that separates the classes until $\gamma$ is large — kernel PCA maximises variance, not class separation.

## The algorithm, step by step

**fit(X) and transform(x)**

1. **Kernel matrix.** $K_{ij}=k(x_i,x_j)$ for all pairs — $O(n^2d)$ time, $O(n^2)$ memory. Store the row means $\bar K_{i\cdot}$ and the grand mean $\bar K$.
2. **Centre.** $\tilde K=K-\mathbf 1_nK-K\mathbf 1_n+\mathbf 1_nK\mathbf 1_n$.
3. **Eigen-decompose.** Top $m$ eigenpairs $(\mu_j,u_j)$ of $\tilde K$: $O(n^3)$ dense, or roughly $O(n^2m)$ with Lanczos/ARPACK or randomized SVD. Discard components with $\mu_j\approx0$.
4. **Normalise.** $\alpha^{(j)}=u_j/\sqrt{\mu_j}$ so each feature-space axis has unit length.
5. **Training scores.** $Z=\tilde K\,[\alpha^{(1)}\cdots\alpha^{(m)}]$, i.e. $z_{ij}=\sqrt{\mu_j}\,u_{ij}$.
6. **New point.** Compute $k(x_i,x)$ against all $n$ training points, centre with the stored means, and take $z_j(x)=\sum_i\alpha^{(j)}_i\tilde k(x_i,x)$ — $O(nd+nm)$.

The cost profile is the opposite of linear PCA's: linear PCA scales with the dimension ($O(nd^2+d^3)$) and stores a $d\times m$ matrix; kernel PCA scales with the number of samples and stores the whole training set. Beyond a few tens of thousands of points the dense kernel matrix no longer fits in memory.

## Kernels, hyperparameters and scaling up

Common kernels and what the components they produce look like.

| Kernel | k(x, x′) | Feature space | Typical use |
| --- | --- | --- | --- |
| Linear | x·x′ | the input space itself | sanity check — reproduces PCA |
| Polynomial | (γ x·x′ + c₀)ᵖ | all monomials up to degree p | interactions, curved but global structure |
| RBF / Gaussian | exp(−γ‖x − x′‖²) | infinite-dimensional | default for manifolds, clusters, rings |
| Sigmoid | tanh(γ x·x′ + c₀) | not always PSD | rarely; can give negative eigenvalues |
| Precomputed | any PSD similarity | whatever it implies | strings, graphs, sets (string / graph kernels) |

**Tuning in practice**

- **Standardise first:** RBF and polynomial kernels depend on raw distances / dot products, so a feature measured in thousands dominates. Scale features before fitting.
- **γ (RBF):** start near 1 / (median squared pairwise distance); scikit-learn's default for KernelPCA is `1 / n_features`. Too small → linear PCA; too large → every eigenvalue ≈ 1.
- **Choosing γ and m:** there is no likelihood to maximise. Either cross-validate a downstream model (kernel PCA inside a Pipeline with GridSearchCV), or minimise reconstruction error via `fit_inverse_transform=True`.
- **n_components:** look at the eigenvalue spectrum for an elbow, but remember the shares are only comparable within one kernel setting.
- **Large n:** Nyström approximation (`sklearn.kernel_approximation.Nystroem`) or random Fourier features (`RBFSampler`) build an explicit low-dimensional φ; then run ordinary PCA on it in O(n·r²).
- **eigen_solver:** `dense` for small n, `arpack` when m ≪ n, `randomized` for large n and few components.

> **A family resemblance:** Classical MDS, Isomap, Laplacian eigenmaps and spectral embedding can all be written as kernel PCA with a particular, data-dependent "kernel" (a double-centred distance or graph matrix). Spectral clustering is close kin too: it takes the leading eigenvectors of a normalised similarity matrix and clusters in that embedding.

## In code

```python
import numpy as np

def rbf(A, B, gamma):
    d2 = ((A[:, None, :] - B[None, :, :]) ** 2).sum(-1)
    return np.exp(-gamma * d2)

class KernelPCA:
    def __init__(self, n_components=2, gamma=1.0):
        self.k, self.gamma = n_components, gamma

    def fit(self, X):
        self.X = X
        K = rbf(X, X, self.gamma)
        self.col_mean, self.all_mean = K.mean(0), K.mean()
        n = len(X)
        H = np.eye(n) - np.ones((n, n)) / n
        Kc = H @ K @ H                                  # centre in feature space
        mu, V = np.linalg.eigh(Kc)                      # ascending order
        mu, V = mu[::-1][: self.k], V[:, ::-1][:, : self.k]
        V *= np.sign((V * np.arange(1, n + 1)[:, None]).sum(0))   # fix the sign
        self.mu = mu
        self.alpha = V / np.sqrt(mu)                    # unit-length feature-space axes
        self.Z = Kc @ self.alpha                        # = V * sqrt(mu)
        return self

    def transform(self, Xnew):
        k = rbf(Xnew, self.X, self.gamma)
        kc = k - k.mean(1, keepdims=True) - self.col_mean + self.all_mean
        return kc @ self.alpha

X = np.array([[1, 0], [0, 1], [-1, 0], [0, -1],      # inner ring A1..A4
              [3, 0], [0, 3], [-3, 0], [0, -3]], float)  # outer ring B1..B4
kp = KernelPCA(n_components=2, gamma=0.5).fit(X)
print("eigenvalues:", kp.mu.round(4))
print("PC1 scores: ", kp.Z[:, 0].round(4))
print("new (0,2.5):", kp.transform(np.array([[0, 2.5]]))[:, 0].round(4))
```

From scratch with NumPy: RBF kernel, double centring, eigen-decomposition, out-of-sample projection. Prints eigenvalues [1.2865 1.0833], scores ±0.401 and 0.2926 for (0, 2.5).

```python
import numpy as np
from sklearn.decomposition import KernelPCA, PCA

X = np.array([[1, 0], [0, 1], [-1, 0], [0, -1],
              [3, 0], [0, 3], [-3, 0], [0, -3]], float)
ring = np.array([0, 0, 0, 0, 1, 1, 1, 1])

pca = PCA(n_components=2).fit(X)
print("linear PCA variances:", pca.explained_variance_.round(4))   # equal: no preferred direction

kpca = KernelPCA(n_components=2, kernel="rbf", gamma=0.5).fit(X)
print("kernel PCA eigenvalues:", kpca.eigenvalues_.round(4))
z = kpca.transform(X)[:, 0]
z = z if z[ring == 1].mean() > 0 else -z        # eigenvector sign is arbitrary
print("PC1 scores:", z.round(4))
new = kpca.transform([[0, 2.5]])[0, 0] * np.sign(kpca.transform(X)[4, 0])
print("new point (0, 2.5) -> PC1 =", round(new, 4))
```

scikit-learn on the worked example. Output: linear PCA variances [2.8571 2.8571] (ties — it uses n−1, so 20/7); kernel PCA eigenvalues [1.2865 1.0833]; PC1 scores ±0.401; new point → 0.2926.

> **Signs and ties:** An eigenvector is only defined up to sign, so libraries may return −z instead of z; fix a convention before comparing runs. When eigenvalues tie (the two 1.0833 components above), any rotation within that 2-D eigenspace is equally valid, so those components can differ completely between implementations.

**Strengths**

- The structure is nonlinear but smooth — rings, curved manifolds, clusters — and linear PCA visibly fails.
- You want nonlinear features for a downstream linear model, or a kernel method that also handles new points.
- Your data are not vectors at all (strings, graphs) but you have a valid similarity kernel.
- n is in the hundreds to low tens of thousands.

**Limitations**

- n is large: the n×n matrix costs O(n²) memory and O(n³) time (use Nyström / random features instead).
- You need interpretable components: there are no loadings on the original features, only weights on training points.
- You need to reconstruct inputs exactly: the pre-image problem only has approximate solutions.
- You only want a 2-D picture of clusters: t-SNE or UMAP usually show local structure better.
- You have labels and want separation: use a supervised method — maximum variance is not maximum separability.

- **Image denoising:** Project noisy images onto the leading kernel components and map back through an approximate pre-image (Mika et al., 1999); noise lives in the discarded low-variance directions.
- **Novelty detection:** The reconstruction error in feature space is large for points unlike the training data, giving a one-class anomaly score (Hoffmann, 2007).
- **Nonlinear feature extraction:** Kernel PCA features fed to a linear classifier or regressor, e.g. face recognition with "kernel eigenfaces" and process monitoring in chemical plants.
- **Structured data:** With string or graph kernels, embed protein sequences or molecules into a few coordinates for visualisation and clustering.

> **Forgetting to centre:** Decomposing the raw $K$ instead of $\tilde K$ is PCA without subtracting the mean: with an RBF kernel all entries are positive, so the top eigenvector of $K$ is roughly the constant direction — it just measures how dense each region is. New points must be centred with the *training* means, not their own.

> **Variance is not separation:** Kernel PCA is unsupervised. On the two-moons data the top component tracks the long axis of the whole cloud and only separates the moons when γ is large enough; on other data the useful direction can be the 5th component. Check several components, and if you have labels, tune γ by downstream validation.

> **γ is in squared input units:** Rescaling the inputs by a factor $s$ is equivalent to multiplying $\gamma$ by $s^2$. A $\gamma$ tuned on unstandardised data is meaningless after standardisation, and vice versa.

**Question:** On the eight-point rings example, linear PCA's two variances are both 2.5. What does that imply?

- A. The data are one-dimensional
- B. Every direction has the same variance, so no 1-D linear projection is preferred or separates the rings
- C. PCA has failed to converge
- D. The first principal component is the radius

**Answer:** B. Every direction has the same variance, so no 1-D linear projection is preferred or separates the rings — A covariance of 2.5·I is isotropic: every unit direction w has variance wᵀCw = 2.5. Linear projections cannot express the radius, which is the nonlinear quantity that separates the rings.

**Question:** An eigenvector $u$ of $\tilde K$ (unit length) has eigenvalue $\mu=4$. What is the norm of the dual coefficient vector $\alpha$ that makes the feature-space axis unit length?

- A. 4
- B. 2
- C. 0.5
- D. 0.25

**Answer:** C. 0.5 — $\lVert v\rVert^2=\alpha^\top\tilde K\alpha=\mu\lVert\alpha\rVert^2=1$, so $\lVert\alpha\rVert=1/\sqrt\mu=1/2$.

**Question:** With an RBF kernel and $\gamma\to\infty$ on $n=8$ distinct points, the centred kernel matrix's eigenvalues approach…

- A. one large value and seven zeros
- B. seven 1s and one 0
- C. all equal to 8
- D. the linear-PCA variances

**Answer:** B. seven 1s and one 0 — Off-diagonal kernel values vanish, so $K\to I$ and $\tilde K\to H=I-\tfrac18\mathbf{1}\mathbf{1}^\top$, which has eigenvalue 1 seven times and 0 once (the constant vector). On the worked example at $\gamma=20$ numpy already gives seven eigenvalues of 1.0000 — every direction is equally important, so nothing is learned.

**Question:** You want kernel PCA on 100,000 points. Roughly how much memory does the dense float64 kernel matrix need?

- A. 800 MB
- B. 8 GB
- C. 80 GB
- D. 800 GB

**Answer:** C. 80 GB — $n^2=10^{10}$ entries × 8 bytes $=8\times10^{10}$ bytes = 80 GB — hence Nyström or random-feature approximations for large $n$.

**Key points**

- Kernel PCA is PCA in a feature space $\phi(x)$ reached only through a kernel $k(x,x')=\phi(x)^\top\phi(x')$.
- Axes lie in the span of the data, $v=\sum_i\alpha_i\phi(x_i)$, so the eigen-problem becomes $\tilde K\alpha=\mu\alpha$ on the $n\times n$ centred kernel matrix; normalise with $\alpha=u/\sqrt\mu$.
- Centre implicitly with $\tilde K=HKH$; project new points with their centred kernel row: $z(x)=\sum_i\alpha_i\tilde k(x_i,x)$.
- A linear kernel reproduces PCA; RBF $\gamma$ interpolates between linear PCA (small $\gamma$) and a useless identity-like spectrum (large $\gamma$).
- Costs $O(n^2)$ memory and $O(n^3)$ time, keeps the training set, has no feature loadings and only approximate pre-images.
- It maximises feature-space variance, not class separation — inspect several components and tune the kernel by downstream validation.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
