Definition
Kernel PCA is principal component analysis carried out in a high- (often infinite-) dimensional feature space that is never built explicitly. Everything is computed from the kernel matrix : centre it, take its top eigenvectors, and each eigenvector becomes a nonlinear projection of the input.
At a glance
- Task
- Unsupervised nonlinear dimensionality reduction / feature extraction
- Input → output
- points → scores per point, plus a map for new points
- Learns
- dual coefficients and eigenvalues of the centred kernel matrix; it must also keep the training set
- Training cost
- to build , for a full eigen-decomposition ( with iterative solvers); memory
- Prediction cost
- per new point — one kernel evaluation against every training point
- Hyperparameters
- kernel (RBF, polynomial, sigmoid…), its parameters (, degree, ), number of components
- Assumes
- the structure you care about is linear in the chosen feature space — i.e. the kernel encodes the right notion of similarity; features on comparable scales
The problem
Ordinary PCA (see the PCA article) looks for straight directions of maximum variance. That is exactly right when the data lie near a flat subspace and exactly wrong when they lie on a curve. Take two concentric rings: an inner ring of radius 1 and an outer ring of radius 3. The one number that describes the structure — the radius — is a nonlinear function of . Worse, because both rings are symmetric, their covariance matrix is a multiple of the identity: every direction has the same variance, PCA has no preferred axis, and any 1-D projection mixes the rings.
The classical fix is to add features by hand — , , — and run PCA on those. For two inputs and degree 2 that is 6 features, but a degree-3 polynomial map of a 784-pixel image has million, and the Gaussian (RBF) kernel corresponds to infinitely many. Kernel PCA gets the result of PCA in that space without ever writing a feature vector down.
The idea
Two observations make it work. First, the principal axes in feature space always lie in the span of the mapped training points, so each axis can be written as — numbers instead of a vector of unknown length. Second, everything PCA needs (variances, projections, normalisation) then reduces to inner products , which the kernel function computes directly from the raw inputs. That is the kernel trick applied to PCA instead of to a classifier (compare the SVM article).
The payoff is that a "projection onto an axis" becomes a curved function on the input space. With an RBF kernel the first component of the rings data is, in effect, a radius detector: its level sets are circles, so all inner-ring points get one score and all outer-ring points another.
Vocabulary
- Feature map φ
- a (possibly infinite-dimensional) embedding ; never computed explicitly
- Kernel k
- a function with — e.g. RBF , polynomial , linear
- Gram / kernel matrix K
- the table of all pairwise kernel values ; symmetric and positive semidefinite
- Centring
- subtracting the feature-space mean from every ; done implicitly as
- Dual coefficients α
- the weights expressing a principal axis as ; one vector per component
- Eigenvalue μ
- eigenvalue of ; the variance captured along that axis is
- Score z
- the coordinate of a point along a kernel principal axis,
- Out-of-sample projection
- scoring a point that was not in the training set, using its kernel values against the training points
- Pre-image
- a point in input space whose is close to a given feature-space vector — needed for denoising/reconstruction, and generally does not exist exactly
The math
Assume for now that the mapped points are centred, . PCA in feature space diagonalises the covariance operator
- the i-th training point mapped into feature space
- feature-space covariance (an operator if the space is infinite-dimensional)
- a principal axis and the variance along it
- number of training points
Step 1 — the axis lives in the span of the data. Expand : . Each bracket is a scalar, so for , — a linear combination of the mapped points. Name the coefficients :
- dual coefficient of training point i for this axis
Step 2 — turn the eigen-problem into one about K. Substitute into and take the inner product of both sides with for every . Every becomes a kernel value, and stacking the equations gives . Any solution of the smaller problem below solves this one (the remaining solutions differ only by null-space directions of , which do not change ):
- the centred n×n kernel matrix
- eigenvalue of ; the feature-space variance is
Step 3 — normalise. PCA axes have unit length. , so a unit eigenvector of must be rescaled:
- unit-length eigenvector of (what numpy's eigh returns)
Step 4 — project. The score of any point is its inner product with the axis, which again needs only kernel values. For the training points it simplifies to .
- score of a (possibly new) point x on this component
- the centred kernel value between training point i and x (below)
Centring without coordinates. We cannot subtract directly, but expanding gives four kernel terms. In matrix form, with the matrix whose entries are all :
- the centring matrix; it subtracts row and column means
- matrix with every entry
- mean of row i of the training kernel matrix (store it at fit time)
Because each is an eigenvector of with , its entries sum to zero; the two terms that do not depend on therefore cancel in , leaving the shortcut , where is the training row mean.
What γ does (RBF). For small , ; after centring the constant drops out and — kernel PCA collapses back towards linear PCA. For large , every off-diagonal entry tends to 0, so and , whose eigenvalues are (with multiplicity ) and : every direction carries the same variance, each point is its own "component", and nothing has been learned. Useful values sit between: should be comparable to the squared distance between points that you want to count as similar.
What the eigenvalues tell you. is the share of feature-space variance on component — but that variance is measured in a space the kernel defined. Changing changes the space, so explained-variance shares cannot be compared across kernels or values, and a high share does not mean the component is useful.
Worked example: two rings, eight points
Four points on a ring of radius 1 (A1–A4) and four on a ring of radius 3 (B1–B4), all on the axes. We use an RBF kernel with , so .
| Point | x₁ | x₂ | radius | ring | PC1 score z₁ |
|---|---|---|---|---|---|
| A1 | 1 | 0 | 1 | inner | −0.4010 |
| A2 | 0 | 1 | 1 | inner | −0.4010 |
| A3 | −1 | 0 | 1 | inner | −0.4010 |
| A4 | 0 | −1 | 1 | inner | −0.4010 |
| B1 | 3 | 0 | 3 | outer | +0.4010 |
| B2 | 0 | 3 | 3 | outer | +0.4010 |
| B3 | −3 | 0 | 3 | outer | +0.4010 |
| B4 | 0 | −3 | 3 | outer | +0.4010 |
Building and centring the kernel matrix
- 1Squared distances. Inner–inner: 2 (neighbours) or 4 (opposite). Outer–outer: 18 or 36. Inner–outer: 4 (same ray, e.g. A1–B1), 10 (a right angle apart) or 16 (opposite).
- 2Kernel values : , , , , , . The matrix is shown in the table below.
- 3Row sums. An inner row: within its ring , towards the outer ring . An outer row: and .
- 4Means for centring. Row (= column) means: for inner points and for outer points; grand mean .
- 5Centre. . For example and (unrounded means).
| k(·,·) | A1 | A2 | A3 | A4 | B1 | B2 | B3 | B4 |
|---|---|---|---|---|---|---|---|---|
| A1 | 1.0000 | 0.3679 | 0.1353 | 0.3679 | 0.1353 | 0.0067 | 0.0003 | 0.0067 |
| A2 | 0.3679 | 1.0000 | 0.3679 | 0.1353 | 0.0067 | 0.1353 | 0.0067 | 0.0003 |
| A3 | 0.1353 | 0.3679 | 1.0000 | 0.3679 | 0.0003 | 0.0067 | 0.1353 | 0.0067 |
| A4 | 0.3679 | 0.1353 | 0.3679 | 1.0000 | 0.0067 | 0.0003 | 0.0067 | 0.1353 |
| B1 | 0.1353 | 0.0067 | 0.0003 | 0.0067 | 1.0000 | 0.0001 | 0.0000 | 0.0001 |
| B2 | 0.0067 | 0.1353 | 0.0067 | 0.0003 | 0.0001 | 1.0000 | 0.0001 | 0.0000 |
| B3 | 0.0003 | 0.0067 | 0.1353 | 0.0067 | 0.0000 | 0.0001 | 1.0000 | 0.0001 |
| B4 | 0.0067 | 0.0003 | 0.0067 | 0.1353 | 0.0001 | 0.0000 | 0.0001 | 1.0000 |
Eigen-decomposition, scores and a new point
- 1Guess the eigenvector by symmetry. Rotating the picture by 90° permutes the points within each ring, so try — "inner versus outer". Its entries sum to 0, so .
- 2Check it. and . subtracts their mean , leaving — exactly . So is an eigenvector with .
- 3Is it the top one? A full decomposition (numpy) gives the spectrum . Yes: the radial component is first, with of the feature-space variance. The tied pair are "angular" modes.
- 4Rescale. (negative for A's, positive for B's).
- 5Training scores. : the whole inner ring collapses to and the outer ring to . One number now separates the rings perfectly — something no linear projection can do.
- 6New point . Squared distances to A1–A4: ; to B1–B4: . Kernel values: (sum 0.3801) and (sum 0.8835).
- 7Project with the shortcut : .
- 8Read it. is much closer to the outer ring's than to , so in this 1-D representation the new point belongs with the outer ring. (The origin, by contrast, projects to : beyond the inner ring.)
Explore it
Things to try: drag the star across the gap between the rings and watch change sign; on the rings, sweep from 0.02 to 20 and watch the separation appear then degrade as each point becomes its own island; on the moons, notice that the top component is not the one that separates the classes until is large — kernel PCA maximises variance, not class separation.
The algorithm, step by step
fit(X) and transform(x)
- 1Kernel matrix. for all pairs — time, memory. Store the row means and the grand mean .
- 2Centre. .
- 3Eigen-decompose. Top eigenpairs of : dense, or roughly with Lanczos/ARPACK or randomized SVD. Discard components with .
- 4Normalise. so each feature-space axis has unit length.
- 5Training scores. , i.e. .
- 6New point. Compute against all training points, centre with the stored means, and take — .
The cost profile is the opposite of linear PCA's: linear PCA scales with the dimension () and stores a matrix; kernel PCA scales with the number of samples and stores the whole training set. Beyond a few tens of thousands of points the dense kernel matrix no longer fits in memory.
Kernels, hyperparameters and scaling up
| Kernel | k(x, x′) | Feature space | Typical use |
|---|---|---|---|
| Linear | x·x′ | the input space itself | sanity check — reproduces PCA |
| Polynomial | (γ x·x′ + c₀)ᵖ | all monomials up to degree p | interactions, curved but global structure |
| RBF / Gaussian | exp(−γ‖x − x′‖²) | infinite-dimensional | default for manifolds, clusters, rings |
| Sigmoid | tanh(γ x·x′ + c₀) | not always PSD | rarely; can give negative eigenvalues |
| Precomputed | any PSD similarity | whatever it implies | strings, graphs, sets (string / graph kernels) |
Tuning in practice
- Standardise first
- RBF and polynomial kernels depend on raw distances / dot products, so a feature measured in thousands dominates. Scale features before fitting.
- γ (RBF)
- start near 1 / (median squared pairwise distance); scikit-learn's default for KernelPCA is
1 / n_features. Too small → linear PCA; too large → every eigenvalue ≈ 1. - Choosing γ and m
- there is no likelihood to maximise. Either cross-validate a downstream model (kernel PCA inside a Pipeline with GridSearchCV), or minimise reconstruction error via
fit_inverse_transform=True. - n_components
- look at the eigenvalue spectrum for an elbow, but remember the shares are only comparable within one kernel setting.
- Large n
- Nyström approximation (
sklearn.kernel_approximation.Nystroem) or random Fourier features (RBFSampler) build an explicit low-dimensional φ; then run ordinary PCA on it in O(n·r²). - eigen_solver
densefor small n,arpackwhen m ≪ n,randomizedfor large n and few components.
In code
1import numpy as np23def rbf(A, B, gamma):4 d2 = ((A[:, None, :] - B[None, :, :]) ** 2).sum(-1)5 return np.exp(-gamma * d2)67class KernelPCA:8 def __init__(self, n_components=2, gamma=1.0):9 self.k, self.gamma = n_components, gamma1011 def fit(self, X):12 self.X = X13 K = rbf(X, X, self.gamma)14 self.col_mean, self.all_mean = K.mean(0), K.mean()15 n = len(X)16 H = np.eye(n) - np.ones((n, n)) / n17 Kc = H @ K @ H # centre in feature space18 mu, V = np.linalg.eigh(Kc) # ascending order19 mu, V = mu[::-1][: self.k], V[:, ::-1][:, : self.k]20 V *= np.sign((V * np.arange(1, n + 1)[:, None]).sum(0)) # fix the sign21 self.mu = mu22 self.alpha = V / np.sqrt(mu) # unit-length feature-space axes23 self.Z = Kc @ self.alpha # = V * sqrt(mu)24 return self2526 def transform(self, Xnew):27 k = rbf(Xnew, self.X, self.gamma)28 kc = k - k.mean(1, keepdims=True) - self.col_mean + self.all_mean29 return kc @ self.alpha3031X = np.array([[1, 0], [0, 1], [-1, 0], [0, -1], # inner ring A1..A432 [3, 0], [0, 3], [-3, 0], [0, -3]], float) # outer ring B1..B433kp = KernelPCA(n_components=2, gamma=0.5).fit(X)34print("eigenvalues:", kp.mu.round(4))35print("PC1 scores: ", kp.Z[:, 0].round(4))36print("new (0,2.5):", kp.transform(np.array([[0, 2.5]]))[:, 0].round(4))1import numpy as np2from sklearn.decomposition import KernelPCA, PCA34X = np.array([[1, 0], [0, 1], [-1, 0], [0, -1],5 [3, 0], [0, 3], [-3, 0], [0, -3]], float)6ring = np.array([0, 0, 0, 0, 1, 1, 1, 1])78pca = PCA(n_components=2).fit(X)9print("linear PCA variances:", pca.explained_variance_.round(4)) # equal: no preferred direction1011kpca = KernelPCA(n_components=2, kernel="rbf", gamma=0.5).fit(X)12print("kernel PCA eigenvalues:", kpca.eigenvalues_.round(4))13z = kpca.transform(X)[:, 0]14z = z if z[ring == 1].mean() > 0 else -z # eigenvector sign is arbitrary15print("PC1 scores:", z.round(4))16new = kpca.transform([[0, 2.5]])[0, 0] * np.sign(kpca.transform(X)[4, 0])17print("new point (0, 2.5) -> PC1 =", round(new, 4))Use when
- +The structure is nonlinear but smooth — rings, curved manifolds, clusters — and linear PCA visibly fails.
- +You want nonlinear features for a downstream linear model, or a kernel method that also handles new points.
- +Your data are not vectors at all (strings, graphs) but you have a valid similarity kernel.
- +n is in the hundreds to low tens of thousands.
Avoid when
- −n is large: the n×n matrix costs O(n²) memory and O(n³) time (use Nyström / random features instead).
- −You need interpretable components: there are no loadings on the original features, only weights on training points.
- −You need to reconstruct inputs exactly: the pre-image problem only has approximate solutions.
- −You only want a 2-D picture of clusters: t-SNE or UMAP usually show local structure better.
- −You have labels and want separation: use a supervised method — maximum variance is not maximum separability.
Where it's used
Image denoising
Project noisy images onto the leading kernel components and map back through an approximate pre-image (Mika et al., 1999); noise lives in the discarded low-variance directions.
Novelty detection
The reconstruction error in feature space is large for points unlike the training data, giving a one-class anomaly score (Hoffmann, 2007).
Nonlinear feature extraction
Kernel PCA features fed to a linear classifier or regressor, e.g. face recognition with "kernel eigenfaces" and process monitoring in chemical plants.
Structured data
With string or graph kernels, embed protein sequences or molecules into a few coordinates for visualisation and clustering.
Check yourself
On the eight-point rings example, linear PCA's two variances are both 2.5. What does that imply?
Check yourself
An eigenvector of (unit length) has eigenvalue . What is the norm of the dual coefficient vector that makes the feature-space axis unit length?
Check yourself
With an RBF kernel and on distinct points, the centred kernel matrix's eigenvalues approach…
Check yourself
You want kernel PCA on 100,000 points. Roughly how much memory does the dense float64 kernel matrix need?
Remember
- ✓Kernel PCA is PCA in a feature space reached only through a kernel .
- ✓Axes lie in the span of the data, , so the eigen-problem becomes on the centred kernel matrix; normalise with .
- ✓Centre implicitly with ; project new points with their centred kernel row: .
- ✓A linear kernel reproduces PCA; RBF interpolates between linear PCA (small ) and a useless identity-like spectrum (large ).
- ✓Costs memory and time, keeps the training set, has no feature loadings and only approximate pre-images.
- ✓It maximises feature-space variance, not class separation — inspect several components and tune the kernel by downstream validation.