Kernel PCA

Run PCA in a kernel-induced feature space using only the kernel matrix, capturing nonlinear structure.

17 min read1698 wordsadvanced8 viewsupdated 2026-10-07

Definition

Kernel PCA is principal component analysis carried out in a high- (often infinite-) dimensional feature space ϕ(x)\phi(x) that is never built explicitly. Everything is computed from the n×nn\times n kernel matrix Kij=k(xi,xj)=ϕ(xi)⊤ϕ(xj)K_{ij}=k(x_i,x_j)=\phi(x_i)^\top\phi(x_j): centre it, take its top eigenvectors, and each eigenvector becomes a nonlinear projection z(x)=∑iαi k(xi,x)z(x)=\sum_i \alpha_i\,k(x_i,x) of the input.

At a glance

Task
Unsupervised nonlinear dimensionality reduction / feature extraction
Input → output
nn points xi∈Rdx_i\in\mathbb R^d → mm scores (z1,…,zm)(z_1,\dots,z_m) per point, plus a map for new points
Learns
dual coefficients α(j)∈Rn\alpha^{(j)}\in\mathbb R^n and eigenvalues μj\mu_j of the centred kernel matrix; it must also keep the training set
Training cost
O(n2d)O(n^2 d) to build KK, O(n3)O(n^3) for a full eigen-decomposition (O(n2m)O(n^2 m) with iterative solvers); O(n2)O(n^2) memory
Prediction cost
O(n d+n m)O(n\,d + n\,m) per new point — one kernel evaluation against every training point
Hyperparameters
kernel (RBF, polynomial, sigmoid…), its parameters (γ\gamma, degree, c0c_0), number of components mm
Assumes
the structure you care about is linear in the chosen feature space — i.e. the kernel encodes the right notion of similarity; features on comparable scales

The problem

Ordinary PCA (see the PCA article) looks for straight directions of maximum variance. That is exactly right when the data lie near a flat subspace and exactly wrong when they lie on a curve. Take two concentric rings: an inner ring of radius 1 and an outer ring of radius 3. The one number that describes the structure — the radius — is a nonlinear function of (x1,x2)(x_1,x_2). Worse, because both rings are symmetric, their covariance matrix is a multiple of the identity: every direction has the same variance, PCA has no preferred axis, and any 1-D projection mixes the rings.

The classical fix is to add features by hand — x12x_1^2, x22x_2^2, x1x2x_1x_2 — and run PCA on those. For two inputs and degree 2 that is 6 features, but a degree-3 polynomial map of a 784-pixel image has (7873)≈81\binom{787}{3}\approx 81 million, and the Gaussian (RBF) kernel corresponds to infinitely many. Kernel PCA gets the result of PCA in that space without ever writing a feature vector down.

The idea

Two observations make it work. First, the principal axes in feature space always lie in the span of the mapped training points, so each axis can be written as v=∑iαiϕ(xi)v=\sum_i\alpha_i\phi(x_i) — nn numbers instead of a vector of unknown length. Second, everything PCA needs (variances, projections, normalisation) then reduces to inner products ϕ(xi)⊤ϕ(xj)\phi(x_i)^\top\phi(x_j), which the kernel function computes directly from the raw inputs. That is the kernel trick applied to PCA instead of to a classifier (compare the SVM article).

The payoff is that a "projection onto an axis" becomes a curved function on the input space. With an RBF kernel the first component of the rings data is, in effect, a radius detector: its level sets are circles, so all inner-ring points get one score and all outer-ring points another.

Vocabulary

Feature map φ
a (possibly infinite-dimensional) embedding ϕ:Rd→H\phi:\mathbb R^d\to\mathcal H; never computed explicitly
Kernel k
a function with k(x,x′)=ϕ(x)⊤ϕ(x′)k(x,x')=\phi(x)^\top\phi(x') — e.g. RBF e−γ∥x−x′∥2e^{-\gamma\lVert x-x'\rVert^2}, polynomial (x⊤x′+c0)p(x^\top x'+c_0)^p, linear x⊤x′x^\top x'
Gram / kernel matrix K
the n×nn\times n table of all pairwise kernel values Kij=k(xi,xj)K_{ij}=k(x_i,x_j); symmetric and positive semidefinite
Centring
subtracting the feature-space mean ϕˉ\bar\phi from every ϕ(xi)\phi(x_i); done implicitly as K~=HKH\tilde K=HKH
Dual coefficients α
the weights expressing a principal axis as v=∑iαiϕ(xi)v=\sum_i\alpha_i\phi(x_i); one vector α(j)\alpha^{(j)} per component
Eigenvalue μ
eigenvalue of K~\tilde K; the variance captured along that axis is μ/n\mu/n
Score z
the coordinate of a point along a kernel principal axis, z=v⊤(ϕ(x)−ϕˉ)z=v^\top(\phi(x)-\bar\phi)
Out-of-sample projection
scoring a point that was not in the training set, using its kernel values against the training points
Pre-image
a point xx in input space whose ϕ(x)\phi(x) is close to a given feature-space vector — needed for denoising/reconstruction, and generally does not exist exactly

The math

Assume for now that the mapped points are centred, ∑iϕ(xi)=0\sum_i\phi(x_i)=0. PCA in feature space diagonalises the covariance operator

C=1n∑i=1nϕ(xi) ϕ(xi)⊤,C v=λ vC=\frac1n\sum_{i=1}^{n}\phi(x_i)\,\phi(x_i)^\top,\qquad C\,v=\lambda\,v
ϕ(xi)\phi(x_i)
the i-th training point mapped into feature space
CC
feature-space covariance (an operator if the space is infinite-dimensional)
v, λv,\ \lambda
a principal axis and the variance along it
nn
number of training points

Step 1 — the axis lives in the span of the data. Expand CvCv:   Cv=1n∑iϕ(xi) (ϕ(xi)⊤v)\;Cv=\frac1n\sum_i\phi(x_i)\,\big(\phi(x_i)^\top v\big). Each bracket is a scalar, so for λ>0\lambda>0, v=1nλ∑i(ϕ(xi)⊤v)ϕ(xi)v=\frac{1}{n\lambda}\sum_i\big(\phi(x_i)^\top v\big)\phi(x_i) — a linear combination of the mapped points. Name the coefficients αi\alpha_i:

v=∑i=1nαi ϕ(xi)v=\sum_{i=1}^{n}\alpha_i\,\phi(x_i)
αi\alpha_i
dual coefficient of training point i for this axis

Step 2 — turn the eigen-problem into one about K. Substitute vv into Cv=λvCv=\lambda v and take the inner product of both sides with ϕ(xk)\phi(x_k) for every kk. Every ϕ⊤ϕ\phi^\top\phi becomes a kernel value, and stacking the nn equations gives 1nK2α=λKα\frac1n K^2\alpha=\lambda K\alpha. Any solution of the smaller problem below solves this one (the remaining solutions differ only by null-space directions of KK, which do not change vv):

K~ α=μ α,μ=nλ\tilde K\,\alpha=\mu\,\alpha,\qquad \mu=n\lambda
K~\tilde K
the centred n×n kernel matrix
μ\mu
eigenvalue of K~\tilde K; the feature-space variance is λ=μ/n\lambda=\mu/n

Step 3 — normalise. PCA axes have unit length. ∥v∥2=∑i,jαiαjϕ(xi)⊤ϕ(xj)=α⊤K~α=μ α⊤α\lVert v\rVert^2=\sum_{i,j}\alpha_i\alpha_j\phi(x_i)^\top\phi(x_j)=\alpha^\top\tilde K\alpha=\mu\,\alpha^\top\alpha, so a unit eigenvector uu of K~\tilde K must be rescaled:

α=uμ⟹∥v∥=1\alpha=\frac{u}{\sqrt{\mu}}\quad\Longrightarrow\quad \lVert v\rVert=1
uu
unit-length eigenvector of K~\tilde K (what numpy's eigh returns)

Step 4 — project. The score of any point is its inner product with the axis, which again needs only kernel values. For the training points it simplifies to z=K~α=μα=μ uz=\tilde K\alpha=\mu\alpha=\sqrt\mu\,u.

z(x)=v⊤(ϕ(x)−ϕˉ)=∑i=1nαi k~(xi,x)z(x)=v^\top\big(\phi(x)-\bar\phi\big)=\sum_{i=1}^{n}\alpha_i\,\tilde k(x_i,x)
z(x)z(x)
score of a (possibly new) point x on this component
k~(xi,x)\tilde k(x_i,x)
the centred kernel value between training point i and x (below)

Centring without coordinates. We cannot subtract ϕˉ=1n∑jϕ(xj)\bar\phi=\frac1n\sum_j\phi(x_j) directly, but expanding (ϕ(xi)−ϕˉ)⊤(ϕ(xj)−ϕˉ)(\phi(x_i)-\bar\phi)^\top(\phi(x_j)-\bar\phi) gives four kernel terms. In matrix form, with 1n\mathbf 1_n the n×nn\times n matrix whose entries are all 1/n1/n:

K~=K−1nK−K1n+1nK1n=HKH,H=I−1n\tilde K = K-\mathbf 1_nK-K\mathbf 1_n+\mathbf 1_nK\mathbf 1_n = HKH,\qquad H=I-\mathbf 1_n
HH
the centring matrix; it subtracts row and column means
1n\mathbf 1_n
n×nn\times n matrix with every entry 1/n1/n
k~(xi,x)=k(xi,x)−1n∑jk(xj,x)−1n∑jKij+1n2∑j,lKjl\tilde k(x_i,x)=k(x_i,x)-\frac1n\sum_{j}k(x_j,x)-\frac1n\sum_{j}K_{ij}+\frac1{n^2}\sum_{j,l}K_{jl}
1n∑jKij\tfrac1n\sum_j K_{ij}
mean of row i of the training kernel matrix (store it at fit time)

Because each α(j)\alpha^{(j)} is an eigenvector of K~\tilde K with μ>0\mu>0, its entries sum to zero; the two terms that do not depend on ii therefore cancel in z(x)z(x), leaving the shortcut z(x)=∑iαi(k(xi,x)−Kˉi⋅)z(x)=\sum_i\alpha_i\big(k(x_i,x)-\bar K_{i\cdot}\big), where Kˉi⋅\bar K_{i\cdot} is the training row mean.

What γ does (RBF). For small γ\gamma, e−γ∥x−x′∥2≈1−γ∥x−x′∥2e^{-\gamma\lVert x-x'\rVert^2}\approx 1-\gamma\lVert x-x'\rVert^2; after centring the constant drops out and K~≈2γ X~X~⊤\tilde K\approx 2\gamma\,\tilde X\tilde X^\top — kernel PCA collapses back towards linear PCA. For large γ\gamma, every off-diagonal entry tends to 0, so K→IK\to I and K~→H\tilde K\to H, whose eigenvalues are 11 (with multiplicity n−1n-1) and 00: every direction carries the same variance, each point is its own "component", and nothing has been learned. Useful values sit between: 1/γ1/\gamma should be comparable to the squared distance between points that you want to count as similar.

What the eigenvalues tell you. μj/∑lμl\mu_j/\sum_l\mu_l is the share of feature-space variance on component jj — but that variance is measured in a space the kernel defined. Changing γ\gamma changes the space, so explained-variance shares cannot be compared across kernels or γ\gamma values, and a high share does not mean the component is useful.

Worked example: two rings, eight points

Four points on a ring of radius 1 (A1–A4) and four on a ring of radius 3 (B1–B4), all on the axes. We use an RBF kernel with γ=0.5\gamma=0.5, so k(x,x′)=e−∥x−x′∥2/2k(x,x')=e^{-\lVert x-x'\rVert^2/2}.

Pointx₁x₂radiusringPC1 score z₁
A1101inner−0.4010
A2011inner−0.4010
A3−101inner−0.4010
A40−11inner−0.4010
B1303outer+0.4010
B2033outer+0.4010
B3−303outer+0.4010
B40−33outer+0.4010
The data, and the first kernel-PCA score each point receives (computed below). Linear PCA cannot separate the rings: the covariance is 2.5·I, so every direction has variance 2.5.

Building and centring the kernel matrix

  1. 1Squared distances. Inner–inner: 2 (neighbours) or 4 (opposite). Outer–outer: 18 or 36. Inner–outer: 4 (same ray, e.g. A1–B1), 10 (a right angle apart) or 16 (opposite).
  2. 2Kernel values e−d2/2e^{-d^2/2}: e−1=0.3679e^{-1}=0.3679, e−2=0.1353e^{-2}=0.1353, e−5=0.0067e^{-5}=0.0067, e−8=0.0003e^{-8}=0.0003, e−9=0.0001e^{-9}=0.0001, e−18≈0e^{-18}\approx 0. The matrix is shown in the table below.
  3. 3Row sums. An inner row: within its ring SAA=1+2(0.3679)+0.1353=1.8711S_{AA}=1+2(0.3679)+0.1353=1.8711, towards the outer ring SAB=0.1353+2(0.0067)+0.0003=0.1491S_{AB}=0.1353+2(0.0067)+0.0003=0.1491. An outer row: SBB=1+2(0.0001)+0≈1.0002S_{BB}=1+2(0.0001)+0\approx1.0002 and SBA=0.1491S_{BA}=0.1491.
  4. 4Means for centring. Row (= column) means: (1.8711+0.1491)/8=0.2525(1.8711+0.1491)/8=0.2525 for inner points and (0.1491+1.0002)/8=0.1437(0.1491+1.0002)/8=0.1437 for outer points; grand mean 0.19810.1981.
  5. 5Centre. K~ij=Kij−Kˉi⋅−Kˉ⋅j+Kˉ\tilde K_{ij}=K_{ij}-\bar K_{i\cdot}-\bar K_{\cdot j}+\bar K. For example K~A1,A1=1−2(0.2525)+0.1981≈0.6930\tilde K_{A1,A1}=1-2(0.2525)+0.1981\approx0.6930 and K~B1,B1=1−2(0.1437)+0.1981≈0.9108\tilde K_{B1,B1}=1-2(0.1437)+0.1981\approx0.9108 (unrounded means).
k(·,·)A1A2A3A4B1B2B3B4
A11.00000.36790.13530.36790.13530.00670.00030.0067
A20.36791.00000.36790.13530.00670.13530.00670.0003
A30.13530.36791.00000.36790.00030.00670.13530.0067
A40.36790.13530.36791.00000.00670.00030.00670.1353
B10.13530.00670.00030.00671.00000.00010.00000.0001
B20.00670.13530.00670.00030.00011.00000.00010.0000
B30.00030.00670.13530.00670.00000.00011.00000.0001
B40.00670.00030.00670.13530.00010.00000.00011.0000
The RBF kernel matrix KK (γ=0.5\gamma=0.5). Each inner point is strongly similar to its ring neighbours (0.3679) and barely to the outer ring; outer points are almost orthogonal to everything — their ring is too spread out for this γ\gamma.

Eigen-decomposition, scores and a new point

  1. 1Guess the eigenvector by symmetry. Rotating the picture by 90° permutes the points within each ring, so try u=18(−1,−1,−1,−1,+1,+1,+1,+1)u=\tfrac{1}{\sqrt8}(-1,-1,-1,-1,+1,+1,+1,+1) — "inner versus outer". Its entries sum to 0, so K~u=HKHu=HKu\tilde Ku=HKHu=HKu.
  2. 2Check it. (Ku)inner=18(−SAA+SAB)=−1.72208(Ku)_{\text{inner}}=\tfrac{1}{\sqrt8}(-S_{AA}+S_{AB})=\tfrac{-1.7220}{\sqrt8} and (Ku)outer=18(−SBA+SBB)=0.85118(Ku)_{\text{outer}}=\tfrac{1}{\sqrt8}(-S_{BA}+S_{BB})=\tfrac{0.8511}{\sqrt8}. HH subtracts their mean −0.4355/8-0.4355/\sqrt8, leaving ∓1.2865/8\mp1.2865/\sqrt8 — exactly 1.2865 u1.2865\,u. So uu is an eigenvector with μ1=SAA+SBB2−SAB=1.2865\mu_1=\tfrac{S_{AA}+S_{BB}}{2}-S_{AB}=1.2865.
  3. 3Is it the top one? A full decomposition (numpy) gives the spectrum 1.2865, 1.0833, 1.0833, 1.0237, 0.7813, 0.7813, 0.3757, 01.2865,\ 1.0833,\ 1.0833,\ 1.0237,\ 0.7813,\ 0.7813,\ 0.3757,\ 0. Yes: the radial component is first, with 1.2865/6.4152=20.1%1.2865/6.4152=20.1\% of the feature-space variance. The tied pair 1.08331.0833 are "angular" modes.
  4. 4Rescale. α=u/μ1=±18×1.2865=±0.3117\alpha=u/\sqrt{\mu_1}=\pm\tfrac{1}{\sqrt{8\times1.2865}}=\pm0.3117 (negative for A's, positive for B's).
  5. 5Training scores. z=μ1α=±1.2865/8=±0.4010z=\mu_1\alpha=\pm\sqrt{1.2865/8}=\pm0.4010: the whole inner ring collapses to −0.4010-0.4010 and the outer ring to +0.4010+0.4010. One number now separates the rings perfectly — something no linear projection can do.
  6. 6New point x∗=(0, 2.5)x^*=(0,\,2.5). Squared distances to A1–A4: 7.25, 2.25, 7.25, 12.257.25,\ 2.25,\ 7.25,\ 12.25; to B1–B4: 15.25, 0.25, 15.25, 30.2515.25,\ 0.25,\ 15.25,\ 30.25. Kernel values: 0.0266, 0.3247, 0.0266, 0.00220.0266,\ 0.3247,\ 0.0266,\ 0.0022 (sum 0.3801) and 0.0005, 0.8825, 0.0005, 0.00000.0005,\ 0.8825,\ 0.0005,\ 0.0000 (sum 0.8835).
  7. 7Project with the shortcut z∗=∑iαi (k(xi,x∗)−Kˉi⋅)z^*=\sum_i\alpha_i\,(k(x_i,x^*)-\bar K_{i\cdot}):   0.3117[(0.8835−4×0.1437)−(0.3801−4×0.2525)]=0.3117×0.9386=0.2926\;0.3117\big[(0.8835-4\times0.1437)-(0.3801-4\times0.2525)\big]=0.3117\times0.9386=0.2926.
  8. 8Read it. z∗=0.2926z^*=0.2926 is much closer to the outer ring's +0.4010+0.4010 than to −0.4010-0.4010, so in this 1-D representation the new point belongs with the outer ring. (The origin, by contrast, projects to −0.6067-0.6067: beyond the inner ring.)

Explore it

▸ kernel PCA
kernelshade by
input space · shade = z1(x) · drag ★A1A2A3A4B1B2B3B4kernel PCA scores · z₁ → · z₂ ↑centred kernel matrix K̃ (8×8)A1A2A3A4B1B2B3B4+−eigenvalues of K̃ · share of total variance20.1%μ116.9%μ216.9%μ316.0%μ412.2%μ512.2%μ65.9%μ70.0%μ8
μ₁ = 1.2865, μ₂ = 1.0833★ (0.00, 2.50) → z = (0.2926, -0.5593)best 1-threshold accuracy: z₁ 100.0% · z₂ 62.5%
fig.Kernel PCA live. Left: input space shaded by the score z₁(x) (warm = positive, blue = negative) — with the RBF kernel at γ = 0.5 on the worked example the shading is a radial bowl, the eight points score ±0.4010, and the star at (0, 2.5) scores 0.2926. Right: every point at its (z₁, z₂) scores; they glide when you change the kernel or γ. Bottom: the centred kernel matrix K̃ that is actually decomposed, and its eigenvalue shares (μ₁ = 20.1%). Switch to the linear kernel and the two eigenvalues tie at 20 with no separating direction; push γ to 20 and all seven nonzero eigenvalues become 1. On the 48-point two rings, γ = 0.5 lets one threshold on z₁ separate the rings 100% versus 66.7% for linear PCA.

Things to try: drag the star across the gap between the rings and watch z1z_1 change sign; on the rings, sweep γ\gamma from 0.02 to 20 and watch the separation appear then degrade as each point becomes its own island; on the moons, notice that the top component is not the one that separates the classes until γ\gamma is large — kernel PCA maximises variance, not class separation.

The algorithm, step by step

fit(X) and transform(x)

  1. 1Kernel matrix. Kij=k(xi,xj)K_{ij}=k(x_i,x_j) for all pairs — O(n2d)O(n^2d) time, O(n2)O(n^2) memory. Store the row means Kˉi⋅\bar K_{i\cdot} and the grand mean Kˉ\bar K.
  2. 2Centre. K~=K−1nK−K1n+1nK1n\tilde K=K-\mathbf 1_nK-K\mathbf 1_n+\mathbf 1_nK\mathbf 1_n.
  3. 3Eigen-decompose. Top mm eigenpairs (μj,uj)(\mu_j,u_j) of K~\tilde K: O(n3)O(n^3) dense, or roughly O(n2m)O(n^2m) with Lanczos/ARPACK or randomized SVD. Discard components with μj≈0\mu_j\approx0.
  4. 4Normalise. α(j)=uj/μj\alpha^{(j)}=u_j/\sqrt{\mu_j} so each feature-space axis has unit length.
  5. 5Training scores. Z=K~ [α(1)⋯α(m)]Z=\tilde K\,[\alpha^{(1)}\cdots\alpha^{(m)}], i.e. zij=μj uijz_{ij}=\sqrt{\mu_j}\,u_{ij}.
  6. 6New point. Compute k(xi,x)k(x_i,x) against all nn training points, centre with the stored means, and take zj(x)=∑iαi(j)k~(xi,x)z_j(x)=\sum_i\alpha^{(j)}_i\tilde k(x_i,x) — O(nd+nm)O(nd+nm).

The cost profile is the opposite of linear PCA's: linear PCA scales with the dimension (O(nd2+d3)O(nd^2+d^3)) and stores a d×md\times m matrix; kernel PCA scales with the number of samples and stores the whole training set. Beyond a few tens of thousands of points the dense kernel matrix no longer fits in memory.

Kernels, hyperparameters and scaling up

Kernelk(x, x′)Feature spaceTypical use
Linearx·x′the input space itselfsanity check — reproduces PCA
Polynomial(γ x·x′ + c₀)ᵖall monomials up to degree pinteractions, curved but global structure
RBF / Gaussianexp(−γ‖x − x′‖²)infinite-dimensionaldefault for manifolds, clusters, rings
Sigmoidtanh(γ x·x′ + c₀)not always PSDrarely; can give negative eigenvalues
Precomputedany PSD similaritywhatever it impliesstrings, graphs, sets (string / graph kernels)
Common kernels and what the components they produce look like.

Tuning in practice

Standardise first
RBF and polynomial kernels depend on raw distances / dot products, so a feature measured in thousands dominates. Scale features before fitting.
γ (RBF)
start near 1 / (median squared pairwise distance); scikit-learn's default for KernelPCA is 1 / n_features. Too small → linear PCA; too large → every eigenvalue ≈ 1.
Choosing γ and m
there is no likelihood to maximise. Either cross-validate a downstream model (kernel PCA inside a Pipeline with GridSearchCV), or minimise reconstruction error via fit_inverse_transform=True.
n_components
look at the eigenvalue spectrum for an elbow, but remember the shares are only comparable within one kernel setting.
Large n
Nyström approximation (sklearn.kernel_approximation.Nystroem) or random Fourier features (RBFSampler) build an explicit low-dimensional φ; then run ordinary PCA on it in O(n·r²).
eigen_solver
dense for small n, arpack when m ≪ n, randomized for large n and few components.

In code

python
1import numpy as np
2
3def rbf(A, B, gamma):
4 d2 = ((A[:, None, :] - B[None, :, :]) ** 2).sum(-1)
5 return np.exp(-gamma * d2)
6
7class KernelPCA:
8 def __init__(self, n_components=2, gamma=1.0):
9 self.k, self.gamma = n_components, gamma
10
11 def fit(self, X):
12 self.X = X
13 K = rbf(X, X, self.gamma)
14 self.col_mean, self.all_mean = K.mean(0), K.mean()
15 n = len(X)
16 H = np.eye(n) - np.ones((n, n)) / n
17 Kc = H @ K @ H # centre in feature space
18 mu, V = np.linalg.eigh(Kc) # ascending order
19 mu, V = mu[::-1][: self.k], V[:, ::-1][:, : self.k]
20 V *= np.sign((V * np.arange(1, n + 1)[:, None]).sum(0)) # fix the sign
21 self.mu = mu
22 self.alpha = V / np.sqrt(mu) # unit-length feature-space axes
23 self.Z = Kc @ self.alpha # = V * sqrt(mu)
24 return self
25
26 def transform(self, Xnew):
27 k = rbf(Xnew, self.X, self.gamma)
28 kc = k - k.mean(1, keepdims=True) - self.col_mean + self.all_mean
29 return kc @ self.alpha
30
31X = np.array([[1, 0], [0, 1], [-1, 0], [0, -1], # inner ring A1..A4
32 [3, 0], [0, 3], [-3, 0], [0, -3]], float) # outer ring B1..B4
33kp = KernelPCA(n_components=2, gamma=0.5).fit(X)
34print("eigenvalues:", kp.mu.round(4))
35print("PC1 scores: ", kp.Z[:, 0].round(4))
36print("new (0,2.5):", kp.transform(np.array([[0, 2.5]]))[:, 0].round(4))
From scratch with NumPy: RBF kernel, double centring, eigen-decomposition, out-of-sample projection. Prints eigenvalues [1.2865 1.0833], scores ±0.401 and 0.2926 for (0, 2.5).
python
1import numpy as np
2from sklearn.decomposition import KernelPCA, PCA
3
4X = np.array([[1, 0], [0, 1], [-1, 0], [0, -1],
5 [3, 0], [0, 3], [-3, 0], [0, -3]], float)
6ring = np.array([0, 0, 0, 0, 1, 1, 1, 1])
7
8pca = PCA(n_components=2).fit(X)
9print("linear PCA variances:", pca.explained_variance_.round(4)) # equal: no preferred direction
10
11kpca = KernelPCA(n_components=2, kernel="rbf", gamma=0.5).fit(X)
12print("kernel PCA eigenvalues:", kpca.eigenvalues_.round(4))
13z = kpca.transform(X)[:, 0]
14z = z if z[ring == 1].mean() > 0 else -z # eigenvector sign is arbitrary
15print("PC1 scores:", z.round(4))
16new = kpca.transform([[0, 2.5]])[0, 0] * np.sign(kpca.transform(X)[4, 0])
17print("new point (0, 2.5) -> PC1 =", round(new, 4))
scikit-learn on the worked example. Output: linear PCA variances [2.8571 2.8571] (ties — it uses n−1, so 20/7); kernel PCA eigenvalues [1.2865 1.0833]; PC1 scores ±0.401; new point → 0.2926.

Use when

  • +The structure is nonlinear but smooth — rings, curved manifolds, clusters — and linear PCA visibly fails.
  • +You want nonlinear features for a downstream linear model, or a kernel method that also handles new points.
  • +Your data are not vectors at all (strings, graphs) but you have a valid similarity kernel.
  • +n is in the hundreds to low tens of thousands.

Avoid when

  • −n is large: the n×n matrix costs O(n²) memory and O(n³) time (use Nyström / random features instead).
  • −You need interpretable components: there are no loadings on the original features, only weights on training points.
  • −You need to reconstruct inputs exactly: the pre-image problem only has approximate solutions.
  • −You only want a 2-D picture of clusters: t-SNE or UMAP usually show local structure better.
  • −You have labels and want separation: use a supervised method — maximum variance is not maximum separability.

Where it's used

Image denoising

Project noisy images onto the leading kernel components and map back through an approximate pre-image (Mika et al., 1999); noise lives in the discarded low-variance directions.

Novelty detection

The reconstruction error in feature space is large for points unlike the training data, giving a one-class anomaly score (Hoffmann, 2007).

Nonlinear feature extraction

Kernel PCA features fed to a linear classifier or regressor, e.g. face recognition with "kernel eigenfaces" and process monitoring in chemical plants.

Structured data

With string or graph kernels, embed protein sequences or molecules into a few coordinates for visualisation and clustering.

Check yourself

On the eight-point rings example, linear PCA's two variances are both 2.5. What does that imply?

Check yourself

An eigenvector uu of K~\tilde K (unit length) has eigenvalue μ=4\mu=4. What is the norm of the dual coefficient vector α\alpha that makes the feature-space axis unit length?

Check yourself

With an RBF kernel and γ→∞\gamma\to\infty on n=8n=8 distinct points, the centred kernel matrix's eigenvalues approach…

Check yourself

You want kernel PCA on 100,000 points. Roughly how much memory does the dense float64 kernel matrix need?

Remember

  • ✓Kernel PCA is PCA in a feature space ϕ(x)\phi(x) reached only through a kernel k(x,x′)=ϕ(x)⊤ϕ(x′)k(x,x')=\phi(x)^\top\phi(x').
  • ✓Axes lie in the span of the data, v=∑iαiϕ(xi)v=\sum_i\alpha_i\phi(x_i), so the eigen-problem becomes K~α=μα\tilde K\alpha=\mu\alpha on the n×nn\times n centred kernel matrix; normalise with α=u/μ\alpha=u/\sqrt\mu.
  • ✓Centre implicitly with K~=HKH\tilde K=HKH; project new points with their centred kernel row: z(x)=∑iαik~(xi,x)z(x)=\sum_i\alpha_i\tilde k(x_i,x).
  • ✓A linear kernel reproduces PCA; RBF γ\gamma interpolates between linear PCA (small γ\gamma) and a useless identity-like spectrum (large γ\gamma).
  • ✓Costs O(n2)O(n^2) memory and O(n3)O(n^3) time, keeps the training set, has no feature loadings and only approximate pre-images.
  • ✓It maximises feature-space variance, not class separation — inspect several components and tune the kernel by downstream validation.
← back to ML Algorithms roadmap