Definition
A radial basis function network is a three-layer feed-forward network whose hidden units respond to the distance between the input and a stored centre, , and whose output is a linear combination of those responses. Broomhead and Lowe introduced it as a neural network in 1988, building on Powell's work on RBF interpolation.
Local bumps instead of global ramps
An MLP hidden unit computes , a ramp that changes along one direction and stays flat along all the others, so it affects the whole input space. An RBF unit is local. It fires near its centre and fades to zero away from it, so each unit is responsible for one region of input space.
The network builds a function by placing bumps where the data lives and scaling each one, like building a surface out of sand piles. That locality makes RBF networks easy to interpret, and it gives them a much simpler training procedure than backprop.
The model
- centre (prototype) of hidden unit k, a point in input space
- width of the Gaussians: how far each unit's influence reaches
- output weights and bias, the only parameters of the linear readout
- number of hidden units (centres)
Other radial functions work too (multiquadric , inverse multiquadric, thin-plate spline ), but the Gaussian is by far the most common. For classification, use one output per class on the same hidden layer.
Once the centres and widths are fixed, is linear in the parameters . Collect the hidden activations of all training points in a design matrix with , and fitting the output layer becomes ordinary (or ridge) least squares:
- design matrix of hidden activations, one row per training example (append a column of ones for the bias)
- ridge penalty; with this is , the Moore–Penrose pseudo-inverse solution
That problem is convex with a closed-form solution, so there are no local minima and no learning rate. The nonlinear part of the problem (where to put the bumps, how wide to make them) is handled separately, usually without looking at at all.
Two-stage training (Moody & Darken, 1989)
- 1Centres. Run k-means on the training inputs with clusters and use the cluster means as . Cheaper alternatives: a random subset of the training points, or a regular grid in low dimensions.
- 2Widths. Set by a heuristic. A common one is , with the largest distance between two centres. Per-unit alternatives use the mean distance to the nearest centres. The aim is bumps that overlap their neighbours. This one errs on the wide side, so pair it with a small .
- 3Output weights. Build (plus a ones column), then solve the least-squares problem above with
lstsqor the normal equations. - 4Optional fine-tuning. Everything is differentiable, so centres, widths and weights can afterwards be trained jointly by gradient descent. Good starting values from stages 1–3 make this much easier.
At the default ( k-means centres, , ) the training MSE is and the test MSE , against a noise floor of , and no weight exceeds in magnitude. Turn on auto σ and the heuristic picks : the test MSE is still , but the weights reach about with opposite signs that nearly cancel. With the largest weight drops to .
Overfitting shows up as K grows. With and , the test error at jumps to ; brings it back to . Too many bumps with nothing to stop them fit the noise.
Exact interpolation and regularization
The original RBF problem from numerical analysis is interpolation: put one centre on every training point (, ) and demand exactly. That gives a square system with . Micchelli (1986) proved that for distinct points the Gaussian matrix is positive definite, so a solution always exists.
Existence is not the same as usefulness. Exact interpolation passes the curve through the noise, and when two inputs are close or is wide, is so ill-conditioned that the weights reach and cancel each other almost perfectly. Poggio and Girosi (1990) replaced it with the regularization network , which trades a little training error for a smooth function.
| λ | weights w = (G + λI)⁻¹y | fitted ŷ at the 3 points | ŷ(0.25) |
|---|---|---|---|
| 0 | (−1.5179, 2.8413, −1.5179) | (0, 1, 0) exactly | 0.6751 |
| 0.1 | (−0.9734, 1.9825, −0.9734) | (0.0973, 0.8017, 0.0973) | 0.5745 |
| 1 | (−0.1716, 0.6041, −0.1716) | (0.1716, 0.3959, 0.1716) | 0.3260 |
Check the first row: . The outer weights are negative so that the wide middle bump is pulled back down to 0 at the ends. Far from the data the output decays toward 0 (the bias), so . A pure RBF network without a bias extrapolates to zero, which may be wrong for your problem.
Why it works: Cover's theorem and kernels
Cover's theorem (1965) says that a pattern set mapped nonlinearly into a higher-dimensional space is more likely to be linearly separable there. In dimensions, points in general position can be split by a hyperplane for every one of the labelings once . The RBF hidden layer is exactly such a nonlinear lift, and the output layer is the linear separator.
| x | XOR | φ₁(x) | φ₂(x) | φ₁ + φ₂ |
|---|---|---|---|---|
| (0, 0) | 0 | 0.1353 | 1 | 1.1353 |
| (1, 1) | 0 | 1 | 0.1353 | 1.1353 |
| (0, 1) | 1 | 0.3679 | 0.3679 | 0.7358 |
| (1, 0) | 1 | 0.3679 | 0.3679 | 0.7358 |
Both XOR-positive inputs land on the same point, and the line separates them from the other two. Here the lift did not even add dimensions; the nonlinearity alone was enough.
The Gaussian is also the RBF kernel . With one centre per point and , the regularization network is exactly kernel ridge regression, and a Gaussian-kernel SVM is an RBF network whose centres are the support vectors. Park and Sandberg (1991) proved that RBF networks are universal approximators: with enough Gaussian units they approximate any continuous function on a compact set arbitrarily well.
1import numpy as np23def design(x, centres, sigma): # Φ[n, k] = exp(-(x_n - c_k)^2 / 2σ^2)4 return np.exp(-(x[:, None] - centres[None, :]) ** 2 / (2 * sigma**2))56x, y = np.array([0.0, 0.5, 1.0]), np.array([0.0, 1.0, 0.0])7w = np.linalg.solve(design(x, x, 0.5), y) # [-1.5179 2.8413 -1.5179]8print(design(np.array([0.25]), x, 0.5) @ w) # [0.6751]910def fit_rbfn(x, y, K, lam=0.0, iters=100):11 c = np.quantile(x, (np.arange(K) + 0.5) / K) # stage 1: k-means centres12 for _ in range(iters):13 lab = np.abs(x[:, None] - c[None, :]).argmin(1)14 c = np.array([x[lab == k].mean() if (lab == k).any() else c[k] for k in range(K)])15 sigma = (c.max() - c.min()) / np.sqrt(2 * K) # stage 2: width heuristic16 feats = lambda q: np.hstack([design(q, c, sigma), np.ones((len(q), 1))])17 Phi = feats(x)18 R = lam * np.eye(K + 1); R[-1, -1] = 0 # don't penalise the bias19 w = np.linalg.solve(Phi.T @ Phi + R, Phi.T @ y) # stage 3: least squares20 return lambda q: feats(q) @ wUse when
- +Fast, reliable training: k-means plus one linear solve, with no learning rate and no local minima in the output layer
- +Interpretable: each hidden unit is a prototype in input space, and the output near a point is driven by the nearby centres
- +Low confidence far from data is easy to detect, since every activation is near 0 there
Avoid when
- −Number of units needed grows quickly with input dimension
- −Sensitive to feature scaling and to the choice of σ
- −Unsupervised centre placement ignores , so centres may land where the target is flat and miss where it varies
Check yourself
With the centres and σ fixed, why can an RBF network's output weights be found without gradient descent?
Check yourself
You place one Gaussian on each of 500 noisy training points and solve Gw = y exactly. Training error is 0 but test error is terrible. What is the standard fix that keeps K = N?
Remember
- ✓Hidden units are local Gaussian bumps ; the output is their weighted sum.
- ✓Two-stage training: k-means for centres, a heuristic such as σ = d_max/√(2K) for widths, then least squares for the output weights.
- ✓With and the network interpolates exactly (the matrix is invertible for distinct points) but fits the noise; is the regularized fix and equals kernel ridge regression.
- ✓Cover's theorem explains why a nonlinear lift helps; Park and Sandberg (1991) proved RBF networks are universal approximators.
- ✓Great for low-dimensional problems, poor in high dimensions, and sensitive to input scaling.