Radial basis function network (RBFN)

One hidden layer of Gaussian bumps centred at prototypes, then a linear readout: place the centres by k-means and the output weights come from one least-squares solve.

9 min read1085 wordscore0 viewsupdated 2026-10-11

Definition

A radial basis function network is a three-layer feed-forward network whose hidden units respond to the distance between the input and a stored centre, φk(x)=exp⁡ ⁣(−∥x−ck∥2/2σk2)\varphi_k(x)=\exp\!\big(-\lVert x-c_k\rVert^2/2\sigma_k^2\big), and whose output is a linear combination of those responses. Broomhead and Lowe introduced it as a neural network in 1988, building on Powell's work on RBF interpolation.

Local bumps instead of global ramps

An MLP hidden unit computes σ(w⋅x+b)\sigma(w\cdot x+b), a ramp that changes along one direction and stays flat along all the others, so it affects the whole input space. An RBF unit is local. It fires near its centre ckc_k and fades to zero away from it, so each unit is responsible for one region of input space.

The network builds a function by placing bumps where the data lives and scaling each one, like building a surface out of sand piles. That locality makes RBF networks easy to interpret, and it gives them a much simpler training procedure than backprop.

The model

y^(x)=b+∑k=1Kwk φ(∥x−ck∥),φ(r)=exp⁡ ⁣(−r22σ2)\hat y(x)=b+\sum_{k=1}^{K}w_k\,\varphi\big(\lVert x-c_k\rVert\big),\qquad \varphi(r)=\exp\!\Big(-\frac{r^2}{2\sigma^2}\Big)
ck∈Rdc_k\in\mathbb{R}^d
centre (prototype) of hidden unit k, a point in input space
σ\sigma
width of the Gaussians: how far each unit's influence reaches
wk, bw_k,\ b
output weights and bias, the only parameters of the linear readout
KK
number of hidden units (centres)

Other radial functions work too (multiquadric r2+c2\sqrt{r^2+c^2}, inverse multiquadric, thin-plate spline r2log⁡rr^2\log r), but the Gaussian is by far the most common. For classification, use one output per class on the same hidden layer.

Once the centres and widths are fixed, y^\hat y is linear in the parameters (w,b)(w,b). Collect the hidden activations of all NN training points in a design matrix Φ∈RN×K\Phi\in\mathbb{R}^{N\times K} with Φnk=φ(∥xn−ck∥)\Phi_{nk}=\varphi(\lVert x_n-c_k\rVert), and fitting the output layer becomes ordinary (or ridge) least squares:

w⋆=arg⁡min⁡w ∥Φw−y∥2+λ∥w∥2=(Φ⊤Φ+λI)−1Φ⊤yw^\star=\arg\min_w\ \lVert \Phi w-y\rVert^2+\lambda\lVert w\rVert^2=\big(\Phi^\top\Phi+\lambda I\big)^{-1}\Phi^\top y
Φ\Phi
design matrix of hidden activations, one row per training example (append a column of ones for the bias)
λ≥0\lambda\ge0
ridge penalty; with λ=0\lambda=0 this is w⋆=Φ+yw^\star=\Phi^{+}y, the Moore–Penrose pseudo-inverse solution

That problem is convex with a closed-form solution, so there are no local minima and no learning rate. The nonlinear part of the problem (where to put the bumps, how wide to make them) is handled separately, usually without looking at yy at all.

Two-stage training (Moody & Darken, 1989)

  1. 1Centres. Run k-means on the training inputs with KK clusters and use the cluster means as c1,…,cKc_1,\dots,c_K. Cheaper alternatives: a random subset of the training points, or a regular grid in low dimensions.
  2. 2Widths. Set σ\sigma by a heuristic. A common one is σ=dmax⁡/2K\sigma=d_{\max}/\sqrt{2K}, with dmax⁡d_{\max} the largest distance between two centres. Per-unit alternatives use the mean distance to the pp nearest centres. The aim is bumps that overlap their neighbours. This one errs on the wide side, so pair it with a small λ\lambda.
  3. 3Output weights. Build Φ\Phi (plus a ones column), then solve the least-squares problem above with lstsq or the normal equations.
  4. 4Optional fine-tuning. Everything is differentiable, so centres, widths and weights can afterwards be trained jointly by gradient descent. Good starting values from stages 1–3 make this much easier.
ŷ(x) = b + Σₖ wₖ·exp(−(x − cₖ)²/2σ²) · K = 12, σ = 0.060, b = -0.909-2-1012● train (24) · test (120) ━ network output ─/─ wₖφₖ(x), wₖ > 0 / < 0 ▲ centres ╌ truthMSE vs K (k-means centres, σ = 0.06, λ = 0) · log scale · noise floor σ²ₙ = 0.041e-21e-111e114812162024━ train ━ test · K →
centres
train MSE 0.026 · test MSE 0.059 · max |wₖ| 2.176
drag σ: too narrow leaves gaps between bumps, too wide makes them nearly collinear.
fig.24 noisy samples (orange) of sin 2πx + 0.3 sin 6πx with noise sd 0.2. Each thin curve is one hidden unit's contribution wₖφₖ(x), and the thick curve is their sum plus the bias. Move K and σ, then switch to one centre per training point and set λ = 0 to see exact interpolation, and raise λ to tame it. The lower chart re-solves the network for every K.

At the default (K=12K=12 k-means centres, σ=0.06\sigma=0.06, λ=0\lambda=0) the training MSE is 0.0260.026 and the test MSE 0.0590.059, against a noise floor of σn2=0.04\sigma_n^2=0.04, and no weight exceeds 2.22.2 in magnitude. Turn on auto σ and the heuristic picks σ=0.192\sigma=0.192: the test MSE is still 0.0810.081, but the weights reach about 21502150 with opposite signs that nearly cancel. With λ=10−3\lambda=10^{-3} the largest weight drops to 5.75.7.

Overfitting shows up as K grows. With σ=0.08\sigma=0.08 and λ=0\lambda=0, the test error at K=20K=20 jumps to 4.994.99; λ=10−3\lambda=10^{-3} brings it back to 0.0570.057. Too many bumps with nothing to stop them fit the noise.

Exact interpolation and regularization

The original RBF problem from numerical analysis is interpolation: put one centre on every training point (K=NK=N, cn=xnc_n=x_n) and demand y^(xn)=yn\hat y(x_n)=y_n exactly. That gives a square system Gw=yGw=y with Gnm=φ(∥xn−xm∥)G_{nm}=\varphi(\lVert x_n-x_m\rVert). Micchelli (1986) proved that for distinct points the Gaussian matrix GG is positive definite, so a solution always exists.

Existence is not the same as usefulness. Exact interpolation passes the curve through the noise, and when two inputs are close or σ\sigma is wide, GG is so ill-conditioned that the weights reach 101210^{12} and cancel each other almost perfectly. Poggio and Girosi (1990) replaced it with the regularization network (G+λI) w=y(G+\lambda I)\,w=y, which trades a little training error for a smooth function.

λweights w = (G + λI)⁻¹yfitted ŷ at the 3 pointsŷ(0.25)
0(−1.5179, 2.8413, −1.5179)(0, 1, 0) exactly0.6751
0.1(−0.9734, 1.9825, −0.9734)(0.0973, 0.8017, 0.0973)0.5745
1(−0.1716, 0.6041, −0.1716)(0.1716, 0.3959, 0.1716)0.3260
Worked example: x=(0, 0.5, 1)x=(0,\,0.5,\,1), y=(0,1,0)y=(0,1,0), σ=0.5\sigma=0.5, centres on the data, no bias. Entries of GG are 11, e−1/2=0.6065e^{-1/2}=0.6065 (neighbours) and e−2=0.1353e^{-2}=0.1353 (the two ends).

Check the first row: y^(0.5)=0.6065⋅(−1.5179)+2.8413+0.6065⋅(−1.5179)=1.0000\hat y(0.5)=0.6065\cdot(-1.5179)+2.8413+0.6065\cdot(-1.5179)=1.0000. The outer weights are negative so that the wide middle bump is pulled back down to 0 at the ends. Far from the data the output decays toward 0 (the bias), so y^(1.5)=−0.5530\hat y(1.5)=-0.5530. A pure RBF network without a bias extrapolates to zero, which may be wrong for your problem.

Why it works: Cover's theorem and kernels

Cover's theorem (1965) says that a pattern set mapped nonlinearly into a higher-dimensional space is more likely to be linearly separable there. In DD dimensions, NN points in general position can be split by a hyperplane for every one of the 2N2^N labelings once D≥N−1D\ge N-1. The RBF hidden layer is exactly such a nonlinear lift, and the output layer is the linear separator.

xXORφ₁(x)φ₂(x)φ₁ + φ₂
(0, 0)00.135311.1353
(1, 1)010.13531.1353
(0, 1)10.36790.36790.7358
(1, 0)10.36790.36790.7358
XOR becomes linearly separable with two Gaussian units, φk(x)=exp⁡(−∥x−ck∥2)\varphi_k(x)=\exp(-\lVert x-c_k\rVert^2), centred at c1=(1,1)c_1=(1,1) and c2=(0,0)c_2=(0,0).

Both XOR-positive inputs land on the same point, and the line φ1+φ2=0.9\varphi_1+\varphi_2=0.9 separates them from the other two. Here the lift did not even add dimensions; the nonlinearity alone was enough.

The Gaussian φ(∥x−x′∥)\varphi(\lVert x-x'\rVert) is also the RBF kernel k(x,x′)k(x,x'). With one centre per point and λ>0\lambda>0, the regularization network (G+λI)w=y(G+\lambda I)w=y is exactly kernel ridge regression, and a Gaussian-kernel SVM is an RBF network whose centres are the support vectors. Park and Sandberg (1991) proved that RBF networks are universal approximators: with enough Gaussian units they approximate any continuous function on a compact set arbitrarily well.

python
1import numpy as np
2
3def design(x, centres, sigma): # Φ[n, k] = exp(-(x_n - c_k)^2 / 2σ^2)
4 return np.exp(-(x[:, None] - centres[None, :]) ** 2 / (2 * sigma**2))
5
6x, y = np.array([0.0, 0.5, 1.0]), np.array([0.0, 1.0, 0.0])
7w = np.linalg.solve(design(x, x, 0.5), y) # [-1.5179 2.8413 -1.5179]
8print(design(np.array([0.25]), x, 0.5) @ w) # [0.6751]
9
10def fit_rbfn(x, y, K, lam=0.0, iters=100):
11 c = np.quantile(x, (np.arange(K) + 0.5) / K) # stage 1: k-means centres
12 for _ in range(iters):
13 lab = np.abs(x[:, None] - c[None, :]).argmin(1)
14 c = np.array([x[lab == k].mean() if (lab == k).any() else c[k] for k in range(K)])
15 sigma = (c.max() - c.min()) / np.sqrt(2 * K) # stage 2: width heuristic
16 feats = lambda q: np.hstack([design(q, c, sigma), np.ones((len(q), 1))])
17 Phi = feats(x)
18 R = lam * np.eye(K + 1); R[-1, -1] = 0 # don't penalise the bias
19 w = np.linalg.solve(Phi.T @ Phi + R, Phi.T @ y) # stage 3: least squares
20 return lambda q: feats(q) @ w
An RBF network in NumPy: exact interpolation on the worked example, then the two-stage recipe for K < N

Use when

  • +Fast, reliable training: k-means plus one linear solve, with no learning rate and no local minima in the output layer
  • +Interpretable: each hidden unit is a prototype in input space, and the output near a point is driven by the nearby centres
  • +Low confidence far from data is easy to detect, since every activation is near 0 there

Avoid when

  • −Number of units needed grows quickly with input dimension
  • −Sensitive to feature scaling and to the choice of σ
  • −Unsupervised centre placement ignores yy, so centres may land where the target is flat and miss where it varies

Check yourself

With the centres and σ fixed, why can an RBF network's output weights be found without gradient descent?

Check yourself

You place one Gaussian on each of 500 noisy training points and solve Gw = y exactly. Training error is 0 but test error is terrible. What is the standard fix that keeps K = N?

Remember

  • ✓Hidden units are local Gaussian bumps exp⁡(−∥x−ck∥2/2σ2)\exp(-\lVert x-c_k\rVert^2/2\sigma^2); the output is their weighted sum.
  • ✓Two-stage training: k-means for centres, a heuristic such as σ = d_max/√(2K) for widths, then least squares for the output weights.
  • ✓With K=NK=N and λ=0\lambda=0 the network interpolates exactly (the matrix is invertible for distinct points) but fits the noise; (G+λI)w=y(G+\lambda I)w=y is the regularized fix and equals kernel ridge regression.
  • ✓Cover's theorem explains why a nonlinear lift helps; Park and Sandberg (1991) proved RBF networks are universal approximators.
  • ✓Great for low-dimensional problems, poor in high dimensions, and sensitive to input scaling.
← back to Deep Learning roadmap