# Radial basis function network (RBFN)

> One hidden layer of Gaussian bumps centred at prototypes, then a linear readout: place the centres by k-means and the output weights come from one least-squares solve.

Source: https://www.mlfoundry.in/learn/radial-basis-function-network-rbfn  
Section: Deep Learning › Neural Network Algorithms  
Level: core · 9 min read · updated 2026-10-11

**A radial basis function network:** is a three-layer feed-forward network whose hidden units respond to the **distance** between the input and a stored centre, $\varphi_k(x)=\exp\!\big(-\lVert x-c_k\rVert^2/2\sigma_k^2\big)$, and whose output is a linear combination of those responses. Broomhead and Lowe introduced it as a neural network in 1988, building on Powell's work on RBF interpolation.

## Local bumps instead of global ramps

An MLP hidden unit computes $\sigma(w\cdot x+b)$, a ramp that changes along one direction and stays flat along all the others, so it affects the whole input space. An RBF unit is **local**. It fires near its centre $c_k$ and fades to zero away from it, so each unit is responsible for one region of input space.

The network builds a function by placing bumps where the data lives and scaling each one, like building a surface out of sand piles. That locality makes RBF networks easy to interpret, and it gives them a much simpler training procedure than backprop.

## The model

$$
\hat y(x)=b+\sum_{k=1}^{K}w_k\,\varphi\big(\lVert x-c_k\rVert\big),\qquad \varphi(r)=\exp\!\Big(-\frac{r^2}{2\sigma^2}\Big)
$$

- $c_k\in\mathbb{R}^d$: centre (prototype) of hidden unit k, a point in input space
- $\sigma$: width of the Gaussians: how far each unit's influence reaches
- $w_k,\ b$: output weights and bias, the only parameters of the linear readout
- $K$: number of hidden units (centres)

Other radial functions work too (multiquadric $\sqrt{r^2+c^2}$, inverse multiquadric, thin-plate spline $r^2\log r$), but the Gaussian is by far the most common. For classification, use one output per class on the same hidden layer.

Once the centres and widths are fixed, $\hat y$ is **linear in the parameters** $(w,b)$. Collect the hidden activations of all $N$ training points in a design matrix $\Phi\in\mathbb{R}^{N\times K}$ with $\Phi_{nk}=\varphi(\lVert x_n-c_k\rVert)$, and fitting the output layer becomes ordinary (or ridge) least squares:

$$
w^\star=\arg\min_w\ \lVert \Phi w-y\rVert^2+\lambda\lVert w\rVert^2=\big(\Phi^\top\Phi+\lambda I\big)^{-1}\Phi^\top y
$$

- $\Phi$: design matrix of hidden activations, one row per training example (append a column of ones for the bias)
- $\lambda\ge0$: ridge penalty; with $\lambda=0$ this is $w^\star=\Phi^{+}y$, the Moore–Penrose pseudo-inverse solution

That problem is convex with a closed-form solution, so there are no local minima and no learning rate. The nonlinear part of the problem (where to put the bumps, how wide to make them) is handled separately, usually without looking at $y$ at all.

**Two-stage training (Moody & Darken, 1989)**

1. **Centres.** Run k-means on the training inputs with $K$ clusters and use the cluster means as $c_1,\dots,c_K$. Cheaper alternatives: a random subset of the training points, or a regular grid in low dimensions.
2. **Widths.** Set $\sigma$ by a heuristic. A common one is $\sigma=d_{\max}/\sqrt{2K}$, with $d_{\max}$ the largest distance between two centres. Per-unit alternatives use the mean distance to the $p$ nearest centres. The aim is bumps that overlap their neighbours. This one errs on the wide side, so pair it with a small $\lambda$.
3. **Output weights.** Build $\Phi$ (plus a ones column), then solve the least-squares problem above with `lstsq` or the normal equations.
4. **Optional fine-tuning.** Everything is differentiable, so centres, widths and weights can afterwards be trained jointly by gradient descent. Good starting values from stages 1–3 make this much easier.

> Figure: 24 noisy samples (orange) of sin 2πx + 0.3 sin 6πx with noise sd 0.2. Each thin curve is one hidden unit's contribution wₖφₖ(x), and the thick curve is their sum plus the bias. Move K and σ, then switch to one centre per training point and set λ = 0 to see exact interpolation, and raise λ to tame it. The lower chart re-solves the network for every K.

At the default ($K=12$ k-means centres, $\sigma=0.06$, $\lambda=0$) the training MSE is $0.026$ and the test MSE $0.059$, against a noise floor of $\sigma_n^2=0.04$, and no weight exceeds $2.2$ in magnitude. Turn on auto σ and the heuristic picks $\sigma=0.192$: the test MSE is still $0.081$, but the weights reach about $2150$ with opposite signs that nearly cancel. With $\lambda=10^{-3}$ the largest weight drops to $5.7$.

Overfitting shows up as K grows. With $\sigma=0.08$ and $\lambda=0$, the test error at $K=20$ jumps to $4.99$; $\lambda=10^{-3}$ brings it back to $0.057$. Too many bumps with nothing to stop them fit the noise.

## Exact interpolation and regularization

The original RBF problem from numerical analysis is **interpolation**: put one centre on every training point ($K=N$, $c_n=x_n$) and demand $\hat y(x_n)=y_n$ exactly. That gives a square system $Gw=y$ with $G_{nm}=\varphi(\lVert x_n-x_m\rVert)$. Micchelli (1986) proved that for distinct points the Gaussian matrix $G$ is positive definite, so a solution always exists.

Existence is not the same as usefulness. Exact interpolation passes the curve through the noise, and when two inputs are close or $\sigma$ is wide, $G$ is so ill-conditioned that the weights reach $10^{12}$ and cancel each other almost perfectly. Poggio and Girosi (1990) replaced it with the **regularization network** $(G+\lambda I)\,w=y$, which trades a little training error for a smooth function.

Worked example: $x=(0,\,0.5,\,1)$, $y=(0,1,0)$, $\sigma=0.5$, centres on the data, no bias. Entries of $G$ are $1$, $e^{-1/2}=0.6065$ (neighbours) and $e^{-2}=0.1353$ (the two ends).

| λ | weights w = (G + λI)⁻¹y | fitted ŷ at the 3 points | ŷ(0.25) |
| --- | --- | --- | --- |
| 0 | (−1.5179, 2.8413, −1.5179) | (0, 1, 0) exactly | 0.6751 |
| 0.1 | (−0.9734, 1.9825, −0.9734) | (0.0973, 0.8017, 0.0973) | 0.5745 |
| 1 | (−0.1716, 0.6041, −0.1716) | (0.1716, 0.3959, 0.1716) | 0.3260 |

Check the first row: $\hat y(0.5)=0.6065\cdot(-1.5179)+2.8413+0.6065\cdot(-1.5179)=1.0000$. The outer weights are negative so that the wide middle bump is pulled back down to 0 at the ends. Far from the data the output decays toward 0 (the bias), so $\hat y(1.5)=-0.5530$. A pure RBF network without a bias extrapolates to zero, which may be wrong for your problem.

## Why it works: Cover's theorem and kernels

**Cover's theorem** (1965) says that a pattern set mapped nonlinearly into a higher-dimensional space is more likely to be linearly separable there. In $D$ dimensions, $N$ points in general position can be split by a hyperplane for every one of the $2^N$ labelings once $D\ge N-1$. The RBF hidden layer is exactly such a nonlinear lift, and the output layer is the linear separator.

XOR becomes linearly separable with two Gaussian units, $\varphi_k(x)=\exp(-\lVert x-c_k\rVert^2)$, centred at $c_1=(1,1)$ and $c_2=(0,0)$.

| x | XOR | φ₁(x) | φ₂(x) | φ₁ + φ₂ |
| --- | --- | --- | --- | --- |
| (0, 0) | 0 | 0.1353 | 1 | 1.1353 |
| (1, 1) | 0 | 1 | 0.1353 | 1.1353 |
| (0, 1) | 1 | 0.3679 | 0.3679 | 0.7358 |
| (1, 0) | 1 | 0.3679 | 0.3679 | 0.7358 |

Both XOR-positive inputs land on the same point, and the line $\varphi_1+\varphi_2=0.9$ separates them from the other two. Here the lift did not even add dimensions; the nonlinearity alone was enough.

The Gaussian $\varphi(\lVert x-x'\rVert)$ is also the RBF **kernel** $k(x,x')$. With one centre per point and $\lambda>0$, the regularization network $(G+\lambda I)w=y$ is exactly kernel ridge regression, and a Gaussian-kernel SVM is an RBF network whose centres are the support vectors. Park and Sandberg (1991) proved that RBF networks are **universal approximators**: with enough Gaussian units they approximate any continuous function on a compact set arbitrarily well.

```python
import numpy as np

def design(x, centres, sigma):                 # Φ[n, k] = exp(-(x_n - c_k)^2 / 2σ^2)
    return np.exp(-(x[:, None] - centres[None, :]) ** 2 / (2 * sigma**2))

x, y = np.array([0.0, 0.5, 1.0]), np.array([0.0, 1.0, 0.0])
w = np.linalg.solve(design(x, x, 0.5), y)      # [-1.5179  2.8413 -1.5179]
print(design(np.array([0.25]), x, 0.5) @ w)    # [0.6751]

def fit_rbfn(x, y, K, lam=0.0, iters=100):
    c = np.quantile(x, (np.arange(K) + 0.5) / K)              # stage 1: k-means centres
    for _ in range(iters):
        lab = np.abs(x[:, None] - c[None, :]).argmin(1)
        c = np.array([x[lab == k].mean() if (lab == k).any() else c[k] for k in range(K)])
    sigma = (c.max() - c.min()) / np.sqrt(2 * K)              # stage 2: width heuristic
    feats = lambda q: np.hstack([design(q, c, sigma), np.ones((len(q), 1))])
    Phi = feats(x)
    R = lam * np.eye(K + 1); R[-1, -1] = 0                    # don't penalise the bias
    w = np.linalg.solve(Phi.T @ Phi + R, Phi.T @ y)           # stage 3: least squares
    return lambda q: feats(q) @ w
```

An RBF network in NumPy: exact interpolation on the worked example, then the two-stage recipe for K < N

> **Scale the inputs first:** One shared $\sigma$ assumes every input dimension is measured in comparable units. If one feature ranges over thousands and another over $[0,1]$, the distance is dominated by the first and the bumps ignore the second. Standardize features, or give each unit a full covariance (which turns it into a Gaussian-mixture-like model).

> **The curse of dimensionality:** Local units need to tile the region where data lives. In $d$ dimensions, covering it at a fixed resolution takes a number of bumps that grows exponentially in $d$. RBF networks shine for low-dimensional inputs (control, time series, interpolating scattered measurements) and lose to MLPs on images or text.

> **σ controls smoothness, K controls capacity:** Small $\sigma$ gives spiky, isolated bumps and gaps between them; large $\sigma$ gives nearly identical columns in $\Phi$, so the least-squares problem becomes ill-conditioned and needs $\lambda$. A good $\sigma$ is roughly the spacing between neighbouring centres. The $d_{\max}/\sqrt{2K}$ heuristic gives about twice that, which is safe only with some ridge penalty.

**Strengths**

- Fast, reliable training: k-means plus one linear solve, with no learning rate and no local minima in the output layer
- Interpretable: each hidden unit is a prototype in input space, and the output near a point is driven by the nearby centres
- Low confidence far from data is easy to detect, since every activation is near 0 there

**Limitations**

- Number of units needed grows quickly with input dimension
- Sensitive to feature scaling and to the choice of σ
- Unsupervised centre placement ignores $y$, so centres may land where the target is flat and miss where it varies

**Question:** With the centres and σ fixed, why can an RBF network's output weights be found without gradient descent?

- A. Because the Gaussian is convex in x
- B. Because the output is linear in the weights, so fitting them is a linear least-squares problem with a closed-form solution
- C. Because k-means already computes the weights
- D. Because the hidden layer has no parameters at all

**Answer:** B. Because the output is linear in the weights, so fitting them is a linear least-squares problem with a closed-form solution — $\hat y=\Phi w$ (plus bias) with $\Phi$ fixed, so minimizing squared error is linear least squares: $w^\star=(\Phi^\top\Phi+\lambda I)^{-1}\Phi^\top y$. The hidden layer does have parameters (centres and widths), but they are chosen beforehand.

**Question:** You place one Gaussian on each of 500 noisy training points and solve Gw = y exactly. Training error is 0 but test error is terrible. What is the standard fix that keeps K = N?

- A. Increase σ until G is the identity
- B. Solve (G + λI)w = y with λ > 0
- C. Use fewer k-means iterations
- D. Remove the bias term

**Answer:** B. Solve (G + λI)w = y with λ > 0 — Exact interpolation fits the noise and is often numerically ill-conditioned. Adding $\lambda I$ (the regularization network, equivalently kernel ridge regression) trades training error for smoothness and small weights. Note that a larger $\sigma$ makes $G$ *more* ill-conditioned, not closer to the identity; that happens as $\sigma\to0$.

**Key points**

- Hidden units are local Gaussian bumps $\exp(-\lVert x-c_k\rVert^2/2\sigma^2)$; the output is their weighted sum.
- Two-stage training: k-means for centres, a heuristic such as σ = d_max/√(2K) for widths, then least squares for the output weights.
- With $K=N$ and $\lambda=0$ the network interpolates exactly (the matrix is invertible for distinct points) but fits the noise; $(G+\lambda I)w=y$ is the regularized fix and equals kernel ridge regression.
- Cover's theorem explains why a nonlinear lift helps; Park and Sandberg (1991) proved RBF networks are universal approximators.
- Great for low-dimensional problems, poor in high dimensions, and sensitive to input scaling.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
