Multilayer perceptron (MLP)

Stack layers of neurons with nonlinear activations between them; a hidden layer bends the input space so a line can separate what one neuron could not.

8 min read682 wordscore0 viewsupdated 2026-10-09

Definition

A multilayer perceptron is a feed-forward neural network made of fully connected layers. Each hidden layer computes h=φ(Wx+b)h=\varphi(Wx+b), a linear map followed by an elementwise nonlinearity, and the last layer reads the prediction off the final hidden representation.

Why one neuron is not enough

A single perceptron draws one straight line, so it cannot compute XOR: the 1s, (0,1)(0,1) and (1,0)(1,0), sit on one diagonal of the square and the 0s on the other (see the perceptron article for the proof).

The fix is to change the coordinates first. A hidden layer computes new features h1,h2h_1,h_2 from xx. If those features put the four points in a position where one line separates them, the output neuron can finish the job. The hidden layer learns a representation; the output layer is still just a linear classifier on top of it.

The layer equation

h(0)=x,h(ℓ)=φ ⁣(W(ℓ)h(ℓ−1)+b(ℓ)),  ℓ=1,…,L−1,y^=W(L)h(L−1)+b(L)h^{(0)}=x,\qquad h^{(\ell)}=\varphi\!\left(W^{(\ell)}h^{(\ell-1)}+b^{(\ell)}\right),\ \ \ell=1,\dots,L-1,\qquad \hat y=W^{(L)}h^{(L-1)}+b^{(L)}
h(ℓ)∈Rnℓh^{(\ell)}\in\mathbb{R}^{n_\ell}
activations of layer ℓ (its n_ℓ neurons)
W(ℓ)∈Rnℓ×nℓ−1W^{(\ell)}\in\mathbb{R}^{n_\ell\times n_{\ell-1}}
weight matrix: row j holds the incoming weights of neuron j
b(ℓ)∈Rnℓb^{(\ell)}\in\mathbb{R}^{n_\ell}
bias vector, one per neuron
φ\varphi
elementwise nonlinearity (ReLU, tanh, …)
y^\hat y
output scores; a sigmoid or softmax turns them into probabilities when needed

Each row of W(ℓ)W^{(\ell)} together with its bias is one perceptron-like neuron. A layer is many of them reading the same input in parallel, and an MLP feeds each layer's outputs to the next. A layer with nℓ−1n_{\ell-1} inputs and nℓn_\ell outputs has nℓ(nℓ−1+1)n_\ell(n_{\ell-1}+1) parameters.

XOR with hand-set weights

A 2-2-1 network with ReLU, φ(z)=max⁡(0,z)\varphi(z)=\max(0,z), solves XOR exactly:

W=(1111),b=(0−1),h=max⁡(0, Wx+b),o=h1−2h2W=\begin{pmatrix}1&1\\1&1\end{pmatrix},\quad b=\begin{pmatrix}0\\-1\end{pmatrix},\quad h=\max(0,\,Wx+b),\quad o=h_1-2h_2
h1=max⁡(0,x1+x2)h_1=\max(0,x_1+x_2)
counts how many inputs are on
h2=max⁡(0,x1+x2−1)h_2=\max(0,x_1+x_2-1)
is nonzero only when both are on (an AND detector)
oo
count minus twice the AND, which is exactly XOR; predict 1 when o > ½
xz = Wx + bh = ReLU(z)o = h₁ − 2h₂XOR
(0, 0)(0, −1)(0, 0)00
(0, 1)(1, 0)(1, 0)11
(1, 0)(1, 0)(1, 0)11
(1, 1)(2, 1)(2, 1)00
Every number checked by direct computation. The hidden layer maps (0,1) and (1,0) to the same point and pushes (1,1) out to (2, 1).

In hidden space the 0s are at (0,0)(0,0) and (2,1)(2,1) and the 1s at (1,0)(1,0). The line h1−2h2=12h_1-2h_2=\tfrac12 separates them. The ReLU on h2h_2 did the work: it clipped z2=−1z_2=-1 at (0,0)(0,0) up to 0 and bent the image of the plane at x1+x2=1x_1+x_2=1. In input space the resulting decision region is the diagonal band 12<x1+x2<32\tfrac12<x_1+x_2<\tfrac32, bounded by two parallel lines.

The step-activation version reads like logic: with biases (−12,−32)(-\tfrac12,-\tfrac32) the hidden units become OR and AND, and the output computes OR and not AND, which is XOR.

input (x₁, x₂) · shaded = 100110110φ(Wx+b)→hidden (h₁, h₂) · line = output010-1-1001122xz = Wx + bh = φ(z)o = h₁ − 2h₂ − ½ŷXOR(0, 0)(0, −1)(0, 0)−0.500 ✓0(0, 1)(1, 0)(1, 0)0.501 ✓1(1, 0)(1, 0)(1, 0)0.501 ✓1(1, 1)(2, 1)(2, 1)−0.500 ✓04/4 on the correct side of the output line · one straight line separates XOR
activation φ:
fig.Left: input space, shaded where the network outputs 1. Right: where the hidden layer sends each point, with the warped input grid in violet and the fixed output line. Slide t from 0 to 1 to morph x into h and watch the left boundary bend from one line into a band. Move the biases to break the solution, then switch the activation to none: the four points collapse onto one straight line with the 1s between the 0s, and 3 of 4 is the best any output line can do.

Why the nonlinearity is required

Remove φ\varphi and two layers collapse into one:

W2(W1x+b1)+b2=(W2W1) x+(W2b1+b2)W_2(W_1x+b_1)+b_2=(W_2W_1)\,x+(W_2b_1+b_2)
W2W1W_2W_1
a single matrix, so the whole stack is one linear layer
W2b1+b2W_2b_1+b_2
a single bias

Any depth of purely linear layers is therefore one linear model, which still draws one line. In the XOR network without ReLU the hidden points are (0,−1),(1,0),(1,0),(2,1)(0,-1),(1,0),(1,0),(2,1), all on the line h2=h1−1h_2=h_1-1 with the 1s in the middle. A linear readout along a line is monotone, so it cannot go low, high, low. The activation functions article covers which nonlinearities are used in practice.

Universal approximation

Statement. Let φ\varphi be continuous and not a polynomial. For every continuous function ff on a compact set K⊂RdK\subset\mathbb{R}^d and every ε>0\varepsilon>0 there is a one-hidden-layer network g(x)=∑j=1maj φ(wj⋅x+bj)g(x)=\sum_{j=1}^{m}a_j\,\varphi(w_j\cdot x+b_j) with ∣f(x)−g(x)∣<ε|f(x)-g(x)|<\varepsilon for all x∈Kx\in K (Cybenko 1989 for sigmoids; Hornik 1991; Leshno et al. 1993 for the non-polynomial condition).

It is an existence result only. It says nothing about how large mm must be, whether gradient descent will find those weights, or how well the network generalises from finite data.

Depth vs width

Width alone is enough in principle; depth is often far more economical. A clean 1-D example uses the tent map T(x)=2 ReLU(x)−4 ReLU(x−12)T(x)=2\,\mathrm{ReLU}(x)-4\,\mathrm{ReLU}(x-\tfrac12) on [0,1][0,1], which needs two ReLU units. Composing it kk times gives a zigzag with 2k2^k linear pieces from only 2k2k hidden units.

A single hidden layer of mm ReLU units on a 1-D input is piecewise linear with at most mm kinks, so at most m+1m+1 pieces. Matching the kk-fold tent needs at least 2k−12^k-1 units: for k=10k=10, 20 units in depth versus at least 1023 in width. Each layer folds the space produced by the previous one, so features are built from features.

Use when

  • +Depth: compositional features, often exponentially fewer units for the same function
  • +Width: easier to optimise, more parallel per layer, and enough by the universal approximation theorem

Avoid when

  • −Deep plain MLPs are hard to train (vanishing or exploding gradients) without careful initialization, normalization or skip connections
  • −Very wide shallow nets can need an impractical number of units and parameters
python
1import torch
2import torch.nn as nn
3
4net = nn.Sequential(nn.Linear(2, 2), nn.ReLU(), nn.Linear(2, 1))
5with torch.no_grad():
6 net[0].weight.copy_(torch.tensor([[1., 1.], [1., 1.]]))
7 net[0].bias.copy_(torch.tensor([0., -1.]))
8 net[2].weight.copy_(torch.tensor([[1., -2.]]))
9 net[2].bias.zero_()
10
11X = torch.tensor([[0., 0.], [0., 1.], [1., 0.], [1., 1.]])
12print(net(X).detach().squeeze()) # tensor([0., 1., 1., 0.])
13print(sum(p.numel() for p in net.parameters())) # 9 = 2·(2+1) + 1·(2+1)
The hand-set 2-2-1 XOR network in PyTorch. nn.Linear stores W with shape (out, in).

Check yourself

An MLP for MNIST has layers 784 → 256 → 10 (weights and biases in both layers). How many parameters does it have?

Check yourself

You delete the ReLU from the 2-2-1 XOR network and are allowed to retrain every weight. What is the best accuracy on the four XOR points?

Remember

  • ✓Layer: h=φ(Wx+b)h=\varphi(Wx+b); an MLP chains such layers and ends with a linear readout.
  • ✓A hidden layer re-maps the inputs; XOR becomes linearly separable in (h₁, h₂) space.
  • ✓Hand-set XOR: W = [[1,1],[1,1]], b = (0, −1), ReLU, o = h₁ − 2h₂ gives 0, 1, 1, 0.
  • ✓Without a nonlinearity any stack of linear layers is a single linear layer.
  • ✓One wide hidden layer can approximate any continuous function, but depth often does it with exponentially fewer units.
← back to Deep Learning roadmap