Definition
A multilayer perceptron is a feed-forward neural network made of fully connected layers. Each hidden layer computes , a linear map followed by an elementwise nonlinearity, and the last layer reads the prediction off the final hidden representation.
Why one neuron is not enough
A single perceptron draws one straight line, so it cannot compute XOR: the 1s, and , sit on one diagonal of the square and the 0s on the other (see the perceptron article for the proof).
The fix is to change the coordinates first. A hidden layer computes new features from . If those features put the four points in a position where one line separates them, the output neuron can finish the job. The hidden layer learns a representation; the output layer is still just a linear classifier on top of it.
The layer equation
- activations of layer ℓ (its n_ℓ neurons)
- weight matrix: row j holds the incoming weights of neuron j
- bias vector, one per neuron
- elementwise nonlinearity (ReLU, tanh, …)
- output scores; a sigmoid or softmax turns them into probabilities when needed
Each row of together with its bias is one perceptron-like neuron. A layer is many of them reading the same input in parallel, and an MLP feeds each layer's outputs to the next. A layer with inputs and outputs has parameters.
XOR with hand-set weights
A 2-2-1 network with ReLU, , solves XOR exactly:
- counts how many inputs are on
- is nonzero only when both are on (an AND detector)
- count minus twice the AND, which is exactly XOR; predict 1 when o > ½
| x | z = Wx + b | h = ReLU(z) | o = h₁ − 2h₂ | XOR |
|---|---|---|---|---|
| (0, 0) | (0, −1) | (0, 0) | 0 | 0 |
| (0, 1) | (1, 0) | (1, 0) | 1 | 1 |
| (1, 0) | (1, 0) | (1, 0) | 1 | 1 |
| (1, 1) | (2, 1) | (2, 1) | 0 | 0 |
In hidden space the 0s are at and and the 1s at . The line separates them. The ReLU on did the work: it clipped at up to 0 and bent the image of the plane at . In input space the resulting decision region is the diagonal band , bounded by two parallel lines.
The step-activation version reads like logic: with biases the hidden units become OR and AND, and the output computes OR and not AND, which is XOR.
Why the nonlinearity is required
Remove and two layers collapse into one:
- a single matrix, so the whole stack is one linear layer
- a single bias
Any depth of purely linear layers is therefore one linear model, which still draws one line. In the XOR network without ReLU the hidden points are , all on the line with the 1s in the middle. A linear readout along a line is monotone, so it cannot go low, high, low. The activation functions article covers which nonlinearities are used in practice.
Universal approximation
Statement. Let be continuous and not a polynomial. For every continuous function on a compact set and every there is a one-hidden-layer network with for all (Cybenko 1989 for sigmoids; Hornik 1991; Leshno et al. 1993 for the non-polynomial condition).
It is an existence result only. It says nothing about how large must be, whether gradient descent will find those weights, or how well the network generalises from finite data.
Depth vs width
Width alone is enough in principle; depth is often far more economical. A clean 1-D example uses the tent map on , which needs two ReLU units. Composing it times gives a zigzag with linear pieces from only hidden units.
A single hidden layer of ReLU units on a 1-D input is piecewise linear with at most kinks, so at most pieces. Matching the -fold tent needs at least units: for , 20 units in depth versus at least 1023 in width. Each layer folds the space produced by the previous one, so features are built from features.
Use when
- +Depth: compositional features, often exponentially fewer units for the same function
- +Width: easier to optimise, more parallel per layer, and enough by the universal approximation theorem
Avoid when
- −Deep plain MLPs are hard to train (vanishing or exploding gradients) without careful initialization, normalization or skip connections
- −Very wide shallow nets can need an impractical number of units and parameters
1import torch2import torch.nn as nn34net = nn.Sequential(nn.Linear(2, 2), nn.ReLU(), nn.Linear(2, 1))5with torch.no_grad():6 net[0].weight.copy_(torch.tensor([[1., 1.], [1., 1.]]))7 net[0].bias.copy_(torch.tensor([0., -1.]))8 net[2].weight.copy_(torch.tensor([[1., -2.]]))9 net[2].bias.zero_()1011X = torch.tensor([[0., 0.], [0., 1.], [1., 0.], [1., 1.]])12print(net(X).detach().squeeze()) # tensor([0., 1., 1., 0.])13print(sum(p.numel() for p in net.parameters())) # 9 = 2·(2+1) + 1·(2+1)Check yourself
An MLP for MNIST has layers 784 → 256 → 10 (weights and biases in both layers). How many parameters does it have?
Check yourself
You delete the ReLU from the 2-2-1 XOR network and are allowed to retrain every weight. What is the best accuracy on the four XOR points?
Remember
- ✓Layer: ; an MLP chains such layers and ends with a linear readout.
- ✓A hidden layer re-maps the inputs; XOR becomes linearly separable in (h₁, h₂) space.
- ✓Hand-set XOR: W = [[1,1],[1,1]], b = (0, −1), ReLU, o = h₁ − 2h₂ gives 0, 1, 1, 0.
- ✓Without a nonlinearity any stack of linear layers is a single linear layer.
- ✓One wide hidden layer can approximate any continuous function, but depth often does it with exponentially fewer units.