# Multilayer perceptron (MLP)

> Stack layers of neurons with nonlinear activations between them; a hidden layer bends the input space so a line can separate what one neuron could not.

Source: https://www.mlfoundry.in/learn/multilayer-perceptron-mlp  
Section: Deep Learning › Deep Learning Fundamentals  
Level: core · 8 min read · updated 2026-10-09

**A multilayer perceptron:** is a feed-forward neural network made of fully connected layers. Each hidden layer computes $h=\varphi(Wx+b)$, a linear map followed by an elementwise nonlinearity, and the last layer reads the prediction off the final hidden representation.

## Why one neuron is not enough

A single perceptron draws one straight line, so it cannot compute XOR: the 1s, $(0,1)$ and $(1,0)$, sit on one diagonal of the square and the 0s on the other (see the perceptron article for the proof).

The fix is to change the coordinates first. A hidden layer computes new features $h_1,h_2$ from $x$. If those features put the four points in a position where one line separates them, the output neuron can finish the job. The hidden layer **learns a representation**; the output layer is still just a linear classifier on top of it.

## The layer equation

$$
h^{(0)}=x,\qquad h^{(\ell)}=\varphi\!\left(W^{(\ell)}h^{(\ell-1)}+b^{(\ell)}\right),\ \ \ell=1,\dots,L-1,\qquad \hat y=W^{(L)}h^{(L-1)}+b^{(L)}
$$

- $h^{(\ell)}\in\mathbb{R}^{n_\ell}$: activations of layer ℓ (its n_ℓ neurons)
- $W^{(\ell)}\in\mathbb{R}^{n_\ell\times n_{\ell-1}}$: weight matrix: row j holds the incoming weights of neuron j
- $b^{(\ell)}\in\mathbb{R}^{n_\ell}$: bias vector, one per neuron
- $\varphi$: elementwise nonlinearity (ReLU, tanh, …)
- $\hat y$: output scores; a sigmoid or softmax turns them into probabilities when needed

Each row of $W^{(\ell)}$ together with its bias is one perceptron-like neuron. A layer is many of them reading the same input in parallel, and an MLP feeds each layer's outputs to the next. A layer with $n_{\ell-1}$ inputs and $n_\ell$ outputs has $n_\ell(n_{\ell-1}+1)$ parameters.

## XOR with hand-set weights

A 2-2-1 network with ReLU, $\varphi(z)=\max(0,z)$, solves XOR exactly:

$$
W=\begin{pmatrix}1&1\\1&1\end{pmatrix},\quad b=\begin{pmatrix}0\\-1\end{pmatrix},\quad h=\max(0,\,Wx+b),\quad o=h_1-2h_2
$$

- $h_1=\max(0,x_1+x_2)$: counts how many inputs are on
- $h_2=\max(0,x_1+x_2-1)$: is nonzero only when both are on (an AND detector)
- $o$: count minus twice the AND, which is exactly XOR; predict 1 when o > ½

Every number checked by direct computation. The hidden layer maps (0,1) and (1,0) to the same point and pushes (1,1) out to (2, 1).

| x | z = Wx + b | h = ReLU(z) | o = h₁ − 2h₂ | XOR |
| --- | --- | --- | --- | --- |
| (0, 0) | (0, −1) | (0, 0) | 0 | 0 |
| (0, 1) | (1, 0) | (1, 0) | 1 | 1 |
| (1, 0) | (1, 0) | (1, 0) | 1 | 1 |
| (1, 1) | (2, 1) | (2, 1) | 0 | 0 |

In hidden space the 0s are at $(0,0)$ and $(2,1)$ and the 1s at $(1,0)$. The line $h_1-2h_2=\tfrac12$ separates them. The ReLU on $h_2$ did the work: it clipped $z_2=-1$ at $(0,0)$ up to 0 and bent the image of the plane at $x_1+x_2=1$. In input space the resulting decision region is the diagonal band $\tfrac12<x_1+x_2<\tfrac32$, bounded by **two** parallel lines.

The step-activation version reads like logic: with biases $(-\tfrac12,-\tfrac32)$ the hidden units become OR and AND, and the output computes OR and not AND, which is XOR.

> Figure: Left: input space, shaded where the network outputs 1. Right: where the hidden layer sends each point, with the warped input grid in violet and the fixed output line. Slide t from 0 to 1 to morph x into h and watch the left boundary bend from one line into a band. Move the biases to break the solution, then switch the activation to none: the four points collapse onto one straight line with the 1s between the 0s, and 3 of 4 is the best any output line can do.

## Why the nonlinearity is required

Remove $\varphi$ and two layers collapse into one:

$$
W_2(W_1x+b_1)+b_2=(W_2W_1)\,x+(W_2b_1+b_2)
$$

- $W_2W_1$: a single matrix, so the whole stack is one linear layer
- $W_2b_1+b_2$: a single bias

Any depth of purely linear layers is therefore one linear model, which still draws one line. In the XOR network without ReLU the hidden points are $(0,-1),(1,0),(1,0),(2,1)$, all on the line $h_2=h_1-1$ with the 1s in the middle. A linear readout along a line is monotone, so it cannot go low, high, low. The activation functions article covers which nonlinearities are used in practice.

## Universal approximation

**Statement.** Let $\varphi$ be continuous and not a polynomial. For every continuous function $f$ on a compact set $K\subset\mathbb{R}^d$ and every $\varepsilon>0$ there is a one-hidden-layer network $g(x)=\sum_{j=1}^{m}a_j\,\varphi(w_j\cdot x+b_j)$ with $|f(x)-g(x)|<\varepsilon$ for all $x\in K$ (Cybenko 1989 for sigmoids; Hornik 1991; Leshno et al. 1993 for the non-polynomial condition).

It is an existence result only. It says nothing about how large $m$ must be, whether gradient descent will find those weights, or how well the network generalises from finite data.

## Depth vs width

Width alone is enough in principle; depth is often far more economical. A clean 1-D example uses the tent map $T(x)=2\,\mathrm{ReLU}(x)-4\,\mathrm{ReLU}(x-\tfrac12)$ on $[0,1]$, which needs two ReLU units. Composing it $k$ times gives a zigzag with $2^k$ linear pieces from only $2k$ hidden units.

A single hidden layer of $m$ ReLU units on a 1-D input is piecewise linear with at most $m$ kinks, so at most $m+1$ pieces. Matching the $k$-fold tent needs at least $2^k-1$ units: for $k=10$, 20 units in depth versus at least 1023 in width. Each layer folds the space produced by the previous one, so features are built from features.

**Strengths**

- Depth: compositional features, often exponentially fewer units for the same function
- Width: easier to optimise, more parallel per layer, and enough by the universal approximation theorem

**Limitations**

- Deep plain MLPs are hard to train (vanishing or exploding gradients) without careful initialization, normalization or skip connections
- Very wide shallow nets can need an impractical number of units and parameters

> **Representable is not the same as learnable:** The 2-2-1 XOR solution exists, but training that tiny network from random weights often fails: a ReLU unit can die or both hidden units can learn the same feature. In a check with He initialization, cross-entropy and 5000 full-batch gradient steps (lr 0.1) over 200 random seeds, 2 hidden units solved XOR in 18.5% of runs, 8 units in 97.5% and 16 units in all of them. Extra width makes the optimisation much easier.

```python
import torch
import torch.nn as nn

net = nn.Sequential(nn.Linear(2, 2), nn.ReLU(), nn.Linear(2, 1))
with torch.no_grad():
    net[0].weight.copy_(torch.tensor([[1., 1.], [1., 1.]]))
    net[0].bias.copy_(torch.tensor([0., -1.]))
    net[2].weight.copy_(torch.tensor([[1., -2.]]))
    net[2].bias.zero_()

X = torch.tensor([[0., 0.], [0., 1.], [1., 0.], [1., 1.]])
print(net(X).detach().squeeze())                    # tensor([0., 1., 1., 0.])
print(sum(p.numel() for p in net.parameters()))     # 9 = 2·(2+1) + 1·(2+1)
```

The hand-set 2-2-1 XOR network in PyTorch. nn.Linear stores W with shape (out, in).

**Question:** An MLP for MNIST has layers 784 → 256 → 10 (weights and biases in both layers). How many parameters does it have?

- A. 200,704
- B. 203,530
- C. 203,264
- D. 2,570

**Answer:** B. 203,530 — Layer 1: $256\cdot(784+1)=200{,}960$. Layer 2: $10\cdot(256+1)=2{,}570$. Total $203{,}530$. The first option forgets the biases and the second layer.

**Question:** You delete the ReLU from the 2-2-1 XOR network and are allowed to retrain every weight. What is the best accuracy on the four XOR points?

- A. 100%
- B. 75%
- C. 50%
- D. 25%

**Answer:** B. 75% — Without the nonlinearity the network is one linear classifier, which cannot separate XOR. It can still get 3 of 4 right (for example by computing OR), so 75% is the ceiling.

**Key points**

- Layer: $h=\varphi(Wx+b)$; an MLP chains such layers and ends with a linear readout.
- A hidden layer re-maps the inputs; XOR becomes linearly separable in (h₁, h₂) space.
- Hand-set XOR: W = [[1,1],[1,1]], b = (0, −1), ReLU, o = h₁ − 2h₂ gives 0, 1, 1, 0.
- Without a nonlinearity any stack of linear layers is a single linear layer.
- One wide hidden layer can approximate any continuous function, but depth often does it with exponentially fewer units.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
