Multilayer perceptron (MLP) training

Training an MLP is one loop repeated thousands of times: sample a mini-batch, forward, loss, backprop, update. Hidden units become learned features, on a non-convex loss surface.

10 min read1027 wordscore0 viewsupdated 2026-10-11

Definition

Training an MLP means choosing the weights θ={W(l),b(l)}\theta=\{W^{(l)},b^{(l)}\} that minimise the average loss 1n∑iℓ(fθ(xi),yi)\frac1n\sum_i\ell\big(f_\theta(x_i),y_i\big) over a training set. In practice this is done by mini-batch stochastic gradient descent: repeatedly take a small random batch, run it forward, measure the loss, backpropagate to get ∇θ\nabla_\theta, and step the weights against the gradient.

The MLP article showed what a hidden layer can represent, using XOR weights set by hand. Nobody sets the weights of a real network by hand. This article is about how they are found: the full loop from data to updated weights, written in batch matrix form, run live on 2-D datasets so you can watch the decision boundary form.

Backpropagation through hidden layers was derived more than once (Linnainmaa's reverse-mode differentiation in 1970, Werbos in 1974). Rumelhart, Hinton and Williams' 1986 paper showed that it makes hidden units learn useful internal representations, and that paper is what made MLP training standard practice.

The training loop

Mini-batch SGD for an MLP

  1. 1Initialise every weight matrix with small random values (Xavier for tanh, He for ReLU) and every bias with 0. Random values are required: see the symmetry pitfall below.
  2. 2Each epoch, shuffle the nn training examples and cut them into mini-batches of size BB.
  3. 3Forward: push the batch through every layer, caching the pre-activations Z(l)Z^{(l)} and activations A(l)A^{(l)}.
  4. 4Loss: compare the outputs with the labels, averaged over the batch.
  5. 5Backward: run backpropagation from the loss down to layer 1, giving the gradient of the batch loss for every W(l)W^{(l)} and b(l)b^{(l)}.
  6. 6Update: θ←θ−η ∇θLbatch\theta\leftarrow\theta-\eta\,\nabla_\theta\mathcal{L}_{\text{batch}} (or an Adam step, etc.). Repeat until the loss on held-out data stops improving.

The loop in matrix form

Stack the batch as rows, X∈RB×dX\in\mathbb{R}^{B\times d}, with labels y∈{0,1}By\in\{0,1\}^B. For one hidden layer of width HH and a sigmoid output, the forward pass is three matrix expressions:

Z1=XW1⊤+1b1⊤,A1=φ(Z1),y^=σ ⁣(A1W2⊤+b2),L=−1B∑i=1B[yilog⁡y^i+(1−yi)log⁡(1−y^i)]Z_1=XW_1^{\top}+\mathbf 1b_1^{\top},\qquad A_1=\varphi(Z_1),\qquad \hat y=\sigma\!\big(A_1W_2^{\top}+b_2\big),\qquad \mathcal{L}=-\frac1B\sum_{i=1}^{B}\big[y_i\log\hat y_i+(1-y_i)\log(1-\hat y_i)\big]
W1∈RH×d, W2∈R1×HW_1\in\mathbb{R}^{H\times d},\ W_2\in\mathbb{R}^{1\times H}
weights, PyTorch's (out, in) layout
Z1,A1∈RB×HZ_1,A_1\in\mathbb{R}^{B\times H}
pre-activations and activations of the hidden layer; row i belongs to example i
1b1⊤\mathbf 1b_1^\top
the bias broadcast to every row
L\mathcal{L}
mean binary cross-entropy over the batch

The backward pass mirrors it. With a mean loss, the output error carries the factor 1/B1/B, and every weight gradient sums the per-example contributions through one matrix product:

Δ2=1B(y^−y),∇W2=Δ2⊤A1,∇b2=1⊤Δ2,Δ1=(Δ2W2)⊙φ′(Z1),∇W1=Δ1⊤X,∇b1=1⊤Δ1\Delta_2=\tfrac1B(\hat y-y),\quad \nabla_{W_2}=\Delta_2^{\top}A_1,\quad \nabla_{b_2}=\mathbf 1^{\top}\Delta_2,\quad \Delta_1=(\Delta_2W_2)\odot\varphi'(Z_1),\quad \nabla_{W_1}=\Delta_1^{\top}X,\quad \nabla_{b_1}=\mathbf 1^{\top}\Delta_1
Δ2∈RB×1\Delta_2\in\mathbb{R}^{B\times 1}
∂L/∂(output logits)\partial\mathcal{L}/\partial(\text{output logits}); sigmoid with cross-entropy collapses to y^−y\hat y-y
Δ2W2\Delta_2W_2
B×HB\times H: the error sent back to each hidden unit through its outgoing weight
φ′(Z1)\varphi'(Z_1)
local slope, 1−A1⊙21-A_1^{\odot2} for tanh, 1[Z1>0]\mathbb 1[Z_1>0] for ReLU
Δ1⊤X\Delta_1^\top X
H×dH\times d, the shape of W1W_1: a sum over the batch of outer products δixi⊤\delta_i x_i^\top

Cost per step: the forward pass is about BdH+BHBdH+BH multiply-adds, and the backward pass about twice that. So one epoch costs roughly 3×3\times a forward pass over the whole dataset, whatever the batch size. Smaller batches give more updates per epoch, each one noisier.

Watch it train

2 → 8 tanh → 1 sigmoid · 33 params · batch 16shade = p(class 1) · dashed: unit lines w·x + b = 0epoch 0 · step 0 · loss 0.842 · acc 18%train loss · accuracy per epoch0.80100 epochshidden units h_j(x) · weight to output−0.63−0.12+0.25−0.15−0.64+1.30−0.11+0.53
data
φinit
fig.A 2-H-1 MLP trained live in your browser by mini-batch SGD (batch 16, 160 points, so 10 steps per epoch) with the backward pass written out above. Shading is the predicted probability, and the violet dashed lines are each hidden unit's line w·x + b = 0. The tiles show what the first eight hidden units compute over the input box, with each unit's outgoing weight underneath. Defaults (moons, tanh, width 8, η = 0.5): the loss sits near 0.29 for about 50 epochs, then drops, and training accuracy first reaches 100% at epoch 128. Try width 2 (stuck at 87.5%), ReLU at width 4 (stuck too, a bad seed), init 'all 0.5' (all tiles stay identical), and spiral with width 16.

Worked example: one step on a batch of two

A 2-3-1 tanh network with W1=[0.2−0.40.70.1−0.50.3]W_1=\begin{bmatrix}0.2&-0.4\\0.7&0.1\\-0.5&0.3\end{bmatrix}, b1=(0, 0.1, −0.1)b_1=(0,\,0.1,\,-0.1), W2=(0.6, −0.3, 0.8)W_2=(0.6,\,-0.3,\,0.8), b2=0b_2=0, on the batch x1=(0.5,−1)x_1=(0.5,-1) with y1=1y_1=1 and x2=(1,0.5)x_2=(1,0.5) with y2=0y_2=0. Learning rate η=0.5\eta=0.5.

QuantityValue
Z1Z_1[0.50.35−0.6500.85−0.45]\begin{bmatrix}0.5&0.35&-0.65\\0&0.85&-0.45\end{bmatrix}
A1=tanh⁡Z1A_1=\tanh Z_1[0.46210.3364−0.571700.6911−0.4219]\begin{bmatrix}0.4621&0.3364&-0.5717\\0&0.6911&-0.4219\end{bmatrix}
y^\hat y(0.4302, 0.3671)(0.4302,\ 0.3671), so L=0.6504\mathcal{L}=0.6504
Δ2=12(y^−y)\Delta_2=\tfrac12(\hat y-y)(−0.2849, 0.1835)(-0.2849,\ 0.1835)
∇W2=Δ2⊤A1\nabla_{W_2}=\Delta_2^\top A_1, ∇b2\nabla_{b_2}(−0.1317, 0.0310, 0.0854)(-0.1317,\ 0.0310,\ 0.0854), −0.1014-0.1014
Δ1=(Δ2W2)⊙(1−A1⊙2)\Delta_1=(\Delta_2W_2)\odot(1-A_1^{\odot2})[−0.13440.0758−0.15340.1101−0.02880.1207]\begin{bmatrix}-0.1344&0.0758&-0.1534\\0.1101&-0.0288&0.1207\end{bmatrix}
∇W1=Δ1⊤X\nabla_{W_1}=\Delta_1^\top X[0.04290.18950.0091−0.09020.04400.2138]\begin{bmatrix}0.0429&0.1895\\0.0091&-0.0902\\0.0440&0.2138\end{bmatrix}
∇b1=1⊤Δ1\nabla_{b_1}=\mathbf 1^\top\Delta_1(−0.0243, 0.0470, −0.0327)(-0.0243,\ 0.0470,\ -0.0327)
Every entry computed directly, rounded to 4 decimals. Rows of Z₁, A₁ and Δ₁ are the two examples.

The two examples pull in opposite directions. Example 1 is under-predicted (Δ2<0\Delta_2<0) and example 2 over-predicted, so their rows of Δ1\Delta_1 have opposite signs, and the weight gradient is their sum. After the step, W2=(0.6658,−0.3155,0.7573)W_2=(0.6658,-0.3155,0.7573), b2=0.0507b_2=0.0507 and y^=(0.4871,0.3637)\hat y=(0.4871,0.3637). The loss falls from 0.65040.6504 to 0.58560.5856, then 0.52520.5252, 0.46590.4659, 0.40770.4077, 0.35250.3525 over the next four steps on the same batch.

What the hidden units learn

In a 2-D input each first-layer unit hj=φ(wj⋅x+bj)h_j=\varphi(w_j\cdot x+b_j) is a ridge: constant along the line wj⋅x+bj=0w_j\cdot x+b_j=0 and changing across it, softly for tanh and as a hinge for ReLU. The output logit ∑jW2,jhj+b2\sum_jW_{2,j}h_j+b_2 is a weighted sum of ridges, so the decision boundary is pieced together from those lines, bent where they cross. Training moves the lines to wherever the data need a bend, and units whose outgoing weight is small hardly matter.

Width is the budget of ridges. With two units no run out of 50 fits the moons, and training stalls near 87.5%. From three tanh units the seeded run reaches 100%. The epoch at which it does so depends on the seed about as much as on the width, but the chance that a random seed succeeds rises steadily with width.

width Htanh: first epoch at 100%ReLU: first epoch at 100%tanh: seeds reaching ≥ 99%
1never (87.5%)never (86.9%)0 / 50
2never (87.5%)never (88.8%)0 / 50
395never (88.8%)43 / 50
4178never (86.9%)48 / 50
81287050 / 50
161148149 / 50
32994350 / 50
Two moons, 160 points, batch 16, η = 0.5, 300 epochs, seeded He/Xavier init (the figure's settings). The last column re-runs tanh with 50 different initial seeds.

A non-convex loss surface

For a linear model with cross-entropy loss the loss is convex in the weights, so there is one basin. An MLP's is not, and the simplest proof is a symmetry. Permute the hidden units, permuting the rows of W1W_1, b1b_1 and the entries of W2W_2 to match, and the network computes exactly the same function. Any minimum therefore has at least H!H! copies (times 2H2^H for tanh, since flipping the signs of a unit's in- and out-weights also changes nothing).

Train the default 2-8-1 tanh network to a loss of 0.01120.0112 and reverse the order of its hidden units. The permuted copy has the identical loss 0.01120.0112. The point halfway between the two in weight space has loss 0.4760.476 and only 79.4% accuracy. A convex function could not be higher at a midpoint than at both endpoints, so the loss surface is not convex.

Non-convexity shows up in training as plateaus: long flat stretches near saddle points, followed by sudden drops. That is the step-shaped loss curve in the figure: about 0.29 at epoch 50, 0.115 at epoch 100, 0.011 at epoch 300. In high dimensions, saddles rather than bad local minima are thought to be the main obstacle (Dauphin et al., 2014), and over-parameterised (wider) networks are empirically much easier to optimise, as the seed column of the table shows.

python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3net = nn.Sequential(nn.Linear(2, 3), nn.Tanh(), nn.Linear(3, 1))
4with torch.no_grad():
5 net[0].weight.copy_(torch.tensor([[0.2, -0.4], [0.7, 0.1], [-0.5, 0.3]]))
6 net[0].bias.copy_(torch.tensor([0., 0.1, -0.1]))
7 net[2].weight.copy_(torch.tensor([[0.6, -0.3, 0.8]])); net[2].bias.zero_()
8X, y = torch.tensor([[0.5, -1.], [1., 0.5]]), torch.tensor([[1.], [0.]])
9loss = F.binary_cross_entropy_with_logits(net(X), y) # 0.6504 (mean over the batch)
10loss.backward()
11print(net[2].weight.grad) # [[-0.1317, 0.0310, 0.0854]]
12
13# the general loop: data -> forward -> loss -> backward -> update
14def train(model, X, y, epochs=300, batch=16, lr=0.5):
15 opt = torch.optim.SGD(model.parameters(), lr=lr)
16 for epoch in range(epochs):
17 perm = torch.randperm(len(X)) # reshuffle every epoch
18 for s in range(0, len(X), batch):
19 idx = perm[s:s + batch]
20 loss = F.binary_cross_entropy_with_logits(model(X[idx]), y[idx])
21 opt.zero_grad()
22 loss.backward() # fills .grad for all 4 tensors
23 opt.step()
24 return model
25
26mlp = nn.Sequential(nn.Linear(2, 8), nn.Tanh(), nn.Linear(8, 1)) # 33 parameters
27# nn.Linear's default init is a scaled uniform; nn.init.xavier_normal_ matches the figure
Check the worked example with autograd, then the generic training loop

Check yourself

In the batch backward pass, what shape is Δ1⊤X\Delta_1^\top X for a batch of B=64B=64 inputs with d=2d=2 features and H=8H=8 hidden units, and what does it hold?

Check yourself

You train the same MLP twice from different random seeds and get two networks with identical training loss but very different weights. Which statement is correct?

Remember

  • ✓Training loop: shuffle → mini-batch → forward (cache Z, A) → loss → backward → update, repeated each epoch.
  • ✓Batch form: Δ2=(y^−y)/B\Delta_2=(\hat y-y)/B, ∇W2=Δ2⊤A1\nabla W_2=\Delta_2^\top A_1, Δ1=(Δ2W2)⊙φ′(Z1)\Delta_1=(\Delta_2W_2)\odot\varphi'(Z_1), ∇W1=Δ1⊤X\nabla W_1=\Delta_1^\top X.
  • ✓Hidden units are ridges along learned lines; the boundary is assembled from them, and width is how many bends are available.
  • ✓The loss is non-convex: permuting units gives equal-loss copies, with a loss of 0.476 halfway between two copies at 0.011.
  • ✓Random init breaks symmetry; identical weights stay identical. The learning rate decides between slow, fast and unstable.
← back to Deep Learning roadmap