Definition
Training an MLP means choosing the weights that minimise the average loss over a training set. In practice this is done by mini-batch stochastic gradient descent: repeatedly take a small random batch, run it forward, measure the loss, backpropagate to get , and step the weights against the gradient.
The MLP article showed what a hidden layer can represent, using XOR weights set by hand. Nobody sets the weights of a real network by hand. This article is about how they are found: the full loop from data to updated weights, written in batch matrix form, run live on 2-D datasets so you can watch the decision boundary form.
Backpropagation through hidden layers was derived more than once (Linnainmaa's reverse-mode differentiation in 1970, Werbos in 1974). Rumelhart, Hinton and Williams' 1986 paper showed that it makes hidden units learn useful internal representations, and that paper is what made MLP training standard practice.
The training loop
Mini-batch SGD for an MLP
- 1Initialise every weight matrix with small random values (Xavier for tanh, He for ReLU) and every bias with 0. Random values are required: see the symmetry pitfall below.
- 2Each epoch, shuffle the training examples and cut them into mini-batches of size .
- 3Forward: push the batch through every layer, caching the pre-activations and activations .
- 4Loss: compare the outputs with the labels, averaged over the batch.
- 5Backward: run backpropagation from the loss down to layer 1, giving the gradient of the batch loss for every and .
- 6Update: (or an Adam step, etc.). Repeat until the loss on held-out data stops improving.
The loop in matrix form
Stack the batch as rows, , with labels . For one hidden layer of width and a sigmoid output, the forward pass is three matrix expressions:
- weights, PyTorch's (out, in) layout
- pre-activations and activations of the hidden layer; row i belongs to example i
- the bias broadcast to every row
- mean binary cross-entropy over the batch
The backward pass mirrors it. With a mean loss, the output error carries the factor , and every weight gradient sums the per-example contributions through one matrix product:
- ; sigmoid with cross-entropy collapses to
- : the error sent back to each hidden unit through its outgoing weight
- local slope, for tanh, for ReLU
- , the shape of : a sum over the batch of outer products
Cost per step: the forward pass is about multiply-adds, and the backward pass about twice that. So one epoch costs roughly a forward pass over the whole dataset, whatever the batch size. Smaller batches give more updates per epoch, each one noisier.
Watch it train
Worked example: one step on a batch of two
A 2-3-1 tanh network with , , , , on the batch with and with . Learning rate .
| Quantity | Value |
|---|---|
| , so | |
| , | , |
The two examples pull in opposite directions. Example 1 is under-predicted () and example 2 over-predicted, so their rows of have opposite signs, and the weight gradient is their sum. After the step, , and . The loss falls from to , then , , , over the next four steps on the same batch.
What the hidden units learn
In a 2-D input each first-layer unit is a ridge: constant along the line and changing across it, softly for tanh and as a hinge for ReLU. The output logit is a weighted sum of ridges, so the decision boundary is pieced together from those lines, bent where they cross. Training moves the lines to wherever the data need a bend, and units whose outgoing weight is small hardly matter.
Width is the budget of ridges. With two units no run out of 50 fits the moons, and training stalls near 87.5%. From three tanh units the seeded run reaches 100%. The epoch at which it does so depends on the seed about as much as on the width, but the chance that a random seed succeeds rises steadily with width.
| width H | tanh: first epoch at 100% | ReLU: first epoch at 100% | tanh: seeds reaching ≥ 99% |
|---|---|---|---|
| 1 | never (87.5%) | never (86.9%) | 0 / 50 |
| 2 | never (87.5%) | never (88.8%) | 0 / 50 |
| 3 | 95 | never (88.8%) | 43 / 50 |
| 4 | 178 | never (86.9%) | 48 / 50 |
| 8 | 128 | 70 | 50 / 50 |
| 16 | 114 | 81 | 49 / 50 |
| 32 | 99 | 43 | 50 / 50 |
A non-convex loss surface
For a linear model with cross-entropy loss the loss is convex in the weights, so there is one basin. An MLP's is not, and the simplest proof is a symmetry. Permute the hidden units, permuting the rows of , and the entries of to match, and the network computes exactly the same function. Any minimum therefore has at least copies (times for tanh, since flipping the signs of a unit's in- and out-weights also changes nothing).
Train the default 2-8-1 tanh network to a loss of and reverse the order of its hidden units. The permuted copy has the identical loss . The point halfway between the two in weight space has loss and only 79.4% accuracy. A convex function could not be higher at a midpoint than at both endpoints, so the loss surface is not convex.
Non-convexity shows up in training as plateaus: long flat stretches near saddle points, followed by sudden drops. That is the step-shaped loss curve in the figure: about 0.29 at epoch 50, 0.115 at epoch 100, 0.011 at epoch 300. In high dimensions, saddles rather than bad local minima are thought to be the main obstacle (Dauphin et al., 2014), and over-parameterised (wider) networks are empirically much easier to optimise, as the seed column of the table shows.
1import torch, torch.nn as nn, torch.nn.functional as F23net = nn.Sequential(nn.Linear(2, 3), nn.Tanh(), nn.Linear(3, 1))4with torch.no_grad():5 net[0].weight.copy_(torch.tensor([[0.2, -0.4], [0.7, 0.1], [-0.5, 0.3]]))6 net[0].bias.copy_(torch.tensor([0., 0.1, -0.1]))7 net[2].weight.copy_(torch.tensor([[0.6, -0.3, 0.8]])); net[2].bias.zero_()8X, y = torch.tensor([[0.5, -1.], [1., 0.5]]), torch.tensor([[1.], [0.]])9loss = F.binary_cross_entropy_with_logits(net(X), y) # 0.6504 (mean over the batch)10loss.backward()11print(net[2].weight.grad) # [[-0.1317, 0.0310, 0.0854]]1213# the general loop: data -> forward -> loss -> backward -> update14def train(model, X, y, epochs=300, batch=16, lr=0.5):15 opt = torch.optim.SGD(model.parameters(), lr=lr)16 for epoch in range(epochs):17 perm = torch.randperm(len(X)) # reshuffle every epoch18 for s in range(0, len(X), batch):19 idx = perm[s:s + batch]20 loss = F.binary_cross_entropy_with_logits(model(X[idx]), y[idx])21 opt.zero_grad()22 loss.backward() # fills .grad for all 4 tensors23 opt.step()24 return model2526mlp = nn.Sequential(nn.Linear(2, 8), nn.Tanh(), nn.Linear(8, 1)) # 33 parameters27# nn.Linear's default init is a scaled uniform; nn.init.xavier_normal_ matches the figureCheck yourself
In the batch backward pass, what shape is for a batch of inputs with features and hidden units, and what does it hold?
Check yourself
You train the same MLP twice from different random seeds and get two networks with identical training loss but very different weights. Which statement is correct?
Remember
- ✓Training loop: shuffle → mini-batch → forward (cache Z, A) → loss → backward → update, repeated each epoch.
- ✓Batch form: , , , .
- ✓Hidden units are ridges along learned lines; the boundary is assembled from them, and width is how many bends are available.
- ✓The loss is non-convex: permuting units gives equal-loss copies, with a loss of 0.476 halfway between two copies at 0.011.
- ✓Random init breaks symmetry; identical weights stay identical. The learning rate decides between slow, fast and unstable.