# Multilayer perceptron (MLP) training

> Training an MLP is one loop repeated thousands of times: sample a mini-batch, forward, loss, backprop, update. Hidden units become learned features, on a non-convex loss surface.

Source: https://www.mlfoundry.in/learn/multilayer-perceptron-mlp-training  
Section: Deep Learning › Neural Network Algorithms  
Level: core · 10 min read · updated 2026-10-11

**Training an MLP:** means choosing the weights $\theta=\{W^{(l)},b^{(l)}\}$ that minimise the average loss $\frac1n\sum_i\ell\big(f_\theta(x_i),y_i\big)$ over a training set. In practice this is done by **mini-batch stochastic gradient descent**: repeatedly take a small random batch, run it forward, measure the loss, backpropagate to get $\nabla_\theta$, and step the weights against the gradient.

The MLP article showed *what* a hidden layer can represent, using XOR weights set by hand. Nobody sets the weights of a real network by hand. This article is about how they are **found**: the full loop from data to updated weights, written in batch matrix form, run live on 2-D datasets so you can watch the decision boundary form.

Backpropagation through hidden layers was derived more than once (Linnainmaa's reverse-mode differentiation in 1970, Werbos in 1974). Rumelhart, Hinton and Williams' 1986 paper showed that it makes hidden units learn useful internal representations, and that paper is what made MLP training standard practice.

## The training loop

**Mini-batch SGD for an MLP**

1. **Initialise** every weight matrix with small random values (Xavier for tanh, He for ReLU) and every bias with 0. Random values are required: see the symmetry pitfall below.
2. **Each epoch, shuffle** the $n$ training examples and cut them into mini-batches of size $B$.
3. **Forward**: push the batch through every layer, caching the pre-activations $Z^{(l)}$ and activations $A^{(l)}$.
4. **Loss**: compare the outputs with the labels, averaged over the batch.
5. **Backward**: run backpropagation from the loss down to layer 1, giving the gradient of the batch loss for every $W^{(l)}$ and $b^{(l)}$.
6. **Update**: $\theta\leftarrow\theta-\eta\,\nabla_\theta\mathcal{L}_{\text{batch}}$ (or an Adam step, etc.). Repeat until the loss on held-out data stops improving.

## The loop in matrix form

Stack the batch as rows, $X\in\mathbb{R}^{B\times d}$, with labels $y\in\{0,1\}^B$. For one hidden layer of width $H$ and a sigmoid output, the forward pass is three matrix expressions:

$$
Z_1=XW_1^{\top}+\mathbf 1b_1^{\top},\qquad A_1=\varphi(Z_1),\qquad \hat y=\sigma\!\big(A_1W_2^{\top}+b_2\big),\qquad \mathcal{L}=-\frac1B\sum_{i=1}^{B}\big[y_i\log\hat y_i+(1-y_i)\log(1-\hat y_i)\big]
$$

- $W_1\in\mathbb{R}^{H\times d},\ W_2\in\mathbb{R}^{1\times H}$: weights, PyTorch's (out, in) layout
- $Z_1,A_1\in\mathbb{R}^{B\times H}$: pre-activations and activations of the hidden layer; row i belongs to example i
- $\mathbf 1b_1^\top$: the bias broadcast to every row
- $\mathcal{L}$: mean binary cross-entropy over the batch

The backward pass mirrors it. With a mean loss, the output error carries the factor $1/B$, and every weight gradient sums the per-example contributions through one matrix product:

$$
\Delta_2=\tfrac1B(\hat y-y),\quad \nabla_{W_2}=\Delta_2^{\top}A_1,\quad \nabla_{b_2}=\mathbf 1^{\top}\Delta_2,\quad \Delta_1=(\Delta_2W_2)\odot\varphi'(Z_1),\quad \nabla_{W_1}=\Delta_1^{\top}X,\quad \nabla_{b_1}=\mathbf 1^{\top}\Delta_1
$$

- $\Delta_2\in\mathbb{R}^{B\times 1}$: $\partial\mathcal{L}/\partial(\text{output logits})$; sigmoid with cross-entropy collapses to $\hat y-y$
- $\Delta_2W_2$: $B\times H$: the error sent back to each hidden unit through its outgoing weight
- $\varphi'(Z_1)$: local slope, $1-A_1^{\odot2}$ for tanh, $\mathbb 1[Z_1>0]$ for ReLU
- $\Delta_1^\top X$: $H\times d$, the shape of $W_1$: a sum over the batch of outer products $\delta_i x_i^\top$

Cost per step: the forward pass is about $BdH+BH$ multiply-adds, and the backward pass about twice that. So one epoch costs roughly $3\times$ a forward pass over the whole dataset, whatever the batch size. Smaller batches give more updates per epoch, each one noisier.

## Watch it train

> Figure: A 2-H-1 MLP trained live in your browser by mini-batch SGD (batch 16, 160 points, so 10 steps per epoch) with the backward pass written out above. Shading is the predicted probability, and the violet dashed lines are each hidden unit's line w·x + b = 0. The tiles show what the first eight hidden units compute over the input box, with each unit's outgoing weight underneath. Defaults (moons, tanh, width 8, η = 0.5): the loss sits near 0.29 for about 50 epochs, then drops, and training accuracy first reaches 100% at epoch 128. Try width 2 (stuck at 87.5%), ReLU at width 4 (stuck too, a bad seed), init 'all 0.5' (all tiles stay identical), and spiral with width 16.

## Worked example: one step on a batch of two

A 2-3-1 tanh network with $W_1=\begin{bmatrix}0.2&-0.4\\0.7&0.1\\-0.5&0.3\end{bmatrix}$, $b_1=(0,\,0.1,\,-0.1)$, $W_2=(0.6,\,-0.3,\,0.8)$, $b_2=0$, on the batch $x_1=(0.5,-1)$ with $y_1=1$ and $x_2=(1,0.5)$ with $y_2=0$. Learning rate $\eta=0.5$.

Every entry computed directly, rounded to 4 decimals. Rows of Z₁, A₁ and Δ₁ are the two examples.

| Quantity | Value |
| --- | --- |
| $Z_1$ | $\begin{bmatrix}0.5&0.35&-0.65\\0&0.85&-0.45\end{bmatrix}$ |
| $A_1=\tanh Z_1$ | $\begin{bmatrix}0.4621&0.3364&-0.5717\\0&0.6911&-0.4219\end{bmatrix}$ |
| $\hat y$ | $(0.4302,\ 0.3671)$, so $\mathcal{L}=0.6504$ |
| $\Delta_2=\tfrac12(\hat y-y)$ | $(-0.2849,\ 0.1835)$ |
| $\nabla_{W_2}=\Delta_2^\top A_1$, $\nabla_{b_2}$ | $(-0.1317,\ 0.0310,\ 0.0854)$, $-0.1014$ |
| $\Delta_1=(\Delta_2W_2)\odot(1-A_1^{\odot2})$ | $\begin{bmatrix}-0.1344&0.0758&-0.1534\\0.1101&-0.0288&0.1207\end{bmatrix}$ |
| $\nabla_{W_1}=\Delta_1^\top X$ | $\begin{bmatrix}0.0429&0.1895\\0.0091&-0.0902\\0.0440&0.2138\end{bmatrix}$ |
| $\nabla_{b_1}=\mathbf 1^\top\Delta_1$ | $(-0.0243,\ 0.0470,\ -0.0327)$ |

The two examples pull in opposite directions. Example 1 is under-predicted ($\Delta_2<0$) and example 2 over-predicted, so their rows of $\Delta_1$ have opposite signs, and the weight gradient is their **sum**. After the step, $W_2=(0.6658,-0.3155,0.7573)$, $b_2=0.0507$ and $\hat y=(0.4871,0.3637)$. The loss falls from $0.6504$ to $0.5856$, then $0.5252$, $0.4659$, $0.4077$, $0.3525$ over the next four steps on the same batch.

## What the hidden units learn

In a 2-D input each first-layer unit $h_j=\varphi(w_j\cdot x+b_j)$ is a **ridge**: constant along the line $w_j\cdot x+b_j=0$ and changing across it, softly for tanh and as a hinge for ReLU. The output logit $\sum_jW_{2,j}h_j+b_2$ is a weighted sum of ridges, so the decision boundary is pieced together from those lines, bent where they cross. Training moves the lines to wherever the data need a bend, and units whose outgoing weight is small hardly matter.

Width is the budget of ridges. With two units no run out of 50 fits the moons, and training stalls near 87.5%. From three tanh units the seeded run reaches 100%. The epoch at which it does so depends on the seed about as much as on the width, but the chance that a random seed succeeds rises steadily with width.

Two moons, 160 points, batch 16, η = 0.5, 300 epochs, seeded He/Xavier init (the figure's settings). The last column re-runs tanh with 50 different initial seeds.

| width H | tanh: first epoch at 100% | ReLU: first epoch at 100% | tanh: seeds reaching ≥ 99% |
| --- | --- | --- | --- |
| 1 | never (87.5%) | never (86.9%) | 0 / 50 |
| 2 | never (87.5%) | never (88.8%) | 0 / 50 |
| 3 | 95 | never (88.8%) | 43 / 50 |
| 4 | 178 | never (86.9%) | 48 / 50 |
| 8 | 128 | 70 | 50 / 50 |
| 16 | 114 | 81 | 49 / 50 |
| 32 | 99 | 43 | 50 / 50 |

## A non-convex loss surface

For a linear model with cross-entropy loss the loss is convex in the weights, so there is one basin. An MLP's is not, and the simplest proof is a **symmetry**. Permute the hidden units, permuting the rows of $W_1$, $b_1$ and the entries of $W_2$ to match, and the network computes exactly the same function. Any minimum therefore has at least $H!$ copies (times $2^H$ for tanh, since flipping the signs of a unit's in- and out-weights also changes nothing).

Train the default 2-8-1 tanh network to a loss of $0.0112$ and reverse the order of its hidden units. The permuted copy has the identical loss $0.0112$. The point halfway between the two in weight space has loss $0.476$ and only 79.4% accuracy. A convex function could not be higher at a midpoint than at both endpoints, so the loss surface is not convex.

Non-convexity shows up in training as **plateaus**: long flat stretches near saddle points, followed by sudden drops. That is the step-shaped loss curve in the figure: about 0.29 at epoch 50, 0.115 at epoch 100, 0.011 at epoch 300. In high dimensions, saddles rather than bad local minima are thought to be the main obstacle (Dauphin et al., 2014), and over-parameterised (wider) networks are empirically much easier to optimise, as the seed column of the table shows.

> **Identical initial weights never become different:** If every hidden unit starts with the same weights, every unit gets the same $\delta$ and therefore the same gradient, so they stay identical forever. The width-8 network then behaves like a width-1 network. Select "all 0.5" in the figure: after 300 epochs all eight tiles are still the same picture and accuracy is stuck at 86.9%. Initial weights must be random to **break symmetry**. All-zero weights are worse still: with $W_2=0$ no error reaches the hidden layer at all.

> **The learning rate is a stability knob:** On moons (tanh, width 8) the loss after 100 epochs is $0.291$ at $\eta=0.05$, $0.115$ at $0.5$, $0.014$ at $2$ and $0.004$ at $4$. On the harder spiral with width 16, $\eta=1$ reaches 100% within 300 epochs, but $\eta=2$ is still at 74% after 1000 epochs, bouncing between configurations instead of descending. The best rate depends on the problem, so tune it first.

```python
import torch, torch.nn as nn, torch.nn.functional as F

net = nn.Sequential(nn.Linear(2, 3), nn.Tanh(), nn.Linear(3, 1))
with torch.no_grad():
    net[0].weight.copy_(torch.tensor([[0.2, -0.4], [0.7, 0.1], [-0.5, 0.3]]))
    net[0].bias.copy_(torch.tensor([0., 0.1, -0.1]))
    net[2].weight.copy_(torch.tensor([[0.6, -0.3, 0.8]])); net[2].bias.zero_()
X, y = torch.tensor([[0.5, -1.], [1., 0.5]]), torch.tensor([[1.], [0.]])
loss = F.binary_cross_entropy_with_logits(net(X), y)    # 0.6504 (mean over the batch)
loss.backward()
print(net[2].weight.grad)                               # [[-0.1317, 0.0310, 0.0854]]

# the general loop: data -> forward -> loss -> backward -> update
def train(model, X, y, epochs=300, batch=16, lr=0.5):
    opt = torch.optim.SGD(model.parameters(), lr=lr)
    for epoch in range(epochs):
        perm = torch.randperm(len(X))                   # reshuffle every epoch
        for s in range(0, len(X), batch):
            idx = perm[s:s + batch]
            loss = F.binary_cross_entropy_with_logits(model(X[idx]), y[idx])
            opt.zero_grad()
            loss.backward()                             # fills .grad for all 4 tensors
            opt.step()
    return model

mlp = nn.Sequential(nn.Linear(2, 8), nn.Tanh(), nn.Linear(8, 1))   # 33 parameters
# nn.Linear's default init is a scaled uniform; nn.init.xavier_normal_ matches the figure
```

Check the worked example with autograd, then the generic training loop

**Question:** In the batch backward pass, what shape is $\Delta_1^\top X$ for a batch of $B=64$ inputs with $d=2$ features and $H=8$ hidden units, and what does it hold?

- A. 64 × 2: one gradient per example
- B. 8 × 2: the shape of W₁, the sum over the 64 examples of each example's outer product δᵢxᵢᵀ
- C. 8 × 64: the error of every unit on every example
- D. 2 × 8: the transpose of W₁

**Answer:** B. 8 × 2: the shape of W₁, the sum over the 64 examples of each example's outer product δᵢxᵢᵀ — $\Delta_1$ is $64\times8$, so $\Delta_1^\top X$ is $8\times2$, the same shape as $W_1$. Its entry $(j,k)$ is $\sum_i\delta_{ij}x_{ik}$: the per-example gradients summed (and already divided by $B$ through $\Delta_2$) in one matrix multiply.

**Question:** You train the same MLP twice from different random seeds and get two networks with identical training loss but very different weights. Which statement is correct?

- A. One of the runs must have a bug, because the loss has a unique minimiser
- B. This is expected: the loss is non-convex and, among other things, permuting hidden units gives different weights with the same function
- C. It only happens with ReLU
- D. It means the learning rate was too high

**Answer:** B. This is expected: the loss is non-convex and, among other things, permuting hidden units gives different weights with the same function — An MLP's loss has many minimisers. Permutation symmetry alone gives $H!$ weight settings for each function, and different seeds also find genuinely different functions that fit the training data equally well. Equal loss with different weights is normal.

**Key points**

- Training loop: shuffle → mini-batch → forward (cache Z, A) → loss → backward → update, repeated each epoch.
- Batch form: $\Delta_2=(\hat y-y)/B$, $\nabla W_2=\Delta_2^\top A_1$, $\Delta_1=(\Delta_2W_2)\odot\varphi'(Z_1)$, $\nabla W_1=\Delta_1^\top X$.
- Hidden units are ridges along learned lines; the boundary is assembled from them, and width is how many bends are available.
- The loss is non-convex: permuting units gives equal-loss copies, with a loss of 0.476 halfway between two copies at 0.011.
- Random init breaks symmetry; identical weights stay identical. The learning rate decides between slow, fast and unstable.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
