Backpropagation

The chain rule applied in reverse, reusing intermediate values to get every gradient in one backward pass.

8 min read624 wordscore2 viewsupdated 2026-10-09

Definition

Backpropagation is the algorithm that computes ∂L/∂θ\partial\mathcal{L}/\partial\theta for every parameter θ\theta of a network by applying the chain rule layer by layer from the loss back to the input, reusing the activations cached by the forward pass and the error signals of the layers already processed.

Gradient descent needs the gradient of one scalar loss with respect to millions of parameters. Estimating each partial derivative by nudging that parameter costs a forward pass per parameter, which is hopeless at scale.

Backprop gets all of them for roughly the price of two extra forward passes. The trick is ordering: every weight's gradient shares the same downstream factors, so compute those once, starting from the loss, and pass them backward.

Backprop is only the gradient computation. The update that uses the gradient is a separate choice (SGD, Adam, …).

The chain rule, organised by layer

A weight in layer ll affects the loss only through z(l)z^{(l)}, which affects it only through z(l+1)z^{(l+1)}, and so on. Define the error signal of a layer as the gradient of the loss with respect to its pre-activations:

δ(l)  ≡  ∂L∂z(l)∈Rnl\delta^{(l)}\;\equiv\;\frac{\partial\mathcal{L}}{\partial z^{(l)}}\in\mathbb{R}^{n_l}
z(l)z^{(l)}
pre-activation of layer ll, z(l)=W(l)a(l−1)+b(l)z^{(l)}=W^{(l)}a^{(l-1)}+b^{(l)}
nln_l
number of units in layer ll

Everything follows from three equations. Start at the output, step backward with the recurrence, and read off the parameter gradients at each layer:

δ(L)=∇a(L)L⊙φ′(z(L)),δ(l)=(W(l+1)⊤ δ(l+1))⊙φ′(z(l))\delta^{(L)}=\nabla_{a^{(L)}}\mathcal{L}\odot\varphi'\big(z^{(L)}\big),\qquad \delta^{(l)}=\Big(W^{(l+1)\top}\,\delta^{(l+1)}\Big)\odot\varphi'\big(z^{(l)}\big)
W(l+1)⊤δ(l+1)W^{(l+1)\top}\delta^{(l+1)}
error sent back through the same weights used going forward, giving ∂L/∂a(l)\partial\mathcal{L}/\partial a^{(l)}
φ′(z(l))\varphi'(z^{(l)})
local slope of the activation, from the cached pre-activations
⊙\odot
elementwise product (each unit scales its own error)
∂L∂W(l)=δ(l) a(l−1)⊤,∂L∂b(l)=δ(l)\frac{\partial\mathcal{L}}{\partial W^{(l)}}=\delta^{(l)}\,a^{(l-1)\top},\qquad \frac{\partial\mathcal{L}}{\partial b^{(l)}}=\delta^{(l)}
δ(l)a(l−1)⊤\delta^{(l)}a^{(l-1)\top}
outer product, shape nl×nl−1n_l\times n_{l-1}, the same as W(l)W^{(l)}: entry (j,i)(j,i) is (error at unit jj) × (input from unit ii)

For a sigmoid output with cross-entropy loss the output error simplifies: ∂L/∂y^=−yy^+1−y1−y^\partial\mathcal{L}/\partial\hat y=-\tfrac{y}{\hat y}+\tfrac{1-y}{1-\hat y} and σ′=y^(1−y^)\sigma'=\hat y(1-\hat y), whose product is just δ(L)=y^−y\delta^{(L)}=\hat y-y. Softmax with cross-entropy gives the same form.

With a mini-batch stored as rows, Δ(l)∈RB×nl\Delta^{(l)}\in\mathbb{R}^{B\times n_l} holds one δ\delta per example and ∂L/∂W(l)=Δ(l)⊤A(l−1)\partial\mathcal{L}/\partial W^{(l)}=\Delta^{(l)\top}A^{(l-1)} sums the per-example outer products (divide by BB if the loss is a mean).

Why backward is the efficient direction

The loss is one number and the parameters are many. Going backward, each layer turns one vector δ(l+1)\delta^{(l+1)} into the next with a single matrix–vector product, and every weight gradient in that layer reuses it. Going forward instead (propagating ∂z/∂θ\partial z/\partial\theta) would need a separate pass for every parameter.

Count the matrix multiplies. A layer's forward pass is one product (WaW a). Its backward pass is two: W⊤δW^\top\delta for the error and δa⊤\delta a^\top for the weight gradient, each the same size. So backward ≈ 2× forward, and a full training step costs about 3 forward passes, whether the network has ten parameters or ten billion. Central finite differences would need 2P2P forward passes.

step 0/4 · forward pass done, activations cachedinputhidden (σ)output (σ)lossx10.50x21.00h10.525h20.657ŷ0.559y = 10.581
ŷ = 0.559, L = −log ŷ = 0.581
cached: a⁽¹⁾ = (0.525, 0.657), x = (0.5, 1.0)
|
loss after 0 SGD steps: 0.581
edge/node colour: positive gradient (step lowers it) · negative (step raises it) · thickness = |∂L/∂w|
fig.Press backward to walk from the loss to the inputs. Each node shows its δ once computed; each edge's colour and thickness show its gradient (rose: positive, so SGD will lower that weight; blue: negative, so SGD will raise it). Then press apply SGD step a few times and watch the loss fall from 0.581. Raise η to take bigger steps.

Worked example: the same 2-2-1 network

Continue the forward-propagation example: W(1)=[0.4−0.20.30.6]W^{(1)}=\begin{bmatrix}0.4&-0.2\\0.3&0.6\end{bmatrix}, b(1)=(0.1,−0.1)b^{(1)}=(0.1,-0.1), W(2)=[0.7−0.5]W^{(2)}=\begin{bmatrix}0.7&-0.5\end{bmatrix}, b(2)=0.2b^{(2)}=0.2, x=(0.5,1)x=(0.5,1), y=1y=1, sigmoid everywhere, cross-entropy loss. The forward pass cached a(1)=(0.5250, 0.6570)a^{(1)}=(0.5250,\,0.6570) and y^=0.5595\hat y=0.5595, with L=0.5808\mathcal{L}=0.5808.

QuantityComputationValue
δ(2)\delta^{(2)}y^−y=0.5595−1\hat y-y=0.5595-1−0.4405
∂L/∂W(2)\partial\mathcal{L}/\partial W^{(2)}δ(2)a(1)⊤=−0.4405⋅(0.5250, 0.6570)\delta^{(2)}a^{(1)\top}=-0.4405\cdot(0.5250,\,0.6570)(−0.2313, −0.2894)
∂L/∂b(2)\partial\mathcal{L}/\partial b^{(2)}δ(2)\delta^{(2)}−0.4405
W(2)⊤δ(2)W^{(2)\top}\delta^{(2)}(0.7, −0.5)⋅(−0.4405)(0.7,\,-0.5)\cdot(-0.4405)(−0.3084, 0.2203)
σ′(z(1))\sigma'(z^{(1)})a(1)⊙(1−a(1))a^{(1)}\odot(1-a^{(1)})(0.2494, 0.2253)
δ(1)\delta^{(1)}(−0.3084⋅0.2494,  0.2203⋅0.2253)(-0.3084\cdot0.2494,\;0.2203\cdot0.2253)(−0.0769, 0.0496)
∂L/∂W(1)\partial\mathcal{L}/\partial W^{(1)}δ(1)x⊤\delta^{(1)}x^\top[−0.0385−0.07690.02480.0496]\begin{bmatrix}-0.0385&-0.0769\\0.0248&0.0496\end{bmatrix}
∂L/∂b(1)\partial\mathcal{L}/\partial b^{(1)}δ(1)\delta^{(1)}(−0.0769, 0.0496)
Backward pass (rounded to 4 decimals; every gradient matches a central finite difference to within 1e-9)

The signs make sense. The output is too low (δ(2)<0\delta^{(2)}<0), so the gradient says: raise w1(2)w^{(2)}_1, and make w2(2)w^{(2)}_2 less negative. Hidden unit 2 feeds the output through a negative weight, so its error flips sign: δ2(1)>0\delta^{(1)}_2>0. The hidden errors are 6–9× smaller than the output error, and the factor σ′≤0.25\sigma'\le0.25 accounts for most of that shrinkage.

One SGD step, η = 0.5 (θ ← θ − η ∂L/∂θ)

  1. 1W(2)W^{(2)}: (0.7, −0.5)→(0.8156, −0.3553)(0.7,\,-0.5)\to(0.8156,\,-0.3553), and b(2)b^{(2)}: 0.2→0.42030.2\to0.4203.
  2. 2W(1)W^{(1)}: [0.4−0.20.30.6]→[0.4192−0.16150.28760.5752]\begin{bmatrix}0.4&-0.2\\0.3&0.6\end{bmatrix}\to\begin{bmatrix}0.4192&-0.1615\\0.2876&0.5752\end{bmatrix}, and b(1)b^{(1)}: (0.1,−0.1)→(0.1385, −0.1248)(0.1,-0.1)\to(0.1385,\,-0.1248).
  3. 3New forward pass: y^=0.6541\hat y=0.6541, so L=0.4245\mathcal{L}=0.4245, down from 0.58080.5808. Further steps give 0.32610.3261, 0.26080.2608, 0.21520.2152.
python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3l1, l2 = nn.Linear(2, 2), nn.Linear(2, 1)
4with torch.no_grad():
5 l1.weight.copy_(torch.tensor([[0.4, -0.2], [0.3, 0.6]])); l1.bias.copy_(torch.tensor([0.1, -0.1]))
6 l2.weight.copy_(torch.tensor([[0.7, -0.5]])); l2.bias.copy_(torch.tensor([0.2]))
7opt = torch.optim.SGD([*l1.parameters(), *l2.parameters()], lr=0.5)
8
9x, y = torch.tensor([[0.5, 1.0]]), torch.tensor([[1.0]])
10loss = F.binary_cross_entropy_with_logits(l2(torch.sigmoid(l1(x))), y) # 0.5808
11opt.zero_grad()
12loss.backward() # one backward pass fills every .grad
13print(l2.weight.grad) # [[-0.2313, -0.2894]]
14print(l1.weight.grad) # [[-0.0385, -0.0769], [ 0.0248, 0.0496]]
15opt.step() # next forward pass gives loss 0.4245
Autograd reproduces the table, then the optimizer applies the step

Frameworks do not hard-code these layer formulas. They record the forward pass as a computational graph and run reverse-mode automatic differentiation on it, which is backprop generalised to any composition of differentiable operations.

Check yourself

A hidden ReLU unit has z = −0.3 on some example. What is its δ for that example?

Check yourself

A network has P = 10⁶ parameters. Roughly how much computation does one exact gradient cost with backprop, versus central finite differences?

Remember

  • ✓δ(l)=∂L/∂z(l)\delta^{(l)}=\partial\mathcal{L}/\partial z^{(l)}; for sigmoid or softmax with cross-entropy, δ(L)=y^−y\delta^{(L)}=\hat y-y.
  • ✓Recurrence: δ(l)=(W(l+1)⊤δ(l+1))⊙φ′(z(l))\delta^{(l)}=(W^{(l+1)\top}\delta^{(l+1)})\odot\varphi'(z^{(l)}).
  • ✓Gradients: ∂L/∂W(l)=δ(l)a(l−1)⊤\partial\mathcal{L}/\partial W^{(l)}=\delta^{(l)}a^{(l-1)\top}, ∂L/∂b(l)=δ(l)\partial\mathcal{L}/\partial b^{(l)}=\delta^{(l)}.
  • ✓Reverse order reuses each δ for a whole layer: backward ≈ 2× forward, independent of parameter count.
  • ✓Worked example: one SGD step at η = 0.5 drops the loss from 0.5808 to 0.4245.
← back to Deep Learning roadmap