The chain rule applied in reverse, reusing intermediate values to get every gradient in one backward pass.
8 min read624 wordscore2 viewsupdated 2026-10-09
Definition
Backpropagation is the algorithm that computes ∂L/∂θ for every parameter θ of a network by applying the chain rule layer by layer from the loss back to the input, reusing the activations cached by the forward pass and the error signals of the layers already processed.
Gradient descent needs the gradient of one scalar loss with respect to millions of parameters. Estimating each partial derivative by nudging that parameter costs a forward pass per parameter, which is hopeless at scale.
Backprop gets all of them for roughly the price of two extra forward passes. The trick is ordering: every weight's gradient shares the same downstream factors, so compute those once, starting from the loss, and pass them backward.
Backprop is only the gradient computation. The update that uses the gradient is a separate choice (SGD, Adam, …).
The chain rule, organised by layer
A weight in layer l affects the loss only through z(l), which affects it only through z(l+1), and so on. Define the error signal of a layer as the gradient of the loss with respect to its pre-activations:
δ(l)≡∂z(l)∂L∈Rnl
z(l)
pre-activation of layer l, z(l)=W(l)a(l−1)+b(l)
nl
number of units in layer l
Everything follows from three equations. Start at the output, step backward with the recurrence, and read off the parameter gradients at each layer:
error sent back through the same weights used going forward, giving ∂L/∂a(l)
φ′(z(l))
local slope of the activation, from the cached pre-activations
⊙
elementwise product (each unit scales its own error)
∂W(l)∂L=δ(l)a(l−1)⊤,∂b(l)∂L=δ(l)
δ(l)a(l−1)⊤
outer product, shape nl×nl−1, the same as W(l): entry (j,i) is (error at unit j) × (input from unit i)
For a sigmoid output with cross-entropy loss the output error simplifies: ∂L/∂y^=−y^y+1−y^1−y and σ′=y^(1−y^), whose product is just δ(L)=y^−y. Softmax with cross-entropy gives the same form.
With a mini-batch stored as rows, Δ(l)∈RB×nl holds one δ per example and ∂L/∂W(l)=Δ(l)⊤A(l−1) sums the per-example outer products (divide by B if the loss is a mean).
Why backward is the efficient direction
The loss is one number and the parameters are many. Going backward, each layer turns one vector δ(l+1) into the next with a single matrix–vector product, and every weight gradient in that layer reuses it. Going forward instead (propagating ∂z/∂θ) would need a separate pass for every parameter.
Count the matrix multiplies. A layer's forward pass is one product (Wa). Its backward pass is two: W⊤δ for the error and δa⊤ for the weight gradient, each the same size. So backward ≈ 2× forward, and a full training step costs about 3 forward passes, whether the network has ten parameters or ten billion. Central finite differences would need 2P forward passes.
fig.Press backward to walk from the loss to the inputs. Each node shows its δ once computed; each edge's colour and thickness show its gradient (rose: positive, so SGD will lower that weight; blue: negative, so SGD will raise it). Then press apply SGD step a few times and watch the loss fall from 0.581. Raise η to take bigger steps.
Worked example: the same 2-2-1 network
Continue the forward-propagation example: W(1)=[0.40.3−0.20.6], b(1)=(0.1,−0.1), W(2)=[0.7−0.5], b(2)=0.2, x=(0.5,1), y=1, sigmoid everywhere, cross-entropy loss. The forward pass cached a(1)=(0.5250,0.6570) and y^=0.5595, with L=0.5808.
Quantity
Computation
Value
δ(2)
y^−y=0.5595−1
−0.4405
∂L/∂W(2)
δ(2)a(1)⊤=−0.4405⋅(0.5250,0.6570)
(−0.2313, −0.2894)
∂L/∂b(2)
δ(2)
−0.4405
W(2)⊤δ(2)
(0.7,−0.5)⋅(−0.4405)
(−0.3084, 0.2203)
σ′(z(1))
a(1)⊙(1−a(1))
(0.2494, 0.2253)
δ(1)
(−0.3084⋅0.2494,0.2203⋅0.2253)
(−0.0769, 0.0496)
∂L/∂W(1)
δ(1)x⊤
[−0.03850.0248−0.07690.0496]
∂L/∂b(1)
δ(1)
(−0.0769, 0.0496)
Backward pass (rounded to 4 decimals; every gradient matches a central finite difference to within 1e-9)
The signs make sense. The output is too low (δ(2)<0), so the gradient says: raise w1(2), and make w2(2) less negative. Hidden unit 2 feeds the output through a negative weight, so its error flips sign: δ2(1)>0. The hidden errors are 6–9× smaller than the output error, and the factor σ′≤0.25 accounts for most of that shrinkage.
One SGD step, η = 0.5 (θ ← θ − η ∂L/∂θ)
1W(2): (0.7,−0.5)→(0.8156,−0.3553), and b(2): 0.2→0.4203.
2W(1): [0.40.3−0.20.6]→[0.41920.2876−0.16150.5752], and b(1): (0.1,−0.1)→(0.1385,−0.1248).
3New forward pass: y^=0.6541, so L=0.4245, down from 0.5808. Further steps give 0.3261, 0.2608, 0.2152.
python
1import torch, torch.nn as nn, torch.nn.functional as F
15opt.step() # next forward pass gives loss 0.4245
Autograd reproduces the table, then the optimizer applies the step
Frameworks do not hard-code these layer formulas. They record the forward pass as a computational graph and run reverse-mode automatic differentiation on it, which is backprop generalised to any composition of differentiable operations.
Check yourself
A hidden ReLU unit has z = −0.3 on some example. What is its δ for that example?
Check yourself
A network has P = 10⁶ parameters. Roughly how much computation does one exact gradient cost with backprop, versus central finite differences?
Remember
✓δ(l)=∂L/∂z(l); for sigmoid or softmax with cross-entropy, δ(L)=y^−y.
✓Recurrence: δ(l)=(W(l+1)⊤δ(l+1))⊙φ′(z(l)).
✓Gradients: ∂L/∂W(l)=δ(l)a(l−1)⊤, ∂L/∂b(l)=δ(l).
✓Reverse order reuses each δ for a whole layer: backward ≈ 2× forward, independent of parameter count.
✓Worked example: one SGD step at η = 0.5 drops the loss from 0.5808 to 0.4245.