Definition
The chain rule states that if and are differentiable, the composition has derivative . With many variables, the derivative of an output with respect to an input is the sum over all paths between them of the product of local derivatives along each path.
At a glance
- Notation
- ; ; Jacobians
- Input → output
- derivatives of the pieces → derivative of the composition
- Key property
- local: each node only needs its own derivative and the gradient arriving from downstream
- Cost (reverse mode)
- all partial derivatives of a scalar loss for a small constant times the cost of one forward pass
- Prerequisites
- derivatives & gradients, function composition
- Used in
- backpropagation, autograd (PyTorch, JAX), sensitivity analysis, change of variables
The intuition
Rates of change multiply. If moves 3 units per unit of , and moves 2 units per unit of , then moves units per unit of . A composition is a pipeline, and each stage scales the incoming wiggle by its own local slope.
A neural network's loss is one long composition: parameters feed a linear layer, then an activation, then another layer, then a loss. To learn how the loss responds to one weight, follow the path from that weight to the loss and multiply the local slopes. When a variable feeds several downstream nodes, its effects add up along every path.
The math
- inner function, applied first
- outer function, evaluated at g(x) — not at x
Why. Linearize each stage: , so the inner output moves by ; then . Substituting gives a change of . For example .
Several variables. If depends on through intermediates , each path contributes its product and the paths add:
- intermediate quantities that depend directly on
- Jacobian of — the matrix of all its first partial derivatives
Backpropagation evaluates this sum from the output backward. Write for the gradient at node (its adjoint). Start with ; at every node, over the nodes that consume . Each adjoint is computed once and reused by every path through that node, so a single backward pass delivers all parameter gradients — instead of one pass per parameter, as forward-mode differentiation would need.
The local derivatives are simple. An addition passes its gradient unchanged to both inputs. A product sends to and to . The sigmoid has , computable from the stored forward value.
Backprop on a computational graph
- 1Forward: evaluate nodes in topological order and store every intermediate value.
- 2Seed the output: .
- 3Visit nodes in reverse order; for each input of node , add to .
- 4Read off for every parameter .
See it
- 1forward:
- 2 z₁ = w · x
- 3 z = z₁ + b
- 4 a = σ(z)
- 5 d = a − y
- 6 L = d²
- 7backward: ∂L/∂L = 1
- 8 ∂L/∂d = ∂L/∂L · 2d
- 9 ∂L/∂a = ∂L/∂d · 1
- 10 ∂L/∂z = ∂L/∂a · a(1 − a)
- 11 ∂L/∂b = ∂L/∂z · 1
- 12 ∂L/∂w = ∂L/∂z · 1 · x
- 13check: finite differences
Inputs: x = 2, target y = 1; parameters w = 0.5, b = −0.3. Forward pass evaluates nodes left to right.
Worked example
with , , , . Forward: , , , .
Backward, multiplying local derivatives right to left: ; ; , so . Since and : and .
Check: with gives . Both gradients are negative, so gradient descent increases and , pushing toward the target 1.
In code
1import torch23x, y = torch.tensor(2.0), torch.tensor(1.0)4w = torch.tensor(0.5, requires_grad=True)5b = torch.tensor(-0.3, requires_grad=True)67a = torch.sigmoid(w * x + b) # forward pass builds the graph8L = (a - y) ** 2 # tensor(0.1101)9L.backward() # reverse-mode chain rule10print(w.grad, b.grad) # tensor(-0.2943) tensor(-0.1471)1112# by hand: product of local derivatives along each path13a_ = a.item()14dz = 2 * (a_ - 1.0) * a_ * (1 - a_) # ∂L/∂d · ∂d/∂a · ∂a/∂z15print(dz * 2.0, dz) # ≈ -0.2943 -0.1471 (∂z/∂w = x, ∂z/∂b = 1)Where it's used
Training neural networks
Backprop is the chain rule applied over the whole network graph; one backward pass yields the gradient for millions or billions of weights.
Autograd frameworks
PyTorch, JAX and TensorFlow store a vector–Jacobian product for each primitive op and chain them, so any composition of ops is differentiable.
Diagnosing vanishing/exploding gradients
A deep gradient is a product of many layer factors. Since , ten sigmoid layers can shrink it by — the reason for ReLU, residual connections and careful initialization.
Reparameterization & change of variables
Training VAEs differentiates through ; densities transform by — the chain rule again.
Check yourself
What is at ?
Check yourself
with , . What is ?
Remember
- ✓: local slopes multiply along a path.
- ✓With several paths, multiply along each and add across them.
- ✓Backprop seeds and propagates in reverse order.
- ✓One backward pass gives every parameter's gradient, at the cost of storing forward values.