The Chain Rule

The derivative of a composition is the product of local derivatives along the path — the basis of backprop.

6 min read582 wordscore5 viewsupdated 2026-10-03

Definition

The chain rule states that if y=g(x)y=g(x) and z=f(y)z=f(y) are differentiable, the composition z=f(g(x))z=f(g(x)) has derivative dzdx=f′(g(x)) g′(x)=dzdydydx\frac{dz}{dx}=f'(g(x))\,g'(x)=\frac{dz}{dy}\frac{dy}{dx}. With many variables, the derivative of an output with respect to an input is the sum over all paths between them of the product of local derivatives along each path.

At a glance

Notation
(f∘g)′=(f′∘g) g′(f\circ g)'=(f'\circ g)\,g'; dzdx=dzdydydx\frac{dz}{dx}=\frac{dz}{dy}\frac{dy}{dx}; Jacobians Jf∘g(x)=Jf(g(x)) Jg(x)J_{f\circ g}(x)=J_f(g(x))\,J_g(x)
Input → output
derivatives of the pieces → derivative of the composition
Key property
local: each node only needs its own derivative and the gradient arriving from downstream
Cost (reverse mode)
all partial derivatives of a scalar loss for a small constant times the cost of one forward pass
Prerequisites
derivatives & gradients, function composition
Used in
backpropagation, autograd (PyTorch, JAX), sensitivity analysis, change of variables

The intuition

Rates of change multiply. If yy moves 3 units per unit of xx, and zz moves 2 units per unit of yy, then zz moves 2×3=62\times3=6 units per unit of xx. A composition is a pipeline, and each stage scales the incoming wiggle by its own local slope.

A neural network's loss is one long composition: parameters feed a linear layer, then an activation, then another layer, then a loss. To learn how the loss responds to one weight, follow the path from that weight to the loss and multiply the local slopes. When a variable feeds several downstream nodes, its effects add up along every path.

The math

ddxf(g(x))=f′(g(x)) g′(x)\frac{d}{dx}f(g(x))=f'(g(x))\,g'(x)
gg
inner function, applied first
ff
outer function, evaluated at g(x) — not at x

Why. Linearize each stage: g(x+h)≈g(x)+g′(x)hg(x+h)\approx g(x)+g'(x)h, so the inner output moves by k=g′(x)hk=g'(x)h; then f(g(x)+k)≈f(g(x))+f′(g(x))kf(g(x)+k)\approx f(g(x))+f'(g(x))k. Substituting kk gives a change of f′(g(x))g′(x) hf'(g(x))g'(x)\,h. For example ddxsin⁡(x2)=cos⁡(x2)⋅2x\frac{d}{dx}\sin(x^2)=\cos(x^2)\cdot2x.

Several variables. If LL depends on ww through intermediates v1,…,vmv_1,\dots,v_m, each path contributes its product and the paths add:

∂L∂w=∑j=1m∂L∂vj∂vj∂w,Jf∘g(x)=Jf(g(x)) Jg(x)\frac{\partial L}{\partial w}=\sum_{j=1}^{m}\frac{\partial L}{\partial v_j}\frac{\partial v_j}{\partial w},\qquad J_{f\circ g}(x)=J_f\big(g(x)\big)\,J_g(x)
vjv_j
intermediate quantities that depend directly on ww
JfJ_f
Jacobian of ff — the matrix of all its first partial derivatives

Backpropagation evaluates this sum from the output backward. Write vˉ=∂L/∂v\bar v=\partial L/\partial v for the gradient at node vv (its adjoint). Start with Lˉ=1\bar L=1; at every node, vˉ=∑ccˉ ∂c/∂v\bar v=\sum_{c}\bar c\,\partial c/\partial v over the nodes cc that consume vv. Each adjoint is computed once and reused by every path through that node, so a single backward pass delivers all parameter gradients — instead of one pass per parameter, as forward-mode differentiation would need.

The local derivatives are simple. An addition z=u+vz=u+v passes its gradient unchanged to both inputs. A product z=uvz=uv sends zˉ v\bar z\,v to uu and zˉ u\bar z\,u to vv. The sigmoid σ(z)=1/(1+e−z)\sigma(z)=1/(1+e^{-z}) has σ′(z)=σ(z)(1−σ(z))\sigma'(z)=\sigma(z)(1-\sigma(z)), computable from the stored forward value.

Backprop on a computational graph

  1. 1Forward: evaluate nodes in topological order and store every intermediate value.
  2. 2Seed the output: Lˉ=1\bar L=1.
  3. 3Visit nodes in reverse order; for each input vv of node cc, add cˉ⋅∂c/∂v\bar c\cdot\partial c/\partial v to vˉ\bar v.
  4. 4Read off wˉ=∂L/∂w\bar w=\partial L/\partial w for every parameter ww.

See it

▸ Backprop through a sigmoid neuron with squared lossstep 1/13
x2.000w0.500×b−0.300+σy1.000−²below: value · above: ∂L/∂node · on edge: local derivative
L = —∂L/∂w = —∂L/∂b = —
  1. 1forward:
  2. 2 z₁ = w · x
  3. 3 z = z₁ + b
  4. 4 a = σ(z)
  5. 5 d = a − y
  6. 6 L = d²
  7. 7backward: ∂L/∂L = 1
  8. 8 ∂L/∂d = ∂L/∂L · 2d
  9. 9 ∂L/∂a = ∂L/∂d · 1
  10. 10 ∂L/∂z = ∂L/∂a · a(1 − a)
  11. 11 ∂L/∂b = ∂L/∂z · 1
  12. 12 ∂L/∂w = ∂L/∂z · 1 · x
  13. 13check: finite differences

Inputs: x = 2, target y = 1; parameters w = 0.5, b = −0.3. Forward pass evaluates nodes left to right.

fig.One sigmoid neuron with squared loss, x = 2, w = 0.5, b = −0.3, y = 1. Forward: z = 0.7, a = σ(0.7) = 0.6682, L = 0.1101. Backward: 2d = −0.6636 times σ′ = 0.2217 gives ∂L/∂z = −0.1471 = ∂L/∂b; times x = 2 gives ∂L/∂w = −0.2943. The final frame matches both against central differences to 6 decimals.

Worked example

L=(σ(wx+b)−y)2L=(\sigma(wx+b)-y)^2 with x=2x=2, w=0.5w=0.5, b=−0.3b=-0.3, y=1y=1. Forward: z=0.5⋅2−0.3=0.7z=0.5\cdot2-0.3=0.7, a=σ(0.7)=0.66819a=\sigma(0.7)=0.66819, d=a−y=−0.33181d=a-y=-0.33181, L=d2=0.11010L=d^2=0.11010.

Backward, multiplying local derivatives right to left: ∂L/∂d=2d=−0.66362\partial L/\partial d=2d=-0.66362; ∂d/∂a=1\partial d/\partial a=1; ∂a/∂z=a(1−a)=0.22171\partial a/\partial z=a(1-a)=0.22171, so ∂L/∂z=−0.66362⋅0.22171=−0.14713\partial L/\partial z=-0.66362\cdot0.22171=-0.14713. Since ∂z/∂b=1\partial z/\partial b=1 and ∂z/∂w=x=2\partial z/\partial w=x=2: ∂L/∂b=−0.14713\partial L/\partial b=-0.14713 and ∂L/∂w=−0.29427\partial L/\partial w=-0.29427.

Check: (L(w+h)−L(w−h))/2h(L(w+h)-L(w-h))/2h with h=10−6h=10^{-6} gives −0.294268-0.294268. Both gradients are negative, so gradient descent increases ww and bb, pushing aa toward the target 1.

In code

python
1import torch
2
3x, y = torch.tensor(2.0), torch.tensor(1.0)
4w = torch.tensor(0.5, requires_grad=True)
5b = torch.tensor(-0.3, requires_grad=True)
6
7a = torch.sigmoid(w * x + b) # forward pass builds the graph
8L = (a - y) ** 2 # tensor(0.1101)
9L.backward() # reverse-mode chain rule
10print(w.grad, b.grad) # tensor(-0.2943) tensor(-0.1471)
11
12# by hand: product of local derivatives along each path
13a_ = a.item()
14dz = 2 * (a_ - 1.0) * a_ * (1 - a_) # ∂L/∂d · ∂d/∂a · ∂a/∂z
15print(dz * 2.0, dz) # ≈ -0.2943 -0.1471 (∂z/∂w = x, ∂z/∂b = 1)
PyTorch records the forward graph and applies the chain rule backward; the manual product gives the same numbers.

Where it's used

Training neural networks

Backprop is the chain rule applied over the whole network graph; one backward pass yields the gradient for millions or billions of weights.

Autograd frameworks

PyTorch, JAX and TensorFlow store a vector–Jacobian product for each primitive op and chain them, so any composition of ops is differentiable.

Diagnosing vanishing/exploding gradients

A deep gradient is a product of many layer factors. Since σ′≤0.25\sigma'\le0.25, ten sigmoid layers can shrink it by 0.2510≈10−60.25^{10}\approx10^{-6} — the reason for ReLU, residual connections and careful initialization.

Reparameterization & change of variables

Training VAEs differentiates through z=μ+σεz=\mu+\sigma\varepsilon; densities transform by pX(x)=pZ(g(x)) ∣g′(x)∣p_X(x)=p_Z(g(x))\,|g'(x)| — the chain rule again.

Check yourself

What is ddx(3x+1)2\frac{d}{dx}(3x+1)^2 at x=1x=1?

Check yourself

L=w x+w2L=w\,x+w^2 with w=3w=3, x=2x=2. What is ∂L/∂w\partial L/\partial w?

Remember

  • ✓(f∘g)′(x)=f′(g(x)) g′(x)(f\circ g)'(x)=f'(g(x))\,g'(x): local slopes multiply along a path.
  • ✓With several paths, multiply along each and add across them.
  • ✓Backprop seeds Lˉ=1\bar L=1 and propagates vˉ=∑ccˉ ∂c/∂v\bar v=\sum_c\bar c\,\partial c/\partial v in reverse order.
  • ✓One backward pass gives every parameter's gradient, at the cost of storing forward values.
← back to Mathematics roadmap