# The Chain Rule

> The derivative of a composition is the product of local derivatives along the path — the basis of backprop.

Source: https://www.mlfoundry.in/learn/chain-rule  
Section: Mathematics › Calculus & Optimization  
Level: core · 6 min read · updated 2026-10-03

**The chain rule:** states that if $y=g(x)$ and $z=f(y)$ are differentiable, the composition $z=f(g(x))$ has derivative $\frac{dz}{dx}=f'(g(x))\,g'(x)=\frac{dz}{dy}\frac{dy}{dx}$. With many variables, the derivative of an output with respect to an input is the **sum over all paths** between them of the **product of local derivatives** along each path.

- **Notation:** $(f\circ g)'=(f'\circ g)\,g'$; $\frac{dz}{dx}=\frac{dz}{dy}\frac{dy}{dx}$; Jacobians $J_{f\circ g}(x)=J_f(g(x))\,J_g(x)$
- **Input → output:** derivatives of the pieces → derivative of the composition
- **Key property:** local: each node only needs its own derivative and the gradient arriving from downstream
- **Cost (reverse mode):** all partial derivatives of a scalar loss for a small constant times the cost of one forward pass
- **Prerequisites:** derivatives & gradients, function composition
- **Used in:** backpropagation, autograd (PyTorch, JAX), sensitivity analysis, change of variables

## The intuition

Rates of change multiply. If $y$ moves 3 units per unit of $x$, and $z$ moves 2 units per unit of $y$, then $z$ moves $2\times3=6$ units per unit of $x$. A composition is a pipeline, and each stage scales the incoming wiggle by its own local slope.

A neural network's loss is one long composition: parameters feed a linear layer, then an activation, then another layer, then a loss. To learn how the loss responds to one weight, follow the path from that weight to the loss and multiply the local slopes. When a variable feeds several downstream nodes, its effects add up along every path.

## The math

$$
\frac{d}{dx}f(g(x))=f'(g(x))\,g'(x)
$$

- $g$: inner function, applied first
- $f$: outer function, evaluated at g(x) — not at x

**Why.** Linearize each stage: $g(x+h)\approx g(x)+g'(x)h$, so the inner output moves by $k=g'(x)h$; then $f(g(x)+k)\approx f(g(x))+f'(g(x))k$. Substituting $k$ gives a change of $f'(g(x))g'(x)\,h$. For example $\frac{d}{dx}\sin(x^2)=\cos(x^2)\cdot2x$.

**Several variables.** If $L$ depends on $w$ through intermediates $v_1,\dots,v_m$, each path contributes its product and the paths add:

$$
\frac{\partial L}{\partial w}=\sum_{j=1}^{m}\frac{\partial L}{\partial v_j}\frac{\partial v_j}{\partial w},\qquad J_{f\circ g}(x)=J_f\big(g(x)\big)\,J_g(x)
$$

- $v_j$: intermediate quantities that depend directly on $w$
- $J_f$: Jacobian of $f$ — the matrix of all its first partial derivatives

**Backpropagation** evaluates this sum from the output backward. Write $\bar v=\partial L/\partial v$ for the gradient at node $v$ (its *adjoint*). Start with $\bar L=1$; at every node, $\bar v=\sum_{c}\bar c\,\partial c/\partial v$ over the nodes $c$ that consume $v$. Each adjoint is computed once and reused by every path through that node, so a single backward pass delivers all parameter gradients — instead of one pass per parameter, as forward-mode differentiation would need.

The local derivatives are simple. An addition $z=u+v$ passes its gradient unchanged to both inputs. A product $z=uv$ sends $\bar z\,v$ to $u$ and $\bar z\,u$ to $v$. The sigmoid $\sigma(z)=1/(1+e^{-z})$ has $\sigma'(z)=\sigma(z)(1-\sigma(z))$, computable from the stored forward value.

**Backprop on a computational graph**

1. Forward: evaluate nodes in topological order and store every intermediate value.
2. Seed the output: $\bar L=1$.
3. Visit nodes in reverse order; for each input $v$ of node $c$, add $\bar c\cdot\partial c/\partial v$ to $\bar v$.
4. Read off $\bar w=\partial L/\partial w$ for every parameter $w$.

## See it

> Figure: One sigmoid neuron with squared loss, x = 2, w = 0.5, b = −0.3, y = 1. Forward: z = 0.7, a = σ(0.7) = 0.6682, L = 0.1101. Backward: 2d = −0.6636 times σ′ = 0.2217 gives ∂L/∂z = −0.1471 = ∂L/∂b; times x = 2 gives ∂L/∂w = −0.2943. The final frame matches both against central differences to 6 decimals.

## Worked example

$L=(\sigma(wx+b)-y)^2$ with $x=2$, $w=0.5$, $b=-0.3$, $y=1$. Forward: $z=0.5\cdot2-0.3=0.7$, $a=\sigma(0.7)=0.66819$, $d=a-y=-0.33181$, $L=d^2=0.11010$.

Backward, multiplying local derivatives right to left: $\partial L/\partial d=2d=-0.66362$; $\partial d/\partial a=1$; $\partial a/\partial z=a(1-a)=0.22171$, so $\partial L/\partial z=-0.66362\cdot0.22171=-0.14713$. Since $\partial z/\partial b=1$ and $\partial z/\partial w=x=2$: $\partial L/\partial b=-0.14713$ and $\partial L/\partial w=-0.29427$.

Check: $(L(w+h)-L(w-h))/2h$ with $h=10^{-6}$ gives $-0.294268$. Both gradients are negative, so gradient descent increases $w$ and $b$, pushing $a$ toward the target 1.

## In code

```python
import torch

x, y = torch.tensor(2.0), torch.tensor(1.0)
w = torch.tensor(0.5, requires_grad=True)
b = torch.tensor(-0.3, requires_grad=True)

a = torch.sigmoid(w * x + b)      # forward pass builds the graph
L = (a - y) ** 2                  # tensor(0.1101)
L.backward()                      # reverse-mode chain rule
print(w.grad, b.grad)             # tensor(-0.2943) tensor(-0.1471)

# by hand: product of local derivatives along each path
a_ = a.item()
dz = 2 * (a_ - 1.0) * a_ * (1 - a_)   # ∂L/∂d · ∂d/∂a · ∂a/∂z
print(dz * 2.0, dz)                   # ≈ -0.2943 -0.1471   (∂z/∂w = x, ∂z/∂b = 1)
```

PyTorch records the forward graph and applies the chain rule backward; the manual product gives the same numbers.

- **Training neural networks:** Backprop is the chain rule applied over the whole network graph; one backward pass yields the gradient for millions or billions of weights.
- **Autograd frameworks:** PyTorch, JAX and TensorFlow store a vector–Jacobian product for each primitive op and chain them, so any composition of ops is differentiable.
- **Diagnosing vanishing/exploding gradients:** A deep gradient is a product of many layer factors. Since $\sigma'\le0.25$, ten sigmoid layers can shrink it by $0.25^{10}\approx10^{-6}$ — the reason for ReLU, residual connections and careful initialization.
- **Reparameterization & change of variables:** Training VAEs differentiates through $z=\mu+\sigma\varepsilon$; densities transform by $p_X(x)=p_Z(g(x))\,|g'(x)|$ — the chain rule again.

> **Forgetting a path:** When a variable is used twice, every use contributes. For $L=wx+w^2$ the gradient is $\partial L/\partial w=x+2w$, the sum over both paths, not either term alone. Autograd handles this by **accumulating** into $\bar w$ — which is also why PyTorch's .grad keeps summing across backward calls until you zero it.

> **Why reverse mode wins for training:** For $f:\mathbb{R}^n\to\mathbb{R}^m$, forward mode costs one pass per input ($n$ passes), reverse mode one pass per output ($m$ passes). A loss has $m=1$ output and $n$ in the millions, so reverse mode — backprop — is millions of times cheaper. The price is memory: the forward pass must store intermediates for the backward pass.

**Question:** What is $\frac{d}{dx}(3x+1)^2$ at $x=1$?

- A. 8
- B. 16
- C. 24
- D. 48

**Answer:** C. 24 — Outer $f(u)=u^2$ has $f'(u)=2u$; inner $u=3x+1$ has $u'=3$. At $x=1$, $u=4$, so the derivative is $2\cdot4\cdot3=24$.

**Question:** $L=w\,x+w^2$ with $w=3$, $x=2$. What is $\partial L/\partial w$?

- A. 2
- B. 6
- C. 8
- D. 12

**Answer:** C. 8 — $w$ reaches $L$ along two paths: through $wx$ (local derivative $x=2$) and through $w^2$ (local derivative $2w=6$). The paths add: $2+6=8$.

**Key points**

- $(f\circ g)'(x)=f'(g(x))\,g'(x)$: local slopes multiply along a path.
- With several paths, multiply along each and add across them.
- Backprop seeds $\bar L=1$ and propagates $\bar v=\sum_c\bar c\,\partial c/\partial v$ in reverse order.
- One backward pass gives every parameter's gradient, at the cost of storing forward values.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
