# Backpropagation

> The chain rule applied in reverse, reusing intermediate values to get every gradient in one backward pass.

Source: https://www.mlfoundry.in/learn/backpropagation  
Section: Deep Learning › Deep Learning Fundamentals  
Level: core · 8 min read · updated 2026-10-09

**Backpropagation:** is the algorithm that computes $\partial\mathcal{L}/\partial\theta$ for every parameter $\theta$ of a network by applying the chain rule layer by layer from the loss back to the input, reusing the activations cached by the forward pass and the error signals of the layers already processed.

Gradient descent needs the gradient of one scalar loss with respect to millions of parameters. Estimating each partial derivative by nudging that parameter costs a forward pass per parameter, which is hopeless at scale.

Backprop gets **all** of them for roughly the price of two extra forward passes. The trick is ordering: every weight's gradient shares the same downstream factors, so compute those once, starting from the loss, and pass them backward.

Backprop is only the gradient computation. The update that uses the gradient is a separate choice (SGD, Adam, …).

## The chain rule, organised by layer

A weight in layer $l$ affects the loss only through $z^{(l)}$, which affects it only through $z^{(l+1)}$, and so on. Define the **error signal** of a layer as the gradient of the loss with respect to its pre-activations:

$$
\delta^{(l)}\;\equiv\;\frac{\partial\mathcal{L}}{\partial z^{(l)}}\in\mathbb{R}^{n_l}
$$

- $z^{(l)}$: pre-activation of layer $l$, $z^{(l)}=W^{(l)}a^{(l-1)}+b^{(l)}$
- $n_l$: number of units in layer $l$

Everything follows from three equations. Start at the output, step backward with the recurrence, and read off the parameter gradients at each layer:

$$
\delta^{(L)}=\nabla_{a^{(L)}}\mathcal{L}\odot\varphi'\big(z^{(L)}\big),\qquad \delta^{(l)}=\Big(W^{(l+1)\top}\,\delta^{(l+1)}\Big)\odot\varphi'\big(z^{(l)}\big)
$$

- $W^{(l+1)\top}\delta^{(l+1)}$: error sent back through the same weights used going forward, giving $\partial\mathcal{L}/\partial a^{(l)}$
- $\varphi'(z^{(l)})$: local slope of the activation, from the cached pre-activations
- $\odot$: elementwise product (each unit scales its own error)

$$
\frac{\partial\mathcal{L}}{\partial W^{(l)}}=\delta^{(l)}\,a^{(l-1)\top},\qquad \frac{\partial\mathcal{L}}{\partial b^{(l)}}=\delta^{(l)}
$$

- $\delta^{(l)}a^{(l-1)\top}$: outer product, shape $n_l\times n_{l-1}$, the same as $W^{(l)}$: entry $(j,i)$ is (error at unit $j$) × (input from unit $i$)

For a sigmoid output with cross-entropy loss the output error simplifies: $\partial\mathcal{L}/\partial\hat y=-\tfrac{y}{\hat y}+\tfrac{1-y}{1-\hat y}$ and $\sigma'=\hat y(1-\hat y)$, whose product is just $\delta^{(L)}=\hat y-y$. Softmax with cross-entropy gives the same form.

With a mini-batch stored as rows, $\Delta^{(l)}\in\mathbb{R}^{B\times n_l}$ holds one $\delta$ per example and $\partial\mathcal{L}/\partial W^{(l)}=\Delta^{(l)\top}A^{(l-1)}$ sums the per-example outer products (divide by $B$ if the loss is a mean).

## Why backward is the efficient direction

The loss is one number and the parameters are many. Going backward, each layer turns one vector $\delta^{(l+1)}$ into the next with a single matrix–vector product, and every weight gradient in that layer reuses it. Going forward instead (propagating $\partial z/\partial\theta$) would need a separate pass for every parameter.

Count the matrix multiplies. A layer's forward pass is one product ($W a$). Its backward pass is two: $W^\top\delta$ for the error and $\delta a^\top$ for the weight gradient, each the same size. So **backward ≈ 2× forward**, and a full training step costs about 3 forward passes, whether the network has ten parameters or ten billion. Central finite differences would need $2P$ forward passes.

> Figure: Press backward to walk from the loss to the inputs. Each node shows its δ once computed; each edge's colour and thickness show its gradient (rose: positive, so SGD will lower that weight; blue: negative, so SGD will raise it). Then press apply SGD step a few times and watch the loss fall from 0.581. Raise η to take bigger steps.

## Worked example: the same 2-2-1 network

Continue the forward-propagation example: $W^{(1)}=\begin{bmatrix}0.4&-0.2\\0.3&0.6\end{bmatrix}$, $b^{(1)}=(0.1,-0.1)$, $W^{(2)}=\begin{bmatrix}0.7&-0.5\end{bmatrix}$, $b^{(2)}=0.2$, $x=(0.5,1)$, $y=1$, sigmoid everywhere, cross-entropy loss. The forward pass cached $a^{(1)}=(0.5250,\,0.6570)$ and $\hat y=0.5595$, with $\mathcal{L}=0.5808$.

Backward pass (rounded to 4 decimals; every gradient matches a central finite difference to within 1e-9)

| Quantity | Computation | Value |
| --- | --- | --- |
| $\delta^{(2)}$ | $\hat y-y=0.5595-1$ | −0.4405 |
| $\partial\mathcal{L}/\partial W^{(2)}$ | $\delta^{(2)}a^{(1)\top}=-0.4405\cdot(0.5250,\,0.6570)$ | (−0.2313, −0.2894) |
| $\partial\mathcal{L}/\partial b^{(2)}$ | $\delta^{(2)}$ | −0.4405 |
| $W^{(2)\top}\delta^{(2)}$ | $(0.7,\,-0.5)\cdot(-0.4405)$ | (−0.3084, 0.2203) |
| $\sigma'(z^{(1)})$ | $a^{(1)}\odot(1-a^{(1)})$ | (0.2494, 0.2253) |
| $\delta^{(1)}$ | $(-0.3084\cdot0.2494,\;0.2203\cdot0.2253)$ | (−0.0769, 0.0496) |
| $\partial\mathcal{L}/\partial W^{(1)}$ | $\delta^{(1)}x^\top$ | $\begin{bmatrix}-0.0385&-0.0769\\0.0248&0.0496\end{bmatrix}$ |
| $\partial\mathcal{L}/\partial b^{(1)}$ | $\delta^{(1)}$ | (−0.0769, 0.0496) |

The signs make sense. The output is too low ($\delta^{(2)}<0$), so the gradient says: raise $w^{(2)}_1$, and make $w^{(2)}_2$ less negative. Hidden unit 2 feeds the output through a negative weight, so its error flips sign: $\delta^{(1)}_2>0$. The hidden errors are 6–9× smaller than the output error, and the factor $\sigma'\le0.25$ accounts for most of that shrinkage.

**One SGD step, η = 0.5 (θ ← θ − η ∂L/∂θ)**

1. $W^{(2)}$: $(0.7,\,-0.5)\to(0.8156,\,-0.3553)$, and $b^{(2)}$: $0.2\to0.4203$.
2. $W^{(1)}$: $\begin{bmatrix}0.4&-0.2\\0.3&0.6\end{bmatrix}\to\begin{bmatrix}0.4192&-0.1615\\0.2876&0.5752\end{bmatrix}$, and $b^{(1)}$: $(0.1,-0.1)\to(0.1385,\,-0.1248)$.
3. New forward pass: $\hat y=0.6541$, so $\mathcal{L}=0.4245$, down from $0.5808$. Further steps give $0.3261$, $0.2608$, $0.2152$.

```python
import torch, torch.nn as nn, torch.nn.functional as F

l1, l2 = nn.Linear(2, 2), nn.Linear(2, 1)
with torch.no_grad():
    l1.weight.copy_(torch.tensor([[0.4, -0.2], [0.3, 0.6]])); l1.bias.copy_(torch.tensor([0.1, -0.1]))
    l2.weight.copy_(torch.tensor([[0.7, -0.5]]));             l2.bias.copy_(torch.tensor([0.2]))
opt = torch.optim.SGD([*l1.parameters(), *l2.parameters()], lr=0.5)

x, y = torch.tensor([[0.5, 1.0]]), torch.tensor([[1.0]])
loss = F.binary_cross_entropy_with_logits(l2(torch.sigmoid(l1(x))), y)   # 0.5808
opt.zero_grad()
loss.backward()                     # one backward pass fills every .grad
print(l2.weight.grad)               # [[-0.2313, -0.2894]]
print(l1.weight.grad)               # [[-0.0385, -0.0769], [ 0.0248,  0.0496]]
opt.step()                          # next forward pass gives loss 0.4245
```

Autograd reproduces the table, then the optimizer applies the step

> **Gradients accumulate:** PyTorch adds each `backward()` into `.grad` rather than overwriting it (useful for gradient accumulation across micro-batches). Forget `opt.zero_grad()` and every step uses the sum of all previous gradients.

> **Products of slopes:** Unroll the recurrence and $\delta^{(l)}$ contains the product $\varphi'(z^{(l)})\,W^{(l+1)\top}\varphi'(z^{(l+1)})\cdots$. Many factors below 1 shrink the signal toward zero; many above 1 blow it up. That is the root of vanishing and exploding gradients.

Frameworks do not hard-code these layer formulas. They record the forward pass as a computational graph and run reverse-mode automatic differentiation on it, which is backprop generalised to any composition of differentiable operations.

**Question:** A hidden ReLU unit has z = −0.3 on some example. What is its δ for that example?

- A. Exactly 0, whatever the downstream error
- B. −0.3 times the downstream error
- C. The same as the downstream error
- D. It depends on the learning rate

**Answer:** A. Exactly 0, whatever the downstream error — $\delta_j=(W^\top\delta^{(l+1)})_j\cdot\varphi'(z_j)$ and $\mathrm{ReLU}'(z)=0$ for $z<0$. So the unit passes no error back, and its incoming weights get zero gradient from this example.

**Question:** A network has P = 10⁶ parameters. Roughly how much computation does one exact gradient cost with backprop, versus central finite differences?

- A. Both about 2·10⁶ forward passes
- B. Backprop ≈ 3 forward passes; finite differences ≈ 2·10⁶
- C. Backprop ≈ 10⁶ forward passes; finite differences ≈ 3
- D. Backprop ≈ 2 forward passes; finite differences ≈ 10⁶ backward passes

**Answer:** B. Backprop ≈ 3 forward passes; finite differences ≈ 2·10⁶ — Backprop's backward pass costs about twice the forward pass (two matrix products per layer instead of one), so forward + backward ≈ 3 forward passes regardless of $P$. Central differences need two forward passes per parameter, $2P=2\cdot10^6$, and are only approximate.

**Key points**

- $\delta^{(l)}=\partial\mathcal{L}/\partial z^{(l)}$; for sigmoid or softmax with cross-entropy, $\delta^{(L)}=\hat y-y$.
- Recurrence: $\delta^{(l)}=(W^{(l+1)\top}\delta^{(l+1)})\odot\varphi'(z^{(l)})$.
- Gradients: $\partial\mathcal{L}/\partial W^{(l)}=\delta^{(l)}a^{(l-1)\top}$, $\partial\mathcal{L}/\partial b^{(l)}=\delta^{(l)}$.
- Reverse order reuses each δ for a whole layer: backward ≈ 2× forward, independent of parameter count.
- Worked example: one SGD step at η = 0.5 drops the loss from 0.5808 to 0.4245.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
