# Automatic differentiation

> Record the operations of a forward pass, then apply the chain rule backwards to get exact gradients at about the cost of one more forward pass.

Source: https://www.mlfoundry.in/learn/automatic-differentiation  
Section: Deep Learning › Deep Learning Fundamentals  
Level: core · 8 min read · updated 2026-10-09

**Automatic differentiation (autodiff):** A technique that computes exact derivatives of a program by applying the chain rule to every elementary operation it executes. It works on code, including loops and branches, not on a symbolic formula, and it has no step size to tune.

Training needs the gradient of a scalar loss with respect to every parameter, often hundreds of millions of them, at every step. How you get that gradient decides whether training is possible at all.

## Three ways to get a derivative

| Method | How | Accuracy | Cost for n inputs |
| --- | --- | --- | --- |
| Symbolic | Manipulate the formula with rules, as a computer algebra system does | Exact | Expressions can blow up in size (expression swell); needs a closed form, not a program |
| Numeric | Finite differences, e.g. $\frac{f(x+h e_i)-f(x-he_i)}{2h}$ | Approximate: truncation error from $h$, round-off error from floating point | $n+1$ (forward) or $2n$ (central) evaluations of $f$ |
| Automatic | Chain rule applied to each executed operation, with numbers rather than expressions | Exact up to floating-point rounding | Reverse mode: a small constant times one evaluation, independent of $n$ |

Finite differences are useless for training a network with $10^8$ parameters ($2\times10^8$ forward passes per step), but they are the standard way to **test** an autodiff implementation on a few coordinates.

## The tape (Wengert list)

Running the program produces a linear sequence of elementary assignments $v_i = \phi_i(v_{j}, v_k)$, each depending only on earlier entries. For $f(x_1,x_2,x_3) = x_1x_2 + x_3\sin x_1$ the tape is $v_4 = v_1v_2$, $v_5 = \sin v_1$, $v_6 = v_3v_5$, $v_7 = v_4 + v_6$. This list is a computational graph laid out in topological order. Autodiff can sweep it in either direction.

## Forward mode: one pass per input

Carry a **tangent** $\dot v_i = \partial v_i/\partial x_k$ alongside every value. Seed $\dot x = e_k$ and apply each op's derivative rule as you go. One pass gives one column of the Jacobian: how every output responds to input $x_k$.

$$
f(a + b\,\varepsilon) = f(a) + f'(a)\,b\,\varepsilon,\qquad \varepsilon^2 = 0
$$

- $a + b\varepsilon$: dual number: value a, tangent b
- $\varepsilon$: formal infinitesimal whose square is zero, so every Taylor term past the first vanishes

Dual numbers are forward mode in algebraic form. With $x_1 = 2 + \varepsilon$, $x_2 = 5$, $x_3 = 3$, ordinary arithmetic gives $f = 12.7279 + 3.7516\,\varepsilon$, so $\partial f/\partial x_1 = 3.7516$. To get $\partial f/\partial x_2$ you must run the whole thing again with the $\varepsilon$ on $x_2$.

## Reverse mode: one pass per output

Run the program once, recording the tape. Then seed $\bar f = 1$ and walk the tape backwards, propagating **adjoints** $\bar v_i = \partial f/\partial v_i$:

$$
\bar v_j \;\mathrel{+}=\; \bar v_i\,\frac{\partial \phi_i}{\partial v_j}\quad\text{for each tape entry } v_i=\phi_i(\dots,v_j,\dots)\text{, in reverse order}
$$

- $\bar v_i = \partial f/\partial v_i$: adjoint of entry i, starting from 1 at the output
- $\partial\phi_i/\partial v_j$: local derivative of the op, evaluated with values saved on the tape
- $\mathrel{+}=$: accumulate: a value used several times collects one contribution per use

One backward sweep yields one row of the Jacobian: how one output depends on **every** input. For $f:\mathbb{R}^n\to\mathbb{R}^m$, forward mode needs $n$ passes and reverse mode $m$. A loss is a single scalar ($m = 1$) of millions of parameters, so reverse mode gets the full gradient in one sweep. Backpropagation is reverse-mode autodiff applied to a neural network.

> Figure: The tape for f = x₁x₂ + x₃ sin x₁ at (2, 5, 3). In forward mode, step through all three passes, one per input. Switch to reverse mode: one recording sweep and one backward sweep fill in all three partials, and v̄₁ is accumulated from two uses.

**Reverse sweep on the example, x = (2, 5, 3)**

1. Forward: $v_4 = 10$, $v_5 = \sin 2 = 0.9093$, $v_6 = 3\cdot0.9093 = 2.7279$, $f = v_7 = 12.7279$.
2. Seed $\bar v_7 = 1$. The add gives $\bar v_4 = \bar v_6 = 1$.
3. Row 6, $v_6 = v_3v_5$: $\bar v_3 = \bar v_6\,v_5 = 0.9093$, $\bar v_5 = \bar v_6\,v_3 = 3$.
4. Row 5, $v_5 = \sin v_1$: $\bar v_1 \mathrel{+}= 3\cos 2 = 3\cdot(-0.4161) = -1.2484$.
5. Row 4, $v_4 = v_1v_2$: $\bar v_1 \mathrel{+}= \bar v_4\,v_2 = 5$, so $\bar v_1 = 3.7516$; $\bar v_2 = \bar v_4\,v_1 = 2$.
6. Result: $\nabla f = (3.7516,\ 2,\ 0.9093)$, matching $(x_2 + x_3\cos x_1,\ x_1,\ \sin x_1)$.

## Why not just use finite differences?

Absolute error in ∂f/∂x₁ at (2, 5, 3), exact value 3.751559. Reverse-mode autodiff matches the analytic derivative to machine precision (~1e−16).

| h | Forward difference error | Central difference error |
| --- | --- | --- |
| 1e−1 | 1.3e−1 | 2.1e−3 |
| 1e−3 | 1.4e−3 | 2.1e−7 |
| 1e−5 | 1.4e−5 | 8.2e−12 |
| 1e−7 | 1.4e−7 | 6.3e−10 |
| 1e−9 | 1.4e−6 | 4.6e−7 |
| 1e−11 | 1.1e−4 | 1.1e−4 |

Shrinking $h$ helps only up to a point. **Truncation error** falls like $h$ (forward) or $h^2$ (central), but **round-off error** grows like $\epsilon_{\text{mach}}/h$, because subtracting two nearly equal numbers cancels most significant digits. The best central difference here is about $10^{-11}$ at $h=10^{-5}$; below that the error climbs again. Autodiff has no $h$ and no cancellation.

## A tiny autograd engine

```python
import math

class Value:
    def __init__(self, data, parents=(), local=()):
        self.data, self.grad = data, 0.0
        self.parents, self.local = parents, local   # inputs and d(self)/d(input)

    def __add__(self, o):
        return Value(self.data + o.data, (self, o), (1.0, 1.0))

    def __mul__(self, o):
        return Value(self.data * o.data, (self, o), (o.data, self.data))

    def sin(self):
        return Value(math.sin(self.data), (self,), (math.cos(self.data),))

    def backward(self):
        order, seen = [], set()
        def visit(v):                       # topological order = the tape
            if v not in seen:
                seen.add(v)
                for p in v.parents:
                    visit(p)
                order.append(v)
        visit(self)
        self.grad = 1.0
        for v in reversed(order):           # sweep the tape backwards
            for p, d in zip(v.parents, v.local):
                p.grad += v.grad * d        # += : fan-out sums

x1, x2, x3 = Value(2.0), Value(5.0), Value(3.0)
f = x1 * x2 + x3 * x1.sin()
f.backward()
print(round(f.data, 4), round(x1.grad, 4), x2.grad, round(x3.grad, 4))
# 12.7279 3.7516 2.0 0.9093
```

Each Value remembers its parents and local derivatives. backward() orders the graph topologically and sweeps it in reverse.

## PyTorch autograd

Tensors created with `requires_grad=True` are leaves that autograd tracks. Every op on them records a node (its `grad_fn`). Calling `.backward()` on a scalar runs reverse mode and **adds** each gradient into the leaf's `.grad`. The graph is freed afterwards unless you pass `retain_graph=True`.

```python
import torch

x = torch.tensor([2.0, 5.0, 3.0], requires_grad=True)

def f(x):
    return x[0] * x[1] + x[2] * torch.sin(x[0])

f(x).backward()
print(x.grad)        # tensor([3.7516, 2.0000, 0.9093])

f(x).backward()      # a second backward without zeroing
print(x.grad)        # tensor([7.5031, 4.0000, 1.8186])  <- accumulated

x.grad.zero_()       # in a training loop: optimizer.zero_grad()

with torch.no_grad():        # inference: record no graph, save memory
    y = f(x)
```

> **Forgetting zero_grad():** Because `.grad` accumulates, a training loop that never calls `optimizer.zero_grad()` updates with the sum of all past gradients, and training diverges or stalls. Accumulation is deliberate: it lets you sum gradients over several small batches before one optimizer step.

> **The cheap gradient principle:** Reverse mode costs a small constant multiple of one function evaluation, typically 2 to 3 times in practice, regardless of how many parameters there are. The catch is memory: the tape must keep every intermediate value until the backward sweep uses it.

**Question:** You need the full Jacobian of a function from 3 inputs to 1000 outputs. Which mode needs fewer passes?

- A. Reverse mode: 1 pass
- B. Reverse mode: 3 passes
- C. Forward mode: 3 passes
- D. Forward mode: 1000 passes

**Answer:** C. Forward mode: 3 passes — Forward mode needs one pass per input (3 here); reverse mode needs one pass per output (1000 here). Losses are the opposite case, one output and many inputs, which is why deep learning uses reverse mode.

**Question:** A central finite difference with h = 1e−12 gives a worse derivative than with h = 1e−5. Why?

- A. Truncation error grows as h shrinks
- B. Round-off error from subtracting nearly equal numbers grows like ε/h
- C. The function is not differentiable at small scales
- D. Central differences only work for h ≥ 1e−5

**Answer:** B. Round-off error from subtracting nearly equal numbers grows like ε/h — Truncation error shrinks with h (like h² for central differences). But f(x+h) and f(x−h) agree in almost all their digits, so their difference keeps few significant bits, and dividing by a tiny 2h magnifies that rounding error roughly like ε/h.

**Key points**

- Autodiff applies the chain rule to the executed operations: exact derivatives, no step size, works through loops and branches.
- Finite differences cost about n evaluations for n inputs and trade truncation error against round-off; use them only to check gradients.
- Forward mode (tangents, dual numbers) needs one pass per input; reverse mode (adjoints on a tape) needs one pass per output.
- A loss has one output, so reverse mode gives the whole gradient for a small constant times the forward cost; backprop is exactly this.
- In PyTorch: requires_grad marks leaves, .backward() fills .grad by accumulation, so zero it every step.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
