Definition
Automatic differentiation (autodiff) A technique that computes exact derivatives of a program by applying the chain rule to every elementary operation it executes. It works on code, including loops and branches, not on a symbolic formula, and it has no step size to tune.
Training needs the gradient of a scalar loss with respect to every parameter, often hundreds of millions of them, at every step. How you get that gradient decides whether training is possible at all.
Three ways to get a derivative
| Method | How | Accuracy | Cost for n inputs |
|---|---|---|---|
| Symbolic | Manipulate the formula with rules, as a computer algebra system does | Exact | Expressions can blow up in size (expression swell); needs a closed form, not a program |
| Numeric | Finite differences, e.g. | Approximate: truncation error from , round-off error from floating point | (forward) or (central) evaluations of |
| Automatic | Chain rule applied to each executed operation, with numbers rather than expressions | Exact up to floating-point rounding | Reverse mode: a small constant times one evaluation, independent of |
Finite differences are useless for training a network with parameters ( forward passes per step), but they are the standard way to test an autodiff implementation on a few coordinates.
The tape (Wengert list)
Running the program produces a linear sequence of elementary assignments , each depending only on earlier entries. For the tape is , , , . This list is a computational graph laid out in topological order. Autodiff can sweep it in either direction.
Forward mode: one pass per input
Carry a tangent alongside every value. Seed and apply each op's derivative rule as you go. One pass gives one column of the Jacobian: how every output responds to input .
- dual number: value a, tangent b
- formal infinitesimal whose square is zero, so every Taylor term past the first vanishes
Dual numbers are forward mode in algebraic form. With , , , ordinary arithmetic gives , so . To get you must run the whole thing again with the on .
Reverse mode: one pass per output
Run the program once, recording the tape. Then seed and walk the tape backwards, propagating adjoints :
- adjoint of entry i, starting from 1 at the output
- local derivative of the op, evaluated with values saved on the tape
- accumulate: a value used several times collects one contribution per use
One backward sweep yields one row of the Jacobian: how one output depends on every input. For , forward mode needs passes and reverse mode . A loss is a single scalar () of millions of parameters, so reverse mode gets the full gradient in one sweep. Backpropagation is reverse-mode autodiff applied to a neural network.
Reverse mode, sweep 1: run the function once and write every operation and its inputs onto the tape.
Reverse sweep on the example, x = (2, 5, 3)
- 1Forward: , , , .
- 2Seed . The add gives .
- 3Row 6, : , .
- 4Row 5, : .
- 5Row 4, : , so ; .
- 6Result: , matching .
Why not just use finite differences?
| h | Forward difference error | Central difference error |
|---|---|---|
| 1e−1 | 1.3e−1 | 2.1e−3 |
| 1e−3 | 1.4e−3 | 2.1e−7 |
| 1e−5 | 1.4e−5 | 8.2e−12 |
| 1e−7 | 1.4e−7 | 6.3e−10 |
| 1e−9 | 1.4e−6 | 4.6e−7 |
| 1e−11 | 1.1e−4 | 1.1e−4 |
Shrinking helps only up to a point. Truncation error falls like (forward) or (central), but round-off error grows like , because subtracting two nearly equal numbers cancels most significant digits. The best central difference here is about at ; below that the error climbs again. Autodiff has no and no cancellation.
A tiny autograd engine
1import math23class Value:4 def __init__(self, data, parents=(), local=()):5 self.data, self.grad = data, 0.06 self.parents, self.local = parents, local # inputs and d(self)/d(input)78 def __add__(self, o):9 return Value(self.data + o.data, (self, o), (1.0, 1.0))1011 def __mul__(self, o):12 return Value(self.data * o.data, (self, o), (o.data, self.data))1314 def sin(self):15 return Value(math.sin(self.data), (self,), (math.cos(self.data),))1617 def backward(self):18 order, seen = [], set()19 def visit(v): # topological order = the tape20 if v not in seen:21 seen.add(v)22 for p in v.parents:23 visit(p)24 order.append(v)25 visit(self)26 self.grad = 1.027 for v in reversed(order): # sweep the tape backwards28 for p, d in zip(v.parents, v.local):29 p.grad += v.grad * d # += : fan-out sums3031x1, x2, x3 = Value(2.0), Value(5.0), Value(3.0)32f = x1 * x2 + x3 * x1.sin()33f.backward()34print(round(f.data, 4), round(x1.grad, 4), x2.grad, round(x3.grad, 4))35# 12.7279 3.7516 2.0 0.9093PyTorch autograd
Tensors created with requires_grad=True are leaves that autograd tracks. Every op on them records a node (its grad_fn). Calling .backward() on a scalar runs reverse mode and adds each gradient into the leaf's .grad. The graph is freed afterwards unless you pass retain_graph=True.
1import torch23x = torch.tensor([2.0, 5.0, 3.0], requires_grad=True)45def f(x):6 return x[0] * x[1] + x[2] * torch.sin(x[0])78f(x).backward()9print(x.grad) # tensor([3.7516, 2.0000, 0.9093])1011f(x).backward() # a second backward without zeroing12print(x.grad) # tensor([7.5031, 4.0000, 1.8186]) <- accumulated1314x.grad.zero_() # in a training loop: optimizer.zero_grad()1516with torch.no_grad(): # inference: record no graph, save memory17 y = f(x)Check yourself
You need the full Jacobian of a function from 3 inputs to 1000 outputs. Which mode needs fewer passes?
Check yourself
A central finite difference with h = 1e−12 gives a worse derivative than with h = 1e−5. Why?
Remember
- ✓Autodiff applies the chain rule to the executed operations: exact derivatives, no step size, works through loops and branches.
- ✓Finite differences cost about n evaluations for n inputs and trade truncation error against round-off; use them only to check gradients.
- ✓Forward mode (tangents, dual numbers) needs one pass per input; reverse mode (adjoints on a tape) needs one pass per output.
- ✓A loss has one output, so reverse mode gives the whole gradient for a small constant times the forward cost; backprop is exactly this.
- ✓In PyTorch: requires_grad marks leaves, .backward() fills .grad by accumulation, so zero it every step.