Automatic differentiation

Record the operations of a forward pass, then apply the chain rule backwards to get exact gradients at about the cost of one more forward pass.

8 min read734 wordscore0 viewsupdated 2026-10-09

Definition

Automatic differentiation (autodiff) A technique that computes exact derivatives of a program by applying the chain rule to every elementary operation it executes. It works on code, including loops and branches, not on a symbolic formula, and it has no step size to tune.

Training needs the gradient of a scalar loss with respect to every parameter, often hundreds of millions of them, at every step. How you get that gradient decides whether training is possible at all.

Three ways to get a derivative

MethodHowAccuracyCost for n inputs
SymbolicManipulate the formula with rules, as a computer algebra system doesExactExpressions can blow up in size (expression swell); needs a closed form, not a program
NumericFinite differences, e.g. f(x+hei)−f(x−hei)2h\frac{f(x+h e_i)-f(x-he_i)}{2h}Approximate: truncation error from hh, round-off error from floating pointn+1n+1 (forward) or 2n2n (central) evaluations of ff
AutomaticChain rule applied to each executed operation, with numbers rather than expressionsExact up to floating-point roundingReverse mode: a small constant times one evaluation, independent of nn

Finite differences are useless for training a network with 10810^8 parameters (2×1082\times10^8 forward passes per step), but they are the standard way to test an autodiff implementation on a few coordinates.

The tape (Wengert list)

Running the program produces a linear sequence of elementary assignments vi=ϕi(vj,vk)v_i = \phi_i(v_{j}, v_k), each depending only on earlier entries. For f(x1,x2,x3)=x1x2+x3sin⁡x1f(x_1,x_2,x_3) = x_1x_2 + x_3\sin x_1 the tape is v4=v1v2v_4 = v_1v_2, v5=sin⁡v1v_5 = \sin v_1, v6=v3v5v_6 = v_3v_5, v7=v4+v6v_7 = v_4 + v_6. This list is a computational graph laid out in topological order. Autodiff can sweep it in either direction.

Forward mode: one pass per input

Carry a tangent v˙i=∂vi/∂xk\dot v_i = \partial v_i/\partial x_k alongside every value. Seed x˙=ek\dot x = e_k and apply each op's derivative rule as you go. One pass gives one column of the Jacobian: how every output responds to input xkx_k.

f(a+b ε)=f(a)+f′(a) b ε,ε2=0f(a + b\,\varepsilon) = f(a) + f'(a)\,b\,\varepsilon,\qquad \varepsilon^2 = 0
a+bεa + b\varepsilon
dual number: value a, tangent b
ε\varepsilon
formal infinitesimal whose square is zero, so every Taylor term past the first vanishes

Dual numbers are forward mode in algebraic form. With x1=2+εx_1 = 2 + \varepsilon, x2=5x_2 = 5, x3=3x_3 = 3, ordinary arithmetic gives f=12.7279+3.7516 εf = 12.7279 + 3.7516\,\varepsilon, so ∂f/∂x1=3.7516\partial f/\partial x_1 = 3.7516. To get ∂f/∂x2\partial f/\partial x_2 you must run the whole thing again with the ε\varepsilon on x2x_2.

Reverse mode: one pass per output

Run the program once, recording the tape. Then seed fˉ=1\bar f = 1 and walk the tape backwards, propagating adjoints vˉi=∂f/∂vi\bar v_i = \partial f/\partial v_i:

vˉj  +=  vˉi ∂ϕi∂vjfor each tape entry vi=ϕi(…,vj,… ), in reverse order\bar v_j \;\mathrel{+}=\; \bar v_i\,\frac{\partial \phi_i}{\partial v_j}\quad\text{for each tape entry } v_i=\phi_i(\dots,v_j,\dots)\text{, in reverse order}
vˉi=∂f/∂vi\bar v_i = \partial f/\partial v_i
adjoint of entry i, starting from 1 at the output
∂ϕi/∂vj\partial\phi_i/\partial v_j
local derivative of the op, evaluated with values saved on the tape
+=\mathrel{+}=
accumulate: a value used several times collects one contribution per use

One backward sweep yields one row of the Jacobian: how one output depends on every input. For f:Rn→Rmf:\mathbb{R}^n\to\mathbb{R}^m, forward mode needs nn passes and reverse mode mm. A loss is a single scalar (m=1m = 1) of millions of parameters, so reverse mode gets the full gradient in one sweep. Backpropagation is reverse-mode autodiff applied to a neural network.

f(x₁, x₂, x₃) = x₁·x₂ + x₃·sin x₁ at x = (2, 5, 3)forward sweep (record tape)tape entryvalueadjoint v̄adjoint rule (on pop)1v₁ = x₁2.00002v₂ = x₂5.00003v₃ = x₃3.00004v₄ = v₁ · v₂v̄₁ += v̄₄·v₂, v̄₂ += v̄₄·v₁5v₅ = sin v₁v̄₁ += v̄₅·cos v₁6v₆ = v₃ · v₅v̄₃ += v̄₆·v₅, v̄₅ += v̄₆·v₃7v₇ = v₄ + v₆v̄₄ += v̄₇, v̄₆ += v̄₇sweep direction ↓ (tape order)sweeps used:forward sweep (record tape)

Reverse mode, sweep 1: run the function once and write every operation and its inputs onto the tape.

|step 1/10
∂f/∂x₁ = —∂f/∂x₂ = —∂f/∂x₃ = —sweeps: 1 (one forward + one backward)
fig.The tape for f = x₁x₂ + x₃ sin x₁ at (2, 5, 3). In forward mode, step through all three passes, one per input. Switch to reverse mode: one recording sweep and one backward sweep fill in all three partials, and v̄₁ is accumulated from two uses.

Reverse sweep on the example, x = (2, 5, 3)

  1. 1Forward: v4=10v_4 = 10, v5=sin⁡2=0.9093v_5 = \sin 2 = 0.9093, v6=3⋅0.9093=2.7279v_6 = 3\cdot0.9093 = 2.7279, f=v7=12.7279f = v_7 = 12.7279.
  2. 2Seed vˉ7=1\bar v_7 = 1. The add gives vˉ4=vˉ6=1\bar v_4 = \bar v_6 = 1.
  3. 3Row 6, v6=v3v5v_6 = v_3v_5: vˉ3=vˉ6 v5=0.9093\bar v_3 = \bar v_6\,v_5 = 0.9093, vˉ5=vˉ6 v3=3\bar v_5 = \bar v_6\,v_3 = 3.
  4. 4Row 5, v5=sin⁡v1v_5 = \sin v_1: vˉ1+=3cos⁡2=3⋅(−0.4161)=−1.2484\bar v_1 \mathrel{+}= 3\cos 2 = 3\cdot(-0.4161) = -1.2484.
  5. 5Row 4, v4=v1v2v_4 = v_1v_2: vˉ1+=vˉ4 v2=5\bar v_1 \mathrel{+}= \bar v_4\,v_2 = 5, so vˉ1=3.7516\bar v_1 = 3.7516; vˉ2=vˉ4 v1=2\bar v_2 = \bar v_4\,v_1 = 2.
  6. 6Result: ∇f=(3.7516, 2, 0.9093)\nabla f = (3.7516,\ 2,\ 0.9093), matching (x2+x3cos⁡x1, x1, sin⁡x1)(x_2 + x_3\cos x_1,\ x_1,\ \sin x_1).

Why not just use finite differences?

hForward difference errorCentral difference error
1e−11.3e−12.1e−3
1e−31.4e−32.1e−7
1e−51.4e−58.2e−12
1e−71.4e−76.3e−10
1e−91.4e−64.6e−7
1e−111.1e−41.1e−4
Absolute error in ∂f/∂x₁ at (2, 5, 3), exact value 3.751559. Reverse-mode autodiff matches the analytic derivative to machine precision (~1e−16).

Shrinking hh helps only up to a point. Truncation error falls like hh (forward) or h2h^2 (central), but round-off error grows like ϵmach/h\epsilon_{\text{mach}}/h, because subtracting two nearly equal numbers cancels most significant digits. The best central difference here is about 10−1110^{-11} at h=10−5h=10^{-5}; below that the error climbs again. Autodiff has no hh and no cancellation.

A tiny autograd engine

python
1import math
2
3class Value:
4 def __init__(self, data, parents=(), local=()):
5 self.data, self.grad = data, 0.0
6 self.parents, self.local = parents, local # inputs and d(self)/d(input)
7
8 def __add__(self, o):
9 return Value(self.data + o.data, (self, o), (1.0, 1.0))
10
11 def __mul__(self, o):
12 return Value(self.data * o.data, (self, o), (o.data, self.data))
13
14 def sin(self):
15 return Value(math.sin(self.data), (self,), (math.cos(self.data),))
16
17 def backward(self):
18 order, seen = [], set()
19 def visit(v): # topological order = the tape
20 if v not in seen:
21 seen.add(v)
22 for p in v.parents:
23 visit(p)
24 order.append(v)
25 visit(self)
26 self.grad = 1.0
27 for v in reversed(order): # sweep the tape backwards
28 for p, d in zip(v.parents, v.local):
29 p.grad += v.grad * d # += : fan-out sums
30
31x1, x2, x3 = Value(2.0), Value(5.0), Value(3.0)
32f = x1 * x2 + x3 * x1.sin()
33f.backward()
34print(round(f.data, 4), round(x1.grad, 4), x2.grad, round(x3.grad, 4))
35# 12.7279 3.7516 2.0 0.9093
Each Value remembers its parents and local derivatives. backward() orders the graph topologically and sweeps it in reverse.

PyTorch autograd

Tensors created with requires_grad=True are leaves that autograd tracks. Every op on them records a node (its grad_fn). Calling .backward() on a scalar runs reverse mode and adds each gradient into the leaf's .grad. The graph is freed afterwards unless you pass retain_graph=True.

python
1import torch
2
3x = torch.tensor([2.0, 5.0, 3.0], requires_grad=True)
4
5def f(x):
6 return x[0] * x[1] + x[2] * torch.sin(x[0])
7
8f(x).backward()
9print(x.grad) # tensor([3.7516, 2.0000, 0.9093])
10
11f(x).backward() # a second backward without zeroing
12print(x.grad) # tensor([7.5031, 4.0000, 1.8186]) <- accumulated
13
14x.grad.zero_() # in a training loop: optimizer.zero_grad()
15
16with torch.no_grad(): # inference: record no graph, save memory
17 y = f(x)

Check yourself

You need the full Jacobian of a function from 3 inputs to 1000 outputs. Which mode needs fewer passes?

Check yourself

A central finite difference with h = 1e−12 gives a worse derivative than with h = 1e−5. Why?

Remember

  • ✓Autodiff applies the chain rule to the executed operations: exact derivatives, no step size, works through loops and branches.
  • ✓Finite differences cost about n evaluations for n inputs and trade truncation error against round-off; use them only to check gradients.
  • ✓Forward mode (tangents, dual numbers) needs one pass per input; reverse mode (adjoints on a tape) needs one pass per output.
  • ✓A loss has one output, so reverse mode gives the whole gradient for a small constant times the forward cost; backprop is exactly this.
  • ✓In PyTorch: requires_grad marks leaves, .backward() fills .grad by accumulation, so zero it every step.
← back to Deep Learning roadmap