Definition
Computational graph A directed acyclic graph whose nodes are elementary operations (add, multiply, exp, matmul, …) and whose edges carry the tensors flowing between them. Evaluating it in topological order computes the function; walking it in reverse computes the gradient.
A neural network's loss is one enormous formula in millions of parameters. Nobody differentiates that by hand. Instead, the formula is broken into a few kinds of simple operations, and each operation only has to know one thing: the derivative of its output with respect to its own inputs.
That local derivative is all a node needs. The chain rule stitches the local pieces together along the edges, so the gradient of the loss with respect to every parameter comes out of one backward sweep through the graph.
Nodes, edges and the order of evaluation
Leaves are the inputs and parameters. Every inner node applies one operation to its parents. The graph must be acyclic: a value cannot depend on itself, so there is always a topological order in which every node comes after its inputs. A recurrent network is no exception; it is unrolled over time into a longer DAG.
The forward pass visits nodes in topological order and caches every intermediate value. The backward pass visits them in reverse order and computes the adjoint of each node from the adjoints of its children.
- the scalar output, usually the loss
- adjoint: upstream gradient arriving at node v
- local derivative of node v with respect to its input u, evaluated at the cached forward values
- every node that consumes u; the sum runs over all of them
Each node multiplies the upstream gradient by its local derivative and sends the result to its inputs. When a value fans out to several consumers, the gradients coming back from them add up. That sum is just the multivariable chain rule: a change in reaches along every path, and the effects add.
Local gradients of the basic gates
| Gate | Forward | Local derivatives | Backward behaviour |
|---|---|---|---|
| add | distributes: both inputs get unchanged | ||
| mul | swaps: gets , gets | ||
| max | for the larger input, for the other | routes: all of goes to the winner | |
| exp | gets , reusing the cached output |
ReLU is a max gate against 0, which is why its gradient is either passed through or blocked. The exp gate shows why the forward cache matters: its derivative is its own output, so nothing has to be recomputed. A sigmoid node does the same trick with .
Max gate: the whole gradient −1 goes to the winner (y), the loser gets 0. y fed two gates, so its gradients add: ∂f/∂y = 2 + −1 = 1.
Worked example: f = (x + y)·z
Take , , and name the intermediate , so .
| Step | Node | Computation | Result |
|---|---|---|---|
| forward | q | ||
| forward | f | ||
| backward | f | seed | |
| backward | z | mul swaps: | |
| backward | q | mul swaps: | |
| backward | x, y | add distributes: |
Check against the closed form : and . The graph got there using only the four local rules and the cached values and .
Static vs dynamic graphs
Define-and-run (static): you first declare the whole graph, then feed data through it many times. TensorFlow 1.x and Theano worked this way. The framework sees the full program in advance, so it can fuse ops, plan memory and compile for accelerators, but Python control flow such as if and for has to be expressed with special graph ops.
Define-by-run (dynamic): the graph is recorded while ordinary code executes, and rebuilt on every forward pass. PyTorch and TensorFlow eager work this way. Loops, branches and debuggers just work, and the graph can differ from one input to the next, at the price of less global optimization. Tools like torch.compile and JAX's jit close the gap by tracing a dynamic program into a static graph.
1import torch23x = torch.tensor(-2.0, requires_grad=True)4y = torch.tensor(5.0, requires_grad=True)5z = torch.tensor(-4.0, requires_grad=True)67q = x + y # q.grad_fn = <AddBackward0>8f = q * z # f.grad_fn = <MulBackward0>9f.backward() # walk the graph in reverse1011print(x.grad, y.grad, z.grad) # tensor(-4.) tensor(-4.) tensor(3.)Check yourself
A max gate computes z = max(x, y) with x = 4, y = 1, and receives upstream gradient 3. What gradients do x and y get?
Check yourself
In f = (x·y + max(y, z))·w with x = −2, y = 3, z = 1, w = −1, what is ∂f/∂y?
Remember
- ✓A computational graph is a DAG of elementary ops; evaluate it in topological order, differentiate it in reverse order.
- ✓Each node only needs its local derivative; the chain rule multiplies them along paths.
- ✓Add distributes, mul swaps, max routes; exp and sigmoid reuse their cached outputs.
- ✓When a value feeds several nodes, its gradients are summed (accumulate with +=).
- ✓Static graphs (define-and-run) optimize better; dynamic graphs (define-by-run) are easier to write and debug.