Computational graphs

Write a computation as a graph of elementary operations; each node knows its local derivative, and gradients flow back along the edges.

7 min read699 wordscore0 viewsupdated 2026-10-09

Definition

Computational graph A directed acyclic graph whose nodes are elementary operations (add, multiply, exp, matmul, …) and whose edges carry the tensors flowing between them. Evaluating it in topological order computes the function; walking it in reverse computes the gradient.

A neural network's loss is one enormous formula in millions of parameters. Nobody differentiates that by hand. Instead, the formula is broken into a few kinds of simple operations, and each operation only has to know one thing: the derivative of its output with respect to its own inputs.

That local derivative is all a node needs. The chain rule stitches the local pieces together along the edges, so the gradient of the loss with respect to every parameter comes out of one backward sweep through the graph.

Nodes, edges and the order of evaluation

Leaves are the inputs and parameters. Every inner node v=g(u1,…,uk)v = g(u_1,\dots,u_k) applies one operation to its parents. The graph must be acyclic: a value cannot depend on itself, so there is always a topological order in which every node comes after its inputs. A recurrent network is no exception; it is unrolled over time into a longer DAG.

The forward pass visits nodes in topological order and caches every intermediate value. The backward pass visits them in reverse order and computes the adjoint vˉ=∂L/∂v\bar v = \partial L/\partial v of each node from the adjoints of its children.

uˉ  =  ∑v ∈ children(u)vˉ ∂v∂u,Lˉ=1\bar u \;=\; \sum_{v\,\in\,\mathrm{children}(u)} \bar v\,\frac{\partial v}{\partial u},\qquad \bar L = 1
LL
the scalar output, usually the loss
vˉ=∂L/∂v\bar v = \partial L/\partial v
adjoint: upstream gradient arriving at node v
∂v/∂u\partial v/\partial u
local derivative of node v with respect to its input u, evaluated at the cached forward values
children(u)\mathrm{children}(u)
every node that consumes u; the sum runs over all of them

Each node multiplies the upstream gradient by its local derivative and sends the result to its inputs. When a value fans out to several consumers, the gradients coming back from them add up. That sum is just the multivariable chain rule: a change in uu reaches LL along every path, and the effects add.

Local gradients of the basic gates

GateForwardLocal derivativesBackward behaviour
addz=x+yz = x + y∂z/∂x=∂z/∂y=1\partial z/\partial x = \partial z/\partial y = 1distributes: both inputs get gˉ\bar g unchanged
mulz=x yz = x\,y∂z/∂x=y, ∂z/∂y=x\partial z/\partial x = y,\ \partial z/\partial y = xswaps: xx gets gˉ y\bar g\,y, yy gets gˉ x\bar g\,x
maxz=max⁡(x,y)z = \max(x, y)11 for the larger input, 00 for the otherroutes: all of gˉ\bar g goes to the winner
expz=exz = e^{x}∂z/∂x=ex=z\partial z/\partial x = e^{x} = zxx gets gˉ z\bar g\,z, reusing the cached output
For an output z and upstream gradient ḡ = ∂L/∂z, what each input receives.

ReLU is a max gate against 0, which is why its gradient is either passed through or blocked. The exp gate shows why the forward cache matters: its derivative is its own output, so nothing has to be recomputed. A sigmoid node does the same trick with σ′(z)=σ(z)(1−σ(z))\sigma'(z) = \sigma(z)(1-\sigma(z)).

f = (x·y + max(y, z)) · wbackward pass · max routes · fan-out sums←−3←2←−1←0←−1←−1←−1←−3x−2∂ −3y3∂ 1z1∂ 0×q = −6∂ −1maxm = 3∂ −1+s = −3∂ −1w−1∂ −3×f = 3∂ 1below: value · above: ∂f/∂node · on edge: gradient flowing back

Max gate: the whole gradient −1 goes to the winner (y), the loser gets 0. y fed two gates, so its gradients add: ∂f/∂y = 2 + −1 = 1.

step 10/10|
f = 3∇f = (∂x, ∂y, ∂z, ∂w) = (−3, 1, 0, −3)
fig.f = (x·y + max(y, z))·w. Step forward to fill in values, then backward to watch each gate apply its rule. y feeds two gates, so its gradient is a sum. Drag z above y and the max gate reroutes its gradient from y to z.

Worked example: f = (x + y)·z

Take x=−2x = -2, y=5y = 5, z=−4z = -4 and name the intermediate q=x+yq = x + y, so f=q zf = q\,z.

StepNodeComputationResult
forwardqx+y=−2+5x + y = -2 + 5q=3q = 3
forwardfq z=3⋅(−4)q\,z = 3\cdot(-4)f=−12f = -12
backwardfseed∂f/∂f=1\partial f/\partial f = 1
backwardzmul swaps: 1⋅q1\cdot q∂f/∂z=3\partial f/\partial z = 3
backwardqmul swaps: 1⋅z1\cdot z∂f/∂q=−4\partial f/\partial q = -4
backwardx, yadd distributes: −4⋅1-4\cdot 1∂f/∂x=∂f/∂y=−4\partial f/\partial x = \partial f/\partial y = -4

Check against the closed form f=(x+y)zf = (x+y)z: ∂f/∂x=z=−4\partial f/\partial x = z = -4 and ∂f/∂z=x+y=3\partial f/\partial z = x + y = 3. The graph got there using only the four local rules and the cached values qq and zz.

Static vs dynamic graphs

Define-and-run (static): you first declare the whole graph, then feed data through it many times. TensorFlow 1.x and Theano worked this way. The framework sees the full program in advance, so it can fuse ops, plan memory and compile for accelerators, but Python control flow such as if and for has to be expressed with special graph ops.

Define-by-run (dynamic): the graph is recorded while ordinary code executes, and rebuilt on every forward pass. PyTorch and TensorFlow eager work this way. Loops, branches and debuggers just work, and the graph can differ from one input to the next, at the price of less global optimization. Tools like torch.compile and JAX's jit close the gap by tracing a dynamic program into a static graph.

python
1import torch
2
3x = torch.tensor(-2.0, requires_grad=True)
4y = torch.tensor(5.0, requires_grad=True)
5z = torch.tensor(-4.0, requires_grad=True)
6
7q = x + y # q.grad_fn = <AddBackward0>
8f = q * z # f.grad_fn = <MulBackward0>
9f.backward() # walk the graph in reverse
10
11print(x.grad, y.grad, z.grad) # tensor(-4.) tensor(-4.) tensor(3.)
PyTorch builds the graph as the code runs. Each result remembers the op that made it (grad_fn).

Check yourself

A max gate computes z = max(x, y) with x = 4, y = 1, and receives upstream gradient 3. What gradients do x and y get?

Check yourself

In f = (x·y + max(y, z))·w with x = −2, y = 3, z = 1, w = −1, what is ∂f/∂y?

Remember

  • ✓A computational graph is a DAG of elementary ops; evaluate it in topological order, differentiate it in reverse order.
  • ✓Each node only needs its local derivative; the chain rule multiplies them along paths.
  • ✓Add distributes, mul swaps, max routes; exp and sigmoid reuse their cached outputs.
  • ✓When a value feeds several nodes, its gradients are summed (accumulate with +=).
  • ✓Static graphs (define-and-run) optimize better; dynamic graphs (define-by-run) are easier to write and debug.
← back to Deep Learning roadmap