# Computational graphs

> Write a computation as a graph of elementary operations; each node knows its local derivative, and gradients flow back along the edges.

Source: https://www.mlfoundry.in/learn/computational-graphs  
Section: Deep Learning › Deep Learning Fundamentals  
Level: core · 7 min read · updated 2026-10-09

**Computational graph:** A directed acyclic graph whose nodes are elementary operations (add, multiply, exp, matmul, …) and whose edges carry the tensors flowing between them. Evaluating it in topological order computes the function; walking it in reverse computes the gradient.

A neural network's loss is one enormous formula in millions of parameters. Nobody differentiates that by hand. Instead, the formula is broken into a few kinds of simple operations, and each operation only has to know one thing: the derivative of its output with respect to its own inputs.

That **local derivative** is all a node needs. The chain rule stitches the local pieces together along the edges, so the gradient of the loss with respect to every parameter comes out of one backward sweep through the graph.

## Nodes, edges and the order of evaluation

Leaves are the inputs and parameters. Every inner node $v = g(u_1,\dots,u_k)$ applies one operation to its parents. The graph must be **acyclic**: a value cannot depend on itself, so there is always a topological order in which every node comes after its inputs. A recurrent network is no exception; it is unrolled over time into a longer DAG.

The **forward pass** visits nodes in topological order and caches every intermediate value. The **backward pass** visits them in reverse order and computes the adjoint $\bar v = \partial L/\partial v$ of each node from the adjoints of its children.

$$
\bar u \;=\; \sum_{v\,\in\,\mathrm{children}(u)} \bar v\,\frac{\partial v}{\partial u},\qquad \bar L = 1
$$

- $L$: the scalar output, usually the loss
- $\bar v = \partial L/\partial v$: adjoint: upstream gradient arriving at node v
- $\partial v/\partial u$: local derivative of node v with respect to its input u, evaluated at the cached forward values
- $\mathrm{children}(u)$: every node that consumes u; the sum runs over all of them

Each node multiplies the upstream gradient by its local derivative and sends the result to its inputs. When a value fans out to several consumers, the gradients coming back from them **add up**. That sum is just the multivariable chain rule: a change in $u$ reaches $L$ along every path, and the effects add.

## Local gradients of the basic gates

For an output z and upstream gradient ḡ = ∂L/∂z, what each input receives.

| Gate | Forward | Local derivatives | Backward behaviour |
| --- | --- | --- | --- |
| add | $z = x + y$ | $\partial z/\partial x = \partial z/\partial y = 1$ | **distributes**: both inputs get $\bar g$ unchanged |
| mul | $z = x\,y$ | $\partial z/\partial x = y,\ \partial z/\partial y = x$ | **swaps**: $x$ gets $\bar g\,y$, $y$ gets $\bar g\,x$ |
| max | $z = \max(x, y)$ | $1$ for the larger input, $0$ for the other | **routes**: all of $\bar g$ goes to the winner |
| exp | $z = e^{x}$ | $\partial z/\partial x = e^{x} = z$ | $x$ gets $\bar g\,z$, reusing the cached output |

ReLU is a max gate against 0, which is why its gradient is either passed through or blocked. The exp gate shows why the forward cache matters: its derivative is its own output, so nothing has to be recomputed. A sigmoid node does the same trick with $\sigma'(z) = \sigma(z)(1-\sigma(z))$.

> Figure: f = (x·y + max(y, z))·w. Step forward to fill in values, then backward to watch each gate apply its rule. y feeds two gates, so its gradient is a sum. Drag z above y and the max gate reroutes its gradient from y to z.

## Worked example: f = (x + y)·z

Take $x = -2$, $y = 5$, $z = -4$ and name the intermediate $q = x + y$, so $f = q\,z$.

| Step | Node | Computation | Result |
| --- | --- | --- | --- |
| forward | q | $x + y = -2 + 5$ | $q = 3$ |
| forward | f | $q\,z = 3\cdot(-4)$ | $f = -12$ |
| backward | f | seed | $\partial f/\partial f = 1$ |
| backward | z | mul swaps: $1\cdot q$ | $\partial f/\partial z = 3$ |
| backward | q | mul swaps: $1\cdot z$ | $\partial f/\partial q = -4$ |
| backward | x, y | add distributes: $-4\cdot 1$ | $\partial f/\partial x = \partial f/\partial y = -4$ |

Check against the closed form $f = (x+y)z$: $\partial f/\partial x = z = -4$ and $\partial f/\partial z = x + y = 3$. The graph got there using only the four local rules and the cached values $q$ and $z$.

> **Fan-out means accumulate, not overwrite:** In $f = x\cdot x$, the mul gate sends $\bar g\,x$ to each of its two input slots, and both are the same $x$. The correct gradient is $x + x = 2x$. An implementation that writes `grad = ...` instead of `grad += ...` returns $x$. This is also why PyTorch accumulates into `.grad` across calls.

## Static vs dynamic graphs

**Define-and-run** (static): you first declare the whole graph, then feed data through it many times. TensorFlow 1.x and Theano worked this way. The framework sees the full program in advance, so it can fuse ops, plan memory and compile for accelerators, but Python control flow such as `if` and `for` has to be expressed with special graph ops.

**Define-by-run** (dynamic): the graph is recorded while ordinary code executes, and rebuilt on every forward pass. PyTorch and TensorFlow eager work this way. Loops, branches and debuggers just work, and the graph can differ from one input to the next, at the price of less global optimization. Tools like `torch.compile` and JAX's `jit` close the gap by tracing a dynamic program into a static graph.

```python
import torch

x = torch.tensor(-2.0, requires_grad=True)
y = torch.tensor(5.0, requires_grad=True)
z = torch.tensor(-4.0, requires_grad=True)

q = x + y            # q.grad_fn = <AddBackward0>
f = q * z            # f.grad_fn = <MulBackward0>
f.backward()         # walk the graph in reverse

print(x.grad, y.grad, z.grad)   # tensor(-4.) tensor(-4.) tensor(3.)
```

PyTorch builds the graph as the code runs. Each result remembers the op that made it (grad_fn).

> **Memory is the real cost:** The backward pass needs the cached forward values, so training memory grows with the number of nodes kept alive, which in a deep network means activations of every layer for every example in the batch. Gradient checkpointing trades compute for memory: it discards some activations and recomputes them during the backward pass.

**Question:** A max gate computes z = max(x, y) with x = 4, y = 1, and receives upstream gradient 3. What gradients do x and y get?

- A. x: 3, y: 3
- B. x: 3, y: 0
- C. x: 1.5, y: 1.5
- D. x: 12, y: 3

**Answer:** B. x: 3, y: 0 — Max routes: the local derivative is 1 for the input that won (x = 4) and 0 for the other, so x receives the whole 3 and y receives 0.

**Question:** In f = (x·y + max(y, z))·w with x = −2, y = 3, z = 1, w = −1, what is ∂f/∂y?

- A. 2
- B. −1
- C. 1
- D. 3

**Answer:** C. 1 — Upstream at the add gate is w = −1. Through the mul gate y gets −1·x = 2; through the max gate (y wins) it gets −1. y fans out, so the two contributions add: 2 + (−1) = 1.

**Key points**

- A computational graph is a DAG of elementary ops; evaluate it in topological order, differentiate it in reverse order.
- Each node only needs its local derivative; the chain rule multiplies them along paths.
- Add distributes, mul swaps, max routes; exp and sigmoid reuse their cached outputs.
- When a value feeds several nodes, its gradients are summed (accumulate with +=).
- Static graphs (define-and-run) optimize better; dynamic graphs (define-by-run) are easier to write and debug.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
