# Message Passing Neural Networks (MPNN)

> MPNN (Gilmer et al., 2017) writes a GNN as T rounds of messages M(h_v, h_w, e_vw) summed over neighbours, an update U and a permutation-invariant readout R; an edge network + GRU + set2set reached chemical accuracy on 11 of 13 QM9 targets.

Source: https://www.mlfoundry.in/learn/message-passing-neural-networks-mpnn  
Section: Deep Learning › Graph Neural Networks (GNNs)  
Level: core · 8 min read · updated 2026-10-21

**Message Passing Neural Network (MPNN):** is a framework, introduced by Gilmer, Schoenholz, Riley, Vinyals and Dahl in "Neural Message Passing for Quantum Chemistry" (ICML 2017). It describes a graph neural network by three learned functions: a **message** function $M_t$ computed on every edge, an **update** function $U_t$ for every node, and a **readout** $R$ that maps the final node states to a graph-level output.

The input is a graph with node features $x_v$ and edge features $e_{vw}$. For a molecule these are atoms (element, charge, hybridisation, …) and bonds (single, double, aromatic, distance). The output here is one vector per graph: 13 quantum-chemical properties of the molecule.

Three constraints shape the model. Graphs vary in size, so the weights cannot depend on the number of nodes. There is no grid, so a node has no "left neighbour" for a convolution kernel to align with. And the atoms of a molecule have no canonical order, so the output must not change when the nodes are renumbered (**permutation invariance**).

Before 2017, several models met these constraints in different notation: molecular fingerprints by Duvenaud et al. (2015), Gated Graph Neural Networks (GG-NN, Li et al., 2016), Interaction Networks (Battaglia et al., 2016), Kearnes et al.'s molecular graph convolutions, Deep Tensor Neural Networks (Schütt et al., 2017), and spectral GCNs. Gilmer et al. showed they are all instances of one scheme. They then searched within that scheme for the best variant on QM9.

## Message, update, readout

Node states start from the features, $h_v^0=x_v$. For $T$ rounds, every node gathers messages from its neighbours and updates its state:

$$
m_v^{t+1}=\sum_{w\in N(v)} M_t\big(h_v^t,\,h_w^t,\,e_{vw}\big),\qquad h_v^{t+1}=U_t\big(h_v^t,\,m_v^{t+1}\big)
$$

- $N(v)$: neighbours of node $v$
- $M_t$: message function, applied to every edge with shared weights
- $U_t$: update function, shared by all nodes
- $\sum$: the aggregator; any symmetric function (sum, mean, max) keeps the result independent of neighbour order

After $T$ rounds a readout turns the set of final node states into the prediction:

$$
\hat y = R\big(\{\,h_v^T \mid v\in G\,\}\big)
$$

- $R$: must be invariant to permutations of the nodes, e.g. a sum, a mean or set2set
- $T$: number of message-passing rounds; $h_v^T$ depends only on nodes within $T$ hops of $v$

> Figure: One round of message passing. Each neighbour w sends M_t(h_v, h_w, e_vw) along its edge. The messages are summed into m_v, and U_t (a GRU in the paper) combines m_v with h_v. After T rounds, R reads out the set of all final states. The 2-hop node u first reaches v at round 2. The table at the bottom writes three models as choices of M, U and R.

Because $M_t$ and $U_t$ are shared across edges and nodes, the parameter count does not depend on graph size. Because the aggregator and $R$ are symmetric, relabelling the nodes permutes the $h_v^t$ but leaves $\hat y$ unchanged. Node-level tasks skip $R$ and read $h_v^T$ directly.

## The variant that won on QM9

Gilmer et al. compared several message and readout functions. The best used an **edge network**: a small MLP maps the edge feature vector to a $d\times d$ matrix that multiplies the neighbour's state. Updates used a **Gated recurrent unit (GRU)** whose weights are shared across all $T$ rounds, as in GG-NN. The readout was **set2set** (Vinyals, Bengio and Kudlur, 2016), an LSTM-driven attention over the node set:

$$
M\big(h_v,h_w,e_{vw}\big)=A(e_{vw})\,h_w,\qquad U=\mathrm{GRU}\big(h_v^t,\,m_v^{t+1}\big),\qquad R=\mathrm{set2set}\big(\{h_v^T\}\big)
$$

- $A(e_{vw})$: MLP output reshaped to $d\times d$; a continuous edge feature such as bond length changes the message smoothly
- $\mathrm{GRU}$: gated update: the message is the GRU input, $h_v^t$ the hidden state

Two further ideas from the paper: a **virtual master node** connected to every atom, which lets information cross the molecule in two hops, and **towers**, which split $h_v$ into $k$ chunks of size $d/k$ with separate message functions to cut the $d^2$ cost of the edge network.

**Setting and results (Gilmer et al., 2017)**

- **Dataset:** QM9: about 134k (133,885) small organic molecules, up to 9 heavy atoms (C, N, O, F); properties computed with DFT
- **Targets:** 13 per molecule, including dipole moment, HOMO, LUMO, gap, zero-point energy, internal energy U₀ and heat capacity
- **Split:** 10,000 validation and 10,000 test molecules chosen at random; the rest for training
- **Metric:** error ratio = MAE / chemical accuracy for each target; below 1 means within chemical accuracy
- **Best model:** edge network + GRU + set2set: chemical accuracy on 11 of the 13 targets

> Figure: Real message passing on a 6-atom graph with 2-d states, edge-network messages A(e) h_w (one fixed matrix per bond type) and U = tanh(W h + V m). Default (t = 1, sum, atom C1): messages [0.20, 0.50] from O0, [0.20, 0.80] from C2 over the double bond and [0.40, 0.30] from N4 sum to m = [0.80, 1.60], so h = [0.88, 0.87]. C1 already depends on 4 of the 6 atoms, and on all 6 from t = 3. The sum readout reads 1.00, 0.53, 0.65 at t = 0, 1, 2. Switch to mean: m = [0.27, 0.53] and h = [0.66, 0.52]. Atom O0 needs t = 4 to see O5, because the graph diameter is 4.

## Worked example: two rounds by hand

Take 4 nodes with scalar states $h^0=(A{:}\,1,\ B{:}\,2,\ C{:}\,0,\ D{:}\,1)$ and edges A–B, B–D, C–D (weight 1) and B–C (weight 0.5). Use $M=a(e_{vw})\,h_w$, $U=\mathrm{ReLU}(0.5\,h_v+0.5\,m_v)$ and a sum readout. The initial readout is $\sum_v h_v^0=4$.

Every message is the edge weight times the neighbour's state from the previous round

| Node | h⁰ | messages, round 1 | m¹ | h¹ | messages, round 2 | m² | h² |
| --- | --- | --- | --- | --- | --- | --- | --- |
| A | 1 | 1·2 | 2 | 1.5 | 1·2 | 2 | 1.75 |
| B | 2 | 1·1 + 0.5·0 + 1·1 | 2 | 2 | 1·1.5 + 0.5·1 + 1·1.5 | 3.5 | 2.75 |
| C | 0 | 0.5·2 + 1·1 | 2 | 1 | 0.5·2 + 1·1.5 | 2.5 | 1.75 |
| D | 1 | 1·2 + 1·0 | 2 | 1.5 | 1·2 + 1·1 | 3 | 2.25 |
| Σ (readout) | 4 |  |  | 6 |  |  | 8.5 |

In round 1 every node happens to receive $m=2$, yet the states differ because $U$ also sees $h_v$. Renumbering the nodes would reorder the rows but leave the readouts 6 and 8.5 unchanged.

```python
import torch
from torch import nn
from torch_geometric.datasets import QM9
from torch_geometric.loader import DataLoader
from torch_geometric.nn import NNConv, Set2Set

class MPNN(nn.Module):
    def __init__(self, d_node=11, d_edge=4, d=64, T=3):
        super().__init__()
        self.T = T
        self.embed = nn.Linear(d_node, d)
        A = nn.Sequential(nn.Linear(d_edge, 128), nn.ReLU(), nn.Linear(128, d * d))  # e_vw -> d×d
        self.msg = NNConv(d, d, A, aggr="add")       # m_v = Σ_w A(e_vw) h_w
        self.gru = nn.GRU(d, d)                       # U, shared across the T rounds
        self.readout = Set2Set(d, processing_steps=3) # R, permutation-invariant
        self.head = nn.Sequential(nn.Linear(2 * d, d), nn.ReLU(), nn.Linear(d, 1))

    def forward(self, g):
        h = torch.relu(self.embed(g.x))
        state = h.unsqueeze(0)
        for _ in range(self.T):
            m = torch.relu(self.msg(h, g.edge_index, g.edge_attr))
            h, state = self.gru(m.unsqueeze(0), state)
            h = h.squeeze(0)
        return self.head(self.readout(h, g.batch)).view(-1)

ds = QM9("data/QM9").shuffle()
test, val, train = ds[:10000], ds[10000:20000], ds[20000:]
model = MPNN()
opt = torch.optim.Adam(model.parameters(), lr=1e-3)
for g in DataLoader(train, batch_size=128, shuffle=True):
    loss = nn.functional.l1_loss(model(g), g.y[:, 0])   # MAE on target 0 (dipole moment)
    opt.zero_grad(); loss.backward(); opt.step()
```

The Gilmer et al. edge-network MPNN on QM9 in PyTorch Geometric, after PyG's qm9_nn_conv example. NNConv is the edge network; not run here, since torch and PyG were not available.

## What the framework implies

**Graph Convolutional Network (GCN)** is the case $M=c_{vw}h_w$ with $c_{vw}=(\deg v\,\deg w)^{-1/2}$ and $U=\sigma(Wm_v)$. **Graph Attention Network (GAT)** makes the weight of each message a learned attention coefficient. **GraphSAGE** swaps the sum for mean, max-pool or LSTM aggregators. **Graph Isomorphism Network (GIN)** keeps the sum and adds $(1+\epsilon)h_v$.

The framework also has a ceiling. Xu et al. (2019) and Morris et al. (2019) proved that an MPNN distinguishes two graphs at most as well as the 1-WL colour-refinement test. Mean and max aggregators can be weaker still, because they lose neighbour counts. Battaglia et al. (2018) generalised the scheme to **graph networks**, which also update edge states and a global state.

> **T is a receptive field, and more is not free:** After $T$ rounds a node sees only its $T$-hop neighbourhood, so long-range effects need large $T$ or a master node. Stacking many rounds makes node states converge (**over-smoothing**). It also squeezes an exponentially growing neighbourhood into one fixed-size vector (**over-squashing**). **Graph transformers** attack both by letting every node attend to every other.

**Question:** Why must the MPNN readout R be a symmetric function such as sum, mean or set2set?

- A. To keep the number of parameters fixed
- B. Atoms have no canonical order, so renumbering the nodes must not change the prediction
- C. Because the GRU update requires it
- D. So that gradients reach every node

**Answer:** B. Atoms have no canonical order, so renumbering the nodes must not change the prediction — Renumbering the nodes permutes the set $\{h_v^T\}$. A symmetric $R$ returns the same $\hat y$ for every ordering. Concatenating the node states in index order would give different outputs for the same molecule.

**Question:** In the interactive figure, why does atom O0's state only start to depend on O5 at t = 4?

- A. The double bond to O5 blocks messages
- B. The mean aggregator discards O5's message
- C. O5 is 4 hops from O0, and each round extends the receptive field by one hop
- D. The readout ignores O5

**Answer:** C. O5 is 4 hops from O0, and each round extends the receptive field by one hop — $h_v^t$ is a function of nodes within $t$ hops. The shortest path O0–C1–C2–C3–O5 has 4 edges, so O5 influences O0 only from round 4.

**Key points**

- MPNN (Gilmer et al., ICML 2017): $m_v^{t+1}=\sum_{w}M_t(h_v^t,h_w^t,e_{vw})$, $h_v^{t+1}=U_t(h_v^t,m_v^{t+1})$, $\hat y=R(\{h_v^T\})$.
- Shared M and U handle graphs of any size. A symmetric aggregator and readout give permutation invariance.
- GCN, GG-NN, Interaction Networks, molecular fingerprints and DTNN are all special cases.
- The best QM9 model: edge network $A(e_{vw})h_w$, a GRU update shared over rounds, and a set2set readout. It reached chemical accuracy on 11 of 13 targets.
- T rounds give a T-hop receptive field. Deep stacks over-smooth and over-squash.
- Expressiveness is bounded by the 1-WL test. GIN reaches that bound with sum aggregation.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
