Message Passing Neural Networks (MPNN)

MPNN (Gilmer et al., 2017) writes a GNN as T rounds of messages M(h_v, h_w, e_vw) summed over neighbours, an update U and a permutation-invariant readout R; an edge network + GRU + set2set reached chemical accuracy on 11 of 13 QM9 targets.

8 min read750 wordscore0 viewsupdated 2026-10-21

Definition

Message Passing Neural Network (MPNN) is a framework, introduced by Gilmer, Schoenholz, Riley, Vinyals and Dahl in "Neural Message Passing for Quantum Chemistry" (ICML 2017). It describes a graph neural network by three learned functions: a message function MtM_t computed on every edge, an update function UtU_t for every node, and a readout RR that maps the final node states to a graph-level output.

The input is a graph with node features xvx_v and edge features evwe_{vw}. For a molecule these are atoms (element, charge, hybridisation, …) and bonds (single, double, aromatic, distance). The output here is one vector per graph: 13 quantum-chemical properties of the molecule.

Three constraints shape the model. Graphs vary in size, so the weights cannot depend on the number of nodes. There is no grid, so a node has no "left neighbour" for a convolution kernel to align with. And the atoms of a molecule have no canonical order, so the output must not change when the nodes are renumbered (permutation invariance).

Before 2017, several models met these constraints in different notation: molecular fingerprints by Duvenaud et al. (2015), Gated Graph Neural Networks (GG-NN, Li et al., 2016), Interaction Networks (Battaglia et al., 2016), Kearnes et al.'s molecular graph convolutions, Deep Tensor Neural Networks (Schütt et al., 2017), and spectral GCNs. Gilmer et al. showed they are all instances of one scheme. They then searched within that scheme for the best variant on QM9.

Message, update, readout

Node states start from the features, hv0=xvh_v^0=x_v. For TT rounds, every node gathers messages from its neighbours and updates its state:

mvt+1=∑w∈N(v)Mt(hvt, hwt, evw),hvt+1=Ut(hvt, mvt+1)m_v^{t+1}=\sum_{w\in N(v)} M_t\big(h_v^t,\,h_w^t,\,e_{vw}\big),\qquad h_v^{t+1}=U_t\big(h_v^t,\,m_v^{t+1}\big)
N(v)N(v)
neighbours of node vv
MtM_t
message function, applied to every edge with shared weights
UtU_t
update function, shared by all nodes
∑\sum
the aggregator; any symmetric function (sum, mean, max) keeps the result independent of neighbour order

After TT rounds a readout turns the set of final node states into the prediction:

y^=R({ hvT∣v∈G })\hat y = R\big(\{\,h_v^T \mid v\in G\,\}\big)
RR
must be invariant to permutations of the nodes, e.g. a sum, a mean or set2set
TT
number of message-passing rounds; hvTh_v^T depends only on nodes within TT hops of vv
MPNN (Gilmer et al., 2017): T message-passing rounds, then a readoutureaches v at t = 2w1w2w3e_vw1e_vw2e_vw3h_w1^th_w2^tveach edge sends M_t(h_v^t, h_w^t, e_vw)Σ_w∈N(v)→ m_v^{t+1}U_t(GRU)h_v^t also enters U_th_v^(t+1): next round, t = 0 … T−1Rset2set{h_v^T : v ∈ G}ŷ (graph)R must not depend on node orderearlier models are choices of M, U and RGCN (Kipf & Welling)M = c_vw · h_w, U = σ(W m_v), c_vw = (deg v · deg w)^−½GG-NN (Li et al.)M = A_{bond type} h_w, U = GRU, R = gated sum over nodesGilmer et al. (best)M = A(e_vw) h_w, A = MLP → d×d, U = GRU, R = set2set
fig.One round of message passing. Each neighbour w sends M_t(h_v, h_w, e_vw) along its edge. The messages are summed into m_v, and U_t (a GRU in the paper) combines m_v with h_v. After T rounds, R reads out the set of all final states. The 2-hop node u first reaches v at round 2. The table at the bottom writes three models as choices of M, U and R.

Because MtM_t and UtU_t are shared across edges and nodes, the parameter count does not depend on graph size. Because the aggregator and RR are symmetric, relabelling the nodes permutes the hvth_v^t but leaves y^\hat y unchanged. Node-level tasks skip RR and read hvTh_v^T directly.

The variant that won on QM9

Gilmer et al. compared several message and readout functions. The best used an edge network: a small MLP maps the edge feature vector to a d×dd\times d matrix that multiplies the neighbour's state. Updates used a Gated recurrent unit (GRU) whose weights are shared across all TT rounds, as in GG-NN. The readout was set2set (Vinyals, Bengio and Kudlur, 2016), an LSTM-driven attention over the node set:

M(hv,hw,evw)=A(evw) hw,U=GRU(hvt, mvt+1),R=set2set({hvT})M\big(h_v,h_w,e_{vw}\big)=A(e_{vw})\,h_w,\qquad U=\mathrm{GRU}\big(h_v^t,\,m_v^{t+1}\big),\qquad R=\mathrm{set2set}\big(\{h_v^T\}\big)
A(evw)A(e_{vw})
MLP output reshaped to d×dd\times d; a continuous edge feature such as bond length changes the message smoothly
GRU\mathrm{GRU}
gated update: the message is the GRU input, hvth_v^t the hidden state

Two further ideas from the paper: a virtual master node connected to every atom, which lets information cross the molecule in two hops, and towers, which split hvh_v into kk chunks of size d/kd/k with separate message functions to cut the d2d^2 cost of the edge network.

Setting and results (Gilmer et al., 2017)

Dataset
QM9: about 134k (133,885) small organic molecules, up to 9 heavy atoms (C, N, O, F); properties computed with DFT
Targets
13 per molecule, including dipole moment, HOMO, LUMO, gap, zero-point energy, internal energy U₀ and heat capacity
Split
10,000 validation and 10,000 test molecules chosen at random; the rest for training
Metric
error ratio = MAE / chemical accuracy for each target; below 1 means within chemical accuracy
Best model
edge network + GRU + set2set: chemical accuracy on 11 of the 13 targets
step t = 1 · aggregator = sum · selected atom 1 (C)M = A(e_vw) h_w · U = tanh(W h_v + V m_v) · bars: h_v^t (dim 1, dim 2)O0C1C2C3N4O5single (1)double (2) · click an atomhalo: atoms within t = 1 hops, which h_1^1 depends on(atom 0 needs t = 4 to see atom 5: the graph diameter is 4)dim 1dim 2 (bars in [−1, 1])atom 1: messages at step 1from O0 (1): [ 0.20, 0.50]from C2 (2): [ 0.20, 0.80]from N4 (1): [ 0.40, 0.30]m_v (sum) = [ 0.80, 1.60]h_v^0 = [ 1.00, 0.00]h_v^1 = [ 0.88, 0.87]depends on 4 of 6 atomsreadout ŷ = Σ_v (h_v,1 − h_v,2)per step t = 0 … 4 (current in accent)1.00t=00.53t=10.65t=20.49t=30.44t=4same weights at every step; only T changes
aggregator
atom v
fig.Real message passing on a 6-atom graph with 2-d states, edge-network messages A(e) h_w (one fixed matrix per bond type) and U = tanh(W h + V m). Default (t = 1, sum, atom C1): messages [0.20, 0.50] from O0, [0.20, 0.80] from C2 over the double bond and [0.40, 0.30] from N4 sum to m = [0.80, 1.60], so h = [0.88, 0.87]. C1 already depends on 4 of the 6 atoms, and on all 6 from t = 3. The sum readout reads 1.00, 0.53, 0.65 at t = 0, 1, 2. Switch to mean: m = [0.27, 0.53] and h = [0.66, 0.52]. Atom O0 needs t = 4 to see O5, because the graph diameter is 4.

Worked example: two rounds by hand

Take 4 nodes with scalar states h0=(A: 1, B: 2, C: 0, D: 1)h^0=(A{:}\,1,\ B{:}\,2,\ C{:}\,0,\ D{:}\,1) and edges A–B, B–D, C–D (weight 1) and B–C (weight 0.5). Use M=a(evw) hwM=a(e_{vw})\,h_w, U=ReLU(0.5 hv+0.5 mv)U=\mathrm{ReLU}(0.5\,h_v+0.5\,m_v) and a sum readout. The initial readout is ∑vhv0=4\sum_v h_v^0=4.

Nodeh⁰messages, round 1m¹h¹messages, round 2m²h²
A11·221.51·221.75
B21·1 + 0.5·0 + 1·1221·1.5 + 0.5·1 + 1·1.53.52.75
C00.5·2 + 1·1210.5·2 + 1·1.52.51.75
D11·2 + 1·021.51·2 + 1·132.25
Σ (readout)468.5
Every message is the edge weight times the neighbour's state from the previous round

In round 1 every node happens to receive m=2m=2, yet the states differ because UU also sees hvh_v. Renumbering the nodes would reorder the rows but leave the readouts 6 and 8.5 unchanged.

python
1import torch
2from torch import nn
3from torch_geometric.datasets import QM9
4from torch_geometric.loader import DataLoader
5from torch_geometric.nn import NNConv, Set2Set
6
7class MPNN(nn.Module):
8 def __init__(self, d_node=11, d_edge=4, d=64, T=3):
9 super().__init__()
10 self.T = T
11 self.embed = nn.Linear(d_node, d)
12 A = nn.Sequential(nn.Linear(d_edge, 128), nn.ReLU(), nn.Linear(128, d * d)) # e_vw -> d×d
13 self.msg = NNConv(d, d, A, aggr="add") # m_v = Σ_w A(e_vw) h_w
14 self.gru = nn.GRU(d, d) # U, shared across the T rounds
15 self.readout = Set2Set(d, processing_steps=3) # R, permutation-invariant
16 self.head = nn.Sequential(nn.Linear(2 * d, d), nn.ReLU(), nn.Linear(d, 1))
17
18 def forward(self, g):
19 h = torch.relu(self.embed(g.x))
20 state = h.unsqueeze(0)
21 for _ in range(self.T):
22 m = torch.relu(self.msg(h, g.edge_index, g.edge_attr))
23 h, state = self.gru(m.unsqueeze(0), state)
24 h = h.squeeze(0)
25 return self.head(self.readout(h, g.batch)).view(-1)
26
27ds = QM9("data/QM9").shuffle()
28test, val, train = ds[:10000], ds[10000:20000], ds[20000:]
29model = MPNN()
30opt = torch.optim.Adam(model.parameters(), lr=1e-3)
31for g in DataLoader(train, batch_size=128, shuffle=True):
32 loss = nn.functional.l1_loss(model(g), g.y[:, 0]) # MAE on target 0 (dipole moment)
33 opt.zero_grad(); loss.backward(); opt.step()
The Gilmer et al. edge-network MPNN on QM9 in PyTorch Geometric, after PyG's qm9_nn_conv example. NNConv is the edge network; not run here, since torch and PyG were not available.

What the framework implies

Graph Convolutional Network (GCN) is the case M=cvwhwM=c_{vw}h_w with cvw=(deg⁡v deg⁡w)−1/2c_{vw}=(\deg v\,\deg w)^{-1/2} and U=σ(Wmv)U=\sigma(Wm_v). Graph Attention Network (GAT) makes the weight of each message a learned attention coefficient. GraphSAGE swaps the sum for mean, max-pool or LSTM aggregators. Graph Isomorphism Network (GIN) keeps the sum and adds (1+ϵ)hv(1+\epsilon)h_v.

The framework also has a ceiling. Xu et al. (2019) and Morris et al. (2019) proved that an MPNN distinguishes two graphs at most as well as the 1-WL colour-refinement test. Mean and max aggregators can be weaker still, because they lose neighbour counts. Battaglia et al. (2018) generalised the scheme to graph networks, which also update edge states and a global state.

Check yourself

Why must the MPNN readout R be a symmetric function such as sum, mean or set2set?

Check yourself

In the interactive figure, why does atom O0's state only start to depend on O5 at t = 4?

Remember

  • ✓MPNN (Gilmer et al., ICML 2017): mvt+1=∑wMt(hvt,hwt,evw)m_v^{t+1}=\sum_{w}M_t(h_v^t,h_w^t,e_{vw}), hvt+1=Ut(hvt,mvt+1)h_v^{t+1}=U_t(h_v^t,m_v^{t+1}), y^=R({hvT})\hat y=R(\{h_v^T\}).
  • ✓Shared M and U handle graphs of any size. A symmetric aggregator and readout give permutation invariance.
  • ✓GCN, GG-NN, Interaction Networks, molecular fingerprints and DTNN are all special cases.
  • ✓The best QM9 model: edge network A(evw)hwA(e_{vw})h_w, a GRU update shared over rounds, and a set2set readout. It reached chemical accuracy on 11 of 13 targets.
  • ✓T rounds give a T-hop receptive field. Deep stacks over-smooth and over-squash.
  • ✓Expressiveness is bounded by the 1-WL test. GIN reaches that bound with sum aggregation.
← back to Deep Learning roadmap