Temporal graph networks

A TGN keeps a GRU-updated memory per node, fed by messages that each timestamped event sends to both endpoints, and embeds a node at time t by attending over its most recent neighbours with a time encoding cos(ωΔt + b).

8 min read785 wordsadvanced0 viewsupdated 2026-10-21

Definition

Temporal graph network (TGN) is a GNN for continuous-time dynamic graphs, given as a stream of timestamped events. It combines a per-node memory, updated by a recurrent cell whenever the node takes part in an event, with a temporal graph attention layer over the node's past neighbours. Introduced by Emanuele Rossi, Ben Chamberlain, Fabrizio Frasca, Davide Eynard, Federico Monti and Michael Bronstein (Twitter), "Temporal Graph Networks for Deep Learning on Dynamic Graphs" (arXiv, 2020).

Many graphs are streams of events rather than fixed objects: users edit Wikipedia pages, post in subreddits, follow accounts, make payments. Each event is eij(t)e_{ij}(t), an interaction between nodes ii and jj at time tt, often with a feature vector. The usual task is future link prediction: given everything before tt, score whether ii and jj interact at tt (see Link prediction). A second is dynamic node classification, for example predicting that a user is about to be banned.

A static GNN such as GraphSAGE sees one frozen graph. Running it on the whole history leaks the future into the past, and rerunning it on every snapshot is expensive and coarse in time. Earlier continuous-time models each solved part of the problem. JODIE (Kumar, Zhang and Leskovec, KDD 2019) keeps an RNN memory per user and item. DyRep (Trivedi et al., ICLR 2019) updates memories with attention over neighbours. TGAT (Xu et al., ICLR 2020) has no memory but attends over temporal neighbours with a learned time encoding. TGN puts these pieces into one framework, in which each earlier model is a special case.

Memory: messages and a recurrent update

mi(t)=[ si(t−) ∥ sj(t−) ∥ ϕ(t−ti−) ∥ eij(t) ],mj(t) likewise from j’s sidemˉi(t)=agg⁡(mi(t1),…,mi(tb)),si(t)=GRU⁡(mˉi(t), si(t−))\begin{aligned}m_i(t)&=\big[\,s_i(t^-)\,\Vert\,s_j(t^-)\,\Vert\,\phi(t-t_i^-)\,\Vert\,e_{ij}(t)\,\big],\qquad m_j(t)\ \text{likewise from }j\text{'s side}\\ \bar m_i(t)&=\operatorname{agg}\big(m_i(t_1),\dots,m_i(t_b)\big),\qquad s_i(t)=\operatorname{GRU}\big(\bar m_i(t),\,s_i(t^-)\big)\end{aligned}
si(t−)s_i(t^-)
memory of node ii just before tt; it starts at zero and changes only when ii takes part in an event
ti−t_i^-
time of ii's last memory update, so t−ti−t-t_i^- says how long ii has been silent
mi(t)m_i(t)
message: the paper's best variant is the identity, a plain concatenation (an MLP is the alternative)
agg⁡\operatorname{agg}
combines the bb messages a node receives in one batch; "keep only the most recent" and "mean" were tested
GRU⁡\operatorname{GRU}
the memory updater: a Gated recurrent unit (GRU) cell, or a Long short-term memory (LSTM) cell
1 · memory: every interaction sends a message to both endpointsevent(i, j, t, e_ij)message m_i(t)s_i ‖ s_j ‖ φ(Δt) ‖ e_ijaggregatelast messagememory updateGRU(m̄_i, s_i)memory Su0u1i0i1…N × d_sm_j(t): same from j's side · Δt = t − t_i⁻ (since i's last update)time encoding φ(Δt) = cos(ω Δt + b), learnable ω and b2 · embed at time t: attend over recent neighboursΔt 2aΔt 5bΔt 9cΔt 30dik = 3 sampled · d too oldtemporal graph attentionq from [s_i ‖ φ(0)]k_j, v_j from [s_j ‖ e_ij ‖ φ(t − t_j)]z_i = MLP(s_i ‖ Σ_j α_ij v_j)sz_i(t)z_j(t)decoder MLPp(i ~ j)vs negative j′ (BCE)3 · one training batch b, in time order (200 events in the paper)① update memorywith b − 1 messages② embed + decodebatch b + negatives③ BCE lossgrads reach GRU via ①④ store raw msgsof batch b for b + 1batch b is predicted before its own events reach memory: no leakage, but in-batch memory is stale
fig.TGN in three parts. 1: an event (i, j, t, e_ij) builds one message per endpoint from both memories, the time encoding of the gap since the last update and the edge features. Messages are aggregated per node and a GRU writes the memory store S (one d_s-dimensional row per node). 2: to embed i at query time t, temporal graph attention attends over its k most recent earlier neighbours (here k = 3; the older d is not sampled), and a decoder MLP scores the pair against a negative. 3: within each batch of 200 events, memory is first updated with the previous batch's stored messages, so no event is used before it is predicted.

Embedding: temporal graph attention

Memory alone has a weakness. A node that has been silent for a long time keeps an old memory, because only its own events update it. The embedding module fixes this by aggregating over the node's temporal neighbours, whose memories may be fresh. Only events before tt are used, so the model never looks ahead.

q=WQ[hi ∥ ϕ(0)],Kj=WK cj,Vj=WV cj,cj=[hj ∥ eij ∥ ϕ(t−tj)],j∈nik([0,t))zi(t)=MLP⁡(hi ∥ ∑jαijVj),αij=softmax⁡j(q⊤Kj/d),ϕ(Δt)=cos⁡(ω Δt+b)\begin{aligned}q&=W_Q\big[h_i\,\Vert\,\phi(0)\big],\qquad K_j=W_K\,c_j,\quad V_j=W_V\,c_j,\quad c_j=\big[h_j\,\Vert\,e_{ij}\,\Vert\,\phi(t-t_j)\big],\quad j\in n^k_i([0,t))\\ z_i(t)&=\operatorname{MLP}\Big(h_i\,\Vert\,\sum_j\alpha_{ij}V_j\Big),\qquad \alpha_{ij}=\operatorname{softmax}_j\big(q^{\top}K_j/\sqrt{d}\big),\qquad \phi(\Delta t)=\cos(\omega\,\Delta t+b)\end{aligned}
hjh_j
input to the layer: memory sj(t)s_j(t) plus the node's own features, if any
nik([0,t))n^k_i([0,t))
kk temporal neighbours of ii: nodes it interacted with before tt, the most recent ones by default
ϕ(Δt)\phi(\Delta t)
TGAT's time encoding, a vector of cosines with learnable frequencies ω\omega and phases bb, so the attention can weigh events by their age
αij\alpha_{ij}
attention weights, computed with multiple heads as in Multi-head attention

For link prediction, a decoder MLP takes [zi(t)∥zj(t)][z_i(t)\Vert z_j(t)] and outputs a probability. Each true event is paired with a negative, the same source and a random destination, and the loss is Binary cross-entropy. The metric is average precision (AP). The paper also tests two cheaper embedding modules: identity, zi=siz_i=s_i, and JODIE's time projection, which scales the memory by 1+w Δt1+w\,\Delta t.

Training without leaking the future

Events are processed in chronological batches of 200. A naive loop would update memory with batch bb and then predict batch bb, so the memory would already contain the answer. TGN reverses the order. It stores each batch's raw messages, and at the start of the next batch it first updates memory with them, then predicts. Because that update runs inside the forward pass, the GRU and message modules get gradients. The cost is staleness: an event cannot see earlier events of its own batch through memory, only through the graph that the embedding module reads.

modelmemorymemory updaterembedding module
JODIEyesRNNtime projection: (1 + w Δt) ∘ s
DyRepyesRNN, messages from attention over neighboursidentity
TGATno—temporal graph attention, 2 layers
TGN-attnyesGRUtemporal graph attention, 1 layer
Earlier models as special cases of TGN (after Table 1 of Rossi et al., 2020)

Paper setting (Rossi et al., 2020)

Wikipedia
users editing pages over one month: 9,227 nodes, 157,474 timestamped edges, 172-d edge features from the edit text; bipartite
Reddit
users posting to subreddits over one month: 10,984 nodes, 672,447 edges, 172-d edge features; bipartite
Splits
chronological 70 % / 15 % / 15 %; the inductive setting scores links involving nodes never seen in training
Main model
TGN-attn: GRU memory, identity message, most-recent aggregator, one attention layer with 2 heads over 10 most recent neighbours, batch 200
Future link prediction (AP)
transductive 98.46 on Wikipedia and 98.70 on Reddit; inductive 97.81 and 97.55
Ablations
removing the memory or replacing attention with the identity embedding both cost accuracy; most-recent neighbour sampling beat uniform; with memory, one attention layer was about as good as two
batch 4 of 6 (B = 6): events 18–23 · memory has absorbed events 0–17in memory (earlier batches) · this batch: stale memory · ringed: endpoint already seen in batchu0u1u2u3u4i0i1i2i3t = 0t = 20t = 40t = 60t = 80u1 embedded at t = 47.8 (dashed): most recent k = 4 of its earlier eventsneighbour · Δtφ(Δt) = cos(ω Δt), ω = 0.1 … 0.001score = φ(0)·φ(Δt)/d · attention αi1 · 9.3 *0.9300.18i1 · 6.0 *0.9700.24i0 · 4.0 *0.9870.28i3 · 2.5 *0.9950.308 frequencies: periods 63 … 6283 time unitsmemory of u1: last updated at t = 4.2 · 4 events in this batch not yet absorbed (*)stale-memory events: 5 of 6 in this batch · 25 of 36 (69%) over the stream at B = 6
sampling
node
fig.36 timestamped user–item events (5 users, 4 items, exponential gaps with mean 2.5), split into batches of B. The default is batch 4 of 6 with B = 6 (events 18–23), viewed from u1 at t = 47.8, the last event of the batch. Memory has absorbed events 0–17, and u1 last appeared there at t = 4.2. But u1's 4 most recent temporal neighbours (Δt = 9.3, 6.0, 4.0, 2.5) all lie inside this batch: the embedding module sees them, memory does not. Scoring them by the untrained time encoding alone (8 frequencies ω from 0.1 to 0.001, softmax with temperature 1/8) gives α = 0.18, 0.24, 0.28, 0.30, favouring recent events. Switching to uniform sampling swaps the Δt = 9.3 event for one at Δt = 43.5, which gets α = 0.01. Over the stream, 25 of 36 events (69 %) see stale memory at B = 6, 9 (25 %) at B = 2, 29 (81 %) at B = 12 and none at B = 1.
eventnodestbatchmemory read (last update)stale?
e1A–B11A: none, B: noneno
e2B–C21B: none (e1 not yet absorbed), C: noneyes, B
e3A–C41A: none, C: noneyes, A and C
e4C–D52C: t = 4, D: noneno
e5A–B72A: t = 4, B: t = 2no
e6B–D82B: t = 2, D: noneyes, B and D
Worked example: six events, batch size 3. An event is stale if an endpoint already appeared earlier in the same batch (checked in Python)

So 3 of the 6 events are stale at batch size 3. With batch size 1 none are and with batch size 6 five are. At the start of batch 3, node B has two stored messages, from e5 and e6. The "most recent" aggregator keeps only e6's, mˉB=mB(8)\bar m_B=m_B(8), and the GRU applies it once. Smaller batches mean fresher memory but more sequential steps, which is why batch size is a speed–accuracy knob.

python
1import torch
2from torch_geometric.datasets import JODIEDataset
3from torch_geometric.loader import TemporalDataLoader
4from torch_geometric.nn import TGNMemory
5from torch_geometric.nn.models.tgn import IdentityMessage, LastAggregator, LastNeighborLoader
6
7data = JODIEDataset("data/JODIE", name="wikipedia")[0] # 157,474 events, 172-d messages
8train, val, test = data.train_val_test_split(val_ratio=0.15, test_ratio=0.15)
9loader = TemporalDataLoader(train, batch_size=200, neg_sampling_ratio=1.0)
10nbrs = LastNeighborLoader(data.num_nodes, size=10) # 10 most recent neighbours
11
12d = 100
13memory = TGNMemory(data.num_nodes, data.msg.size(-1), d, d,
14 message_module=IdentityMessage(data.msg.size(-1), d, d),
15 aggregator_module=LastAggregator())
16
17for batch in loader: # chronological batches
18 opt.zero_grad()
19 n_id, edge_index, e_id = nbrs(batch.n_id)
20 assoc[n_id] = torch.arange(n_id.size(0))
21 s, last_update = memory(n_id) # applies the previous batch's stored messages
22 z = gnn(s, last_update, edge_index, data.t[e_id], data.msg[e_id])
23 pos = link_pred(z[assoc[batch.src]], z[assoc[batch.dst]])
24 neg = link_pred(z[assoc[batch.src]], z[assoc[batch.neg_dst]])
25 loss = bce(pos, torch.ones_like(pos)) + bce(neg, torch.zeros_like(neg))
26 memory.update_state(batch.src, batch.dst, batch.t, batch.msg) # store this batch
27 nbrs.insert(batch.src, batch.dst)
28 loss.backward()
29 opt.step()
30 memory.detach() # truncate backprop through time
TGN on JODIE's Wikipedia data with PyTorch Geometric, condensed from PyG's examples/tgn.py (not run here; gnn, link_pred, opt, bce and the assoc index buffer are defined as in that example: gnn is a TransformerConv layer fed the time encoding, link_pred an MLP on the pair)

Where it's used

Social and content platforms

predicting the next follow, post or interaction from an event stream; the authors were at Twitter

Transactions

fraud and anomaly detection, where the timing and recency of payments between accounts carry the signal

Recommendation

user–item interaction streams, the setting of JODIE's Wikipedia, Reddit and LastFM data

Check yourself

Why does TGN update memory with the previous batch's messages at the start of each batch, rather than with the current batch's messages?

Check yourself

At query time t=50t=50, node ii's past events are at t=10,30,44,48t=10, 30, 44, 48. With k=2k=2 most-recent sampling, which neighbours does attention see and with which time encodings?

Remember

  • ✓Continuous-time dynamic graph = stream of timestamped events; main task is future link prediction, scored by AP against sampled negatives.
  • ✓Memory: messages [si∥sj∥ϕ(Δt)∥eij][s_i\Vert s_j\Vert\phi(\Delta t)\Vert e_{ij}] to both endpoints, aggregated per batch, then si←GRU⁡(mˉi,si)s_i\leftarrow\operatorname{GRU}(\bar m_i,s_i).
  • ✓Embedding: attention over the kk most recent earlier neighbours, keys with ϕ(t−tj)=cos⁡(ω(t−tj)+b)\phi(t-t_j)=\cos(\omega(t-t_j)+b); it counters stale memory.
  • ✓Batches of 200: apply the previous batch's messages, predict, then store this batch's messages. No leakage, but in-batch memory is stale.
  • ✓JODIE, DyRep and TGAT are special cases. TGN-attn: 98.46 / 98.70 transductive AP on Wikipedia / Reddit.
← back to Deep Learning roadmap