# Temporal graph networks

> A TGN keeps a GRU-updated memory per node, fed by messages that each timestamped event sends to both endpoints, and embeds a node at time t by attending over its most recent neighbours with a time encoding cos(ωΔt + b).

Source: https://www.mlfoundry.in/learn/temporal-graph-networks  
Section: Deep Learning › Graph Neural Networks (GNNs)  
Level: advanced · 8 min read · updated 2026-10-21

**Temporal graph network (TGN):** is a GNN for continuous-time dynamic graphs, given as a stream of timestamped events. It combines a per-node **memory**, updated by a recurrent cell whenever the node takes part in an event, with a **temporal graph attention** layer over the node's past neighbours. Introduced by Emanuele Rossi, Ben Chamberlain, Fabrizio Frasca, Davide Eynard, Federico Monti and Michael Bronstein (Twitter), "Temporal Graph Networks for Deep Learning on Dynamic Graphs" (arXiv, 2020).

Many graphs are streams of events rather than fixed objects: users edit Wikipedia pages, post in subreddits, follow accounts, make payments. Each event is $e_{ij}(t)$, an interaction between nodes $i$ and $j$ at time $t$, often with a feature vector. The usual task is **future link prediction**: given everything before $t$, score whether $i$ and $j$ interact at $t$ (see **Link prediction**). A second is dynamic node classification, for example predicting that a user is about to be banned.

A static GNN such as **GraphSAGE** sees one frozen graph. Running it on the whole history leaks the future into the past, and rerunning it on every snapshot is expensive and coarse in time. Earlier continuous-time models each solved part of the problem. JODIE (Kumar, Zhang and Leskovec, KDD 2019) keeps an RNN memory per user and item. DyRep (Trivedi et al., ICLR 2019) updates memories with attention over neighbours. TGAT (Xu et al., ICLR 2020) has no memory but attends over temporal neighbours with a learned time encoding. TGN puts these pieces into one framework, in which each earlier model is a special case.

## Memory: messages and a recurrent update

$$
\begin{aligned}m_i(t)&=\big[\,s_i(t^-)\,\Vert\,s_j(t^-)\,\Vert\,\phi(t-t_i^-)\,\Vert\,e_{ij}(t)\,\big],\qquad m_j(t)\ \text{likewise from }j\text{'s side}\\ \bar m_i(t)&=\operatorname{agg}\big(m_i(t_1),\dots,m_i(t_b)\big),\qquad s_i(t)=\operatorname{GRU}\big(\bar m_i(t),\,s_i(t^-)\big)\end{aligned}
$$

- $s_i(t^-)$: memory of node $i$ just before $t$; it starts at zero and changes only when $i$ takes part in an event
- $t_i^-$: time of $i$'s last memory update, so $t-t_i^-$ says how long $i$ has been silent
- $m_i(t)$: message: the paper's best variant is the identity, a plain concatenation (an MLP is the alternative)
- $\operatorname{agg}$: combines the $b$ messages a node receives in one batch; "keep only the most recent" and "mean" were tested
- $\operatorname{GRU}$: the memory updater: a **Gated recurrent unit (GRU)** cell, or a **Long short-term memory (LSTM)** cell

> Figure: TGN in three parts. 1: an event (i, j, t, e_ij) builds one message per endpoint from both memories, the time encoding of the gap since the last update and the edge features. Messages are aggregated per node and a GRU writes the memory store S (one d_s-dimensional row per node). 2: to embed i at query time t, temporal graph attention attends over its k most recent earlier neighbours (here k = 3; the older d is not sampled), and a decoder MLP scores the pair against a negative. 3: within each batch of 200 events, memory is first updated with the previous batch's stored messages, so no event is used before it is predicted.

## Embedding: temporal graph attention

Memory alone has a weakness. A node that has been silent for a long time keeps an old memory, because only its own events update it. The embedding module fixes this by aggregating over the node's temporal neighbours, whose memories may be fresh. Only events before $t$ are used, so the model never looks ahead.

$$
\begin{aligned}q&=W_Q\big[h_i\,\Vert\,\phi(0)\big],\qquad K_j=W_K\,c_j,\quad V_j=W_V\,c_j,\quad c_j=\big[h_j\,\Vert\,e_{ij}\,\Vert\,\phi(t-t_j)\big],\quad j\in n^k_i([0,t))\\ z_i(t)&=\operatorname{MLP}\Big(h_i\,\Vert\,\sum_j\alpha_{ij}V_j\Big),\qquad \alpha_{ij}=\operatorname{softmax}_j\big(q^{\top}K_j/\sqrt{d}\big),\qquad \phi(\Delta t)=\cos(\omega\,\Delta t+b)\end{aligned}
$$

- $h_j$: input to the layer: memory $s_j(t)$ plus the node's own features, if any
- $n^k_i([0,t))$: $k$ temporal neighbours of $i$: nodes it interacted with before $t$, the most recent ones by default
- $\phi(\Delta t)$: TGAT's time encoding, a vector of cosines with learnable frequencies $\omega$ and phases $b$, so the attention can weigh events by their age
- $\alpha_{ij}$: attention weights, computed with multiple heads as in **Multi-head attention**

For link prediction, a decoder MLP takes $[z_i(t)\Vert z_j(t)]$ and outputs a probability. Each true event is paired with a negative, the same source and a random destination, and the loss is **Binary cross-entropy**. The metric is average precision (AP). The paper also tests two cheaper embedding modules: identity, $z_i=s_i$, and JODIE's time projection, which scales the memory by $1+w\,\Delta t$.

## Training without leaking the future

Events are processed in chronological batches of 200. A naive loop would update memory with batch $b$ and then predict batch $b$, so the memory would already contain the answer. TGN reverses the order. It stores each batch's raw messages, and at the start of the next batch it first updates memory with them, then predicts. Because that update runs inside the forward pass, the GRU and message modules get gradients. The cost is staleness: an event cannot see earlier events of its own batch through memory, only through the graph that the embedding module reads.

Earlier models as special cases of TGN (after Table 1 of Rossi et al., 2020)

| model | memory | memory updater | embedding module |
| --- | --- | --- | --- |
| JODIE | yes | RNN | time projection: (1 + w Δt) ∘ s |
| DyRep | yes | RNN, messages from attention over neighbours | identity |
| TGAT | no | — | temporal graph attention, 2 layers |
| TGN-attn | yes | GRU | temporal graph attention, 1 layer |

**Paper setting (Rossi et al., 2020)**

- **Wikipedia:** users editing pages over one month: 9,227 nodes, 157,474 timestamped edges, 172-d edge features from the edit text; bipartite
- **Reddit:** users posting to subreddits over one month: 10,984 nodes, 672,447 edges, 172-d edge features; bipartite
- **Splits:** chronological 70 % / 15 % / 15 %; the inductive setting scores links involving nodes never seen in training
- **Main model:** TGN-attn: GRU memory, identity message, most-recent aggregator, one attention layer with 2 heads over 10 most recent neighbours, batch 200
- **Future link prediction (AP):** transductive 98.46 on Wikipedia and 98.70 on Reddit; inductive 97.81 and 97.55
- **Ablations:** removing the memory or replacing attention with the identity embedding both cost accuracy; most-recent neighbour sampling beat uniform; with memory, one attention layer was about as good as two

> Figure: 36 timestamped user–item events (5 users, 4 items, exponential gaps with mean 2.5), split into batches of B. The default is batch 4 of 6 with B = 6 (events 18–23), viewed from u1 at t = 47.8, the last event of the batch. Memory has absorbed events 0–17, and u1 last appeared there at t = 4.2. But u1's 4 most recent temporal neighbours (Δt = 9.3, 6.0, 4.0, 2.5) all lie inside this batch: the embedding module sees them, memory does not. Scoring them by the untrained time encoding alone (8 frequencies ω from 0.1 to 0.001, softmax with temperature 1/8) gives α = 0.18, 0.24, 0.28, 0.30, favouring recent events. Switching to uniform sampling swaps the Δt = 9.3 event for one at Δt = 43.5, which gets α = 0.01. Over the stream, 25 of 36 events (69 %) see stale memory at B = 6, 9 (25 %) at B = 2, 29 (81 %) at B = 12 and none at B = 1.

Worked example: six events, batch size 3. An event is stale if an endpoint already appeared earlier in the same batch (checked in Python)

| event | nodes | t | batch | memory read (last update) | stale? |
| --- | --- | --- | --- | --- | --- |
| e1 | A–B | 1 | 1 | A: none, B: none | no |
| e2 | B–C | 2 | 1 | B: none (e1 not yet absorbed), C: none | yes, B |
| e3 | A–C | 4 | 1 | A: none, C: none | yes, A and C |
| e4 | C–D | 5 | 2 | C: t = 4, D: none | no |
| e5 | A–B | 7 | 2 | A: t = 4, B: t = 2 | no |
| e6 | B–D | 8 | 2 | B: t = 2, D: none | yes, B and D |

So 3 of the 6 events are stale at batch size 3. With batch size 1 none are and with batch size 6 five are. At the start of batch 3, node B has two stored messages, from e5 and e6. The "most recent" aggregator keeps only e6's, $\bar m_B=m_B(8)$, and the GRU applies it once. Smaller batches mean fresher memory but more sequential steps, which is why batch size is a speed–accuracy knob.

```python
import torch
from torch_geometric.datasets import JODIEDataset
from torch_geometric.loader import TemporalDataLoader
from torch_geometric.nn import TGNMemory
from torch_geometric.nn.models.tgn import IdentityMessage, LastAggregator, LastNeighborLoader

data = JODIEDataset("data/JODIE", name="wikipedia")[0]      # 157,474 events, 172-d messages
train, val, test = data.train_val_test_split(val_ratio=0.15, test_ratio=0.15)
loader = TemporalDataLoader(train, batch_size=200, neg_sampling_ratio=1.0)
nbrs = LastNeighborLoader(data.num_nodes, size=10)          # 10 most recent neighbours

d = 100
memory = TGNMemory(data.num_nodes, data.msg.size(-1), d, d,
                   message_module=IdentityMessage(data.msg.size(-1), d, d),
                   aggregator_module=LastAggregator())

for batch in loader:                       # chronological batches
    opt.zero_grad()
    n_id, edge_index, e_id = nbrs(batch.n_id)
    assoc[n_id] = torch.arange(n_id.size(0))
    s, last_update = memory(n_id)          # applies the previous batch's stored messages
    z = gnn(s, last_update, edge_index, data.t[e_id], data.msg[e_id])
    pos = link_pred(z[assoc[batch.src]], z[assoc[batch.dst]])
    neg = link_pred(z[assoc[batch.src]], z[assoc[batch.neg_dst]])
    loss = bce(pos, torch.ones_like(pos)) + bce(neg, torch.zeros_like(neg))
    memory.update_state(batch.src, batch.dst, batch.t, batch.msg)   # store this batch
    nbrs.insert(batch.src, batch.dst)
    loss.backward()
    opt.step()
    memory.detach()                        # truncate backprop through time
```

TGN on JODIE's Wikipedia data with PyTorch Geometric, condensed from PyG's examples/tgn.py (not run here; gnn, link_pred, opt, bce and the assoc index buffer are defined as in that example: gnn is a TransformerConv layer fed the time encoding, link_pred an MLP on the pair)

> **Leakage through the memory or the negatives:** Never update memory with an event before scoring it. Reset memory at the start of each training epoch; evaluation continues from the memory state at the end of the training stream, and each validation or test event updates memory only after it has been scored. Easy negatives also inflate scores: with random destinations, models reach AP near 99 on Wikipedia and Reddit. Poursafaei et al. (NeurIPS 2022) added historical negatives, pairs that did interact but at other times, and model rankings changed.

> **Memory and attention cover each other's gaps:** Memory compresses a node's whole history into one vector at constant cost per event, but it goes stale while the node is silent. Attention over recent neighbours reads their current memories at query time, so a silent node is refreshed by its active neighbours. That is why, with memory, one attention layer was about as good as two, while TGAT, which has no memory, relies on two layers.

- **Social and content platforms:** predicting the next follow, post or interaction from an event stream; the authors were at Twitter
- **Transactions:** fraud and anomaly detection, where the timing and recency of payments between accounts carry the signal
- **Recommendation:** user–item interaction streams, the setting of JODIE's Wikipedia, Reddit and LastFM data

**Question:** Why does TGN update memory with the previous batch's messages at the start of each batch, rather than with the current batch's messages?

- A. It is faster on GPUs
- B. Updating with the current batch would put the events being predicted into memory first, leaking the answer; deferring the update avoids that and still trains the memory modules
- C. The GRU cannot take more than one message at a time
- D. Messages need a full batch of delay to compute Δt

**Answer:** B. Updating with the current batch would put the events being predicted into memory first, leaking the answer; deferring the update avoids that and still trains the memory modules — If batch b's events were written into memory before predicting batch b, the memory would contain the target edges. Storing the raw messages and applying them at the start of batch b + 1 keeps causality, and because the update runs inside the forward pass, the message and GRU parameters get gradients.

**Question:** At query time $t=50$, node $i$'s past events are at $t=10, 30, 44, 48$. With $k=2$ most-recent sampling, which neighbours does attention see and with which time encodings?

- A. Events at 10 and 30, with φ(40) and φ(20)
- B. Events at 44 and 48, with φ(6) and φ(2)
- C. Events at 44 and 48, with φ(44) and φ(48)
- D. All four, with φ(0)

**Answer:** B. Events at 44 and 48, with φ(6) and φ(2) — The two most recent events before $t$ are at 48 and 44. Their keys use $\phi(t-t_j)$, the age of each event: $\phi(2)$ and $\phi(6)$. The query uses $\phi(0)$.

**Key points**

- Continuous-time dynamic graph = stream of timestamped events; main task is future link prediction, scored by AP against sampled negatives.
- Memory: messages $[s_i\Vert s_j\Vert\phi(\Delta t)\Vert e_{ij}]$ to both endpoints, aggregated per batch, then $s_i\leftarrow\operatorname{GRU}(\bar m_i,s_i)$.
- Embedding: attention over the $k$ most recent earlier neighbours, keys with $\phi(t-t_j)=\cos(\omega(t-t_j)+b)$; it counters stale memory.
- Batches of 200: apply the previous batch's messages, predict, then store this batch's messages. No leakage, but in-batch memory is stale.
- JODIE, DyRep and TGAT are special cases. TGN-attn: 98.46 / 98.70 transductive AP on Wikipedia / Reddit.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
