Definition
Graph Attention Network (GAT) A message-passing layer in which node aggregates its neighbours with learned, input-dependent weights , computed by a small attention function and normalised by a softmax over . Several independent attention heads run in parallel. Introduced by Veličković, Cucurull, Casanova, Romero, Liò and Bengio, "Graph Attention Networks" (ICLR 2018). GATv2 (Brody, Alon and Yahav, "How Attentive are Graph Attention Networks?", ICLR 2022) fixes a limitation of the original scoring function.
The task is usually node classification: a graph with node features , labels on some nodes, predict the rest. A layer must work for any number of neighbours and must not depend on the order in which they are listed. So it has to aggregate with a permutation-invariant operation such as a sum.
Graph Convolutional Network (GCN) sums neighbours with fixed weights that depend only on degrees. Every neighbour of a given degree counts the same, whatever its features. GAT keeps the masked sum over edges but lets the features decide the weights, in the spirit of Self-attention. Unlike spectral methods, it needs no eigendecomposition and no knowledge of the whole graph, so it applies to unseen graphs (inductive learning).
The attention layer
- shared linear map, applied to every node
- attention vector; is concatenation, so
- negative slope 0.2
- softmax only over the edges of (masked attention), self-loop included; each row sums to 1
- nonlinearity, ELU in the paper
- number of heads, each with its own and ; hidden output size
- the prediction layer averages heads instead of concatenating, then applies softmax or sigmoid
Cost per head is : one matrix product for all nodes, then one dot product per edge. That is the same order as GCN. Memory is for the coefficients, never , because only existing edges are scored.
Static attention and GATv2
Because splits into a query part plus a key part, and LeakyReLU and exp are monotonic, the ranking of keys by is the ranking by alone. It is the same for every query node. Brody et al. call this static attention: one GAT head cannot make node A prefer neighbour X while node B prefers Y when both see X and Y. GATv2 moves the nonlinearity inside:
- applied to the concatenation, so query and key mix before the nonlinearity
- applied after LeakyReLU; the score is no longer a sum of a query term and a key term, so rankings can depend on (dynamic attention)
| Dataset | Nodes / edges | Features / classes | Setting | GAT | Comparison |
|---|---|---|---|---|---|
| Cora | 2,708 / 5,429 | 1,433 / 7 | transductive, 140 labels | 83.0 ± 0.7% | GCN 81.5% |
| Citeseer | 3,327 / 4,732 | 3,703 / 6 | transductive, 120 labels | 72.5 ± 0.7% | GCN 70.3% |
| Pubmed | 19,717 / 44,338 | 500 / 3 | transductive, 60 labels | 79.0 ± 0.3% | GCN 79.0% |
| PPI | 56,944 / 818,716 (24 graphs) | 50 / 121 multi-label | inductive: 20 train, 2 val, 2 test graphs | 0.973 ± 0.002 micro-F1 | GraphSAGE-LSTM 0.612; Const-GAT 0.934 |
Configurations in the paper
- Cora, Citeseer
- 2 layers: 8 heads × 8 features (ELU), then 1 head over the classes with softmax; dropout 0.6 on layer inputs and on the attention coefficients; L2 5·10⁻⁴; Adam, lr 0.005
- Pubmed
- same, but 8 output heads averaged, L2 10⁻³, lr 0.01
- PPI
- 3 layers: 4 heads × 256 features, 4 heads × 256, then 6 heads × 121 averaged with a sigmoid; skip connection across the middle layer; no dropout or L2
- Const-GAT
- the same PPI model with all αᵢⱼ uniform; the drop from 0.973 to 0.934 isolates the value of attention
Worked example: one head on node v0
Features , , , ; edges v0–v1, v0–v2, v0–v3. Take , for the query half and for the key half. The query term is .
| j | a₁ᵀz₀ + a₂ᵀzⱼ | eⱼ = LeakyReLU | exp(eⱼ) | α₀ⱼ |
|---|---|---|---|---|
| v0 (self) | 0.5 + 0.5 = 1.0 | 1.0 | 2.718 | 0.175 |
| v1 | 0.5 + 1.0 = 1.5 | 1.5 | 4.482 | 0.289 |
| v2 | 0.5 + 1.5 = 2.0 | 2.0 | 7.389 | 0.477 |
| v3 | 0.5 − 1.0 = −0.5 | −0.1 | 0.905 | 0.058 |
| sum | 15.494 | 1.000 |
The aggregate is , before . A plain mean over the same four nodes gives : v3's large negative first coordinate pulls it to 0, while GAT gives v3 only 5.8%. Note that the query term 0.5 is added to every score and cancels in the softmax. With this scoring function the query only matters through LeakyReLU's kink, which is the static-attention limitation in miniature.
1import torch, torch.nn.functional as F2from torch_geometric.datasets import Planetoid3from torch_geometric.nn import GATConv # GATv2Conv has the same interface45data = Planetoid("data", "Cora")[0] # 2708 nodes, 1433 features, 7 classes67class GAT(torch.nn.Module):8 def __init__(self, d_in, n_cls):9 super().__init__()10 self.l1 = GATConv(d_in, 8, heads=8, dropout=0.6) # concat -> 6411 self.l2 = GATConv(8 * 8, n_cls, heads=1, concat=False, dropout=0.6)12 def forward(self, x, edge_index):13 x = F.dropout(x, 0.6, self.training)14 x = F.elu(self.l1(x, edge_index))15 x = F.dropout(x, 0.6, self.training)16 return self.l2(x, edge_index)1718model = GAT(data.num_features, 7)19opt = torch.optim.Adam(model.parameters(), lr=0.005, weight_decay=5e-4)20for epoch in range(200):21 model.train(); opt.zero_grad()22 loss = F.cross_entropy(model(data.x, data.edge_index)[data.train_mask], data.y[data.train_mask])23 loss.backward(); opt.step()2425model.eval()26acc = (model(data.x, data.edge_index).argmax(1) == data.y)[data.test_mask].float().mean()27_, (ei, alpha) = model.l1(data.x, data.edge_index, return_attention_weights=True)28print(f"test acc {acc:.3f}; alpha {tuple(alpha.shape)}") # one row per edge incl. self-loops, 8 headsUse when
- +Feature-dependent neighbour weights at O(|E|) cost, no eigendecomposition
- +Inductive: the same W and a apply to any unseen graph
- +Multi-head attention stabilises training and gives several views per layer
- +Big gains when neighbours are not equally useful (PPI: 0.973 vs 0.934 with constant attention)
Avoid when
- −Original scoring is static: one global key ranking per head (fixed by GATv2)
- −K heads cost K times the memory of a GCN layer; per-edge coefficients dominate on large graphs
- −Little or no gain over GCN on small homophilous benchmarks
- −Still 1-WL-bounded and prone to over-smoothing as layers stack, like other message-passing layers
Check yourself
Nodes A and B both have neighbours X and Y. In one original GAT head, A gives X more weight than Y. What can B do?
Check yourself
Why does GAT's output layer on Pubmed and PPI average its heads instead of concatenating them?
Remember
- ✓, softmax over , then .
- ✓Masked attention: only existing edges are scored, so cost is O(|V|FF′ + |E|F′) per head, like GCN.
- ✓K heads: concatenate in hidden layers (Cora: 8 × 8 = 64), average in the output layer.
- ✓Results: Cora 83.0%, Citeseer 72.5%, Pubmed 79.0%, PPI 0.973 micro-F1 (0.934 with uniform attention).
- ✓Original GAT attention is static (one key ranking per head); GATv2 applies a after the nonlinearity to make it dynamic.