Graph Attention Network (GAT)

GAT replaces GCN's fixed 1/√(dᵢdⱼ) edge weights with learned attention: each node scores its neighbours with LeakyReLU(aᵀ[Whᵢ‖Whⱼ]), softmaxes over its neighbourhood and sums, using several heads. 83.0% on Cora, 0.973 micro-F1 on PPI.

8 min read591 wordscore0 viewsupdated 2026-10-21

Definition

Graph Attention Network (GAT) A message-passing layer in which node ii aggregates its neighbours with learned, input-dependent weights αij\alpha_{ij}, computed by a small attention function and normalised by a softmax over N(i)∪{i}\mathcal N(i)\cup\{i\}. Several independent attention heads run in parallel. Introduced by Veličković, Cucurull, Casanova, Romero, Liò and Bengio, "Graph Attention Networks" (ICLR 2018). GATv2 (Brody, Alon and Yahav, "How Attentive are Graph Attention Networks?", ICLR 2022) fixes a limitation of the original scoring function.

The task is usually node classification: a graph with node features hi∈RFh_i\in\mathbb R^F, labels on some nodes, predict the rest. A layer must work for any number of neighbours and must not depend on the order in which they are listed. So it has to aggregate with a permutation-invariant operation such as a sum.

Graph Convolutional Network (GCN) sums neighbours with fixed weights 1/d~id~j1/\sqrt{\tilde d_i\tilde d_j} that depend only on degrees. Every neighbour of a given degree counts the same, whatever its features. GAT keeps the masked sum over edges but lets the features decide the weights, in the spirit of Self-attention. Unlike spectral methods, it needs no eigendecomposition and no knowledge of the whole graph, so it applies to unseen graphs (inductive learning).

The attention layer

eij=LeakyReLU ⁣(a⊤[Whi ∥ Whj]),αij=exp⁡(eij)∑k∈N(i)∪{i}exp⁡(eik),hi′=σ(∑j∈N(i)∪{i}αij Whj)e_{ij}=\mathrm{LeakyReLU}\!\big(\mathbf a^{\top}[\mathbf W h_i\,\Vert\,\mathbf W h_j]\big),\qquad \alpha_{ij}=\frac{\exp(e_{ij})}{\sum_{k\in\mathcal N(i)\cup\{i\}}\exp(e_{ik})},\qquad h_i'=\sigma\Big(\sum_{j\in\mathcal N(i)\cup\{i\}}\alpha_{ij}\,\mathbf W h_j\Big)
W∈RF′×F\mathbf W\in\mathbb R^{F'\times F}
shared linear map, applied to every node
a∈R2F′\mathbf a\in\mathbb R^{2F'}
attention vector; ∥\Vert is concatenation, so a⊤[⋅∥⋅]=a1⊤Whi+a2⊤Whj\mathbf a^\top[\cdot\Vert\cdot]=\mathbf a_1^\top\mathbf W h_i+\mathbf a_2^\top\mathbf W h_j
LeakyReLU\mathrm{LeakyReLU}
negative slope 0.2
αij\alpha_{ij}
softmax only over the edges of ii (masked attention), self-loop included; each row sums to 1
σ\sigma
nonlinearity, ELU in the paper
hi′=∥k=1K σ(∑jαijkWkhj)  (hidden layers),hi′=σ(1K∑k=1K∑jαijkWkhj)  (output layer)h_i'=\big\Vert_{k=1}^{K}\,\sigma\Big(\sum_{j}\alpha_{ij}^{k}\mathbf W^{k}h_j\Big)\ \ \text{(hidden layers)},\qquad h_i'=\sigma\Big(\frac1K\sum_{k=1}^{K}\sum_{j}\alpha_{ij}^{k}\mathbf W^{k}h_j\Big)\ \ \text{(output layer)}
KK
number of heads, each with its own Wk\mathbf W^k and ak\mathbf a^k; hidden output size KF′KF'
1K∑\tfrac1K\sum
the prediction layer averages heads instead of concatenating, then applies softmax or sigmoid
one attention head: each node scores its own neighboursv0v1v2v3h=(0,1)h=(1,1)h=(−2,0)h=(1,0)0.2890.4770.0580.175 (self)W = I, a = (0.5, 0 | 0.5, 1): α sums to 11 z_j = W h_jshared linear map F → F′, every node2 [z_i ‖ z_j]concatenate query and key (2F′)3 e_ij = LeakyReLU(aᵀ[z_i ‖ z_j])slope 0.2; one scalar per edge4 α_ij = softmax over j ∈ N(i) ∪ {i}masked: only edges of the graph5 h_i′ = σ(Σ_j α_ij z_j)weighted sum of transformed neighboursmulti-head, as on Cora: 1433 features → 8 heads × 8 → concat 64 → 1 head × 7 classesx_i14338 independent heads (W, a each)‖ concat64 · ELUlayer 2: 1 head64 → 7softmax7 classesoutput layer averages its heads (1 on Cora)
fig.One head on the worked example below. Node v0 scores itself and its three neighbours; the edge widths are the resulting α (0.175 self, 0.289, 0.477, 0.058), which sum to 1. The five steps on the right run on every edge in parallel. Bottom: the Cora model, 8 heads of 8 features concatenated to 64, then one output head over the 7 classes.

Cost per head is O(∣V∣FF′+∣E∣F′)O(|V|FF'+|E|F'): one matrix product for all nodes, then one dot product per edge. That is the same order as GCN. Memory is O(∣E∣)O(|E|) for the coefficients, never O(∣V∣2)O(|V|^2), because only existing edges are scored.

Static attention and GATv2

Because a⊤[Whi∥Whj]\mathbf a^\top[\mathbf Wh_i\Vert\mathbf Wh_j] splits into a query part plus a key part, and LeakyReLU and exp are monotonic, the ranking of keys by eije_{ij} is the ranking by a2⊤Whj\mathbf a_2^\top\mathbf Wh_j alone. It is the same for every query node. Brody et al. call this static attention: one GAT head cannot make node A prefer neighbour X while node B prefers Y when both see X and Y. GATv2 moves the nonlinearity inside:

eijGATv2=a⊤ LeakyReLU ⁣(W[hi ∥ hj])e_{ij}^{\text{GATv2}}=\mathbf a^{\top}\,\mathrm{LeakyReLU}\!\big(\mathbf W[h_i\,\Vert\,h_j]\big)
W∈RF′×2F\mathbf W\in\mathbb R^{F'\times 2F}
applied to the concatenation, so query and key mix before the nonlinearity
a∈RF′\mathbf a\in\mathbb R^{F'}
applied after LeakyReLU; the score is no longer a sum of a query term and a key term, so rankings can depend on ii (dynamic attention)
DatasetNodes / edgesFeatures / classesSettingGATComparison
Cora2,708 / 5,4291,433 / 7transductive, 140 labels83.0 ± 0.7%GCN 81.5%
Citeseer3,327 / 4,7323,703 / 6transductive, 120 labels72.5 ± 0.7%GCN 70.3%
Pubmed19,717 / 44,338500 / 3transductive, 60 labels79.0 ± 0.3%GCN 79.0%
PPI56,944 / 818,716 (24 graphs)50 / 121 multi-labelinductive: 20 train, 2 val, 2 test graphs0.973 ± 0.002 micro-F1GraphSAGE-LSTM 0.612; Const-GAT 0.934
Benchmarks and results from the GAT paper (Veličković et al., 2018). Mean ± std over 100 runs (transductive) or 10 runs (PPI).

Configurations in the paper

Cora, Citeseer
2 layers: 8 heads × 8 features (ELU), then 1 head over the classes with softmax; dropout 0.6 on layer inputs and on the attention coefficients; L2 5·10⁻⁴; Adam, lr 0.005
Pubmed
same, but 8 output heads averaged, L2 10⁻³, lr 0.01
PPI
3 layers: 4 heads × 256 features, 4 heads × 256, then 6 heads × 121 averaged with a sigmoid; skip connection across the middle layer; no dropout or L2
Const-GAT
the same PPI model with all αᵢⱼ uniform; the drop from 0.973 to 0.934 isolates the value of attention
GAT layer, K = 3 heads, F = 4 → F′ = 4 · query v0, |N(v0) ∪ {v0}| = 5v00.16v10.16v20.12v30.42v40.14v5v6v7v8click a node · edge width ∝ α (mean of heads)α(v0, j) per headv0*0.2770.0060.210v10.2540.0070.206v20.1480.0190.196v30.1380.9380.188v40.1820.0300.201* self-loop · each head sums to 1entropy 1.479 / ln 5 = 1.609dense test: top-scoring key over all 9 nodes, for queries v0…v8 (scale 1.00)head 1: v6 v6 v6 v6 v6 v6 v6 v6 v6 → one global ranking (static attention)head 2: v3 v3 v3 v3 v3 v3 v3 v3 v3 → one global ranking (static attention)head 3: v0 v0 v0 v0 v0 v0 v0 v0 v0 → one global ranking (static attention)
scoring
head
query node
fig.One untrained layer (seeded features and weights), 3 heads, on a 9-node graph. Default: query v0 over {v0, v1, v2, v3, v4}. Head 2 puts 0.938 on v3, head 3 is nearly uniform (0.188 to 0.210), and the mean of heads is 0.164, 0.156, 0.121, 0.421, 0.138, entropy 1.479 against ln 5 = 1.609. Scale of a = 0 gives exactly 0.200 everywhere: GAT degenerates to a mean over the neighbourhood. Scale 3 sharpens head 2 to 1.000 on v3. The dense test scores all 9 keys for every query: GAT heads pick v6, v3 and v0 for every query (static attention). Switch to GATv2: head 2 picks v4 or v1 and head 3 picks v4 or v3, depending on the query.

Worked example: one head on node v0

Features h0=(1,0)h_0=(1,0), h1=(0,1)h_1=(0,1), h2=(1,1)h_2=(1,1), h3=(−2,0)h_3=(-2,0); edges v0–v1, v0–v2, v0–v3. Take W=I\mathbf W=I, a1=(0.5,0)\mathbf a_1=(0.5,0) for the query half and a2=(0.5,1)\mathbf a_2=(0.5,1) for the key half. The query term is a1⊤h0=0.5\mathbf a_1^\top h_0=0.5.

ja₁ᵀz₀ + a₂ᵀzⱼeⱼ = LeakyReLUexp(eⱼ)α₀ⱼ
v0 (self)0.5 + 0.5 = 1.01.02.7180.175
v10.5 + 1.0 = 1.51.54.4820.289
v20.5 + 1.5 = 2.02.07.3890.477
v30.5 − 1.0 = −0.5−0.10.9050.058
sum15.4941.000
Computed in Python

The aggregate is ∑jα0jhj=(0.536, 0.766)\sum_j\alpha_{0j}h_j=(0.536,\,0.766), before σ\sigma. A plain mean over the same four nodes gives (0, 0.5)(0,\,0.5): v3's large negative first coordinate pulls it to 0, while GAT gives v3 only 5.8%. Note that the query term 0.5 is added to every score and cancels in the softmax. With this scoring function the query only matters through LeakyReLU's kink, which is the static-attention limitation in miniature.

python
1import torch, torch.nn.functional as F
2from torch_geometric.datasets import Planetoid
3from torch_geometric.nn import GATConv # GATv2Conv has the same interface
4
5data = Planetoid("data", "Cora")[0] # 2708 nodes, 1433 features, 7 classes
6
7class GAT(torch.nn.Module):
8 def __init__(self, d_in, n_cls):
9 super().__init__()
10 self.l1 = GATConv(d_in, 8, heads=8, dropout=0.6) # concat -> 64
11 self.l2 = GATConv(8 * 8, n_cls, heads=1, concat=False, dropout=0.6)
12 def forward(self, x, edge_index):
13 x = F.dropout(x, 0.6, self.training)
14 x = F.elu(self.l1(x, edge_index))
15 x = F.dropout(x, 0.6, self.training)
16 return self.l2(x, edge_index)
17
18model = GAT(data.num_features, 7)
19opt = torch.optim.Adam(model.parameters(), lr=0.005, weight_decay=5e-4)
20for epoch in range(200):
21 model.train(); opt.zero_grad()
22 loss = F.cross_entropy(model(data.x, data.edge_index)[data.train_mask], data.y[data.train_mask])
23 loss.backward(); opt.step()
24
25model.eval()
26acc = (model(data.x, data.edge_index).argmax(1) == data.y)[data.test_mask].float().mean()
27_, (ei, alpha) = model.l1(data.x, data.edge_index, return_attention_weights=True)
28print(f"test acc {acc:.3f}; alpha {tuple(alpha.shape)}") # one row per edge incl. self-loops, 8 heads
The paper's Cora model in PyTorch Geometric, plus reading out attention weights (not run here: no PyG in this sandbox)

Use when

  • +Feature-dependent neighbour weights at O(|E|) cost, no eigendecomposition
  • +Inductive: the same W and a apply to any unseen graph
  • +Multi-head attention stabilises training and gives several views per layer
  • +Big gains when neighbours are not equally useful (PPI: 0.973 vs 0.934 with constant attention)

Avoid when

  • −Original scoring is static: one global key ranking per head (fixed by GATv2)
  • −K heads cost K times the memory of a GCN layer; per-edge coefficients dominate on large graphs
  • −Little or no gain over GCN on small homophilous benchmarks
  • −Still 1-WL-bounded and prone to over-smoothing as layers stack, like other message-passing layers

Check yourself

Nodes A and B both have neighbours X and Y. In one original GAT head, A gives X more weight than Y. What can B do?

Check yourself

Why does GAT's output layer on Pubmed and PPI average its heads instead of concatenating them?

Remember

  • ✓eij=LeakyReLU(a⊤[Whi∥Whj])e_{ij}=\mathrm{LeakyReLU}(\mathbf a^\top[\mathbf Wh_i\Vert\mathbf Wh_j]), softmax over N(i)∪{i}\mathcal N(i)\cup\{i\}, then hi′=σ(∑jαijWhj)h_i'=\sigma(\sum_j\alpha_{ij}\mathbf Wh_j).
  • ✓Masked attention: only existing edges are scored, so cost is O(|V|FF′ + |E|F′) per head, like GCN.
  • ✓K heads: concatenate in hidden layers (Cora: 8 × 8 = 64), average in the output layer.
  • ✓Results: Cora 83.0%, Citeseer 72.5%, Pubmed 79.0%, PPI 0.973 micro-F1 (0.934 with uniform attention).
  • ✓Original GAT attention is static (one key ranking per head); GATv2 applies a after the nonlinearity to make it dynamic.
← back to Deep Learning roadmap