# Graph Attention Network (GAT)

> GAT replaces GCN's fixed 1/√(dᵢdⱼ) edge weights with learned attention: each node scores its neighbours with LeakyReLU(aᵀ[Whᵢ‖Whⱼ]), softmaxes over its neighbourhood and sums, using several heads. 83.0% on Cora, 0.973 micro-F1 on PPI.

Source: https://www.mlfoundry.in/learn/graph-attention-network-gat  
Section: Deep Learning › Graph Neural Networks (GNNs)  
Level: core · 8 min read · updated 2026-10-21

**Graph Attention Network (GAT):** A message-passing layer in which node $i$ aggregates its neighbours with **learned, input-dependent weights** $\alpha_{ij}$, computed by a small attention function and normalised by a softmax over $\mathcal N(i)\cup\{i\}$. Several independent attention heads run in parallel. Introduced by Veličković, Cucurull, Casanova, Romero, Liò and Bengio, "Graph Attention Networks" (ICLR 2018). GATv2 (Brody, Alon and Yahav, "How Attentive are Graph Attention Networks?", ICLR 2022) fixes a limitation of the original scoring function.

The task is usually node classification: a graph with node features $h_i\in\mathbb R^F$, labels on some nodes, predict the rest. A layer must work for any number of neighbours and must not depend on the order in which they are listed. So it has to aggregate with a permutation-invariant operation such as a sum.

**Graph Convolutional Network (GCN)** sums neighbours with fixed weights $1/\sqrt{\tilde d_i\tilde d_j}$ that depend only on degrees. Every neighbour of a given degree counts the same, whatever its features. GAT keeps the masked sum over edges but lets the features decide the weights, in the spirit of **Self-attention**. Unlike spectral methods, it needs no eigendecomposition and no knowledge of the whole graph, so it applies to unseen graphs (inductive learning).

## The attention layer

$$
e_{ij}=\mathrm{LeakyReLU}\!\big(\mathbf a^{\top}[\mathbf W h_i\,\Vert\,\mathbf W h_j]\big),\qquad \alpha_{ij}=\frac{\exp(e_{ij})}{\sum_{k\in\mathcal N(i)\cup\{i\}}\exp(e_{ik})},\qquad h_i'=\sigma\Big(\sum_{j\in\mathcal N(i)\cup\{i\}}\alpha_{ij}\,\mathbf W h_j\Big)
$$

- $\mathbf W\in\mathbb R^{F'\times F}$: shared linear map, applied to every node
- $\mathbf a\in\mathbb R^{2F'}$: attention vector; $\Vert$ is concatenation, so $\mathbf a^\top[\cdot\Vert\cdot]=\mathbf a_1^\top\mathbf W h_i+\mathbf a_2^\top\mathbf W h_j$
- $\mathrm{LeakyReLU}$: negative slope 0.2
- $\alpha_{ij}$: softmax only over the edges of $i$ (masked attention), self-loop included; each row sums to 1
- $\sigma$: nonlinearity, ELU in the paper

$$
h_i'=\big\Vert_{k=1}^{K}\,\sigma\Big(\sum_{j}\alpha_{ij}^{k}\mathbf W^{k}h_j\Big)\ \ \text{(hidden layers)},\qquad h_i'=\sigma\Big(\frac1K\sum_{k=1}^{K}\sum_{j}\alpha_{ij}^{k}\mathbf W^{k}h_j\Big)\ \ \text{(output layer)}
$$

- $K$: number of heads, each with its own $\mathbf W^k$ and $\mathbf a^k$; hidden output size $KF'$
- $\tfrac1K\sum$: the prediction layer averages heads instead of concatenating, then applies softmax or sigmoid

> Figure: One head on the worked example below. Node v0 scores itself and its three neighbours; the edge widths are the resulting α (0.175 self, 0.289, 0.477, 0.058), which sum to 1. The five steps on the right run on every edge in parallel. Bottom: the Cora model, 8 heads of 8 features concatenated to 64, then one output head over the 7 classes.

Cost per head is $O(|V|FF'+|E|F')$: one matrix product for all nodes, then one dot product per edge. That is the same order as GCN. Memory is $O(|E|)$ for the coefficients, never $O(|V|^2)$, because only existing edges are scored.

## Static attention and GATv2

Because $\mathbf a^\top[\mathbf Wh_i\Vert\mathbf Wh_j]$ splits into a query part plus a key part, and LeakyReLU and exp are monotonic, the ranking of keys by $e_{ij}$ is the ranking by $\mathbf a_2^\top\mathbf Wh_j$ alone. It is the same for every query node. Brody et al. call this **static attention**: one GAT head cannot make node A prefer neighbour X while node B prefers Y when both see X and Y. GATv2 moves the nonlinearity inside:

$$
e_{ij}^{\text{GATv2}}=\mathbf a^{\top}\,\mathrm{LeakyReLU}\!\big(\mathbf W[h_i\,\Vert\,h_j]\big)
$$

- $\mathbf W\in\mathbb R^{F'\times 2F}$: applied to the concatenation, so query and key mix before the nonlinearity
- $\mathbf a\in\mathbb R^{F'}$: applied after LeakyReLU; the score is no longer a sum of a query term and a key term, so rankings can depend on $i$ (dynamic attention)

Benchmarks and results from the GAT paper (Veličković et al., 2018). Mean ± std over 100 runs (transductive) or 10 runs (PPI).

| Dataset | Nodes / edges | Features / classes | Setting | GAT | Comparison |
| --- | --- | --- | --- | --- | --- |
| Cora | 2,708 / 5,429 | 1,433 / 7 | transductive, 140 labels | 83.0 ± 0.7% | GCN 81.5% |
| Citeseer | 3,327 / 4,732 | 3,703 / 6 | transductive, 120 labels | 72.5 ± 0.7% | GCN 70.3% |
| Pubmed | 19,717 / 44,338 | 500 / 3 | transductive, 60 labels | 79.0 ± 0.3% | GCN 79.0% |
| PPI | 56,944 / 818,716 (24 graphs) | 50 / 121 multi-label | inductive: 20 train, 2 val, 2 test graphs | 0.973 ± 0.002 micro-F1 | GraphSAGE-LSTM 0.612; Const-GAT 0.934 |

**Configurations in the paper**

- **Cora, Citeseer:** 2 layers: 8 heads × 8 features (ELU), then 1 head over the classes with softmax; dropout 0.6 on layer inputs and on the attention coefficients; L2 5·10⁻⁴; Adam, lr 0.005
- **Pubmed:** same, but 8 output heads averaged, L2 10⁻³, lr 0.01
- **PPI:** 3 layers: 4 heads × 256 features, 4 heads × 256, then 6 heads × 121 averaged with a sigmoid; skip connection across the middle layer; no dropout or L2
- **Const-GAT:** the same PPI model with all αᵢⱼ uniform; the drop from 0.973 to 0.934 isolates the value of attention

> Figure: One untrained layer (seeded features and weights), 3 heads, on a 9-node graph. Default: query v0 over {v0, v1, v2, v3, v4}. Head 2 puts 0.938 on v3, head 3 is nearly uniform (0.188 to 0.210), and the mean of heads is 0.164, 0.156, 0.121, 0.421, 0.138, entropy 1.479 against ln 5 = 1.609. Scale of a = 0 gives exactly 0.200 everywhere: GAT degenerates to a mean over the neighbourhood. Scale 3 sharpens head 2 to 1.000 on v3. The dense test scores all 9 keys for every query: GAT heads pick v6, v3 and v0 for every query (static attention). Switch to GATv2: head 2 picks v4 or v1 and head 3 picks v4 or v3, depending on the query.

## Worked example: one head on node v0

Features $h_0=(1,0)$, $h_1=(0,1)$, $h_2=(1,1)$, $h_3=(-2,0)$; edges v0–v1, v0–v2, v0–v3. Take $\mathbf W=I$, $\mathbf a_1=(0.5,0)$ for the query half and $\mathbf a_2=(0.5,1)$ for the key half. The query term is $\mathbf a_1^\top h_0=0.5$.

Computed in Python

| j | a₁ᵀz₀ + a₂ᵀzⱼ | eⱼ = LeakyReLU | exp(eⱼ) | α₀ⱼ |
| --- | --- | --- | --- | --- |
| v0 (self) | 0.5 + 0.5 = 1.0 | 1.0 | 2.718 | 0.175 |
| v1 | 0.5 + 1.0 = 1.5 | 1.5 | 4.482 | 0.289 |
| v2 | 0.5 + 1.5 = 2.0 | 2.0 | 7.389 | 0.477 |
| v3 | 0.5 − 1.0 = −0.5 | −0.1 | 0.905 | 0.058 |
| sum |  |  | 15.494 | 1.000 |

The aggregate is $\sum_j\alpha_{0j}h_j=(0.536,\,0.766)$, before $\sigma$. A plain mean over the same four nodes gives $(0,\,0.5)$: v3's large negative first coordinate pulls it to 0, while GAT gives v3 only 5.8%. Note that the query term 0.5 is added to every score and cancels in the softmax. With this scoring function the query only matters through LeakyReLU's kink, which is the static-attention limitation in miniature.

```python
import torch, torch.nn.functional as F
from torch_geometric.datasets import Planetoid
from torch_geometric.nn import GATConv          # GATv2Conv has the same interface

data = Planetoid("data", "Cora")[0]             # 2708 nodes, 1433 features, 7 classes

class GAT(torch.nn.Module):
    def __init__(self, d_in, n_cls):
        super().__init__()
        self.l1 = GATConv(d_in, 8, heads=8, dropout=0.6)                  # concat -> 64
        self.l2 = GATConv(8 * 8, n_cls, heads=1, concat=False, dropout=0.6)
    def forward(self, x, edge_index):
        x = F.dropout(x, 0.6, self.training)
        x = F.elu(self.l1(x, edge_index))
        x = F.dropout(x, 0.6, self.training)
        return self.l2(x, edge_index)

model = GAT(data.num_features, 7)
opt = torch.optim.Adam(model.parameters(), lr=0.005, weight_decay=5e-4)
for epoch in range(200):
    model.train(); opt.zero_grad()
    loss = F.cross_entropy(model(data.x, data.edge_index)[data.train_mask], data.y[data.train_mask])
    loss.backward(); opt.step()

model.eval()
acc = (model(data.x, data.edge_index).argmax(1) == data.y)[data.test_mask].float().mean()
_, (ei, alpha) = model.l1(data.x, data.edge_index, return_attention_weights=True)
print(f"test acc {acc:.3f}; alpha {tuple(alpha.shape)}")   # one row per edge incl. self-loops, 8 heads
```

The paper's Cora model in PyTorch Geometric, plus reading out attention weights (not run here: no PyG in this sandbox)

> **Attention weights are not explanations:** The $\alpha_{ij}$ of one head are a mixing weight, not evidence of importance. Heads disagree, later layers re-mix, and on homophilous citation graphs learned attention is often close to uniform; that is why GAT beats GCN by only about 1.5 points on Cora. Also keep the heavy regularisation: dropout 0.6 on the attention coefficients means each node sees a random subset of its neighbours at every step, and without it GAT overfits 140 labels quickly.

> **Uniform attention is a mean, not GCN:** With $\mathbf a=0$ every $\alpha_{ij}=1/(d_i+1)$, a row-normalised mean over the neighbourhood with self-loop. GCN's $1/\sqrt{\tilde d_i\tilde d_j}$ is symmetric and also down-weights high-degree neighbours. Neither weighting is learned from features; GAT's is.

**Strengths**

- Feature-dependent neighbour weights at O(|E|) cost, no eigendecomposition
- Inductive: the same W and a apply to any unseen graph
- Multi-head attention stabilises training and gives several views per layer
- Big gains when neighbours are not equally useful (PPI: 0.973 vs 0.934 with constant attention)

**Limitations**

- Original scoring is static: one global key ranking per head (fixed by GATv2)
- K heads cost K times the memory of a GCN layer; per-edge coefficients dominate on large graphs
- Little or no gain over GCN on small homophilous benchmarks
- Still 1-WL-bounded and prone to over-smoothing as layers stack, like other message-passing layers

**Question:** Nodes A and B both have neighbours X and Y. In one original GAT head, A gives X more weight than Y. What can B do?

- A. Anything; attention depends on the query
- B. B must also rank X above Y (ties aside)
- C. B must rank Y above X
- D. B gives X and Y equal weight

**Answer:** B. B must also rank X above Y (ties aside) — $e_{ij}=\mathrm{LeakyReLU}(\mathbf a_1^\top\mathbf Wh_i+\mathbf a_2^\top\mathbf Wh_j)$. For a fixed query the first term is a constant, and LeakyReLU is monotonic, so the order of keys is set by $\mathbf a_2^\top\mathbf Wh_j$ for every query. Sharpness can differ between A and B, but not the order. GATv2's $\mathbf a^\top\mathrm{LeakyReLU}(\mathbf W[h_i\Vert h_j])$ removes this.

**Question:** Why does GAT's output layer on Pubmed and PPI average its heads instead of concatenating them?

- A. Averaging is cheaper to backpropagate
- B. The output must have exactly one value per class, so heads are averaged before the softmax or sigmoid
- C. Concatenation breaks permutation invariance
- D. Averaging removes the need for a self-loop

**Answer:** B. The output must have exactly one value per class, so heads are averaged before the softmax or sigmoid — Concatenating K heads of C outputs gives K·C numbers, which no longer map to C class scores. Averaging keeps C outputs and still pools the heads. Hidden layers concatenate, since the next layer can use the wider vector.

**Key points**

- $e_{ij}=\mathrm{LeakyReLU}(\mathbf a^\top[\mathbf Wh_i\Vert\mathbf Wh_j])$, softmax over $\mathcal N(i)\cup\{i\}$, then $h_i'=\sigma(\sum_j\alpha_{ij}\mathbf Wh_j)$.
- Masked attention: only existing edges are scored, so cost is O(|V|FF′ + |E|F′) per head, like GCN.
- K heads: concatenate in hidden layers (Cora: 8 × 8 = 64), average in the output layer.
- Results: Cora 83.0%, Citeseer 72.5%, Pubmed 79.0%, PPI 0.973 micro-F1 (0.934 with uniform attention).
- Original GAT attention is static (one key ranking per head); GATv2 applies a after the nonlinearity to make it dynamic.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
