Definition
Variational graph autoencoder (VGAE) is a latent-variable model of a graph's edges. Each node gets a Gaussian latent , the probability of an edge is , and a Graph Convolutional Network (GCN) encoder infers the posterior over from the features and the graph. It is trained like a Variational autoencoder (VAE), by maximising the evidence lower bound (ELBO). Introduced by Thomas Kipf and Max Welling, "Variational Graph Auto-Encoders" (NIPS 2016 Workshop on Bayesian Deep Learning), together with its deterministic version, Graph autoencoders.
The task is the same as for the GAE: given and features , learn node embeddings that predict missing edges. The difference is the probabilistic reading. In the generative model, each node draws , and every pair is linked independently with probability . Training fits an approximate posterior . A KL penalty keeps it close to the prior. The result is a smooth latent space with an uncertainty per node, rather than a single point per node.
Inference model: two GCN heads
- , built from training edges only
- first GCN layer, shared by both heads ( in the paper)
- separate second layers (): and are
Generative model and objective
- logistic sigmoid; the decoder has no parameters, exactly as in the GAE
- estimated with one sample , , per step; it is the weighted Binary cross-entropy of the GAE, with a sign flip
- closed form for diagonal Gaussians, summed over nodes and latent dimensions (see KL divergence loss)
The sample is written as , so gradients reach and through a deterministic function. This is the reparameterisation trick of the VAE. Positive edges are reweighted as in the GAE. The reference code averages the reconstruction term over the entries and the KL term over nodes, then scales the KL by , so the two terms stay comparable in size. At test time the embeddings are the means , with no sampling.
What changes from the GAE
- Parameters
- one extra matrix ; the decoder still has none
- Embedding
- a distribution per node; is used at test time
- Loss
- the same weighted reconstruction BCE, on a sample Z, plus the KL term
- Cost
- one Gaussian sample per step; the O(N²) full decoder dominates either way
| Model | Cora AUC | Cora AP | Citeseer AUC | Citeseer AP | PubMed AUC | PubMed AP |
|---|---|---|---|---|---|---|
| GAE* | 84.3 | 88.1 | 78.7 | 84.1 | 82.2 | 87.4 |
| VGAE* | 84.0 | 87.7 | 78.9 | 84.1 | 82.7 | 87.5 |
| GAE | 91.0 | 92.0 | 89.5 | 89.9 | 96.4 | 96.5 |
| VGAE | 91.4 | 92.6 | 90.8 | 92.0 | 94.4 | 94.7 |
The variational model wins on Cora and Citeseer and loses on PubMed, all by 2 points or less. Most of the gain over the spectral and DeepWalk baselines comes from the GCN encoder and the features, not from the variational treatment. The value of the VGAE lies elsewhere: a regularised latent space, a per-node , and a proper generative model of edges.
Try it: the KL weight
Worked example: one node, d = 2
- 1Encoder output. and , so .
- 2Sample. Draw . Then .
- 3KL to the prior. Dimension 1: . Dimension 2: . Total nats for this node.
- 4Decode one pair. For a neighbour with : , so . Using the mean instead, and .
- 5Gradient path. and , so the reconstruction loss trains both heads. The KL gradient pushes toward 0 and toward 1.
1import torch2from torch_geometric.nn import VGAE, GCNConv34class VariationalGCNEncoder(torch.nn.Module):5 def __init__(self, f_in, hidden=32, d=16):6 super().__init__()7 self.conv1 = GCNConv(f_in, hidden) # shared first layer (W0)8 self.conv_mu = GCNConv(hidden, d) # W_mu9 self.conv_logstd = GCNConv(hidden, d) # W_sigma10 def forward(self, x, edge_index):11 h = self.conv1(x, edge_index).relu()12 return self.conv_mu(h, edge_index), self.conv_logstd(h, edge_index)1314# train_data / test_data from RandomLinkSplit, as in the GAE example15model = VGAE(VariationalGCNEncoder(train_data.num_features))16opt = torch.optim.Adam(model.parameters(), lr=0.01)17for epoch in range(200):18 model.train(); opt.zero_grad()19 z = model.encode(train_data.x, train_data.edge_index) # z = mu + sigma * eps20 loss = model.recon_loss(z, train_data.pos_edge_label_index) # -E_q[log p(A | Z)], sampled negatives21 loss = loss + (1 / train_data.num_nodes) * model.kl_loss() # KL to N(0, I)22 loss.backward(); opt.step()2324model.eval()25with torch.no_grad():26 mu = model.encode(test_data.x, test_data.edge_index) # eval mode: the means27 auc, ap = model.test(mu, test_data.pos_edge_label_index, test_data.neg_edge_label_index)Several later models build on this one. ARVGA (Pan et al., IJCAI 2018) adds an adversarial regulariser that matches to the prior. GraphVAE (Simonovsky and Komodakis, 2018) changes the task from modelling one large graph to generating whole small graphs, such as molecules, from one latent vector each. Edge-level evaluation for all of them follows Link prediction.
Where it's used
Link prediction
Ranking missing edges in citation, social or interaction graphs, scored by σ(μᵢ·μⱼ).
Unsupervised node embeddings
μ as features for clustering or for a downstream classifier when labels are scarce.
Uncertainty
σᵢ is a per-node spread in latent space; sampling Z gives a distribution over predicted edges instead of one score.
Use when
- +Principled generative model of edges, with a per-node uncertainty σ
- +The KL term regularises the latent space; slightly better AUC than the GAE on Cora and Citeseer
- +Drop-in change from a GAE: one extra GCN head plus a KL term
Avoid when
- −Gaussian prior conflicts with the inner-product decoder; gains over the GAE are small and not uniform (PubMed −2 AUC)
- −Sensitive to the KL weight: too high collapses the posterior, too low reduces it to a GAE
- −Still O(N²) in the full decoder, and still symmetric with no edge types
Check yourself
After training a VGAE, every node has σ ≈ 1, μ ≈ 0, and test AUC ≈ 0.5. What happened?
Check yourself
Which embedding does the VGAE use to score test pairs?
Remember
- ✓ from a shared GCN layer and two GCN heads; .
- ✓, prior ; maximise .
- ✓The KL is scaled 1/N relative to the per-entry reconstruction; too much KL collapses q to the prior, none makes it a GAE.
- ✓Cora / Citeseer / PubMed AUC: VGAE 91.4 / 90.8 / 94.4 vs GAE 91.0 / 89.5 / 96.4.
- ✓A Gaussian prior pulls embeddings to the origin, where the inner-product decoder gives about 0.5 for every pair: a known mismatch.