Variational graph autoencoders

A VGAE gives each node a Gaussian posterior N(μᵢ, σᵢ²) from two GCN heads, samples z = μ + σ⊙ε, decodes edges as σ(zᵢ·zⱼ) and maximises the ELBO: reconstruction of A minus KL to N(0, I) (Kipf & Welling, 2016).

8 min read678 wordsadvanced0 viewsupdated 2026-10-21

Definition

Variational graph autoencoder (VGAE) is a latent-variable model of a graph's edges. Each node gets a Gaussian latent ziz_i, the probability of an edge is σ(zi⊤zj)\sigma(z_i^\top z_j), and a Graph Convolutional Network (GCN) encoder infers the posterior over ZZ from the features and the graph. It is trained like a Variational autoencoder (VAE), by maximising the evidence lower bound (ELBO). Introduced by Thomas Kipf and Max Welling, "Variational Graph Auto-Encoders" (NIPS 2016 Workshop on Bayesian Deep Learning), together with its deterministic version, Graph autoencoders.

The task is the same as for the GAE: given AA and features XX, learn node embeddings that predict missing edges. The difference is the probabilistic reading. In the generative model, each node draws zi∼N(0,I)z_i\sim\mathcal N(0,I), and every pair is linked independently with probability σ(zi⊤zj)\sigma(z_i^\top z_j). Training fits an approximate posterior q(Z∣X,A)q(Z\mid X,A). A KL penalty keeps it close to the prior. The result is a smooth latent space with an uncertainty per node, rather than a single point per node.

Inference model: two GCN heads

q(Z∣X,A)=∏i=1NN ⁣(zi∣μi,diag⁡(σi2)),μ=A^ Xˉ Wμ,log⁡σ=A^ Xˉ Wσ,Xˉ=ReLU(A^XW0)q(Z\mid X,A)=\prod_{i=1}^{N}\mathcal N\!\big(z_i\mid\mu_i,\operatorname{diag}(\sigma_i^2)\big),\qquad \mu=\hat{\mathcal A}\,\bar X\,W_\mu,\quad \log\sigma=\hat{\mathcal A}\,\bar X\,W_\sigma,\quad \bar X=\mathrm{ReLU}(\hat{\mathcal A}XW_0)
A^\hat{\mathcal A}
D~−1/2(A+I)D~−1/2\tilde D^{-1/2}(A+I)\tilde D^{-1/2}, built from training edges only
W0W_0
first GCN layer, shared by both heads (F×32F\times32 in the paper)
Wμ, WσW_\mu,\ W_\sigma
separate second layers (32×1632\times16): μ\mu and log⁡σ\log\sigma are N×16N\times16
VGAE (Kipf & Welling, 2016): Gaussian q(Z | X, A) from a GCN, inner-product p(A | Z)inputA, XN × Fshared GCNReLU(ÂXW₀)H: N × 32mean headμ = ÂHW_μN × 16log-std headlog σ = ÂHW_σN × 16samplez = μ + σ⊙εε ~ N(0, I)decoderσ(Z Zᵀ)N × N edge probsmaximise ELBO = E_q[log p(A | Z)] − KL(q(Z | X, A) ‖ ∏ᵢ N(zᵢ | 0, I))latent (d = 2 shown), [−2, 2]²dashed: prior, 1 stdzμone node, d = 2μ(0.80, −0.30)log σ(−1.00, −0.50) → σ = (0.37, 0.61)ε(0.50, −1.00) drawn from N(0, I)z = μ + σ⊙ε(0.98, −0.91)KL to N(0, I)½Σ(μ² + σ² − 1 − 2 log σ) = 1.12 natsgradients reach μ and σ through z (reparameterisation trick)at test time the embeddings are the means μ
fig.VGAE with the paper's sizes. One shared GCN layer (32 units) feeds a mean head and a log-std head (16 each). The sample z = μ + σ⊙ε goes through the inner-product decoder σ(ZZᵀ). The ELBO trades the reconstruction of A against the KL divergence to the prior, whose 1-std circle is dashed in the latent inset. Below right, one node: μ = (0.80, −0.30), σ = (0.37, 0.61) and ε = (0.50, −1.00) give z = (0.98, −0.91), with a KL cost of 1.12 nats.

Generative model and objective

p(A∣Z)=∏i=1N∏j=1Np(Aij∣zi,zj),p(Aij=1∣zi,zj)=σ(zi⊤zj),p(Z)=∏iN(zi∣0,I)p(A\mid Z)=\prod_{i=1}^{N}\prod_{j=1}^{N}p(A_{ij}\mid z_i,z_j),\qquad p(A_{ij}=1\mid z_i,z_j)=\sigma\big(z_i^\top z_j\big),\qquad p(Z)=\prod_i\mathcal N(z_i\mid0,I)
σ\sigma
logistic sigmoid; the decoder has no parameters, exactly as in the GAE
L=Eq(Z∣X,A)[log⁡p(A∣Z)]−KL(q(Z∣X,A) ∥ p(Z)),KL=∑i∑k12(μik2+σik2−1−2log⁡σik)\mathcal L=\mathbb E_{q(Z\mid X,A)}\big[\log p(A\mid Z)\big]-\mathrm{KL}\big(q(Z\mid X,A)\,\big\|\,p(Z)\big),\qquad \mathrm{KL}=\sum_{i}\sum_{k}\tfrac12\big(\mu_{ik}^2+\sigma_{ik}^2-1-2\log\sigma_{ik}\big)
Eq[log⁡p(A∣Z)]\mathbb E_q[\log p(A\mid Z)]
estimated with one sample Z=μ+σ⊙εZ=\mu+\sigma\odot\varepsilon, ε∼N(0,I)\varepsilon\sim\mathcal N(0,I), per step; it is the weighted Binary cross-entropy of the GAE, with a sign flip
KL\mathrm{KL}
closed form for diagonal Gaussians, summed over nodes ii and latent dimensions kk (see KL divergence loss)

The sample is written as z=μ+σ⊙εz=\mu+\sigma\odot\varepsilon, so gradients reach μ\mu and σ\sigma through a deterministic function. This is the reparameterisation trick of the VAE. Positive edges are reweighted as in the GAE. The reference code averages the reconstruction term over the N2N^2 entries and the KL term over nodes, then scales the KL by 1/N1/N, so the two terms stay comparable in size. At test time the embeddings are the means μ\mu, with no sampling.

What changes from the GAE

Parameters
one extra 32×1632\times16 matrix WσW_\sigma; the decoder still has none
Embedding
a distribution N(μi,σi2)\mathcal N(\mu_i,\sigma_i^2) per node; μi\mu_i is used at test time
Loss
the same weighted reconstruction BCE, on a sample Z, plus the KL term
Cost
one Gaussian sample per step; the O(N²) full decoder dominates either way
ModelCora AUCCora APCiteseer AUCCiteseer APPubMed AUCPubMed AP
GAE*84.388.178.784.182.287.4
VGAE*84.087.778.984.182.787.5
GAE91.092.089.589.996.496.5
VGAE91.492.690.892.094.494.7
Link prediction on the 85/5/10 edge split, AUC / AP in % (Kipf & Welling, 2016). Starred rows use no features (X = I). Same setting as the GAE: 32 hidden, 16 latent, Adam lr 0.01, 200 iterations

The variational model wins on Cora and Citeseer and loses on PubMed, all by 2 points or less. Most of the gain over the spectral and DeepWalk baselines comes from the GCN encoder and the features, not from the variational treatment. The value of the VGAE lies elsewhere: a regularised latent space, a per-node σ\sigma, and a proper generative model of edges.

Try it: the KL weight

VGAE · 40 nodes, 4 communities · p_in 0.6, p_out 0.02 · X = I · β = 1 × 1/Nq(zᵢ) = N(μᵢ, σᵢ²) · prior N(0, I) dashedupper: σ(zᵢ·zⱼ) of sample #1 · lower: Abox [−3, 3]² · dots: μ · ticks: z = μ + σ⊙εrecon 0.458 · KL 2.597 nats/node · mean σ 0.295test AUC: decode μ 0.896 (AP 0.853) · decode sample #1 0.86917 held-out edges (rose) vs 17 non-edges · trained 200 epochs, Adam, lr 0.01KL keeps σ away from 0 and pulls μ toward 0; reconstruction keeps the groups apart
decode
fig.The 40-node, four-community graph from the GAE article (p_out = 0.02, 17 held-out edges), now with a 2-d Gaussian per node, trained for 200 epochs. β multiplies the paper's KL weight. At β = 1, mean σ is 0.295, KL is 2.597 nats per node, test AUC is 0.896 when μ is decoded (AP 0.853) and 0.869 for sample #1. At β = 0 nothing holds σ up, so it shrinks to 0.051 and AUC is 0.851: a GAE with a little noise. At β = 2, σ grows to 0.485 and AUC falls to 0.813. At β = 8, μ moves toward the origin (KL 0.119 nats per node): μ still ranks the test edges at 0.734, but sample #1 scores only 0.491. At β = 16 the posterior is almost the prior (σ = 0.851) and AUC is 0.536.

Worked example: one node, d = 2

  1. 1Encoder output. μi=(0.80, −0.30)\mu_i=(0.80,\,-0.30) and log⁡σi=(−1.00, −0.50)\log\sigma_i=(-1.00,\,-0.50), so σi=(e−1,e−0.5)=(0.368, 0.607)\sigma_i=(e^{-1},e^{-0.5})=(0.368,\,0.607).
  2. 2Sample. Draw ε=(0.50, −1.00)\varepsilon=(0.50,\,-1.00). Then zi=μi+σi⊙ε=(0.80+0.184,  −0.30−0.607)=(0.984, −0.907)z_i=\mu_i+\sigma_i\odot\varepsilon=(0.80+0.184,\;-0.30-0.607)=(0.984,\,-0.907).
  3. 3KL to the prior. Dimension 1: 12(0.640+0.135−1+2.000)=0.888\tfrac12(0.640+0.135-1+2.000)=0.888. Dimension 2: 12(0.090+0.368−1+1.000)=0.229\tfrac12(0.090+0.368-1+1.000)=0.229. Total 1.1171.117 nats for this node.
  4. 4Decode one pair. For a neighbour with zj=(1.00, −0.50)z_j=(1.00,\,-0.50): zi⊤zj=0.984+0.453=1.437z_i^\top z_j=0.984+0.453=1.437, so p(Aij=1)=σ(1.437)=0.808p(A_{ij}=1)=\sigma(1.437)=0.808. Using the mean instead, μi⊤zj=0.95\mu_i^\top z_j=0.95 and σ(0.95)=0.721\sigma(0.95)=0.721.
  5. 5Gradient path. ∂zi/∂μi=1\partial z_i/\partial\mu_i=1 and ∂zi/∂log⁡σi=σi⊙ε=(0.184, −0.607)\partial z_i/\partial\log\sigma_i=\sigma_i\odot\varepsilon=(0.184,\,-0.607), so the reconstruction loss trains both heads. The KL gradient pushes μi\mu_i toward 0 and σi\sigma_i toward 1.
python
1import torch
2from torch_geometric.nn import VGAE, GCNConv
3
4class VariationalGCNEncoder(torch.nn.Module):
5 def __init__(self, f_in, hidden=32, d=16):
6 super().__init__()
7 self.conv1 = GCNConv(f_in, hidden) # shared first layer (W0)
8 self.conv_mu = GCNConv(hidden, d) # W_mu
9 self.conv_logstd = GCNConv(hidden, d) # W_sigma
10 def forward(self, x, edge_index):
11 h = self.conv1(x, edge_index).relu()
12 return self.conv_mu(h, edge_index), self.conv_logstd(h, edge_index)
13
14# train_data / test_data from RandomLinkSplit, as in the GAE example
15model = VGAE(VariationalGCNEncoder(train_data.num_features))
16opt = torch.optim.Adam(model.parameters(), lr=0.01)
17for epoch in range(200):
18 model.train(); opt.zero_grad()
19 z = model.encode(train_data.x, train_data.edge_index) # z = mu + sigma * eps
20 loss = model.recon_loss(z, train_data.pos_edge_label_index) # -E_q[log p(A | Z)], sampled negatives
21 loss = loss + (1 / train_data.num_nodes) * model.kl_loss() # KL to N(0, I)
22 loss.backward(); opt.step()
23
24model.eval()
25with torch.no_grad():
26 mu = model.encode(test_data.x, test_data.edge_index) # eval mode: the means
27 auc, ap = model.test(mu, test_data.pos_edge_label_index, test_data.neg_edge_label_index)
VGAE in PyTorch Geometric (not run here: no torch in this environment). encode() samples z in training mode and returns μ in eval mode; kl_loss() averages over nodes, hence the 1/N factor

Several later models build on this one. ARVGA (Pan et al., IJCAI 2018) adds an adversarial regulariser that matches q(Z)q(Z) to the prior. GraphVAE (Simonovsky and Komodakis, 2018) changes the task from modelling one large graph to generating whole small graphs, such as molecules, from one latent vector each. Edge-level evaluation for all of them follows Link prediction.

Where it's used

Link prediction

Ranking missing edges in citation, social or interaction graphs, scored by σ(μᵢ·μⱼ).

Unsupervised node embeddings

μ as features for clustering or for a downstream classifier when labels are scarce.

Uncertainty

σᵢ is a per-node spread in latent space; sampling Z gives a distribution over predicted edges instead of one score.

Use when

  • +Principled generative model of edges, with a per-node uncertainty σ
  • +The KL term regularises the latent space; slightly better AUC than the GAE on Cora and Citeseer
  • +Drop-in change from a GAE: one extra GCN head plus a KL term

Avoid when

  • −Gaussian prior conflicts with the inner-product decoder; gains over the GAE are small and not uniform (PubMed −2 AUC)
  • −Sensitive to the KL weight: too high collapses the posterior, too low reduces it to a GAE
  • −Still O(N²) in the full decoder, and still symmetric with no edge types

Check yourself

After training a VGAE, every node has σ ≈ 1, μ ≈ 0, and test AUC ≈ 0.5. What happened?

Check yourself

Which embedding does the VGAE use to score test pairs?

Remember

  • ✓q(zi)=N(μi,σi2)q(z_i)=\mathcal N(\mu_i,\sigma_i^2) from a shared GCN layer and two GCN heads; z=μ+σ⊙εz=\mu+\sigma\odot\varepsilon.
  • ✓p(Aij=1∣zi,zj)=σ(zi⊤zj)p(A_{ij}=1\mid z_i,z_j)=\sigma(z_i^\top z_j), prior N(0,I)\mathcal N(0,I); maximise Eq[log⁡p(A∣Z)]−KL(q ∥ p)\mathbb E_q[\log p(A\mid Z)]-\mathrm{KL}(q\,\|\,p).
  • ✓The KL is scaled 1/N relative to the per-entry reconstruction; too much KL collapses q to the prior, none makes it a GAE.
  • ✓Cora / Citeseer / PubMed AUC: VGAE 91.4 / 90.8 / 94.4 vs GAE 91.0 / 89.5 / 96.4.
  • ✓A Gaussian prior pulls embeddings to the origin, where the inner-product decoder gives about 0.5 for every pair: a known mismatch.
← back to Deep Learning roadmap