# Variational graph autoencoders

> A VGAE gives each node a Gaussian posterior N(μᵢ, σᵢ²) from two GCN heads, samples z = μ + σ⊙ε, decodes edges as σ(zᵢ·zⱼ) and maximises the ELBO: reconstruction of A minus KL to N(0, I) (Kipf & Welling, 2016).

Source: https://www.mlfoundry.in/learn/variational-graph-autoencoders  
Section: Deep Learning › Graph Neural Networks (GNNs)  
Level: advanced · 8 min read · updated 2026-10-21

**Variational graph autoencoder (VGAE):** is a latent-variable model of a graph's edges. Each node gets a Gaussian latent $z_i$, the probability of an edge is $\sigma(z_i^\top z_j)$, and a **Graph Convolutional Network (GCN)** encoder infers the posterior over $Z$ from the features and the graph. It is trained like a **Variational autoencoder (VAE)**, by maximising the evidence lower bound (ELBO). Introduced by Thomas Kipf and Max Welling, "Variational Graph Auto-Encoders" (NIPS 2016 Workshop on Bayesian Deep Learning), together with its deterministic version, **Graph autoencoders**.

The task is the same as for the GAE: given $A$ and features $X$, learn node embeddings that predict missing edges. The difference is the probabilistic reading. In the generative model, each node draws $z_i\sim\mathcal N(0,I)$, and every pair is linked independently with probability $\sigma(z_i^\top z_j)$. Training fits an approximate posterior $q(Z\mid X,A)$. A KL penalty keeps it close to the prior. The result is a smooth latent space with an uncertainty per node, rather than a single point per node.

## Inference model: two GCN heads

$$
q(Z\mid X,A)=\prod_{i=1}^{N}\mathcal N\!\big(z_i\mid\mu_i,\operatorname{diag}(\sigma_i^2)\big),\qquad \mu=\hat{\mathcal A}\,\bar X\,W_\mu,\quad \log\sigma=\hat{\mathcal A}\,\bar X\,W_\sigma,\quad \bar X=\mathrm{ReLU}(\hat{\mathcal A}XW_0)
$$

- $\hat{\mathcal A}$: $\tilde D^{-1/2}(A+I)\tilde D^{-1/2}$, built from training edges only
- $W_0$: first GCN layer, shared by both heads ($F\times32$ in the paper)
- $W_\mu,\ W_\sigma$: separate second layers ($32\times16$): $\mu$ and $\log\sigma$ are $N\times16$

> Figure: VGAE with the paper's sizes. One shared GCN layer (32 units) feeds a mean head and a log-std head (16 each). The sample z = μ + σ⊙ε goes through the inner-product decoder σ(ZZᵀ). The ELBO trades the reconstruction of A against the KL divergence to the prior, whose 1-std circle is dashed in the latent inset. Below right, one node: μ = (0.80, −0.30), σ = (0.37, 0.61) and ε = (0.50, −1.00) give z = (0.98, −0.91), with a KL cost of 1.12 nats.

## Generative model and objective

$$
p(A\mid Z)=\prod_{i=1}^{N}\prod_{j=1}^{N}p(A_{ij}\mid z_i,z_j),\qquad p(A_{ij}=1\mid z_i,z_j)=\sigma\big(z_i^\top z_j\big),\qquad p(Z)=\prod_i\mathcal N(z_i\mid0,I)
$$

- $\sigma$: logistic sigmoid; the decoder has no parameters, exactly as in the GAE

$$
\mathcal L=\mathbb E_{q(Z\mid X,A)}\big[\log p(A\mid Z)\big]-\mathrm{KL}\big(q(Z\mid X,A)\,\big\|\,p(Z)\big),\qquad \mathrm{KL}=\sum_{i}\sum_{k}\tfrac12\big(\mu_{ik}^2+\sigma_{ik}^2-1-2\log\sigma_{ik}\big)
$$

- $\mathbb E_q[\log p(A\mid Z)]$: estimated with one sample $Z=\mu+\sigma\odot\varepsilon$, $\varepsilon\sim\mathcal N(0,I)$, per step; it is the weighted **Binary cross-entropy** of the GAE, with a sign flip
- $\mathrm{KL}$: closed form for diagonal Gaussians, summed over nodes $i$ and latent dimensions $k$ (see **KL divergence loss**)

The sample is written as $z=\mu+\sigma\odot\varepsilon$, so gradients reach $\mu$ and $\sigma$ through a deterministic function. This is the reparameterisation trick of the VAE. Positive edges are reweighted as in the GAE. The reference code averages the reconstruction term over the $N^2$ entries and the KL term over nodes, then scales the KL by $1/N$, so the two terms stay comparable in size. At test time the embeddings are the means $\mu$, with no sampling.

**What changes from the GAE**

- **Parameters:** one extra $32\times16$ matrix $W_\sigma$; the decoder still has none
- **Embedding:** a distribution $\mathcal N(\mu_i,\sigma_i^2)$ per node; $\mu_i$ is used at test time
- **Loss:** the same weighted reconstruction BCE, on a sample Z, plus the KL term
- **Cost:** one Gaussian sample per step; the O(N²) full decoder dominates either way

Link prediction on the 85/5/10 edge split, AUC / AP in % (Kipf & Welling, 2016). Starred rows use no features (X = I). Same setting as the GAE: 32 hidden, 16 latent, Adam lr 0.01, 200 iterations

| Model | Cora AUC | Cora AP | Citeseer AUC | Citeseer AP | PubMed AUC | PubMed AP |
| --- | --- | --- | --- | --- | --- | --- |
| GAE* | 84.3 | 88.1 | 78.7 | 84.1 | 82.2 | 87.4 |
| VGAE* | 84.0 | 87.7 | 78.9 | 84.1 | 82.7 | 87.5 |
| GAE | 91.0 | 92.0 | 89.5 | 89.9 | 96.4 | 96.5 |
| VGAE | 91.4 | 92.6 | 90.8 | 92.0 | 94.4 | 94.7 |

The variational model wins on Cora and Citeseer and loses on PubMed, all by 2 points or less. Most of the gain over the spectral and DeepWalk baselines comes from the GCN encoder and the features, not from the variational treatment. The value of the VGAE lies elsewhere: a regularised latent space, a per-node $\sigma$, and a proper generative model of edges.

## Try it: the KL weight

> Figure: The 40-node, four-community graph from the GAE article (p_out = 0.02, 17 held-out edges), now with a 2-d Gaussian per node, trained for 200 epochs. β multiplies the paper's KL weight. At β = 1, mean σ is 0.295, KL is 2.597 nats per node, test AUC is 0.896 when μ is decoded (AP 0.853) and 0.869 for sample #1. At β = 0 nothing holds σ up, so it shrinks to 0.051 and AUC is 0.851: a GAE with a little noise. At β = 2, σ grows to 0.485 and AUC falls to 0.813. At β = 8, μ moves toward the origin (KL 0.119 nats per node): μ still ranks the test edges at 0.734, but sample #1 scores only 0.491. At β = 16 the posterior is almost the prior (σ = 0.851) and AUC is 0.536.

**Worked example: one node, d = 2**

1. **Encoder output.** $\mu_i=(0.80,\,-0.30)$ and $\log\sigma_i=(-1.00,\,-0.50)$, so $\sigma_i=(e^{-1},e^{-0.5})=(0.368,\,0.607)$.
2. **Sample.** Draw $\varepsilon=(0.50,\,-1.00)$. Then $z_i=\mu_i+\sigma_i\odot\varepsilon=(0.80+0.184,\;-0.30-0.607)=(0.984,\,-0.907)$.
3. **KL to the prior.** Dimension 1: $\tfrac12(0.640+0.135-1+2.000)=0.888$. Dimension 2: $\tfrac12(0.090+0.368-1+1.000)=0.229$. Total $1.117$ nats for this node.
4. **Decode one pair.** For a neighbour with $z_j=(1.00,\,-0.50)$: $z_i^\top z_j=0.984+0.453=1.437$, so $p(A_{ij}=1)=\sigma(1.437)=0.808$. Using the mean instead, $\mu_i^\top z_j=0.95$ and $\sigma(0.95)=0.721$.
5. **Gradient path.** $\partial z_i/\partial\mu_i=1$ and $\partial z_i/\partial\log\sigma_i=\sigma_i\odot\varepsilon=(0.184,\,-0.607)$, so the reconstruction loss trains both heads. The KL gradient pushes $\mu_i$ toward 0 and $\sigma_i$ toward 1.

```python
import torch
from torch_geometric.nn import VGAE, GCNConv

class VariationalGCNEncoder(torch.nn.Module):
    def __init__(self, f_in, hidden=32, d=16):
        super().__init__()
        self.conv1 = GCNConv(f_in, hidden)            # shared first layer (W0)
        self.conv_mu = GCNConv(hidden, d)             # W_mu
        self.conv_logstd = GCNConv(hidden, d)         # W_sigma
    def forward(self, x, edge_index):
        h = self.conv1(x, edge_index).relu()
        return self.conv_mu(h, edge_index), self.conv_logstd(h, edge_index)

# train_data / test_data from RandomLinkSplit, as in the GAE example
model = VGAE(VariationalGCNEncoder(train_data.num_features))
opt = torch.optim.Adam(model.parameters(), lr=0.01)
for epoch in range(200):
    model.train(); opt.zero_grad()
    z = model.encode(train_data.x, train_data.edge_index)        # z = mu + sigma * eps
    loss = model.recon_loss(z, train_data.pos_edge_label_index)  # -E_q[log p(A | Z)], sampled negatives
    loss = loss + (1 / train_data.num_nodes) * model.kl_loss()   # KL to N(0, I)
    loss.backward(); opt.step()

model.eval()
with torch.no_grad():
    mu = model.encode(test_data.x, test_data.edge_index)         # eval mode: the means
    auc, ap = model.test(mu, test_data.pos_edge_label_index, test_data.neg_edge_label_index)
```

VGAE in PyTorch Geometric (not run here: no torch in this environment). encode() samples z in training mode and returns μ in eval mode; kl_loss() averages over nodes, hence the 1/N factor

> **The prior and the decoder pull in opposite directions:** An inner-product decoder can only give high edge probability through large, aligned embeddings. The $\mathcal N(0,I)$ prior pulls every $z_i$ toward the origin, where $\sigma(z_i^\top z_j)\approx0.5$ for every pair. Kipf and Welling point this out themselves: a Gaussian prior is a questionable match for an inner-product decoder. Hyperspherical latents (S-VGAE, Davidson et al., 2018), which use von Mises–Fisher distributions, were one response. The figure shows the conflict: raising β shrinks $\mu$ and erases the community structure.

> **Check how the KL is scaled:** The KL weight depends on how each term is averaged. PyG's `kl_loss()` is a mean over nodes, and `recon_loss` is a mean over the sampled pairs. Adding them without the $1/N$ factor makes the KL about $N$ times stronger than in the paper. For Cora that is 2,708 times, which is enough to collapse $q$ onto the prior, the β = 16 regime of the figure. The opposite error, a KL weight near zero, gives $\sigma\to0$: a GAE with extra parameters.

Several later models build on this one. ARVGA (Pan et al., IJCAI 2018) adds an adversarial regulariser that matches $q(Z)$ to the prior. GraphVAE (Simonovsky and Komodakis, 2018) changes the task from modelling one large graph to generating whole small graphs, such as molecules, from one latent vector each. Edge-level evaluation for all of them follows **Link prediction**.

- **Link prediction:** Ranking missing edges in citation, social or interaction graphs, scored by σ(μᵢ·μⱼ).
- **Unsupervised node embeddings:** μ as features for clustering or for a downstream classifier when labels are scarce.
- **Uncertainty:** σᵢ is a per-node spread in latent space; sampling Z gives a distribution over predicted edges instead of one score.

**Strengths**

- Principled generative model of edges, with a per-node uncertainty σ
- The KL term regularises the latent space; slightly better AUC than the GAE on Cora and Citeseer
- Drop-in change from a GAE: one extra GCN head plus a KL term

**Limitations**

- Gaussian prior conflicts with the inner-product decoder; gains over the GAE are small and not uniform (PubMed −2 AUC)
- Sensitive to the KL weight: too high collapses the posterior, too low reduces it to a GAE
- Still O(N²) in the full decoder, and still symmetric with no edge types

**Question:** After training a VGAE, every node has σ ≈ 1, μ ≈ 0, and test AUC ≈ 0.5. What happened?

- A. The learning rate was too low for the decoder's weights
- B. The KL term dominated, so the posterior collapsed onto the N(0, I) prior
- C. The test edges leaked into the encoder's adjacency
- D. The reparameterisation trick blocks gradients to μ

**Answer:** B. The KL term dominated, so the posterior collapsed onto the N(0, I) prior — With KL = 0, q equals the prior and carries no information about the graph, so every pair scores about σ(0) = 0.5 on average. A common cause is a missing 1/N factor on a node-averaged KL. The decoder has no weights to tune, and leakage would raise AUC, not lower it.

**Question:** Which embedding does the VGAE use to score test pairs?

- A. A single random sample z
- B. The mean μ of q(z | X, A)
- C. The log-std head log σ
- D. An average of ZZᵀ over 100 samples

**Answer:** B. The mean μ of q(z | X, A) — At test time the reference implementation and PyG both use μ, which is deterministic. In the figure at β = 1, decoding μ gives 0.896 test AUC, against 0.869 for one sample.

**Key points**

- $q(z_i)=\mathcal N(\mu_i,\sigma_i^2)$ from a shared GCN layer and two GCN heads; $z=\mu+\sigma\odot\varepsilon$.
- $p(A_{ij}=1\mid z_i,z_j)=\sigma(z_i^\top z_j)$, prior $\mathcal N(0,I)$; maximise $\mathbb E_q[\log p(A\mid Z)]-\mathrm{KL}(q\,\|\,p)$.
- The KL is scaled 1/N relative to the per-entry reconstruction; too much KL collapses q to the prior, none makes it a GAE.
- Cora / Citeseer / PubMed AUC: VGAE 91.4 / 90.8 / 94.4 vs GAE 91.0 / 89.5 / 96.4.
- A Gaussian prior pulls embeddings to the origin, where the inner-product decoder gives about 0.5 for every pair: a known mismatch.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
