Deep belief network (DBN)

A stack of RBMs trained greedily one layer at a time, each modelling the codes of the one below; the 2006 recipe that made deep nets trainable.

11 min read955 wordsadvanced0 viewsupdated 2026-10-11

Definition

Deep belief network is a generative model with several layers of binary hidden units. The top two layers form an undirected RBM; every layer below is a directed sigmoid belief net that generates the layer beneath it. It is trained greedily: fit an RBM to the data, then fit another RBM to the first one's hidden codes, and so on.

In 2006, deep networks trained from random weights with backprop usually ended up worse than shallow ones: sigmoid units saturated, gradients vanished, and nobody knew how to initialise them. Hinton, Osindero and Teh showed that you could build a deep model one layer at a time without labels, each layer an RBM trained with contrastive divergence, and then fine-tune it. Their DBN reached 1.25% error on permutation-invariant MNIST, better than the SVMs and backprop-trained nets of the time.

Together with Hinton & Salakhutdinov's deep autoencoders (Science, 2006) and Bengio et al.'s greedy layer-wise study (2007), this kicked off the deep learning revival. DBN-pretrained networks gave the first big deep-learning wins in speech recognition (2009–2012).

The model

Write h(0)=vh^{(0)}=v for the data and h(1),…,h(L)h^{(1)},\dots,h^{(L)} for the hidden layers. The joint distribution is an RBM at the top times a chain of sigmoid layers going down:

p(v,h(1),…,h(L))=p(h(L−1),h(L))∏l=0L−2p(h(l)∣h(l+1)),p(hi(l)=1∣h(l+1))=σ(ai(l)+∑jWij(l+1)hj(l+1))p\big(v,h^{(1)},\dots,h^{(L)}\big)=p\big(h^{(L-1)},h^{(L)}\big)\prod_{l=0}^{L-2}p\big(h^{(l)}\mid h^{(l+1)}\big),\qquad p\big(h^{(l)}_i=1\mid h^{(l+1)}\big)=\sigma\Big(a^{(l)}_i+\sum_j W^{(l+1)}_{ij}h^{(l+1)}_j\Big)
p(h(L−1),h(L))p(h^{(L-1)},h^{(L)})
the top-level RBM: an undirected associative memory
p(h(l)∣h(l+1))p(h^{(l)}\mid h^{(l+1)})
directed generative weights: each lower layer is a factorised sigmoid of the layer above
W(l+1)W^{(l+1)}
the weights of the RBM between layers ll and l+1l+1, reused top-down

Generating a sample is ancestral: run block Gibbs in the top RBM until it settles, then make one stochastic down-pass through the sigmoid layers. Recognition goes the other way: an up-pass with q(hj(l+1)=1∣h(l))=σ(bj+h(l)⊤W:,j)q(h^{(l+1)}_j=1\mid h^{(l)})=\sigma(b_j+h^{(l)\top}W_{:,j}), the same weights read bottom-up. The up-pass is only an approximation to the true posterior, because the layers above change the prior (more on that below).

Greedy layer-wise training

Pretraining a DBN with L hidden layers

  1. 1Train RBM 1 between vv and h(1)h^{(1)} on the data with CD-k.
  2. 2Freeze it. Push every training vector up once to get codes h(1)∼q(h(1)∣v)h^{(1)}\sim q(h^{(1)}\mid v) (samples or probabilities).
  3. 3Train RBM 2 between h(1)h^{(1)} and h(2)h^{(2)} on those codes, treating them as data.
  4. 4Repeat up the stack. Each new RBM only ever sees the layer directly below, so each training problem is as cheap and stable as a single RBM.
  5. 5After stacking, keep the top RBM undirected and turn every lower RBM into directed top-down weights (plus its bottom-up copy for recognition).
  6. 6Fine-tune. For a generative model, the up-down algorithm (a contrastive form of wake–sleep) adjusts the recognition and generative weights separately. For classification, which is how most DBNs were used, unroll the up-pass into a feed-forward sigmoid net, add a softmax layer and train with backprop.

Why adding a layer helps: the variational bound

For any distribution q(h∣v)q(h\mid v) over the first hidden layer, Jensen's inequality gives a lower bound on the log-likelihood of a training vector:

log⁡p(v)  ≥  ∑hq(h∣v)[log⁡p(h)+log⁡p(v∣h)]+H(q(⋅∣v))\log p(v)\;\ge\;\sum_{h}q(h\mid v)\Big[\log p(h)+\log p(v\mid h)\Big]+\mathcal{H}\big(q(\cdot\mid v)\big)
q(h∣v)q(h\mid v)
the up-pass of RBM 1, ∏jσ(bj+v⊤W:,j)\prod_j\sigma(b_j+v^\top W_{:,j})
p(v∣h)p(v\mid h)
RBM 1's top-down sigmoid layer (kept fixed)
p(h)p(h)
the prior over h(1)h^{(1)}: RBM 1's own marginal, or the RBM stacked on top
H(q)\mathcal{H}(q)
entropy of qq; the gap in the bound is KL(q(h∣v) ∥ p(h∣v))\mathrm{KL}\big(q(h\mid v)\,\Vert\,p(h\mid v)\big)

With RBM 1 alone, qq is its exact posterior, so the bound is tight. Now replace the prior p(h)p(h) by a second RBM over hh, initialised as RBM 1 turned upside down (W2=W1⊤W_2=W_1^\top, biases swapped). Its marginal over hh equals RBM 1's, so nothing changes yet. Only the term ∑hq(h∣v)log⁡p(h)\sum_h q(h\mid v)\log p(h) involves the new RBM, and averaged over the data it is the log-likelihood of the aggregated posterior 1N∑nq(h∣vn)\tfrac1N\sum_n q(h\mid v_n) under RBM 2. Training RBM 2 on samples of hh is maximum likelihood on exactly those samples, so it raises the bound.

This is the 2006 guarantee: done this way (with exact maximum-likelihood training), each extra layer cannot lower the bound. It does not promise to raise the likelihood itself, and CD only approximates maximum likelihood, so in practice it is a strong heuristic rather than a theorem about the networks people actually trained.

up-pass of image 10h2h1vv ↔ h1 undirected (an ordinary RBM)training images (click to inspect)Stage 1 · train RBM 1 (v ↔ h1) with CD-1epoch 0 on the 14 images η = 0.50reconstruction error 0.248 / pixelexact log p(v) = -6.249 nats (best −ln 14 = -2.639)Train for ~1000 epochs, then press stage 2.reconstruction error · RBM 10.30greedy epochs 1000mean log p(v) · exact bound-2.5-6.5greedy epochs 1000−ln 14RBM 1: 9 ↔ 6 units · RBM 2: 6 ↔ 9 · log p(v) summed exactly over all 2⁶ states of h1edges: w > 0 w < 0 · unit shade = probability (stages 1–2) or sampled state (stage 3)right chart: dashed vertical line = RBM 2 stacked on the frozen RBM 1
fig.A 9–6–9 DBN on 3×3 bars-and-stripes, small enough that log p(v) can be summed exactly. Stage 1: play RBM 1 (CD-1, η = 0.5) to about epoch 1000; its log-likelihood reaches −3.381. Stage 2: stack RBM 2 with W₂ = W₁ᵀ. The violet bound starts exactly at −3.381, then rises above it as RBM 2 models the codes (−2.983 by epoch 500 at η = 0.2), with the exact DBN likelihood (dashed) just above. Switch the init to random and the bound starts far lower, at −4.230. Stage 3 samples by top-level Gibbs plus a down-pass.

Worked example: the figure's numbers

ModelBoundExact log p(v)Valid samplesPatterns hit
RBM 1 (9–6), 1000 CD-1 epochs= exact−3.38180%13 / 14
+ RBM 2, W2=W1⊤W_2=W_1^\top, epoch 0−3.381−3.381——
+ RBM 2, W2=W1⊤W_2=W_1^\top, 500 epochs−2.983−2.97683%14 / 14
+ RBM 2, random init, epoch 0−4.230−4.027——
+ RBM 2, random init, 1500 epochs−3.469−3.42749%14 / 14
Mean over the 14 training images, in nats (log of a probability; the best possible is −ln 14 = −2.639, a uniform model over 512 images gives −6.238). Samples: 200 chains from noise, 50 Gibbs sweeps, then a down-pass; "valid" means a bars-and-stripes image

Reading the table

  1. 1Row 2 checks the argument. The flipped RBM 2 has the same marginal over h(1)h^{(1)} as RBM 1, so the bound, and the exact likelihood, equal RBM 1's to every printed digit.
  2. 2Row 3 is the payoff: +0.40+0.40 nats per image, i.e. the model puts e0.40≈1.5×e^{0.40}\approx1.5\times more probability on each training image. The gap between bound and exact value, the KL term, is only 0.0070.007 nats, so the up-pass is a good posterior here.
  3. 3Rows 4–5: a randomly initialised RBM 2 starts with a much worse prior and has not caught up after three times as long. Initialisation is part of the guarantee.
  4. 4Sample quality moves less than the likelihood (80% → 83% valid) because RBM 1 already modelled this tiny dataset well. Extra layers matter more when the data has structure at several scales.
python
1import torch, torch.nn as nn
2
3def train_rbm(X, nh, epochs=50, lr=0.05, bs=64):
4 nv = X.shape[1]
5 W = 0.01 * torch.randn(nv, nh); a = torch.zeros(nv); b = torch.zeros(nh)
6 for _ in range(epochs):
7 for v0 in X[torch.randperm(len(X))].split(bs):
8 ph0 = torch.sigmoid(v0 @ W + b)
9 v1 = torch.sigmoid(torch.bernoulli(ph0) @ W.T + a) # mean-field reconstruction
10 ph1 = torch.sigmoid(v1 @ W + b)
11 W += lr * (v0.T @ ph0 - v1.T @ ph1) / len(v0)
12 a += lr * (v0 - v1).mean(0); b += lr * (ph0 - ph1).mean(0)
13 return W, b
14
15def pretrain_dbn(X, sizes=(500, 500, 2000)): # Hinton et al. 2006 used 784-500-500-2000
16 layers, H = [], X # X: (N, 784) pixels in [0, 1]
17 for nh in sizes:
18 W, b = train_rbm(H, nh)
19 lin = nn.Linear(H.shape[1], nh)
20 with torch.no_grad():
21 lin.weight.copy_(W.T); lin.bias.copy_(b)
22 layers += [lin, nn.Sigmoid()]
23 H = torch.sigmoid(H @ W + b) # codes become the next RBM's data
24 return nn.Sequential(*layers)
25
26body = pretrain_dbn(X_train)
27model = nn.Sequential(body, nn.Linear(2000, 10)) # fine-tune everything with cross-entropy
28opt = torch.optim.SGD(model.parameters(), lr=0.1)
Greedy pretraining with CD-1, then unroll into a classifier and fine-tune with backprop (the common 2006–2011 recipe)

Use when

  • +Learns deep features from unlabelled data, one cheap, stable problem at a time.
  • +A real generative model: can sample, fill in missing inputs, and score data.
  • +Historically a reliable initialisation for deep sigmoid nets when labelled data was small.

Avoid when

  • −Inference is approximate, and the likelihood is intractable for real sizes.
  • −Many stages and hyperparameters (CD-k, epochs and learning rate per layer, fine-tuning).
  • −Outperformed by end-to-end training with modern init/normalisation, and by VAEs, GANs and diffusion for generation.

Check yourself

In greedy DBN training, what data does the second RBM learn from?

Check yourself

Why initialise the new top RBM with W₂ = W₁ᵀ in the bound argument?

Remember

  • ✓DBN: undirected RBM at the top, directed sigmoid layers below; generate by top-level Gibbs then one down-pass.
  • ✓Greedy training: each RBM is trained on the hidden codes of the RBM below, without labels.
  • ✓Variational bound: log⁡p(v)≥Eq[log⁡p(h)+log⁡p(v∣h)]+H(q)\log p(v)\ge\mathbb{E}_q[\log p(h)+\log p(v\mid h)]+\mathcal H(q); a new layer improves the prior term.
  • ✓With W2=W1⊤W_2=W_1^\top the bound starts tight; in the figure it then rises from −3.381 to −2.983 nats.
  • ✓Fine-tune with up-down (generative) or unroll and backprop (discriminative).
  • ✓Sparked the 2006 revival; superseded by good init, ReLU, normalisation and plain end-to-end training.
← back to Deep Learning roadmap