# Deep belief network (DBN)

> A stack of RBMs trained greedily one layer at a time, each modelling the codes of the one below; the 2006 recipe that made deep nets trainable.

Source: https://www.mlfoundry.in/learn/deep-belief-network-dbn  
Section: Deep Learning › Neural Network Algorithms  
Level: advanced · 11 min read · updated 2026-10-11

**Deep belief network:** is a generative model with several layers of binary hidden units. The **top two layers** form an undirected RBM; every layer below is a **directed sigmoid belief net** that generates the layer beneath it. It is trained **greedily**: fit an RBM to the data, then fit another RBM to the first one's hidden codes, and so on.

In 2006, deep networks trained from random weights with backprop usually ended up worse than shallow ones: sigmoid units saturated, gradients vanished, and nobody knew how to initialise them. Hinton, Osindero and Teh showed that you could build a deep model **one layer at a time** without labels, each layer an RBM trained with contrastive divergence, and then fine-tune it. Their DBN reached 1.25% error on permutation-invariant MNIST, better than the SVMs and backprop-trained nets of the time.

Together with Hinton & Salakhutdinov's deep autoencoders (*Science*, 2006) and Bengio et al.'s greedy layer-wise study (2007), this kicked off the deep learning revival. DBN-pretrained networks gave the first big deep-learning wins in speech recognition (2009–2012).

## The model

Write $h^{(0)}=v$ for the data and $h^{(1)},\dots,h^{(L)}$ for the hidden layers. The joint distribution is an RBM at the top times a chain of sigmoid layers going down:

$$
p\big(v,h^{(1)},\dots,h^{(L)}\big)=p\big(h^{(L-1)},h^{(L)}\big)\prod_{l=0}^{L-2}p\big(h^{(l)}\mid h^{(l+1)}\big),\qquad p\big(h^{(l)}_i=1\mid h^{(l+1)}\big)=\sigma\Big(a^{(l)}_i+\sum_j W^{(l+1)}_{ij}h^{(l+1)}_j\Big)
$$

- $p(h^{(L-1)},h^{(L)})$: the top-level RBM: an undirected associative memory
- $p(h^{(l)}\mid h^{(l+1)})$: directed generative weights: each lower layer is a factorised sigmoid of the layer above
- $W^{(l+1)}$: the weights of the RBM between layers $l$ and $l+1$, reused top-down

Generating a sample is ancestral: run block Gibbs in the top RBM until it settles, then make one stochastic **down-pass** through the sigmoid layers. Recognition goes the other way: an **up-pass** with $q(h^{(l+1)}_j=1\mid h^{(l)})=\sigma(b_j+h^{(l)\top}W_{:,j})$, the same weights read bottom-up. The up-pass is only an approximation to the true posterior, because the layers above change the prior (more on that below).

> **Not a deep Boltzmann machine:** A **deep Boltzmann machine** (Salakhutdinov & Hinton 2009) makes *every* layer undirected, so inference combines top-down and bottom-up signals. A DBN is a hybrid: only the top link is undirected. That asymmetry is exactly what makes greedy training easy to justify.

## Greedy layer-wise training

**Pretraining a DBN with L hidden layers**

1. Train RBM 1 between $v$ and $h^{(1)}$ on the data with CD-k.
2. Freeze it. Push every training vector up once to get codes $h^{(1)}\sim q(h^{(1)}\mid v)$ (samples or probabilities).
3. Train RBM 2 between $h^{(1)}$ and $h^{(2)}$ on those codes, treating them as data.
4. Repeat up the stack. Each new RBM only ever sees the layer directly below, so each training problem is as cheap and stable as a single RBM.
5. After stacking, keep the top RBM undirected and turn every lower RBM into directed top-down weights (plus its bottom-up copy for recognition).
6. **Fine-tune.** For a generative model, the *up-down* algorithm (a contrastive form of wake–sleep) adjusts the recognition and generative weights separately. For classification, which is how most DBNs were used, unroll the up-pass into a feed-forward sigmoid net, add a softmax layer and train with backprop.

## Why adding a layer helps: the variational bound

For any distribution $q(h\mid v)$ over the first hidden layer, Jensen's inequality gives a lower bound on the log-likelihood of a training vector:

$$
\log p(v)\;\ge\;\sum_{h}q(h\mid v)\Big[\log p(h)+\log p(v\mid h)\Big]+\mathcal{H}\big(q(\cdot\mid v)\big)
$$

- $q(h\mid v)$: the up-pass of RBM 1, $\prod_j\sigma(b_j+v^\top W_{:,j})$
- $p(v\mid h)$: RBM 1's top-down sigmoid layer (kept fixed)
- $p(h)$: the prior over $h^{(1)}$: RBM 1's own marginal, or the RBM stacked on top
- $\mathcal{H}(q)$: entropy of $q$; the gap in the bound is $\mathrm{KL}\big(q(h\mid v)\,\Vert\,p(h\mid v)\big)$

With RBM 1 alone, $q$ is its exact posterior, so the bound is tight. Now replace the prior $p(h)$ by a second RBM over $h$, **initialised as RBM 1 turned upside down** ($W_2=W_1^\top$, biases swapped). Its marginal over $h$ equals RBM 1's, so nothing changes yet. Only the term $\sum_h q(h\mid v)\log p(h)$ involves the new RBM, and averaged over the data it is the log-likelihood of the **aggregated posterior** $\tfrac1N\sum_n q(h\mid v_n)$ under RBM 2. Training RBM 2 on samples of $h$ is maximum likelihood on exactly those samples, so it raises the bound.

This is the 2006 guarantee: done this way (with exact maximum-likelihood training), each extra layer cannot lower the bound. It does not promise to raise the likelihood itself, and CD only approximates maximum likelihood, so in practice it is a strong heuristic rather than a theorem about the networks people actually trained.

> Figure: A 9–6–9 DBN on 3×3 bars-and-stripes, small enough that log p(v) can be summed exactly. Stage 1: play RBM 1 (CD-1, η = 0.5) to about epoch 1000; its log-likelihood reaches −3.381. Stage 2: stack RBM 2 with W₂ = W₁ᵀ. The violet bound starts exactly at −3.381, then rises above it as RBM 2 models the codes (−2.983 by epoch 500 at η = 0.2), with the exact DBN likelihood (dashed) just above. Switch the init to random and the bound starts far lower, at −4.230. Stage 3 samples by top-level Gibbs plus a down-pass.

## Worked example: the figure's numbers

Mean over the 14 training images, in nats (log of a probability; the best possible is −ln 14 = −2.639, a uniform model over 512 images gives −6.238). Samples: 200 chains from noise, 50 Gibbs sweeps, then a down-pass; "valid" means a bars-and-stripes image

| Model | Bound | Exact log p(v) | Valid samples | Patterns hit |
| --- | --- | --- | --- | --- |
| RBM 1 (9–6), 1000 CD-1 epochs | = exact | −3.381 | 80% | 13 / 14 |
| + RBM 2, $W_2=W_1^\top$, epoch 0 | −3.381 | −3.381 | — | — |
| + RBM 2, $W_2=W_1^\top$, 500 epochs | −2.983 | −2.976 | 83% | 14 / 14 |
| + RBM 2, random init, epoch 0 | −4.230 | −4.027 | — | — |
| + RBM 2, random init, 1500 epochs | −3.469 | −3.427 | 49% | 14 / 14 |

**Reading the table**

1. **Row 2 checks the argument.** The flipped RBM 2 has the same marginal over $h^{(1)}$ as RBM 1, so the bound, and the exact likelihood, equal RBM 1's to every printed digit.
2. **Row 3** is the payoff: $+0.40$ nats per image, i.e. the model puts $e^{0.40}\approx1.5\times$ more probability on each training image. The gap between bound and exact value, the KL term, is only $0.007$ nats, so the up-pass is a good posterior here.
3. **Rows 4–5**: a randomly initialised RBM 2 starts with a much worse prior and has not caught up after three times as long. Initialisation is part of the guarantee.
4. Sample quality moves less than the likelihood (80% → 83% valid) because RBM 1 already modelled this tiny dataset well. Extra layers matter more when the data has structure at several scales.

```python
import torch, torch.nn as nn

def train_rbm(X, nh, epochs=50, lr=0.05, bs=64):
    nv = X.shape[1]
    W = 0.01 * torch.randn(nv, nh); a = torch.zeros(nv); b = torch.zeros(nh)
    for _ in range(epochs):
        for v0 in X[torch.randperm(len(X))].split(bs):
            ph0 = torch.sigmoid(v0 @ W + b)
            v1 = torch.sigmoid(torch.bernoulli(ph0) @ W.T + a)    # mean-field reconstruction
            ph1 = torch.sigmoid(v1 @ W + b)
            W += lr * (v0.T @ ph0 - v1.T @ ph1) / len(v0)
            a += lr * (v0 - v1).mean(0); b += lr * (ph0 - ph1).mean(0)
    return W, b

def pretrain_dbn(X, sizes=(500, 500, 2000)):          # Hinton et al. 2006 used 784-500-500-2000
    layers, H = [], X                                  # X: (N, 784) pixels in [0, 1]
    for nh in sizes:
        W, b = train_rbm(H, nh)
        lin = nn.Linear(H.shape[1], nh)
        with torch.no_grad():
            lin.weight.copy_(W.T); lin.bias.copy_(b)
        layers += [lin, nn.Sigmoid()]
        H = torch.sigmoid(H @ W + b)                   # codes become the next RBM's data
    return nn.Sequential(*layers)

body = pretrain_dbn(X_train)
model = nn.Sequential(body, nn.Linear(2000, 10))       # fine-tune everything with cross-entropy
opt = torch.optim.SGD(model.parameters(), lr=0.1)
```

Greedy pretraining with CD-1, then unroll into a classifier and fine-tune with backprop (the common 2006–2011 recipe)

> **The up-pass is not the posterior:** Once RBM 2 replaces the prior, the true posterior $p(h^{(1)}\mid v)$ is no longer factorial: directed layers create **explaining away** between hidden units. The DBN keeps using RBM 1's factorial $q$ anyway. That is what the KL gap in the bound measures, and why the up-down fine-tuning stage adjusts the recognition weights separately from the generative ones.

> **Why pretraining stopped mattering:** Pretraining put the weights in a sensible region before backprop, which mattered when sigmoid nets started badly and data was scarce. Better initialisation (Glorot & Bengio 2010; He et al. 2015), ReLUs (2010–2011), dropout (2012), batch normalisation (2015), residual connections and far larger labelled datasets fixed the same problems directly. AlexNet (2012) used no pretraining at all. Today the idea survives as **self-supervised pretraining**, not as stacked RBMs.

**Strengths**

- Learns deep features from unlabelled data, one cheap, stable problem at a time.
- A real generative model: can sample, fill in missing inputs, and score data.
- Historically a reliable initialisation for deep sigmoid nets when labelled data was small.

**Limitations**

- Inference is approximate, and the likelihood is intractable for real sizes.
- Many stages and hyperparameters (CD-k, epochs and learning rate per layer, fine-tuning).
- Outperformed by end-to-end training with modern init/normalisation, and by VAEs, GANs and diffusion for generation.

**Question:** In greedy DBN training, what data does the second RBM learn from?

- A. The original images again
- B. The hidden codes of the first RBM, obtained by an up-pass of each training image
- C. The reconstructions produced by the first RBM
- D. Samples generated from the top layer

**Answer:** B. The hidden codes of the first RBM, obtained by an up-pass of each training image — RBM 2 models $q(h^{(1)}\mid v)$ over the training set (the aggregated posterior). Making it a better model of those codes is exactly what raises the variational bound on $\log p(v)$.

**Question:** Why initialise the new top RBM with W₂ = W₁ᵀ in the bound argument?

- A. It makes CD exact
- B. So that its marginal over h⁽¹⁾ equals RBM 1's, so the bound starts at RBM 1's log-likelihood and can only improve from there
- C. It makes the posterior factorial
- D. It halves the number of parameters

**Answer:** B. So that its marginal over h⁽¹⁾ equals RBM 1's, so the bound starts at RBM 1's log-likelihood and can only improve from there — Turned upside down, RBM 1 defines the same distribution over $h^{(1)}$, so at initialisation the DBN equals RBM 1 and the bound is tight. In the figure it starts at $-3.381$ either way you compute it; a random init starts at $-4.230$.

**Key points**

- DBN: undirected RBM at the top, directed sigmoid layers below; generate by top-level Gibbs then one down-pass.
- Greedy training: each RBM is trained on the hidden codes of the RBM below, without labels.
- Variational bound: $\log p(v)\ge\mathbb{E}_q[\log p(h)+\log p(v\mid h)]+\mathcal H(q)$; a new layer improves the prior term.
- With $W_2=W_1^\top$ the bound starts tight; in the figure it then rises from −3.381 to −2.983 nats.
- Fine-tune with up-down (generative) or unroll and backprop (discriminative).
- Sparked the 2006 revival; superseded by good init, ReLU, normalisation and plain end-to-end training.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
