Definition
Deep belief network is a generative model with several layers of binary hidden units. The top two layers form an undirected RBM; every layer below is a directed sigmoid belief net that generates the layer beneath it. It is trained greedily: fit an RBM to the data, then fit another RBM to the first one's hidden codes, and so on.
In 2006, deep networks trained from random weights with backprop usually ended up worse than shallow ones: sigmoid units saturated, gradients vanished, and nobody knew how to initialise them. Hinton, Osindero and Teh showed that you could build a deep model one layer at a time without labels, each layer an RBM trained with contrastive divergence, and then fine-tune it. Their DBN reached 1.25% error on permutation-invariant MNIST, better than the SVMs and backprop-trained nets of the time.
Together with Hinton & Salakhutdinov's deep autoencoders (Science, 2006) and Bengio et al.'s greedy layer-wise study (2007), this kicked off the deep learning revival. DBN-pretrained networks gave the first big deep-learning wins in speech recognition (2009–2012).
The model
Write for the data and for the hidden layers. The joint distribution is an RBM at the top times a chain of sigmoid layers going down:
- the top-level RBM: an undirected associative memory
- directed generative weights: each lower layer is a factorised sigmoid of the layer above
- the weights of the RBM between layers and , reused top-down
Generating a sample is ancestral: run block Gibbs in the top RBM until it settles, then make one stochastic down-pass through the sigmoid layers. Recognition goes the other way: an up-pass with , the same weights read bottom-up. The up-pass is only an approximation to the true posterior, because the layers above change the prior (more on that below).
Greedy layer-wise training
Pretraining a DBN with L hidden layers
- 1Train RBM 1 between and on the data with CD-k.
- 2Freeze it. Push every training vector up once to get codes (samples or probabilities).
- 3Train RBM 2 between and on those codes, treating them as data.
- 4Repeat up the stack. Each new RBM only ever sees the layer directly below, so each training problem is as cheap and stable as a single RBM.
- 5After stacking, keep the top RBM undirected and turn every lower RBM into directed top-down weights (plus its bottom-up copy for recognition).
- 6Fine-tune. For a generative model, the up-down algorithm (a contrastive form of wake–sleep) adjusts the recognition and generative weights separately. For classification, which is how most DBNs were used, unroll the up-pass into a feed-forward sigmoid net, add a softmax layer and train with backprop.
Why adding a layer helps: the variational bound
For any distribution over the first hidden layer, Jensen's inequality gives a lower bound on the log-likelihood of a training vector:
- the up-pass of RBM 1,
- RBM 1's top-down sigmoid layer (kept fixed)
- the prior over : RBM 1's own marginal, or the RBM stacked on top
- entropy of ; the gap in the bound is
With RBM 1 alone, is its exact posterior, so the bound is tight. Now replace the prior by a second RBM over , initialised as RBM 1 turned upside down (, biases swapped). Its marginal over equals RBM 1's, so nothing changes yet. Only the term involves the new RBM, and averaged over the data it is the log-likelihood of the aggregated posterior under RBM 2. Training RBM 2 on samples of is maximum likelihood on exactly those samples, so it raises the bound.
This is the 2006 guarantee: done this way (with exact maximum-likelihood training), each extra layer cannot lower the bound. It does not promise to raise the likelihood itself, and CD only approximates maximum likelihood, so in practice it is a strong heuristic rather than a theorem about the networks people actually trained.
Worked example: the figure's numbers
| Model | Bound | Exact log p(v) | Valid samples | Patterns hit |
|---|---|---|---|---|
| RBM 1 (9–6), 1000 CD-1 epochs | = exact | −3.381 | 80% | 13 / 14 |
| + RBM 2, , epoch 0 | −3.381 | −3.381 | — | — |
| + RBM 2, , 500 epochs | −2.983 | −2.976 | 83% | 14 / 14 |
| + RBM 2, random init, epoch 0 | −4.230 | −4.027 | — | — |
| + RBM 2, random init, 1500 epochs | −3.469 | −3.427 | 49% | 14 / 14 |
Reading the table
- 1Row 2 checks the argument. The flipped RBM 2 has the same marginal over as RBM 1, so the bound, and the exact likelihood, equal RBM 1's to every printed digit.
- 2Row 3 is the payoff: nats per image, i.e. the model puts more probability on each training image. The gap between bound and exact value, the KL term, is only nats, so the up-pass is a good posterior here.
- 3Rows 4–5: a randomly initialised RBM 2 starts with a much worse prior and has not caught up after three times as long. Initialisation is part of the guarantee.
- 4Sample quality moves less than the likelihood (80% → 83% valid) because RBM 1 already modelled this tiny dataset well. Extra layers matter more when the data has structure at several scales.
1import torch, torch.nn as nn23def train_rbm(X, nh, epochs=50, lr=0.05, bs=64):4 nv = X.shape[1]5 W = 0.01 * torch.randn(nv, nh); a = torch.zeros(nv); b = torch.zeros(nh)6 for _ in range(epochs):7 for v0 in X[torch.randperm(len(X))].split(bs):8 ph0 = torch.sigmoid(v0 @ W + b)9 v1 = torch.sigmoid(torch.bernoulli(ph0) @ W.T + a) # mean-field reconstruction10 ph1 = torch.sigmoid(v1 @ W + b)11 W += lr * (v0.T @ ph0 - v1.T @ ph1) / len(v0)12 a += lr * (v0 - v1).mean(0); b += lr * (ph0 - ph1).mean(0)13 return W, b1415def pretrain_dbn(X, sizes=(500, 500, 2000)): # Hinton et al. 2006 used 784-500-500-200016 layers, H = [], X # X: (N, 784) pixels in [0, 1]17 for nh in sizes:18 W, b = train_rbm(H, nh)19 lin = nn.Linear(H.shape[1], nh)20 with torch.no_grad():21 lin.weight.copy_(W.T); lin.bias.copy_(b)22 layers += [lin, nn.Sigmoid()]23 H = torch.sigmoid(H @ W + b) # codes become the next RBM's data24 return nn.Sequential(*layers)2526body = pretrain_dbn(X_train)27model = nn.Sequential(body, nn.Linear(2000, 10)) # fine-tune everything with cross-entropy28opt = torch.optim.SGD(model.parameters(), lr=0.1)Use when
- +Learns deep features from unlabelled data, one cheap, stable problem at a time.
- +A real generative model: can sample, fill in missing inputs, and score data.
- +Historically a reliable initialisation for deep sigmoid nets when labelled data was small.
Avoid when
- −Inference is approximate, and the likelihood is intractable for real sizes.
- −Many stages and hyperparameters (CD-k, epochs and learning rate per layer, fine-tuning).
- −Outperformed by end-to-end training with modern init/normalisation, and by VAEs, GANs and diffusion for generation.
Check yourself
In greedy DBN training, what data does the second RBM learn from?
Check yourself
Why initialise the new top RBM with W₂ = W₁ᵀ in the bound argument?
Remember
- ✓DBN: undirected RBM at the top, directed sigmoid layers below; generate by top-level Gibbs then one down-pass.
- ✓Greedy training: each RBM is trained on the hidden codes of the RBM below, without labels.
- ✓Variational bound: ; a new layer improves the prior term.
- ✓With the bound starts tight; in the figure it then rises from −3.381 to −2.983 nats.
- ✓Fine-tune with up-down (generative) or unroll and backprop (discriminative).
- ✓Sparked the 2006 revival; superseded by good init, ReLU, normalisation and plain end-to-end training.