Definition
Restricted Boltzmann machine is an undirected generative model with a layer of visible units and a layer of hidden units , connected by a weight matrix , with no connections inside a layer. It assigns every joint state an energy and a probability . Training makes the training data low-energy, and so probable.
A full Boltzmann machine lets every unit talk to every other unit. It is expressive, but even computing the probability that one hidden unit is on requires a long Markov chain. Smolensky's 1986 "harmonium" removed the visible–visible and hidden–hidden links, leaving a bipartite graph. That one restriction makes the hidden units independent given the visibles, and vice versa, so a whole layer can be inferred or sampled in a single parallel step.
Training was still slow until Hinton's contrastive divergence (2002) replaced the expensive model expectation with a one-step approximation. RBMs became the building block of the 2006 deep belief networks that restarted deep learning, and a strong collaborative-filtering model in the Netflix Prize.
Energy and probability
- binary visible (data) and hidden (feature) units
- visible and hidden biases
- weight between visible and hidden ; positive means "these two like to be on together"
- partition function: a sum over all joint states, intractable for realistic sizes
The energy has no or terms, so it is a sum of terms that each involve one hidden unit. Fix , and factorises over ; each factor is a logistic unit:
- logistic sigmoid
- the same pre-activation a sigmoid neuron would compute: the up-pass is a feed-forward layer
Summing the hidden units out gives the marginal of a visible vector through its free energy, which costs only :
- free energy, ; lower means a more probable
- softplus: each hidden unit contributes independently
The learning rule
Differentiate the log-likelihood of a training vector. The term pulls the data's energy down; the term pushes everyone else's up. The result is a difference of two correlations:
- averaged over training vectors: exact and cheap
- the same correlation under the model's own distribution: needs samples from
- same pattern with and
The model term needs samples. The bipartite structure gives block Gibbs sampling: sample all of from , then all of from , and repeat. Run long enough, the chain samples , but "long enough" can mean thousands of sweeps per update.
Contrastive divergence, CD-k (one update)
- 1Positive phase. Clamp a data vector ; compute and sample from it.
- 2Gibbs steps. For : sample , then . CD-1 stops after one step, giving the "reconstruction" .
- 3Update. , , , averaged over the mini-batch. Using probabilities instead of samples where possible reduces noise.
- 4Persistent CD (PCD, Tieleman 2008). Instead of restarting the negative chain at the data, keep a set of "fantasy" chains alive across updates and advance each by steps. With a small learning rate the chains stay close to the model distribution, and the gradient is much less biased than CD's.
Starting the chain at the data is what makes CD cheap and biased: it measures how the model wants to move away from the data in one step, not the full model expectation. It is not the gradient of any objective, but it works well in practice for learning features, and trades bias for compute.
Worked example: one CD-1 update
Take , , , , , training vector and . With only visible states, can be summed exactly, so we can check that the update helps.
| Quantity | Computation | Value |
|---|---|---|
| (2, −1) | ||
| (0.8808, 0.2689) | ||
| −2.4402 | ||
| 0.2083 | ||
| sample | one draw from the line above | (1, 0) |
| (0.8808, 0.7311, 0.2689) | ||
| sample | one draw: the reconstruction drops pixel 2 | (1, 0, 0) |
| (0.7311, 0.1192) | ||
| , | (0, 0.1, 0), (0.0150, 0.0150) |
The reconstruction "forgot" visible unit 2, so the update strengthens exactly that unit's links to the hidden units and raises its bias. Row 3 is zero because is off in both phases. After the step, drops to and rises from to .
1import torch23def bars_and_stripes(n=3):4 imgs = set()5 for m in range(2 ** n):6 line = torch.tensor([(m >> k) & 1 for k in range(n)], dtype=torch.float)7 imgs.add(tuple(line.repeat_interleave(n).tolist())) # rows on8 imgs.add(tuple(line.repeat(n).tolist())) # columns on9 return torch.tensor(sorted(imgs)) # 14 images for n = 31011V = bars_and_stripes() # (14, 9)12nv, nh, lr = 9, 6, 0.513W = 0.1 * torch.randn(nv, nh); a = torch.zeros(nv); b = torch.zeros(nh)1415for epoch in range(2000):16 ph0 = torch.sigmoid(V @ W + b) # positive phase, p(h | data)17 h0 = torch.bernoulli(ph0)18 v1 = torch.bernoulli(torch.sigmoid(h0 @ W.T + a)) # one Gibbs step19 ph1 = torch.sigmoid(v1 @ W + b) # negative phase20 W += lr * (V.T @ ph0 - v1.T @ ph1) / len(V)21 a += lr * (V - v1).mean(0)22 b += lr * (ph0 - ph1).mean(0)2324def free_energy(v): # F(v) = −aᵀv − Σ softplus(b + vW)25 return -(v @ a) - torch.nn.functional.softplus(v @ W + b).sum(-1)Variants and uses
Gaussian–Bernoulli RBMs handle real-valued inputs such as pixel intensities or audio features by replacing the visible term with and scaling the interaction by . Then , while stays logistic. They need much smaller learning rates and usually standardised inputs.
Where it's used
Feature learning
is a nonlinear feature map learned without labels; stacking these is the deep belief network.
Collaborative filtering
Salakhutdinov, Mnih & Hinton (2007) gave each user an RBM over the movies they rated, with softmax visible units for the 1–5 stars and shared weights. It slightly beat SVD on Netflix data, and blending the two helped further; RBMs were part of the prize-winning ensembles.
Pretraining
Initialising deep nets layer by layer (2006–2011), before ReLUs and better initialisation made it unnecessary.
Physics and topic models
Neural-network quantum states and the replicated-softmax topic model use the same energy.
Use when
- +Exact, parallel inference: is one matrix multiply.
- +Unnormalised density via the free energy, so it can score inputs and fill in missing values.
- +A principled generative model that also yields features.
Avoid when
- − is intractable, so likelihood cannot be monitored and the gradient is approximate.
- −CD is biased; Gibbs chains mix poorly between distant modes.
- −Largely superseded by VAEs, GANs, diffusion and autoregressive models for generation.
Check yourself
Why can all hidden units of an RBM be sampled in parallel given the visible units?
Check yourself
In CD-1, what replaces the model expectation ⟨vᵢhⱼ⟩_model?
Remember
- ✓Energy ; bipartite, so and symmetrically for .
- ✓Free energy gives up to the intractable .
- ✓Gradient: ; block Gibbs supplies the model samples.
- ✓CD-k restarts a k-step chain at the data (cheap, biased); PCD keeps persistent chains (less biased, needs small steps).
- ✓Low reconstruction error does not imply high likelihood.
- ✓Gaussian–Bernoulli RBMs model real-valued inputs; RBMs powered DBN pretraining and Netflix-prize collaborative filtering.