Restricted Boltzmann machine (RBM)

A two-layer energy-based model with no within-layer links, so each layer is easy to sample given the other; trained by contrastive divergence.

11 min read832 wordsadvanced0 viewsupdated 2026-10-11

Definition

Restricted Boltzmann machine is an undirected generative model with a layer of visible units vv and a layer of hidden units hh, connected by a weight matrix WW, with no connections inside a layer. It assigns every joint state an energy E(v,h)E(v,h) and a probability p(v,h)∝e−E(v,h)p(v,h)\propto e^{-E(v,h)}. Training makes the training data low-energy, and so probable.

A full Boltzmann machine lets every unit talk to every other unit. It is expressive, but even computing the probability that one hidden unit is on requires a long Markov chain. Smolensky's 1986 "harmonium" removed the visible–visible and hidden–hidden links, leaving a bipartite graph. That one restriction makes the hidden units independent given the visibles, and vice versa, so a whole layer can be inferred or sampled in a single parallel step.

Training was still slow until Hinton's contrastive divergence (2002) replaced the expensive model expectation with a one-step approximation. RBMs became the building block of the 2006 deep belief networks that restarted deep learning, and a strong collaborative-filtering model in the Netflix Prize.

Energy and probability

E(v,h)=−a⊤v−b⊤h−v⊤Wh,p(v,h)=e−E(v,h)Z,Z=∑v,he−E(v,h)E(v,h)=-a^\top v-b^\top h-v^\top W h,\qquad p(v,h)=\frac{e^{-E(v,h)}}{Z},\qquad Z=\sum_{v,h}e^{-E(v,h)}
v∈{0,1}nv, h∈{0,1}nhv\in\{0,1\}^{n_v},\ h\in\{0,1\}^{n_h}
binary visible (data) and hidden (feature) units
a, ba,\ b
visible and hidden biases
WijW_{ij}
weight between visible ii and hidden jj; positive means "these two like to be on together"
ZZ
partition function: a sum over all 2nv+nh2^{n_v+n_h} joint states, intractable for realistic sizes

The energy has no hjhkh_jh_k or vivkv_iv_k terms, so it is a sum of terms that each involve one hidden unit. Fix vv, and p(h∣v)p(h\mid v) factorises over jj; each factor is a logistic unit:

p(hj=1∣v)=σ(bj+∑iviWij),p(vi=1∣h)=σ(ai+∑jWijhj)p(h_j=1\mid v)=\sigma\Big(b_j+\sum_i v_iW_{ij}\Big),\qquad p(v_i=1\mid h)=\sigma\Big(a_i+\sum_j W_{ij}h_j\Big)
σ\sigma
logistic sigmoid 1/(1+e−x)1/(1+e^{-x})
bj+v⊤W:,jb_j+v^\top W_{:,j}
the same pre-activation a sigmoid neuron would compute: the up-pass is a feed-forward layer

Summing the hidden units out gives the marginal of a visible vector through its free energy, which costs only O(nvnh)O(n_vn_h):

p(v)=e−F(v)Z,F(v)=−a⊤v−∑j=1nhlog⁡ ⁣(1+e bj+v⊤W:,j)p(v)=\frac{e^{-F(v)}}{Z},\qquad F(v)=-a^\top v-\sum_{j=1}^{n_h}\log\!\Big(1+e^{\,b_j+v^\top W_{:,j}}\Big)
F(v)F(v)
free energy, −log⁡∑he−E(v,h)-\log\sum_h e^{-E(v,h)}; lower FF means a more probable vv
log⁡(1+ex)\log(1+e^x)
softplus: each hidden unit contributes independently

The learning rule

Differentiate the log-likelihood of a training vector. The −F(v)-F(v) term pulls the data's energy down; the −log⁡Z-\log Z term pushes everyone else's up. The result is a difference of two correlations:

∂log⁡p(v)∂Wij=⟨vihj⟩data⏟positive phase−⟨vihj⟩model⏟negative phase\frac{\partial\log p(v)}{\partial W_{ij}}=\underbrace{\langle v_ih_j\rangle_{\text{data}}}_{\text{positive phase}}-\underbrace{\langle v_ih_j\rangle_{\text{model}}}_{\text{negative phase}}
⟨vihj⟩data\langle v_ih_j\rangle_{\text{data}}
vi p(hj=1∣v)v_i\,p(h_j{=}1\mid v) averaged over training vectors: exact and cheap
⟨vihj⟩model\langle v_ih_j\rangle_{\text{model}}
the same correlation under the model's own distribution: needs samples from p(v,h)p(v,h)
ai, bja_i,\ b_j
same pattern with ⟨vi⟩\langle v_i\rangle and ⟨hj⟩\langle h_j\rangle

The model term needs samples. The bipartite structure gives block Gibbs sampling: sample all of hh from p(h∣v)p(h\mid v), then all of vv from p(v∣h)p(v\mid h), and repeat. Run long enough, the chain samples p(v,h)p(v,h), but "long enough" can mean thousands of sweeps per update.

Contrastive divergence, CD-k (one update)

  1. 1Positive phase. Clamp a data vector v(0)v^{(0)}; compute p(h∣v(0))p(h\mid v^{(0)}) and sample h(0)h^{(0)} from it.
  2. 2Gibbs steps. For t=1..kt=1..k: sample v(t)∼p(v∣h(t−1))v^{(t)}\sim p(v\mid h^{(t-1)}), then h(t)∼p(h∣v(t))h^{(t)}\sim p(h\mid v^{(t)}). CD-1 stops after one step, giving the "reconstruction" v(1)v^{(1)}.
  3. 3Update. ΔW=η(v(0)p(h∣v(0))⊤−v(k)p(h∣v(k))⊤)\Delta W=\eta\big(v^{(0)}p(h\mid v^{(0)})^\top-v^{(k)}p(h\mid v^{(k)})^\top\big), Δa=η(v(0)−v(k))\Delta a=\eta(v^{(0)}-v^{(k)}), Δb=η(p(h∣v(0))−p(h∣v(k)))\Delta b=\eta\big(p(h\mid v^{(0)})-p(h\mid v^{(k)})\big), averaged over the mini-batch. Using probabilities instead of samples where possible reduces noise.
  4. 4Persistent CD (PCD, Tieleman 2008). Instead of restarting the negative chain at the data, keep a set of "fantasy" chains alive across updates and advance each by kk steps. With a small learning rate the chains stay close to the model distribution, and the gradient is much less biased than CD's.

Starting the chain at the data is what makes CD cheap and biased: it measures how the model wants to move away from the data in one step, not the full model expectation. It is not the gradient of any objective, but it works well in practice for learning features, and kk trades bias for compute.

training data: all 14 bars-and-stripes images (click one to feed it in)input v (editable)training image 10→p(hⱼ=1 | v)h1h2h3h4h5h6→p(v | h) reconerror 0.261/pixelweights into each hidden unith1 b 0.0h2 b 0.0h3 b 0.0h4 b 0.0h5 b 0.0h6 b 0.0■ w > 0 ■ w < 0 · max |w| 1.0reconstruction error / pixel 0.2480epoch 2000.30exact mean log p(v), nats -6.249best: −ln 140epoch 200-2-8model samples (30 Gibbs sweeps from noise): 2% valid, 2/14 patterns seen in 120
epoch 0
negativeshidden
fig.A 9-visible, 6-hidden RBM trained live on all 14 images of 3×3 bars-and-stripes (full batch, seeded sampling). Press play: from reset with CD-1 at η = 0.5 the reconstruction error falls from 0.248 to 0.003 by epoch 500, while the exact log-likelihood climbs from −6.25 (≈ uniform over 512 images) to −3.38 by epoch 1000, still well short of the best possible −ln 14 = −2.64. Click cells of the input to corrupt it and watch the reconstruction repair it. Reset with PCD at η = 0.5: reconstruction error still goes to 0, yet the likelihood collapses (−17.7 at epoch 4000), because the persistent chains cannot keep up with such large weight changes.

Worked example: one CD-1 update

Take nv=3n_v=3, nh=2n_h=2, a=0a=0, b=(−1,−1)b=(-1,-1), W=[2−111−12]W=\begin{bmatrix}2&-1\\1&1\\-1&2\end{bmatrix}, training vector v(0)=(1,1,0)v^{(0)}=(1,1,0) and η=0.1\eta=0.1. With only 232^3 visible states, Z=55.09Z=55.09 can be summed exactly, so we can check that the update helps.

QuantityComputationValue
b+W⊤v(0)b+W^\top v^{(0)}(−1+2+1,  −1−1+1)(-1+2+1,\;-1-1+1)(2, −1)
p(h∣v(0))p(h\mid v^{(0)})(σ(2), σ(−1))(\sigma(2),\,\sigma(-1))(0.8808, 0.2689)
F(v(0))F(v^{(0)})−log⁡(1+e2)−log⁡(1+e−1)-\log(1+e^{2})-\log(1+e^{-1})−2.4402
p(v(0))p(v^{(0)})e2.4402/55.09e^{2.4402}/55.090.2083
sample h(0)h^{(0)}one draw from the line above(1, 0)
p(v∣h(0))p(v\mid h^{(0)})σ(a+W:,1)=σ(2,1,−1)\sigma(a+W_{:,1})=\sigma(2,1,-1)(0.8808, 0.7311, 0.2689)
sample v(1)v^{(1)}one draw: the reconstruction drops pixel 2(1, 0, 0)
p(h∣v(1))p(h\mid v^{(1)})(σ(1), σ(−2))(\sigma(1),\,\sigma(-2))(0.7311, 0.1192)
ΔW\Delta W0.1 (v(0)p(h∣v(0))⊤−v(1)p(h∣v(1))⊤)0.1\,\big(v^{(0)}p(h\mid v^{(0)})^\top-v^{(1)}p(h\mid v^{(1)})^\top\big)[0.01500.01500.08810.026900]\begin{bmatrix}0.0150&0.0150\\0.0881&0.0269\\0&0\end{bmatrix}
Δa, Δb\Delta a,\ \Delta b0.1(v(0)−v(1))0.1(v^{(0)}-v^{(1)}), 0.1(p0−p1)0.1(p_0-p_1)(0, 0.1, 0), (0.0150, 0.0150)
Every number computed exactly (rounded to 4 decimals); the two sampled states are given, not drawn

The reconstruction "forgot" visible unit 2, so the update strengthens exactly that unit's links to the hidden units and raises its bias. Row 3 is zero because v3v_3 is off in both phases. After the step, F(v(0))F(v^{(0)}) drops to −2.6605-2.6605 and p(v(0))p(v^{(0)}) rises from 0.20830.2083 to 0.22140.2214.

python
1import torch
2
3def bars_and_stripes(n=3):
4 imgs = set()
5 for m in range(2 ** n):
6 line = torch.tensor([(m >> k) & 1 for k in range(n)], dtype=torch.float)
7 imgs.add(tuple(line.repeat_interleave(n).tolist())) # rows on
8 imgs.add(tuple(line.repeat(n).tolist())) # columns on
9 return torch.tensor(sorted(imgs)) # 14 images for n = 3
10
11V = bars_and_stripes() # (14, 9)
12nv, nh, lr = 9, 6, 0.5
13W = 0.1 * torch.randn(nv, nh); a = torch.zeros(nv); b = torch.zeros(nh)
14
15for epoch in range(2000):
16 ph0 = torch.sigmoid(V @ W + b) # positive phase, p(h | data)
17 h0 = torch.bernoulli(ph0)
18 v1 = torch.bernoulli(torch.sigmoid(h0 @ W.T + a)) # one Gibbs step
19 ph1 = torch.sigmoid(v1 @ W + b) # negative phase
20 W += lr * (V.T @ ph0 - v1.T @ ph1) / len(V)
21 a += lr * (V - v1).mean(0)
22 b += lr * (ph0 - ph1).mean(0)
23
24def free_energy(v): # F(v) = −aᵀv − Σ softplus(b + vW)
25 return -(v @ a) - torch.nn.functional.softplus(v @ W + b).sum(-1)
A binary RBM trained with CD-1 on 3×3 bars-and-stripes (the data the figure uses)

Variants and uses

Gaussian–Bernoulli RBMs handle real-valued inputs such as pixel intensities or audio features by replacing the visible term with ∑i(vi−ai)2/2σi2\sum_i(v_i-a_i)^2/2\sigma_i^2 and scaling the interaction by vi/σiv_i/\sigma_i. Then p(vi∣h)=N(ai+σi∑jWijhj, σi2)p(v_i\mid h)=\mathcal N\big(a_i+\sigma_i\sum_jW_{ij}h_j,\ \sigma_i^2\big), while p(h∣v)p(h\mid v) stays logistic. They need much smaller learning rates and usually standardised inputs.

Where it's used

Feature learning

p(h∣v)p(h\mid v) is a nonlinear feature map learned without labels; stacking these is the deep belief network.

Collaborative filtering

Salakhutdinov, Mnih & Hinton (2007) gave each user an RBM over the movies they rated, with softmax visible units for the 1–5 stars and shared weights. It slightly beat SVD on Netflix data, and blending the two helped further; RBMs were part of the prize-winning ensembles.

Pretraining

Initialising deep nets layer by layer (2006–2011), before ReLUs and better initialisation made it unnecessary.

Physics and topic models

Neural-network quantum states and the replicated-softmax topic model use the same energy.

Use when

  • +Exact, parallel inference: p(h∣v)p(h\mid v) is one matrix multiply.
  • +Unnormalised density via the free energy, so it can score inputs and fill in missing values.
  • +A principled generative model that also yields features.

Avoid when

  • −ZZ is intractable, so likelihood cannot be monitored and the gradient is approximate.
  • −CD is biased; Gibbs chains mix poorly between distant modes.
  • −Largely superseded by VAEs, GANs, diffusion and autoregressive models for generation.

Check yourself

Why can all hidden units of an RBM be sampled in parallel given the visible units?

Check yourself

In CD-1, what replaces the model expectation ⟨vᵢhⱼ⟩_model?

Remember

  • ✓Energy E(v,h)=−a⊤v−b⊤h−v⊤WhE(v,h)=-a^\top v-b^\top h-v^\top Wh; bipartite, so p(hj=1∣v)=σ(bj+v⊤W:,j)p(h_j{=}1\mid v)=\sigma(b_j+v^\top W_{:,j}) and symmetrically for vv.
  • ✓Free energy F(v)=−a⊤v−∑jsoftplus(bj+v⊤W:,j)F(v)=-a^\top v-\sum_j\mathrm{softplus}(b_j+v^\top W_{:,j}) gives p(v)p(v) up to the intractable ZZ.
  • ✓Gradient: ⟨vh⊤⟩data−⟨vh⊤⟩model\langle vh^\top\rangle_{\text{data}}-\langle vh^\top\rangle_{\text{model}}; block Gibbs supplies the model samples.
  • ✓CD-k restarts a k-step chain at the data (cheap, biased); PCD keeps persistent chains (less biased, needs small steps).
  • ✓Low reconstruction error does not imply high likelihood.
  • ✓Gaussian–Bernoulli RBMs model real-valued inputs; RBMs powered DBN pretraining and Netflix-prize collaborative filtering.
← back to Deep Learning roadmap