# Restricted Boltzmann machine (RBM)

> A two-layer energy-based model with no within-layer links, so each layer is easy to sample given the other; trained by contrastive divergence.

Source: https://www.mlfoundry.in/learn/restricted-boltzmann-machine-rbm  
Section: Deep Learning › Neural Network Algorithms  
Level: advanced · 11 min read · updated 2026-10-11

**Restricted Boltzmann machine:** is an undirected generative model with a layer of **visible** units $v$ and a layer of **hidden** units $h$, connected by a weight matrix $W$, with **no connections inside a layer**. It assigns every joint state an energy $E(v,h)$ and a probability $p(v,h)\propto e^{-E(v,h)}$. Training makes the training data low-energy, and so probable.

A full Boltzmann machine lets every unit talk to every other unit. It is expressive, but even computing the probability that one hidden unit is on requires a long Markov chain. Smolensky's 1986 "harmonium" removed the visible–visible and hidden–hidden links, leaving a **bipartite** graph. That one restriction makes the hidden units independent given the visibles, and vice versa, so a whole layer can be inferred or sampled in a single parallel step.

Training was still slow until Hinton's **contrastive divergence** (2002) replaced the expensive model expectation with a one-step approximation. RBMs became the building block of the 2006 deep belief networks that restarted deep learning, and a strong collaborative-filtering model in the Netflix Prize.

## Energy and probability

$$
E(v,h)=-a^\top v-b^\top h-v^\top W h,\qquad p(v,h)=\frac{e^{-E(v,h)}}{Z},\qquad Z=\sum_{v,h}e^{-E(v,h)}
$$

- $v\in\{0,1\}^{n_v},\ h\in\{0,1\}^{n_h}$: binary visible (data) and hidden (feature) units
- $a,\ b$: visible and hidden biases
- $W_{ij}$: weight between visible $i$ and hidden $j$; positive means "these two like to be on together"
- $Z$: partition function: a sum over all $2^{n_v+n_h}$ joint states, intractable for realistic sizes

The energy has no $h_jh_k$ or $v_iv_k$ terms, so it is a sum of terms that each involve one hidden unit. Fix $v$, and $p(h\mid v)$ factorises over $j$; each factor is a logistic unit:

$$
p(h_j=1\mid v)=\sigma\Big(b_j+\sum_i v_iW_{ij}\Big),\qquad p(v_i=1\mid h)=\sigma\Big(a_i+\sum_j W_{ij}h_j\Big)
$$

- $\sigma$: logistic sigmoid $1/(1+e^{-x})$
- $b_j+v^\top W_{:,j}$: the same pre-activation a sigmoid neuron would compute: the up-pass is a feed-forward layer

Summing the hidden units out gives the marginal of a visible vector through its **free energy**, which costs only $O(n_vn_h)$:

$$
p(v)=\frac{e^{-F(v)}}{Z},\qquad F(v)=-a^\top v-\sum_{j=1}^{n_h}\log\!\Big(1+e^{\,b_j+v^\top W_{:,j}}\Big)
$$

- $F(v)$: free energy, $-\log\sum_h e^{-E(v,h)}$; lower $F$ means a more probable $v$
- $\log(1+e^x)$: softplus: each hidden unit contributes independently

## The learning rule

Differentiate the log-likelihood of a training vector. The $-F(v)$ term pulls the data's energy down; the $-\log Z$ term pushes everyone else's up. The result is a difference of two correlations:

$$
\frac{\partial\log p(v)}{\partial W_{ij}}=\underbrace{\langle v_ih_j\rangle_{\text{data}}}_{\text{positive phase}}-\underbrace{\langle v_ih_j\rangle_{\text{model}}}_{\text{negative phase}}
$$

- $\langle v_ih_j\rangle_{\text{data}}$: $v_i\,p(h_j{=}1\mid v)$ averaged over training vectors: exact and cheap
- $\langle v_ih_j\rangle_{\text{model}}$: the same correlation under the model's own distribution: needs samples from $p(v,h)$
- $a_i,\ b_j$: same pattern with $\langle v_i\rangle$ and $\langle h_j\rangle$

The model term needs samples. The bipartite structure gives **block Gibbs sampling**: sample all of $h$ from $p(h\mid v)$, then all of $v$ from $p(v\mid h)$, and repeat. Run long enough, the chain samples $p(v,h)$, but "long enough" can mean thousands of sweeps per update.

**Contrastive divergence, CD-k (one update)**

1. **Positive phase.** Clamp a data vector $v^{(0)}$; compute $p(h\mid v^{(0)})$ and sample $h^{(0)}$ from it.
2. **Gibbs steps.** For $t=1..k$: sample $v^{(t)}\sim p(v\mid h^{(t-1)})$, then $h^{(t)}\sim p(h\mid v^{(t)})$. CD-1 stops after one step, giving the "reconstruction" $v^{(1)}$.
3. **Update.** $\Delta W=\eta\big(v^{(0)}p(h\mid v^{(0)})^\top-v^{(k)}p(h\mid v^{(k)})^\top\big)$, $\Delta a=\eta(v^{(0)}-v^{(k)})$, $\Delta b=\eta\big(p(h\mid v^{(0)})-p(h\mid v^{(k)})\big)$, averaged over the mini-batch. Using probabilities instead of samples where possible reduces noise.
4. **Persistent CD (PCD, Tieleman 2008).** Instead of restarting the negative chain at the data, keep a set of "fantasy" chains alive across updates and advance each by $k$ steps. With a small learning rate the chains stay close to the model distribution, and the gradient is much less biased than CD's.

Starting the chain at the data is what makes CD cheap and biased: it measures how the model wants to *move away from the data* in one step, not the full model expectation. It is not the gradient of any objective, but it works well in practice for learning features, and $k$ trades bias for compute.

> Figure: A 9-visible, 6-hidden RBM trained live on all 14 images of 3×3 bars-and-stripes (full batch, seeded sampling). Press play: from reset with CD-1 at η = 0.5 the reconstruction error falls from 0.248 to 0.003 by epoch 500, while the exact log-likelihood climbs from −6.25 (≈ uniform over 512 images) to −3.38 by epoch 1000, still well short of the best possible −ln 14 = −2.64. Click cells of the input to corrupt it and watch the reconstruction repair it. Reset with PCD at η = 0.5: reconstruction error still goes to 0, yet the likelihood collapses (−17.7 at epoch 4000), because the persistent chains cannot keep up with such large weight changes.

## Worked example: one CD-1 update

Take $n_v=3$, $n_h=2$, $a=0$, $b=(-1,-1)$, $W=\begin{bmatrix}2&-1\\1&1\\-1&2\end{bmatrix}$, training vector $v^{(0)}=(1,1,0)$ and $\eta=0.1$. With only $2^3$ visible states, $Z=55.09$ can be summed exactly, so we can check that the update helps.

Every number computed exactly (rounded to 4 decimals); the two sampled states are given, not drawn

| Quantity | Computation | Value |
| --- | --- | --- |
| $b+W^\top v^{(0)}$ | $(-1+2+1,\;-1-1+1)$ | (2, −1) |
| $p(h\mid v^{(0)})$ | $(\sigma(2),\,\sigma(-1))$ | (0.8808, 0.2689) |
| $F(v^{(0)})$ | $-\log(1+e^{2})-\log(1+e^{-1})$ | −2.4402 |
| $p(v^{(0)})$ | $e^{2.4402}/55.09$ | 0.2083 |
| sample $h^{(0)}$ | one draw from the line above | (1, 0) |
| $p(v\mid h^{(0)})$ | $\sigma(a+W_{:,1})=\sigma(2,1,-1)$ | (0.8808, 0.7311, 0.2689) |
| sample $v^{(1)}$ | one draw: the reconstruction drops pixel 2 | (1, 0, 0) |
| $p(h\mid v^{(1)})$ | $(\sigma(1),\,\sigma(-2))$ | (0.7311, 0.1192) |
| $\Delta W$ | $0.1\,\big(v^{(0)}p(h\mid v^{(0)})^\top-v^{(1)}p(h\mid v^{(1)})^\top\big)$ | $\begin{bmatrix}0.0150&0.0150\\0.0881&0.0269\\0&0\end{bmatrix}$ |
| $\Delta a,\ \Delta b$ | $0.1(v^{(0)}-v^{(1)})$, $0.1(p_0-p_1)$ | (0, 0.1, 0), (0.0150, 0.0150) |

The reconstruction "forgot" visible unit 2, so the update strengthens exactly that unit's links to the hidden units and raises its bias. Row 3 is zero because $v_3$ is off in both phases. After the step, $F(v^{(0)})$ drops to $-2.6605$ and $p(v^{(0)})$ rises from $0.2083$ to $0.2214$.

```python
import torch

def bars_and_stripes(n=3):
    imgs = set()
    for m in range(2 ** n):
        line = torch.tensor([(m >> k) & 1 for k in range(n)], dtype=torch.float)
        imgs.add(tuple(line.repeat_interleave(n).tolist()))   # rows on
        imgs.add(tuple(line.repeat(n).tolist()))              # columns on
    return torch.tensor(sorted(imgs))                         # 14 images for n = 3

V = bars_and_stripes()                       # (14, 9)
nv, nh, lr = 9, 6, 0.5
W = 0.1 * torch.randn(nv, nh); a = torch.zeros(nv); b = torch.zeros(nh)

for epoch in range(2000):
    ph0 = torch.sigmoid(V @ W + b)           # positive phase, p(h | data)
    h0 = torch.bernoulli(ph0)
    v1 = torch.bernoulli(torch.sigmoid(h0 @ W.T + a))   # one Gibbs step
    ph1 = torch.sigmoid(v1 @ W + b)          # negative phase
    W += lr * (V.T @ ph0 - v1.T @ ph1) / len(V)
    a += lr * (V - v1).mean(0)
    b += lr * (ph0 - ph1).mean(0)

def free_energy(v):                          # F(v) = −aᵀv − Σ softplus(b + vW)
    return -(v @ a) - torch.nn.functional.softplus(v @ W + b).sum(-1)
```

A binary RBM trained with CD-1 on 3×3 bars-and-stripes (the data the figure uses)

> **Reconstruction error is not the objective:** The quantity everyone plots is the easy one, $\lVert v-\hat v\rVert^2$, but CD does not minimise it and a near-zero value says little about $p(v)$. In the figure, PCD at a high learning rate keeps a perfect reconstruction while the log-likelihood gets worse than the untrained model's. On real problems $Z$ is intractable; use annealed importance sampling, sample quality, or a downstream task to judge an RBM.

> **An RBM is a product of experts:** $e^{-F(v)}=e^{a^\top v}\prod_j\big(1+e^{b_j+v^\top W_{:,j}}\big)$: each hidden unit is an "expert" that multiplies in its own factor, which ranges from 1 (the unit is off for this image) to very large (it is strongly on). An image must satisfy many experts at once to score well. A product of soft constraints can be far sharper than any one of them, which is why six hidden units already put most of the sample mass (87% at epoch 2000 in the figure) on the 14 valid images out of 512. A mixture model would need one component per image.

## Variants and uses

**Gaussian–Bernoulli RBMs** handle real-valued inputs such as pixel intensities or audio features by replacing the visible term with $\sum_i(v_i-a_i)^2/2\sigma_i^2$ and scaling the interaction by $v_i/\sigma_i$. Then $p(v_i\mid h)=\mathcal N\big(a_i+\sigma_i\sum_jW_{ij}h_j,\ \sigma_i^2\big)$, while $p(h\mid v)$ stays logistic. They need much smaller learning rates and usually standardised inputs.

- **Feature learning:** $p(h\mid v)$ is a nonlinear feature map learned without labels; stacking these is the deep belief network.
- **Collaborative filtering:** Salakhutdinov, Mnih & Hinton (2007) gave each user an RBM over the movies they rated, with softmax visible units for the 1–5 stars and shared weights. It slightly beat SVD on Netflix data, and blending the two helped further; RBMs were part of the prize-winning ensembles.
- **Pretraining:** Initialising deep nets layer by layer (2006–2011), before ReLUs and better initialisation made it unnecessary.
- **Physics and topic models:** Neural-network quantum states and the replicated-softmax topic model use the same energy.

**Strengths**

- Exact, parallel inference: $p(h\mid v)$ is one matrix multiply.
- Unnormalised density via the free energy, so it can score inputs and fill in missing values.
- A principled generative model that also yields features.

**Limitations**

- $Z$ is intractable, so likelihood cannot be monitored and the gradient is approximate.
- CD is biased; Gibbs chains mix poorly between distant modes.
- Largely superseded by VAEs, GANs, diffusion and autoregressive models for generation.

**Question:** Why can all hidden units of an RBM be sampled in parallel given the visible units?

- A. Because the weights are symmetric
- B. Because there are no hidden–hidden connections, so p(h|v) factorises over hidden units
- C. Because CD-1 uses only one step
- D. Because the hidden units are Gaussian

**Answer:** B. Because there are no hidden–hidden connections, so p(h|v) factorises over hidden units — With no $h_jh_k$ terms in the energy, $E$ is a sum of terms each touching one $h_j$ for fixed $v$, so $p(h\mid v)=\prod_j p(h_j\mid v)$ with $p(h_j{=}1\mid v)=\sigma(b_j+v^\top W_{:,j})$. A general Boltzmann machine loses this.

**Question:** In CD-1, what replaces the model expectation ⟨vᵢhⱼ⟩_model?

- A. The data correlation, scaled by η
- B. The correlation measured after one Gibbs step started from the data vector
- C. An exact sum over all states
- D. The correlation under a long chain started from noise

**Answer:** B. The correlation measured after one Gibbs step started from the data vector — CD-1 starts the chain at $v^{(0)}$, takes one step to $v^{(1)}$ and uses $v^{(1)}p(h\mid v^{(1)})^\top$. A long chain from noise would approximate the true model term; PCD keeps such chains running between updates.

**Key points**

- Energy $E(v,h)=-a^\top v-b^\top h-v^\top Wh$; bipartite, so $p(h_j{=}1\mid v)=\sigma(b_j+v^\top W_{:,j})$ and symmetrically for $v$.
- Free energy $F(v)=-a^\top v-\sum_j\mathrm{softplus}(b_j+v^\top W_{:,j})$ gives $p(v)$ up to the intractable $Z$.
- Gradient: $\langle vh^\top\rangle_{\text{data}}-\langle vh^\top\rangle_{\text{model}}$; block Gibbs supplies the model samples.
- CD-k restarts a k-step chain at the data (cheap, biased); PCD keeps persistent chains (less biased, needs small steps).
- Low reconstruction error does not imply high likelihood.
- Gaussian–Bernoulli RBMs model real-valued inputs; RBMs powered DBN pretraining and Netflix-prize collaborative filtering.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
