# Boltzmann machine

> A stochastic Hopfield net that samples states with probability ∝ exp(−E/T) and learns by matching pairwise statistics of data and model, which is exact but painfully slow.

Source: https://www.mlfoundry.in/learn/boltzmann-machine  
Section: Deep Learning › Neural Network Algorithms  
Level: advanced · 10 min read · updated 2026-10-11

**A Boltzmann machine:** is a network of symmetrically connected binary units, some visible and some hidden, that switch on **at random** with a probability set by their input. Run long enough, it visits each joint state $s$ with probability $p(s)\propto e^{-E(s)/T}$, the Boltzmann distribution of statistical physics. Learning adjusts the weights so that the distribution over the visible units matches the data. Hinton and Sejnowski introduced it in 1983–85.

## From a memory to a generative model

A Hopfield network slides deterministically downhill in energy and stops in the nearest minimum, which may be a spurious one. It stores patterns, but it cannot say how probable a pattern is, and it cannot learn hidden features.

The Boltzmann machine makes two changes. Units update **stochastically**, so the network can climb out of poor minima and, at equilibrium, spends time in each state in proportion to $e^{-E/T}$. That turns the energy into a probability model. It also adds **hidden units**, which are never observed but let the model represent dependencies the visible units alone cannot.

## Energy and the Boltzmann distribution

$$
E(s)=-\sum_{i<j}w_{ij}s_is_j-\sum_i b_is_i,\qquad p(s)=\frac{e^{-E(s)/T}}{Z},\qquad Z=\sum_{s'}e^{-E(s')/T}
$$

- $s_i\in\{0,1\}$: state of unit $i$; $s=(v,h)$ splits into visible and hidden units
- $w_{ij}=w_{ji},\ w_{ii}=0$: symmetric weights, as in a Hopfield network; bᵢ are biases
- $T$: temperature: high T flattens the distribution, T → 0 concentrates it on the minimum-energy states
- $Z$: partition function, a sum over all 2ᴺ joint states

The model of the data is the marginal over the hidden units, $p(v)=\sum_h p(v,h)$. Lower energy means higher probability, and the ratio of two states' probabilities, $e^{-(E(s)-E(s'))/T}$, does not involve $Z$ at all.

## Gibbs sampling: the stochastic update

Pick a unit $i$ and hold all others fixed. Only two joint states are possible, $s_i=1$ and $s_i=0$, and they differ in energy by the **energy gap** $\Delta E_i$. Their probability ratio is $e^{\Delta E_i/T}$, which gives:

$$
p(s_i=1\mid s_{-i})=\sigma\!\Big(\frac{\Delta E_i}{T}\Big),\qquad \Delta E_i=E(s_i{=}0)-E(s_i{=}1)=b_i+\sum_{j}w_{ij}s_j
$$

- $s_{-i}$: all units except i
- $\sigma(x)=1/(1+e^{-x})$: the logistic sigmoid; ΔEᵢ is the Hopfield local field plus the bias

Updating units one at a time with this rule is **Gibbs sampling**. It satisfies detailed balance with respect to $p(s)$, so for any $T>0$ the chain converges to the Boltzmann distribution from any start. As $T\to0$ the sigmoid becomes a step and the rule becomes the deterministic Hopfield update.

For example, in the figure's network with $v_2=v_3=1$ and both hidden units off, unit $v_1$ has $\Delta E_1=-1.5+2+2=2.5$. It turns on with probability $\sigma(2.5)=0.924$ at $T=1$, $\sigma(2.5/4)=0.651$ at $T=4$, and $0.99976$ at $T=0.3$.

> Figure: Five units (3 visible, 2 hidden), fully connected. Play runs Gibbs sweeps; the bars are how often each of the 32 states was visited and the black ticks are the exact Boltzmann probabilities, computed by enumerating all states. The chain starts in 00011, a deep minimum that is not the global one (11100). Lower T and watch the chain get stuck; then try anneal.

At $T=1$ the exact probabilities of the three best states are $0.400$ (11100), $0.296$ (00011) and $0.089$ (00000), with $\log Z=2.417$, and 5,000 sweeps get the histogram within a total-variation distance of about $0.02$. At $T=0.3$ the ground state should get $72.7\%$ of the mass, but the chain started in 00011 almost never crosses the barrier, so after 1,000 sweeps the distance is still about $0.73$. The same sampler is correct in the limit and useless in practice.

## Temperature and simulated annealing

High temperature makes every update close to a coin flip: the chain mixes fast but the distribution is nearly flat. Low temperature makes the distribution sharp but the chain sticky. **Simulated annealing** (Kirkpatrick, Gelatt and Vecchi, 1983) uses both: start hot, cool slowly, and the state settles into a low-energy region. Hinton and Sejnowski first used Boltzmann machines this way, for constraint satisfaction.

Annealing finds the global minimum with probability approaching 1 only under a logarithmically slow schedule (Geman and Geman, 1984). The figure's fast geometric schedule ($T\leftarrow0.95T$ every 25 sweeps, from 4 to 0.3) ends in the ground state 11100 in about 65% of runs. That is close to its equilibrium share at $T=0.5$ ($0.613$), roughly the temperature where the chain stops crossing between the two coalitions.

## Learning: two phases

Maximize the average log-likelihood of the training vectors, $\frac1M\sum_m\log p(v^{(m)})$. Differentiating $\log p(v)=\log\sum_h e^{-E(v,h)/T}-\log Z$ gives a strikingly simple, local rule (Ackley, Hinton and Sejnowski, 1985):

$$
\frac{\partial\,\overline{\log p(v)}}{\partial w_{ij}}=\frac1T\Big(\langle s_is_j\rangle_{\text{data}}-\langle s_is_j\rangle_{\text{model}}\Big),\qquad \Delta w_{ij}=\eta\Big(\langle s_is_j\rangle_{\text{data}}-\langle s_is_j\rangle_{\text{model}}\Big)
$$

- $\langle s_is_j\rangle_{\text{data}}$: **positive phase**: clamp the visible units to a training vector, sample the hidden units from $p(h\mid v)$, and average how often $i$ and $j$ are both on, over the training set
- $\langle s_is_j\rangle_{\text{model}}$: **negative phase**: let the whole network run freely to equilibrium and measure the same co-occurrence
- $\Delta b_i$: the same with $\langle s_i\rangle$ in place of $\langle s_is_j\rangle$

The rule is Hebbian learning on the data minus Hebbian **un**learning on the model's own fantasies. It is local: each weight only needs statistics of the two units it connects. And it stops exactly when the model reproduces the pairwise statistics of the data.

**Worked example: two visible units, no hidden units, T = 1, η = 1**

1. Data: 8 vectors, $(1,1)\times3$, $(0,0)\times2$, $(1,0)\times2$, $(0,1)\times1$. Positive phase: $\langle s_1s_2\rangle_{\text{data}}=3/8=0.375$, $\langle s_1\rangle=0.625$, $\langle s_2\rangle=0.5$.
2. Start at $w=b_1=b_2=0$. All four states have $E=0$, so the model is uniform: $\langle s_1s_2\rangle_{\text{model}}=0.25$, $\langle s_1\rangle=\langle s_2\rangle=0.5$. Average log-likelihood $\ln\tfrac14=-1.3863$.
3. First step: $\Delta w=0.375-0.25=0.125$, $\Delta b_1=0.125$, $\Delta b_2=0$. The model now gives $p(1,1)=0.2907$ and the log-likelihood rises to $-1.3605$.
4. After 500 steps: $w=1.0986=\ln3$, $b_1=0$, $b_2=-0.6931=\ln\tfrac12$, and the model matches the data exactly, $p=(0.25,\,0.125,\,0.25,\,0.375)$, log-likelihood $-1.3209$ (the data's entropy). Check: $p(1,1)\,p(0,0)/\big(p(1,0)\,p(0,1)\big)=e^{w}=3$.

```python
import itertools, numpy as np

S = np.array(list(itertools.product([0, 1], repeat=2)), dtype=float)   # all 2^n states
data = np.array([[1, 1]] * 3 + [[0, 0]] * 2 + [[1, 0]] * 2 + [[0, 1]], dtype=float)

def probs(W, b, T=1.0):                     # exact Boltzmann distribution
    E = -0.5 * np.einsum("ni,ij,nj->n", S, W, S) - S @ b   # W symmetric, zero diagonal
    p = np.exp(-E / T)
    return p / p.sum()                      # the sum is Z: 2^n terms

W, b = np.zeros((2, 2)), np.zeros(2)
for _ in range(500):
    p = probs(W, b)
    pos = data.T @ data / len(data)         # <s_i s_j>_data   (positive phase)
    neg = S.T @ (p[:, None] * S)            # <s_i s_j>_model  (negative phase)
    dW = pos - neg; np.fill_diagonal(dW, 0)
    W += 1.0 * dW
    b += 1.0 * (data.mean(0) - p @ S)
print(W[0, 1], b)                           # 1.0986 (= ln 3), [ 0. -0.6931]

def gibbs_sweep(W, b, s, T, rng):           # for real sizes: sample instead of enumerate
    for i in range(len(s)):
        gap = b[i] + W[i] @ s               # energy gap ΔE_i (W[i, i] = 0)
        s[i] = rng.random() < 1 / (1 + np.exp(-gap / T))
    return s
```

Exact Boltzmann learning by enumerating states (only possible for tiny nets), plus the Gibbs sweep that replaces enumeration in practice

## Why it is so slow

- **Intractable Z:** $Z$ sums $2^N$ terms; for 100 units that is about $1.27\times10^{30}$. So $\langle s_is_j\rangle_{\text{model}}$ cannot be computed exactly and must be estimated by sampling.
- **Equilibrium per step:** each gradient step needs samples from the equilibrium distribution, in both phases (the hidden units are coupled to each other even when the visible ones are clamped). Every weight update hides a full MCMC run.
- **Slow mixing:** as learning succeeds, the weights grow and the distribution develops separated modes. Exactly as in the figure at low T, the chain then takes exponentially long to move between them, so the negative-phase estimates become biased.
- **Noisy differences:** the gradient is the difference of two noisy estimates of similar size, so it has high variance and needs small learning rates.

> **The fix: restrict the connections:** Remove the visible–visible and hidden–hidden connections and you get the **restricted Boltzmann machine** (Smolensky's 1986 harmonium). The hidden units become conditionally independent given the visible ones, so the positive phase is exact in one step, and Hinton's contrastive divergence (2002) replaces the negative-phase equilibrium with a few steps of block Gibbs sampling. Stacking RBMs started the deep-learning revival of 2006.

> **A short chain is not an equilibrium sample:** Running Gibbs for a few sweeps from a convenient start gives samples biased toward that start, as the figure shows at low T. Burn-in, persistent chains (keep the negative-phase chains running across updates) and tempering (swap states with hotter copies) all exist to work around this.

**Strengths**

- A complete probabilistic model: it can sample, fill in missing visible units by clamping the rest, and score configurations up to Z
- Hidden units learn features without labels, with a learning rule that is local and Hebbian
- Exact in principle: with equilibrium statistics the rule follows the true likelihood gradient

**Limitations**

- Every gradient step needs equilibrium samples, so training general Boltzmann machines is impractical beyond small sizes
- $Z$ is intractable, so even the likelihood of a test vector can only be estimated (e.g. by annealed importance sampling)
- Superseded in practice by RBMs, then by VAEs, GANs, autoregressive and diffusion models

**Question:** At temperature T, unit i has energy gap ΔEᵢ = 0. What is p(sᵢ = 1 | all other units)?

- A. 0
- B. 0.5
- C. 1
- D. It depends on T

**Answer:** B. 0.5 — $p(s_i=1\mid s_{-i})=\sigma(\Delta E_i/T)=\sigma(0)=0.5$ for every $T>0$: both values of $s_i$ give the same energy, so they are equally likely.

**Question:** Training stops changing w_ij when…

- A. the energy of every training vector is zero
- B. ⟨sᵢsⱼ⟩ with the visible units clamped to data equals ⟨sᵢsⱼ⟩ when the network runs freely
- C. the network reaches a fixed point
- D. the temperature reaches 0

**Answer:** B. ⟨sᵢsⱼ⟩ with the visible units clamped to data equals ⟨sᵢsⱼ⟩ when the network runs freely — $\Delta w_{ij}\propto\langle s_is_j\rangle_{\text{data}}-\langle s_is_j\rangle_{\text{model}}$, so the weights settle when the model's own pairwise statistics match those seen with data clamped. In the worked example that happens at $w=\ln3$.

**Key points**

- $p(s)\propto e^{-E(s)/T}$ with the Hopfield energy; each unit turns on with probability $\sigma(\Delta E_i/T)$, $\Delta E_i=b_i+\sum_jw_{ij}s_j$.
- Gibbs sampling converges to the Boltzmann distribution from any start, but at low temperature it can take exponentially long to mix between modes.
- Simulated annealing (start hot, cool slowly) finds low-energy states; a guaranteed global minimum needs a logarithmic schedule.
- Learning: $\Delta w_{ij}=\eta(\langle s_is_j\rangle_{\text{data}}-\langle s_is_j\rangle_{\text{model}})$, a positive (clamped) and a negative (free-running) phase.
- The intractable partition function and the need for equilibrium in both phases make it slow; RBMs with contrastive divergence were the practical fix.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
