Boltzmann machine

A stochastic Hopfield net that samples states with probability ∝ exp(−E/T) and learns by matching pairwise statistics of data and model, which is exact but painfully slow.

10 min read864 wordsadvanced0 viewsupdated 2026-10-11

Definition

A Boltzmann machine is a network of symmetrically connected binary units, some visible and some hidden, that switch on at random with a probability set by their input. Run long enough, it visits each joint state ss with probability p(s)∝e−E(s)/Tp(s)\propto e^{-E(s)/T}, the Boltzmann distribution of statistical physics. Learning adjusts the weights so that the distribution over the visible units matches the data. Hinton and Sejnowski introduced it in 1983–85.

From a memory to a generative model

A Hopfield network slides deterministically downhill in energy and stops in the nearest minimum, which may be a spurious one. It stores patterns, but it cannot say how probable a pattern is, and it cannot learn hidden features.

The Boltzmann machine makes two changes. Units update stochastically, so the network can climb out of poor minima and, at equilibrium, spends time in each state in proportion to e−E/Te^{-E/T}. That turns the energy into a probability model. It also adds hidden units, which are never observed but let the model represent dependencies the visible units alone cannot.

Energy and the Boltzmann distribution

E(s)=−∑i<jwijsisj−∑ibisi,p(s)=e−E(s)/TZ,Z=∑s′e−E(s′)/TE(s)=-\sum_{i<j}w_{ij}s_is_j-\sum_i b_is_i,\qquad p(s)=\frac{e^{-E(s)/T}}{Z},\qquad Z=\sum_{s'}e^{-E(s')/T}
si∈{0,1}s_i\in\{0,1\}
state of unit ii; s=(v,h)s=(v,h) splits into visible and hidden units
wij=wji, wii=0w_{ij}=w_{ji},\ w_{ii}=0
symmetric weights, as in a Hopfield network; bᵢ are biases
TT
temperature: high T flattens the distribution, T → 0 concentrates it on the minimum-energy states
ZZ
partition function, a sum over all 2ᴺ joint states

The model of the data is the marginal over the hidden units, p(v)=∑hp(v,h)p(v)=\sum_h p(v,h). Lower energy means higher probability, and the ratio of two states' probabilities, e−(E(s)−E(s′))/Te^{-(E(s)-E(s'))/T}, does not involve ZZ at all.

Gibbs sampling: the stochastic update

Pick a unit ii and hold all others fixed. Only two joint states are possible, si=1s_i=1 and si=0s_i=0, and they differ in energy by the energy gap ΔEi\Delta E_i. Their probability ratio is eΔEi/Te^{\Delta E_i/T}, which gives:

p(si=1∣s−i)=σ ⁣(ΔEiT),ΔEi=E(si=0)−E(si=1)=bi+∑jwijsjp(s_i=1\mid s_{-i})=\sigma\!\Big(\frac{\Delta E_i}{T}\Big),\qquad \Delta E_i=E(s_i{=}0)-E(s_i{=}1)=b_i+\sum_{j}w_{ij}s_j
s−is_{-i}
all units except i
σ(x)=1/(1+e−x)\sigma(x)=1/(1+e^{-x})
the logistic sigmoid; ΔEᵢ is the Hopfield local field plus the bias

Updating units one at a time with this rule is Gibbs sampling. It satisfies detailed balance with respect to p(s)p(s), so for any T>0T>0 the chain converges to the Boltzmann distribution from any start. As T→0T\to0 the sigmoid becomes a step and the rule becomes the deterministic Hopfield update.

For example, in the figure's network with v2=v3=1v_2=v_3=1 and both hidden units off, unit v1v_1 has ΔE1=−1.5+2+2=2.5\Delta E_1=-1.5+2+2=2.5. It turns on with probability σ(2.5)=0.924\sigma(2.5)=0.924 at T=1T=1, σ(2.5/4)=0.651\sigma(2.5/4)=0.651 at T=4T=4, and 0.999760.99976 at T=0.3T=0.3.

s = 00011 · E(s) = -1.2v₁p(on) 0.01b −1.5v₂p(on) 0.01b −1.5v₃p(on) 0.00b −1.5h₁p(on) 0.93b −1.4h₂p(on) 0.93b −1.4━ w > 0 ━ w < 0 · solid = visible, dashed = hiddenp(s), 32 states by energy · T = 1.00111000.400000110.296000000.0891100010100011000001000001100000100000100100110101100111010101000110010001100100100101111101110111010011101100110101101100110111011101110111111111■ sampled ━ exact exp(−E/T)/Ztotal-variation distance between sampled and exact p(s) · 0 samples · TV = 1.0000.250.50.75101001k10k100ksamples (log scale) →
|
one sweep = update v₁…h₂ in turn, each set to 1 with probability σ(ΔEᵢ/T), ΔEᵢ = bᵢ + Σⱼ wᵢⱼsⱼ · log Z = 2.417
fig.Five units (3 visible, 2 hidden), fully connected. Play runs Gibbs sweeps; the bars are how often each of the 32 states was visited and the black ticks are the exact Boltzmann probabilities, computed by enumerating all states. The chain starts in 00011, a deep minimum that is not the global one (11100). Lower T and watch the chain get stuck; then try anneal.

At T=1T=1 the exact probabilities of the three best states are 0.4000.400 (11100), 0.2960.296 (00011) and 0.0890.089 (00000), with log⁡Z=2.417\log Z=2.417, and 5,000 sweeps get the histogram within a total-variation distance of about 0.020.02. At T=0.3T=0.3 the ground state should get 72.7%72.7\% of the mass, but the chain started in 00011 almost never crosses the barrier, so after 1,000 sweeps the distance is still about 0.730.73. The same sampler is correct in the limit and useless in practice.

Temperature and simulated annealing

High temperature makes every update close to a coin flip: the chain mixes fast but the distribution is nearly flat. Low temperature makes the distribution sharp but the chain sticky. Simulated annealing (Kirkpatrick, Gelatt and Vecchi, 1983) uses both: start hot, cool slowly, and the state settles into a low-energy region. Hinton and Sejnowski first used Boltzmann machines this way, for constraint satisfaction.

Annealing finds the global minimum with probability approaching 1 only under a logarithmically slow schedule (Geman and Geman, 1984). The figure's fast geometric schedule (T←0.95TT\leftarrow0.95T every 25 sweeps, from 4 to 0.3) ends in the ground state 11100 in about 65% of runs. That is close to its equilibrium share at T=0.5T=0.5 (0.6130.613), roughly the temperature where the chain stops crossing between the two coalitions.

Learning: two phases

Maximize the average log-likelihood of the training vectors, 1M∑mlog⁡p(v(m))\frac1M\sum_m\log p(v^{(m)}). Differentiating log⁡p(v)=log⁡∑he−E(v,h)/T−log⁡Z\log p(v)=\log\sum_h e^{-E(v,h)/T}-\log Z gives a strikingly simple, local rule (Ackley, Hinton and Sejnowski, 1985):

∂ log⁡p(v)‾∂wij=1T(⟨sisj⟩data−⟨sisj⟩model),Δwij=η(⟨sisj⟩data−⟨sisj⟩model)\frac{\partial\,\overline{\log p(v)}}{\partial w_{ij}}=\frac1T\Big(\langle s_is_j\rangle_{\text{data}}-\langle s_is_j\rangle_{\text{model}}\Big),\qquad \Delta w_{ij}=\eta\Big(\langle s_is_j\rangle_{\text{data}}-\langle s_is_j\rangle_{\text{model}}\Big)
⟨sisj⟩data\langle s_is_j\rangle_{\text{data}}
positive phase: clamp the visible units to a training vector, sample the hidden units from p(h∣v)p(h\mid v), and average how often ii and jj are both on, over the training set
⟨sisj⟩model\langle s_is_j\rangle_{\text{model}}
negative phase: let the whole network run freely to equilibrium and measure the same co-occurrence
Δbi\Delta b_i
the same with ⟨si⟩\langle s_i\rangle in place of ⟨sisj⟩\langle s_is_j\rangle

The rule is Hebbian learning on the data minus Hebbian unlearning on the model's own fantasies. It is local: each weight only needs statistics of the two units it connects. And it stops exactly when the model reproduces the pairwise statistics of the data.

Worked example: two visible units, no hidden units, T = 1, η = 1

  1. 1Data: 8 vectors, (1,1)×3(1,1)\times3, (0,0)×2(0,0)\times2, (1,0)×2(1,0)\times2, (0,1)×1(0,1)\times1. Positive phase: ⟨s1s2⟩data=3/8=0.375\langle s_1s_2\rangle_{\text{data}}=3/8=0.375, ⟨s1⟩=0.625\langle s_1\rangle=0.625, ⟨s2⟩=0.5\langle s_2\rangle=0.5.
  2. 2Start at w=b1=b2=0w=b_1=b_2=0. All four states have E=0E=0, so the model is uniform: ⟨s1s2⟩model=0.25\langle s_1s_2\rangle_{\text{model}}=0.25, ⟨s1⟩=⟨s2⟩=0.5\langle s_1\rangle=\langle s_2\rangle=0.5. Average log-likelihood ln⁡14=−1.3863\ln\tfrac14=-1.3863.
  3. 3First step: Δw=0.375−0.25=0.125\Delta w=0.375-0.25=0.125, Δb1=0.125\Delta b_1=0.125, Δb2=0\Delta b_2=0. The model now gives p(1,1)=0.2907p(1,1)=0.2907 and the log-likelihood rises to −1.3605-1.3605.
  4. 4After 500 steps: w=1.0986=ln⁡3w=1.0986=\ln3, b1=0b_1=0, b2=−0.6931=ln⁡12b_2=-0.6931=\ln\tfrac12, and the model matches the data exactly, p=(0.25, 0.125, 0.25, 0.375)p=(0.25,\,0.125,\,0.25,\,0.375), log-likelihood −1.3209-1.3209 (the data's entropy). Check: p(1,1) p(0,0)/(p(1,0) p(0,1))=ew=3p(1,1)\,p(0,0)/\big(p(1,0)\,p(0,1)\big)=e^{w}=3.
python
1import itertools, numpy as np
2
3S = np.array(list(itertools.product([0, 1], repeat=2)), dtype=float) # all 2^n states
4data = np.array([[1, 1]] * 3 + [[0, 0]] * 2 + [[1, 0]] * 2 + [[0, 1]], dtype=float)
5
6def probs(W, b, T=1.0): # exact Boltzmann distribution
7 E = -0.5 * np.einsum("ni,ij,nj->n", S, W, S) - S @ b # W symmetric, zero diagonal
8 p = np.exp(-E / T)
9 return p / p.sum() # the sum is Z: 2^n terms
10
11W, b = np.zeros((2, 2)), np.zeros(2)
12for _ in range(500):
13 p = probs(W, b)
14 pos = data.T @ data / len(data) # <s_i s_j>_data (positive phase)
15 neg = S.T @ (p[:, None] * S) # <s_i s_j>_model (negative phase)
16 dW = pos - neg; np.fill_diagonal(dW, 0)
17 W += 1.0 * dW
18 b += 1.0 * (data.mean(0) - p @ S)
19print(W[0, 1], b) # 1.0986 (= ln 3), [ 0. -0.6931]
20
21def gibbs_sweep(W, b, s, T, rng): # for real sizes: sample instead of enumerate
22 for i in range(len(s)):
23 gap = b[i] + W[i] @ s # energy gap ΔE_i (W[i, i] = 0)
24 s[i] = rng.random() < 1 / (1 + np.exp(-gap / T))
25 return s
Exact Boltzmann learning by enumerating states (only possible for tiny nets), plus the Gibbs sweep that replaces enumeration in practice

Why it is so slow

At a glance

Intractable Z
ZZ sums 2N2^N terms; for 100 units that is about 1.27×10301.27\times10^{30}. So ⟨sisj⟩model\langle s_is_j\rangle_{\text{model}} cannot be computed exactly and must be estimated by sampling.
Equilibrium per step
each gradient step needs samples from the equilibrium distribution, in both phases (the hidden units are coupled to each other even when the visible ones are clamped). Every weight update hides a full MCMC run.
Slow mixing
as learning succeeds, the weights grow and the distribution develops separated modes. Exactly as in the figure at low T, the chain then takes exponentially long to move between them, so the negative-phase estimates become biased.
Noisy differences
the gradient is the difference of two noisy estimates of similar size, so it has high variance and needs small learning rates.

Use when

  • +A complete probabilistic model: it can sample, fill in missing visible units by clamping the rest, and score configurations up to Z
  • +Hidden units learn features without labels, with a learning rule that is local and Hebbian
  • +Exact in principle: with equilibrium statistics the rule follows the true likelihood gradient

Avoid when

  • −Every gradient step needs equilibrium samples, so training general Boltzmann machines is impractical beyond small sizes
  • −ZZ is intractable, so even the likelihood of a test vector can only be estimated (e.g. by annealed importance sampling)
  • −Superseded in practice by RBMs, then by VAEs, GANs, autoregressive and diffusion models

Check yourself

At temperature T, unit i has energy gap ΔEᵢ = 0. What is p(sᵢ = 1 | all other units)?

Check yourself

Training stops changing w_ij when…

Remember

  • ✓p(s)∝e−E(s)/Tp(s)\propto e^{-E(s)/T} with the Hopfield energy; each unit turns on with probability σ(ΔEi/T)\sigma(\Delta E_i/T), ΔEi=bi+∑jwijsj\Delta E_i=b_i+\sum_jw_{ij}s_j.
  • ✓Gibbs sampling converges to the Boltzmann distribution from any start, but at low temperature it can take exponentially long to mix between modes.
  • ✓Simulated annealing (start hot, cool slowly) finds low-energy states; a guaranteed global minimum needs a logarithmic schedule.
  • ✓Learning: Δwij=η(⟨sisj⟩data−⟨sisj⟩model)\Delta w_{ij}=\eta(\langle s_is_j\rangle_{\text{data}}-\langle s_is_j\rangle_{\text{model}}), a positive (clamped) and a negative (free-running) phase.
  • ✓The intractable partition function and the need for equilibrium in both phases make it slow; RBMs with contrastive divergence were the practical fix.
← back to Deep Learning roadmap