Definition
A Boltzmann machine is a network of symmetrically connected binary units, some visible and some hidden, that switch on at random with a probability set by their input. Run long enough, it visits each joint state with probability , the Boltzmann distribution of statistical physics. Learning adjusts the weights so that the distribution over the visible units matches the data. Hinton and Sejnowski introduced it in 1983–85.
From a memory to a generative model
A Hopfield network slides deterministically downhill in energy and stops in the nearest minimum, which may be a spurious one. It stores patterns, but it cannot say how probable a pattern is, and it cannot learn hidden features.
The Boltzmann machine makes two changes. Units update stochastically, so the network can climb out of poor minima and, at equilibrium, spends time in each state in proportion to . That turns the energy into a probability model. It also adds hidden units, which are never observed but let the model represent dependencies the visible units alone cannot.
Energy and the Boltzmann distribution
- state of unit ; splits into visible and hidden units
- symmetric weights, as in a Hopfield network; bᵢ are biases
- temperature: high T flattens the distribution, T → 0 concentrates it on the minimum-energy states
- partition function, a sum over all 2ᴺ joint states
The model of the data is the marginal over the hidden units, . Lower energy means higher probability, and the ratio of two states' probabilities, , does not involve at all.
Gibbs sampling: the stochastic update
Pick a unit and hold all others fixed. Only two joint states are possible, and , and they differ in energy by the energy gap . Their probability ratio is , which gives:
- all units except i
- the logistic sigmoid; ΔEᵢ is the Hopfield local field plus the bias
Updating units one at a time with this rule is Gibbs sampling. It satisfies detailed balance with respect to , so for any the chain converges to the Boltzmann distribution from any start. As the sigmoid becomes a step and the rule becomes the deterministic Hopfield update.
For example, in the figure's network with and both hidden units off, unit has . It turns on with probability at , at , and at .
At the exact probabilities of the three best states are (11100), (00011) and (00000), with , and 5,000 sweeps get the histogram within a total-variation distance of about . At the ground state should get of the mass, but the chain started in 00011 almost never crosses the barrier, so after 1,000 sweeps the distance is still about . The same sampler is correct in the limit and useless in practice.
Temperature and simulated annealing
High temperature makes every update close to a coin flip: the chain mixes fast but the distribution is nearly flat. Low temperature makes the distribution sharp but the chain sticky. Simulated annealing (Kirkpatrick, Gelatt and Vecchi, 1983) uses both: start hot, cool slowly, and the state settles into a low-energy region. Hinton and Sejnowski first used Boltzmann machines this way, for constraint satisfaction.
Annealing finds the global minimum with probability approaching 1 only under a logarithmically slow schedule (Geman and Geman, 1984). The figure's fast geometric schedule ( every 25 sweeps, from 4 to 0.3) ends in the ground state 11100 in about 65% of runs. That is close to its equilibrium share at (), roughly the temperature where the chain stops crossing between the two coalitions.
Learning: two phases
Maximize the average log-likelihood of the training vectors, . Differentiating gives a strikingly simple, local rule (Ackley, Hinton and Sejnowski, 1985):
- positive phase: clamp the visible units to a training vector, sample the hidden units from , and average how often and are both on, over the training set
- negative phase: let the whole network run freely to equilibrium and measure the same co-occurrence
- the same with in place of
The rule is Hebbian learning on the data minus Hebbian unlearning on the model's own fantasies. It is local: each weight only needs statistics of the two units it connects. And it stops exactly when the model reproduces the pairwise statistics of the data.
Worked example: two visible units, no hidden units, T = 1, η = 1
- 1Data: 8 vectors, , , , . Positive phase: , , .
- 2Start at . All four states have , so the model is uniform: , . Average log-likelihood .
- 3First step: , , . The model now gives and the log-likelihood rises to .
- 4After 500 steps: , , , and the model matches the data exactly, , log-likelihood (the data's entropy). Check: .
1import itertools, numpy as np23S = np.array(list(itertools.product([0, 1], repeat=2)), dtype=float) # all 2^n states4data = np.array([[1, 1]] * 3 + [[0, 0]] * 2 + [[1, 0]] * 2 + [[0, 1]], dtype=float)56def probs(W, b, T=1.0): # exact Boltzmann distribution7 E = -0.5 * np.einsum("ni,ij,nj->n", S, W, S) - S @ b # W symmetric, zero diagonal8 p = np.exp(-E / T)9 return p / p.sum() # the sum is Z: 2^n terms1011W, b = np.zeros((2, 2)), np.zeros(2)12for _ in range(500):13 p = probs(W, b)14 pos = data.T @ data / len(data) # <s_i s_j>_data (positive phase)15 neg = S.T @ (p[:, None] * S) # <s_i s_j>_model (negative phase)16 dW = pos - neg; np.fill_diagonal(dW, 0)17 W += 1.0 * dW18 b += 1.0 * (data.mean(0) - p @ S)19print(W[0, 1], b) # 1.0986 (= ln 3), [ 0. -0.6931]2021def gibbs_sweep(W, b, s, T, rng): # for real sizes: sample instead of enumerate22 for i in range(len(s)):23 gap = b[i] + W[i] @ s # energy gap ΔE_i (W[i, i] = 0)24 s[i] = rng.random() < 1 / (1 + np.exp(-gap / T))25 return sWhy it is so slow
At a glance
- Intractable Z
- sums terms; for 100 units that is about . So cannot be computed exactly and must be estimated by sampling.
- Equilibrium per step
- each gradient step needs samples from the equilibrium distribution, in both phases (the hidden units are coupled to each other even when the visible ones are clamped). Every weight update hides a full MCMC run.
- Slow mixing
- as learning succeeds, the weights grow and the distribution develops separated modes. Exactly as in the figure at low T, the chain then takes exponentially long to move between them, so the negative-phase estimates become biased.
- Noisy differences
- the gradient is the difference of two noisy estimates of similar size, so it has high variance and needs small learning rates.
Use when
- +A complete probabilistic model: it can sample, fill in missing visible units by clamping the rest, and score configurations up to Z
- +Hidden units learn features without labels, with a learning rule that is local and Hebbian
- +Exact in principle: with equilibrium statistics the rule follows the true likelihood gradient
Avoid when
- −Every gradient step needs equilibrium samples, so training general Boltzmann machines is impractical beyond small sizes
- − is intractable, so even the likelihood of a test vector can only be estimated (e.g. by annealed importance sampling)
- −Superseded in practice by RBMs, then by VAEs, GANs, autoregressive and diffusion models
Check yourself
At temperature T, unit i has energy gap ΔEᵢ = 0. What is p(sᵢ = 1 | all other units)?
Check yourself
Training stops changing w_ij when…
Remember
- ✓ with the Hopfield energy; each unit turns on with probability , .
- ✓Gibbs sampling converges to the Boltzmann distribution from any start, but at low temperature it can take exponentially long to mix between modes.
- ✓Simulated annealing (start hot, cool slowly) finds low-energy states; a guaranteed global minimum needs a logarithmic schedule.
- ✓Learning: , a positive (clamped) and a negative (free-running) phase.
- ✓The intractable partition function and the need for equilibrium in both phases make it slow; RBMs with contrastive divergence were the practical fix.