Definition
A Hopfield network is a fully connected recurrent network of binary units with symmetric weights and no self-connections. Units update one at a time to the sign of their input. Stored patterns are written into the weights by a Hebbian rule and become stable states the network falls into, so it works as a content-addressable memory. John Hopfield introduced it in 1982.
Memory you address by content
Computer memory is addressed by location: give an address, get the bits stored there. Human memory works differently. A fragment (a few notes of a song, half a face) is enough to bring back the whole thing.
Hopfield built a network that does this. Each stored pattern is a valley in an energy landscape over all states. Start the network at a corrupted version of a pattern and its dynamics roll downhill into the nearest valley, filling in the missing or wrong bits. Nothing is looked up; the answer is wherever the dynamics stop.
Storage: the Hebbian rule
To store patterns , set each weight to the average correlation of the two units it connects, across the patterns. "Units that fire together wire together" (Hebb, 1949):
- stored pattern , one entry per unit
- : positive if units usually agree in the patterns, negative if they usually disagree
- no self-connections; needed for the energy argument below
This is one-shot learning: there is no training loop, and adding a pattern just adds one more outer product. is symmetric by construction.
Recall: asynchronous updates
- local field: the weighted vote of all other units on unit i
- hard threshold; thresholds θᵢ can be added as hᵢ − θᵢ but are 0 here
Asynchronous means one unit at a time, in random order, each seeing the latest values of the others. Why a stored pattern is stable: at the field splits into a signal and a crosstalk term,
- signal: pushes unit i toward its value in the pattern
- interference from the other patterns; for random patterns roughly Gaussian with variance
A bit flips when the crosstalk beats the signal. For random patterns that happens with probability about : at and at . These are first-step errors only. Each wrong bit raises the crosstalk on the others, and above a critical load the errors avalanche.
The energy can only go down
- a scalar for every one of the 2ᴺ states; low when units connected by positive weights agree
Proof that asynchronous updates never increase E
- 1Update only unit , from to , and let . Only the terms of containing change.
- 2Those terms are . With they equal , and means does not depend on .
- 3So . The rule sets , so when the unit flips, has the sign of and . Otherwise .
- 4 is bounded below (there are finitely many states), and every flip lowers it by a strictly positive amount. So after finitely many flips no unit wants to change: the network has reached a fixed point, a local minimum of .
Both assumptions matter. Without symmetry the "energy" is not a Lyapunov function and the network can oscillate or wander. With synchronous updates (all units at once) the energy argument fails too, and the network can cycle between two states forever.
At the default (4 patterns, H with 25% of pixels flipped) the energy falls from to and H is recalled exactly within two sweeps. With the glyph L is no longer a fixed point (one of its pixels disagrees with its field), and at neither H nor L is. The glyphs fail first because they are correlated with each other, which adds systematic crosstalk on top of the random kind.
Worked example: N = 4, one stored pattern
Store , so off the diagonal: . Probe with , which has unit 2 flipped.
| unit i | sᵢ | hᵢ = Σⱼ wᵢⱼ sⱼ | sᵢhᵢ | update |
|---|---|---|---|---|
| 1 | +1 | (−1 + 1 + 1)/4 = 0.25 | 0.25 | stays |
| 2 | +1 | (−1 − 1 − 1)/4 = −0.75 | −0.75 | flips to −1 |
| 3 | +1 | (1 − 1 + 1)/4 = 0.25 | 0.25 | stays |
| 4 | −1 | (−1 + 1 − 1)/4 = −0.25 | 0.25 | stays |
- 1Energy of the probe: .
- 2Flip unit 2: , so .
- 3Now and every field is , so no unit wants to change. Recall is complete after a single flip.
- 4The reversed pattern also has : depends on only through products , so every stored pattern brings its mirror image along as a second attractor.
Capacity and spurious states
Amit, Gutfreund and Sompolinsky (1985) computed the capacity of the Hebbian network for random patterns in the large- limit: retrieval works up to . Below that load each stored pattern has a nearby attractor with about 1.6% of bits wrong or fewer. Above it, retrieval collapses abruptly. If every bit of every pattern must be exact, the limit is much lower, about (McEliece et al., 1987): only patterns for .
Attractors that were never stored
- Reversed states
- has exactly the same energy as , because is quadratic in .
- Mixture states
- sign combinations of an odd number of patterns, e.g. , are also stable. They sit at higher energy than the pure patterns.
- Spin-glass states
- at high load, local minima appear that are correlated with no stored pattern at all.
1import numpy as np2rng = np.random.default_rng(0)34N, P = 100, 8 # load P/N = 0.08, well below 0.1385xi = rng.choice([-1, 1], size=(P, N))6W = xi.T @ xi / N # Hebbian outer products7np.fill_diagonal(W, 0)89def energy(s):10 return -0.5 * s @ W @ s1112def recall(s, max_sweeps=30):13 s = s.copy()14 for _ in range(max_sweeps):15 flips = 016 for i in rng.permutation(N): # asynchronous, random order17 h = W[i] @ s18 if h != 0 and np.sign(h) != s[i]:19 s[i] = -s[i]; flips += 120 if flips == 0: # fixed point reached21 return s22 return s2324probe = xi[0].copy()25probe[rng.choice(N, 20, replace=False)] *= -1 # flip 20 of the 100 bits26out = recall(probe)27print(probe @ xi[0] / N, out @ xi[0] / N) # overlap 0.6 -> usually 1.0 at this load28print(energy(probe), energy(out)) # the second is always lowerHopfield's 1982 paper connected neural computation to the statistical physics of magnets (the Ising model and spin glasses), and that link led directly to the Boltzmann machine. He shared the 2024 Nobel Prize in Physics with Geoffrey Hinton for this line of work.
Check yourself
Why must the weights be symmetric (w_ij = w_ji) for the convergence proof?
Check yourself
A Hopfield network has N = 1000 units. Roughly how many independent random patterns can it store and still retrieve (with a few bits wrong)?
Remember
- ✓Store with , ; recall with asynchronous .
- ✓ and , so the dynamics always stop at a local minimum. This needs symmetric weights and one-at-a-time updates.
- ✓Capacity is about for random patterns, much less for correlated ones.
- ✓Spurious attractors: reversed patterns , odd mixtures, and spin-glass states.
- ✓Modern Hopfield networks (Ramsauer et al., 2020) use a log-sum-exp energy whose update rule is softmax attention.