Hopfield network

A recurrent net of ±1 units with symmetric Hebbian weights whose energy can only fall, so noisy inputs slide into stored memories; it holds about 0.138N random patterns.

9 min read921 wordscore0 viewsupdated 2026-10-11

Definition

A Hopfield network is a fully connected recurrent network of binary units si∈{−1,+1}s_i\in\{-1,+1\} with symmetric weights wij=wjiw_{ij}=w_{ji} and no self-connections. Units update one at a time to the sign of their input. Stored patterns are written into the weights by a Hebbian rule and become stable states the network falls into, so it works as a content-addressable memory. John Hopfield introduced it in 1982.

Memory you address by content

Computer memory is addressed by location: give an address, get the bits stored there. Human memory works differently. A fragment (a few notes of a song, half a face) is enough to bring back the whole thing.

Hopfield built a network that does this. Each stored pattern is a valley in an energy landscape over all 2N2^N states. Start the network at a corrupted version of a pattern and its dynamics roll downhill into the nearest valley, filling in the missing or wrong bits. Nothing is looked up; the answer is wherever the dynamics stop.

Storage: the Hebbian rule

To store PP patterns ξ1,…,ξP∈{−1,+1}N\xi^1,\dots,\xi^P\in\{-1,+1\}^N, set each weight to the average correlation of the two units it connects, across the patterns. "Units that fire together wire together" (Hebb, 1949):

W=1N∑μ=1Pξμξμ⊤,wii=0W=\frac1N\sum_{\mu=1}^{P}\xi^\mu\xi^{\mu\top},\qquad w_{ii}=0
ξμ\xi^\mu
stored pattern μ\mu, one ±1\pm1 entry per unit
wijw_{ij}
1N∑μξiμξjμ\frac1N\sum_\mu\xi^\mu_i\xi^\mu_j: positive if units i,ji,j usually agree in the patterns, negative if they usually disagree
wii=0w_{ii}=0
no self-connections; needed for the energy argument below

This is one-shot learning: there is no training loop, and adding a pattern just adds one more outer product. WW is symmetric by construction.

Recall: asynchronous updates

hi=∑jwijsj,si←sign⁡(hi)(keep si if hi=0)h_i=\sum_{j}w_{ij}s_j,\qquad s_i\leftarrow\operatorname{sign}(h_i)\quad(\text{keep } s_i \text{ if } h_i=0)
hih_i
local field: the weighted vote of all other units on unit i
sign⁡\operatorname{sign}
hard threshold; thresholds θᵢ can be added as hᵢ − θᵢ but are 0 here

Asynchronous means one unit at a time, in random order, each seeing the latest values of the others. Why a stored pattern is stable: at s=ξνs=\xi^\nu the field splits into a signal and a crosstalk term,

hi=N−1N ξiν  +  1N∑μ≠νξiμ∑j≠iξjμξjνh_i=\frac{N-1}{N}\,\xi^\nu_i\;+\;\frac1N\sum_{\mu\ne\nu}\xi^\mu_i\sum_{j\ne i}\xi^\mu_j\xi^\nu_j
N−1Nξiν\tfrac{N-1}{N}\xi^\nu_i
signal: pushes unit i toward its value in the pattern
crosstalk\text{crosstalk}
interference from the other P−1P-1 patterns; for random patterns roughly Gaussian with variance ≈P/N\approx P/N

A bit flips when the crosstalk beats the signal. For random patterns that happens with probability about Φ(−N/P)\Phi(-\sqrt{N/P}): 0.00080.0008 at P/N=0.1P/N=0.1 and 0.00360.0036 at P/N=0.138P/N=0.138. These are first-step errors only. Each wrong bit raises the crosstalk on the others, and above a critical load the errors avalanche.

The energy can only go down

E(s)=−12∑i≠jwijsisj=−12 s⊤WsE(s)=-\frac12\sum_{i\ne j}w_{ij}s_is_j=-\frac12\,s^\top W s
EE
a scalar for every one of the 2ᴺ states; low when units connected by positive weights agree

Proof that asynchronous updates never increase E

  1. 1Update only unit ii, from sis_i to si′s_i', and let Δsi=si′−si∈{0,±2}\Delta s_i=s_i'-s_i\in\{0,\pm2\}. Only the terms of EE containing sis_i change.
  2. 2Those terms are −12(∑jwijsisj+∑jwjisjsi)-\frac12\big(\sum_j w_{ij}s_is_j+\sum_j w_{ji}s_js_i\big). With wij=wjiw_{ij}=w_{ji} they equal −si∑jwijsj=−sihi-s_i\sum_j w_{ij}s_j=-s_ih_i, and wii=0w_{ii}=0 means hih_i does not depend on sis_i.
  3. 3So ΔE=−Δsi hi\Delta E=-\Delta s_i\,h_i. The rule sets si′=sign⁡(hi)s_i'=\operatorname{sign}(h_i), so when the unit flips, Δsi\Delta s_i has the sign of hih_i and ΔE=−2∣hi∣<0\Delta E=-2|h_i|<0. Otherwise ΔE=0\Delta E=0.
  4. 4EE is bounded below (there are finitely many states), and every flip lowers it by a strictly positive amount. So after finitely many flips no unit wants to change: the network has reached a fixed point, a local minimum of EE.

Both assumptions matter. Without symmetry the "energy" is not a Lyapunov function and the network can oscillate or wander. With synchronous updates (all units at once) the energy argument fails too, and the network can cycle between two states forever.

state s · update 0/200overlap with H: m = 0.50energy E = −½ sᵀWs -12.94 · flips 0E(H) = -52.180updates → (gridlines = sweeps of 100)start -12.94 → -12.94: E never goes uppress play or stepP = 4 stored (0.04N) · fixed points: 4/4stored ξ¹…ξᴾ (4 glyphs, then random) · bars: overlap with the state
probe
0.138N ≈ 14
fig.A 100-unit network (10×10 pixels) storing the glyphs H, T, O, L plus extra random patterns. The probe is a glyph with a fraction of its pixels flipped. Play runs asynchronous updates one unit at a time (orange square) while the energy curve only ever steps down. Push stored P past about 10 and the glyphs stop being fixed points, then recall ends in nearby spurious states.

At the default (4 patterns, H with 25% of pixels flipped) the energy falls from −12.94-12.94 to −52.18-52.18 and H is recalled exactly within two sweeps. With P=10P=10 the glyph L is no longer a fixed point (one of its pixels disagrees with its field), and at P=12P=12 neither H nor L is. The glyphs fail first because they are correlated with each other, which adds systematic crosstalk on top of the random kind.

Worked example: N = 4, one stored pattern

Store ξ=(+1,−1,+1,−1)\xi=(+1,-1,+1,-1), so wij=ξiξj/4w_{ij}=\xi_i\xi_j/4 off the diagonal: ±0.25\pm0.25. Probe with s=(+1,+1,+1,−1)s=(+1,+1,+1,-1), which has unit 2 flipped.

unit isᵢhᵢ = Σⱼ wᵢⱼ sⱼsᵢhᵢupdate
1+1(−1 + 1 + 1)/4 = 0.250.25stays
2+1(−1 − 1 − 1)/4 = −0.75−0.75flips to −1
3+1(1 − 1 + 1)/4 = 0.250.25stays
4−1(−1 + 1 − 1)/4 = −0.250.25stays
Local fields at the probe. Only unit 2 disagrees with its field (sihi<0s_ih_i<0), so it is the only unit that will flip.
  1. 1Energy of the probe: E=−12∑isihi=−12(0.25−0.75+0.25+0.25)=0E=-\frac12\sum_i s_ih_i=-\frac12(0.25-0.75+0.25+0.25)=0.
  2. 2Flip unit 2: ΔE=−Δs2 h2=−(−2)(−0.75)=−1.5\Delta E=-\Delta s_2\,h_2=-(-2)(-0.75)=-1.5, so E=−1.5E=-1.5.
  3. 3Now s=ξs=\xi and every field is hi=34ξih_i=\tfrac34\xi_i, so no unit wants to change. Recall is complete after a single flip.
  4. 4The reversed pattern −ξ=(−1,+1,−1,+1)-\xi=(-1,+1,-1,+1) also has E=−1.5E=-1.5: EE depends on ss only through products sisjs_is_j, so every stored pattern brings its mirror image along as a second attractor.

Capacity and spurious states

Amit, Gutfreund and Sompolinsky (1985) computed the capacity of the Hebbian network for random patterns in the large-NN limit: retrieval works up to P≈0.138NP\approx0.138N. Below that load each stored pattern has a nearby attractor with about 1.6% of bits wrong or fewer. Above it, retrieval collapses abruptly. If every bit of every pattern must be exact, the limit is much lower, about N/(4ln⁡N)N/(4\ln N) (McEliece et al., 1987): only 5.45.4 patterns for N=100N=100.

Attractors that were never stored

Reversed states
−ξμ-\xi^\mu has exactly the same energy as ξμ\xi^\mu, because EE is quadratic in ss.
Mixture states
sign combinations of an odd number of patterns, e.g. sign⁡(ξ1+ξ2+ξ3)\operatorname{sign}(\xi^1+\xi^2+\xi^3), are also stable. They sit at higher energy than the pure patterns.
Spin-glass states
at high load, local minima appear that are correlated with no stored pattern at all.
python
1import numpy as np
2rng = np.random.default_rng(0)
3
4N, P = 100, 8 # load P/N = 0.08, well below 0.138
5xi = rng.choice([-1, 1], size=(P, N))
6W = xi.T @ xi / N # Hebbian outer products
7np.fill_diagonal(W, 0)
8
9def energy(s):
10 return -0.5 * s @ W @ s
11
12def recall(s, max_sweeps=30):
13 s = s.copy()
14 for _ in range(max_sweeps):
15 flips = 0
16 for i in rng.permutation(N): # asynchronous, random order
17 h = W[i] @ s
18 if h != 0 and np.sign(h) != s[i]:
19 s[i] = -s[i]; flips += 1
20 if flips == 0: # fixed point reached
21 return s
22 return s
23
24probe = xi[0].copy()
25probe[rng.choice(N, 20, replace=False)] *= -1 # flip 20 of the 100 bits
26out = recall(probe)
27print(probe @ xi[0] / N, out @ xi[0] / N) # overlap 0.6 -> usually 1.0 at this load
28print(energy(probe), energy(out)) # the second is always lower
Storing and recalling random patterns in NumPy

Hopfield's 1982 paper connected neural computation to the statistical physics of magnets (the Ising model and spin glasses), and that link led directly to the Boltzmann machine. He shared the 2024 Nobel Prize in Physics with Geoffrey Hinton for this line of work.

Check yourself

Why must the weights be symmetric (w_ij = w_ji) for the convergence proof?

Check yourself

A Hopfield network has N = 1000 units. Roughly how many independent random patterns can it store and still retrieve (with a few bits wrong)?

Remember

  • ✓Store with W=1N∑μξμξμ⊤W=\frac1N\sum_\mu\xi^\mu\xi^{\mu\top}, wii=0w_{ii}=0; recall with asynchronous si←sign⁡(hi)s_i\leftarrow\operatorname{sign}(h_i).
  • ✓E=−12s⊤WsE=-\frac12 s^\top Ws and ΔE=−Δsihi≤0\Delta E=-\Delta s_ih_i\le0, so the dynamics always stop at a local minimum. This needs symmetric weights and one-at-a-time updates.
  • ✓Capacity is about 0.138N0.138N for random patterns, much less for correlated ones.
  • ✓Spurious attractors: reversed patterns −ξ-\xi, odd mixtures, and spin-glass states.
  • ✓Modern Hopfield networks (Ramsauer et al., 2020) use a log-sum-exp energy whose update rule is softmax attention.
← back to Deep Learning roadmap