# Hopfield network

> A recurrent net of ±1 units with symmetric Hebbian weights whose energy can only fall, so noisy inputs slide into stored memories; it holds about 0.138N random patterns.

Source: https://www.mlfoundry.in/learn/hopfield-network  
Section: Deep Learning › Neural Network Algorithms  
Level: core · 9 min read · updated 2026-10-11

**A Hopfield network:** is a fully connected recurrent network of binary units $s_i\in\{-1,+1\}$ with symmetric weights $w_{ij}=w_{ji}$ and no self-connections. Units update one at a time to the sign of their input. Stored patterns are written into the weights by a Hebbian rule and become stable states the network falls into, so it works as a **content-addressable memory**. John Hopfield introduced it in 1982.

## Memory you address by content

Computer memory is addressed by location: give an address, get the bits stored there. Human memory works differently. A fragment (a few notes of a song, half a face) is enough to bring back the whole thing.

Hopfield built a network that does this. Each stored pattern is a valley in an **energy landscape** over all $2^N$ states. Start the network at a corrupted version of a pattern and its dynamics roll downhill into the nearest valley, filling in the missing or wrong bits. Nothing is looked up; the answer is wherever the dynamics stop.

## Storage: the Hebbian rule

To store $P$ patterns $\xi^1,\dots,\xi^P\in\{-1,+1\}^N$, set each weight to the average correlation of the two units it connects, across the patterns. "Units that fire together wire together" (Hebb, 1949):

$$
W=\frac1N\sum_{\mu=1}^{P}\xi^\mu\xi^{\mu\top},\qquad w_{ii}=0
$$

- $\xi^\mu$: stored pattern $\mu$, one $\pm1$ entry per unit
- $w_{ij}$: $\frac1N\sum_\mu\xi^\mu_i\xi^\mu_j$: positive if units $i,j$ usually agree in the patterns, negative if they usually disagree
- $w_{ii}=0$: no self-connections; needed for the energy argument below

This is one-shot learning: there is no training loop, and adding a pattern just adds one more outer product. $W$ is symmetric by construction.

## Recall: asynchronous updates

$$
h_i=\sum_{j}w_{ij}s_j,\qquad s_i\leftarrow\operatorname{sign}(h_i)\quad(\text{keep } s_i \text{ if } h_i=0)
$$

- $h_i$: local field: the weighted vote of all other units on unit i
- $\operatorname{sign}$: hard threshold; thresholds θᵢ can be added as hᵢ − θᵢ but are 0 here

**Asynchronous** means one unit at a time, in random order, each seeing the latest values of the others. Why a stored pattern is stable: at $s=\xi^\nu$ the field splits into a signal and a crosstalk term,

$$
h_i=\frac{N-1}{N}\,\xi^\nu_i\;+\;\frac1N\sum_{\mu\ne\nu}\xi^\mu_i\sum_{j\ne i}\xi^\mu_j\xi^\nu_j
$$

- $\tfrac{N-1}{N}\xi^\nu_i$: signal: pushes unit i toward its value in the pattern
- $\text{crosstalk}$: interference from the other $P-1$ patterns; for random patterns roughly Gaussian with variance $\approx P/N$

A bit flips when the crosstalk beats the signal. For random patterns that happens with probability about $\Phi(-\sqrt{N/P})$: $0.0008$ at $P/N=0.1$ and $0.0036$ at $P/N=0.138$. These are first-step errors only. Each wrong bit raises the crosstalk on the others, and above a critical load the errors avalanche.

## The energy can only go down

$$
E(s)=-\frac12\sum_{i\ne j}w_{ij}s_is_j=-\frac12\,s^\top W s
$$

- $E$: a scalar for every one of the 2ᴺ states; low when units connected by positive weights agree

**Proof that asynchronous updates never increase E**

1. Update only unit $i$, from $s_i$ to $s_i'$, and let $\Delta s_i=s_i'-s_i\in\{0,\pm2\}$. Only the terms of $E$ containing $s_i$ change.
2. Those terms are $-\frac12\big(\sum_j w_{ij}s_is_j+\sum_j w_{ji}s_js_i\big)$. With $w_{ij}=w_{ji}$ they equal $-s_i\sum_j w_{ij}s_j=-s_ih_i$, and $w_{ii}=0$ means $h_i$ does not depend on $s_i$.
3. So $\Delta E=-\Delta s_i\,h_i$. The rule sets $s_i'=\operatorname{sign}(h_i)$, so when the unit flips, $\Delta s_i$ has the sign of $h_i$ and $\Delta E=-2|h_i|<0$. Otherwise $\Delta E=0$.
4. $E$ is bounded below (there are finitely many states), and every flip lowers it by a strictly positive amount. So after finitely many flips no unit wants to change: the network has reached a **fixed point**, a local minimum of $E$.

Both assumptions matter. Without symmetry the "energy" is not a Lyapunov function and the network can oscillate or wander. With **synchronous** updates (all units at once) the energy argument fails too, and the network can cycle between two states forever.

> Figure: A 100-unit network (10×10 pixels) storing the glyphs H, T, O, L plus extra random patterns. The probe is a glyph with a fraction of its pixels flipped. Play runs asynchronous updates one unit at a time (orange square) while the energy curve only ever steps down. Push stored P past about 10 and the glyphs stop being fixed points, then recall ends in nearby spurious states.

At the default (4 patterns, H with 25% of pixels flipped) the energy falls from $-12.94$ to $-52.18$ and H is recalled exactly within two sweeps. With $P=10$ the glyph L is no longer a fixed point (one of its pixels disagrees with its field), and at $P=12$ neither H nor L is. The glyphs fail first because they are correlated with each other, which adds systematic crosstalk on top of the random kind.

## Worked example: N = 4, one stored pattern

Store $\xi=(+1,-1,+1,-1)$, so $w_{ij}=\xi_i\xi_j/4$ off the diagonal: $\pm0.25$. Probe with $s=(+1,+1,+1,-1)$, which has unit 2 flipped.

Local fields at the probe. Only unit 2 disagrees with its field ($s_ih_i<0$), so it is the only unit that will flip.

| unit i | sᵢ | hᵢ = Σⱼ wᵢⱼ sⱼ | sᵢhᵢ | update |
| --- | --- | --- | --- | --- |
| 1 | +1 | (−1 + 1 + 1)/4 = 0.25 | 0.25 | stays |
| 2 | +1 | (−1 − 1 − 1)/4 = −0.75 | −0.75 | flips to −1 |
| 3 | +1 | (1 − 1 + 1)/4 = 0.25 | 0.25 | stays |
| 4 | −1 | (−1 + 1 − 1)/4 = −0.25 | 0.25 | stays |

1. Energy of the probe: $E=-\frac12\sum_i s_ih_i=-\frac12(0.25-0.75+0.25+0.25)=0$.
2. Flip unit 2: $\Delta E=-\Delta s_2\,h_2=-(-2)(-0.75)=-1.5$, so $E=-1.5$.
3. Now $s=\xi$ and every field is $h_i=\tfrac34\xi_i$, so no unit wants to change. Recall is complete after a single flip.
4. The reversed pattern $-\xi=(-1,+1,-1,+1)$ also has $E=-1.5$: $E$ depends on $s$ only through products $s_is_j$, so every stored pattern brings its mirror image along as a second attractor.

## Capacity and spurious states

Amit, Gutfreund and Sompolinsky (1985) computed the capacity of the Hebbian network for random patterns in the large-$N$ limit: retrieval works up to $P\approx0.138N$. Below that load each stored pattern has a nearby attractor with about 1.6% of bits wrong or fewer. Above it, retrieval collapses abruptly. If every bit of every pattern must be exact, the limit is much lower, about $N/(4\ln N)$ (McEliece et al., 1987): only $5.4$ patterns for $N=100$.

**Attractors that were never stored**

- **Reversed states:** $-\xi^\mu$ has exactly the same energy as $\xi^\mu$, because $E$ is quadratic in $s$.
- **Mixture states:** sign combinations of an odd number of patterns, e.g. $\operatorname{sign}(\xi^1+\xi^2+\xi^3)$, are also stable. They sit at higher energy than the pure patterns.
- **Spin-glass states:** at high load, local minima appear that are correlated with no stored pattern at all.

```python
import numpy as np
rng = np.random.default_rng(0)

N, P = 100, 8                                   # load P/N = 0.08, well below 0.138
xi = rng.choice([-1, 1], size=(P, N))
W = xi.T @ xi / N                               # Hebbian outer products
np.fill_diagonal(W, 0)

def energy(s):
    return -0.5 * s @ W @ s

def recall(s, max_sweeps=30):
    s = s.copy()
    for _ in range(max_sweeps):
        flips = 0
        for i in rng.permutation(N):            # asynchronous, random order
            h = W[i] @ s
            if h != 0 and np.sign(h) != s[i]:
                s[i] = -s[i]; flips += 1
        if flips == 0:                          # fixed point reached
            return s
    return s

probe = xi[0].copy()
probe[rng.choice(N, 20, replace=False)] *= -1   # flip 20 of the 100 bits
out = recall(probe)
print(probe @ xi[0] / N, out @ xi[0] / N)       # overlap 0.6 -> usually 1.0 at this load
print(energy(probe), energy(out))               # the second is always lower
```

Storing and recalling random patterns in NumPy

> **Correlated patterns break the capacity estimate:** $0.138N$ assumes independent random patterns with half the bits on. Real images are correlated (lots of shared background), and their crosstalk adds up instead of averaging out, so capacity can drop to a handful of patterns. In the figure, the four glyphs overlap by up to $0.30$ and start failing near $P=10$. Fixes include the pseudo-inverse (projection) rule, Storkey's rule, or subtracting the mean pattern before storing.

> **Modern Hopfield networks are attention:** Krotov and Hopfield (2016) replaced the quadratic energy with sharper interaction functions and got capacity that grows faster than $N$. Ramsauer et al. (2020, "Hopfield Networks is All You Need") used a log-sum-exp energy over continuous states; its update $\xi^{\text{new}}=X\,\operatorname{softmax}(\beta X^\top\xi)$ is exactly transformer attention, with the stored patterns $X$ as keys and values and the state $\xi$ as the query. It can store exponentially many patterns in the dimension and usually retrieves in one update.

Hopfield's 1982 paper connected neural computation to the statistical physics of magnets (the Ising model and spin glasses), and that link led directly to the Boltzmann machine. He shared the 2024 Nobel Prize in Physics with Geoffrey Hinton for this line of work.

**Question:** Why must the weights be symmetric (w_ij = w_ji) for the convergence proof?

- A. So that W is invertible
- B. So that the change in energy from flipping unit i is exactly −Δsᵢ·hᵢ, which the update rule makes non-positive
- C. So that the Hebbian rule can be computed in one pass
- D. So that the patterns are orthogonal

**Answer:** B. So that the change in energy from flipping unit i is exactly −Δsᵢ·hᵢ, which the update rule makes non-positive — The two energy terms containing $s_i$ are $\sum_j w_{ij}s_is_j$ and $\sum_j w_{ji}s_js_i$. Only when $w_{ij}=w_{ji}$ do they combine into $-s_ih_i$, so that $\Delta E=-\Delta s_ih_i\le0$. With asymmetric weights the dynamics can cycle.

**Question:** A Hopfield network has N = 1000 units. Roughly how many independent random patterns can it store and still retrieve (with a few bits wrong)?

- A. About 14
- B. About 138
- C. About 500
- D. About 1000

**Answer:** B. About 138 — The capacity of the Hebbian rule is $P\approx0.138N=138$ random patterns. Requiring every bit of every pattern to be exact lowers it to about $N/(4\ln N)\approx36$.

**Key points**

- Store with $W=\frac1N\sum_\mu\xi^\mu\xi^{\mu\top}$, $w_{ii}=0$; recall with asynchronous $s_i\leftarrow\operatorname{sign}(h_i)$.
- $E=-\frac12 s^\top Ws$ and $\Delta E=-\Delta s_ih_i\le0$, so the dynamics always stop at a local minimum. This needs symmetric weights and one-at-a-time updates.
- Capacity is about $0.138N$ for random patterns, much less for correlated ones.
- Spurious attractors: reversed patterns $-\xi$, odd mixtures, and spin-glass states.
- Modern Hopfield networks (Ramsauer et al., 2020) use a log-sum-exp energy whose update rule is softmax attention.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
