Definition
Capsule network is a neural network whose units are capsules: small groups of neurons that output a vector instead of a scalar. The length is the probability that an entity (a digit, a face, an eye) is present, and the direction encodes its instantiation parameters, or pose (position, orientation, scale, stroke thickness …). Lower capsules send their output to higher capsules by routing-by-agreement.
Capsules were Geoffrey Hinton's long-running answer to what he saw as a flaw in convolutional networks. The version that made them famous is Dynamic Routing Between Capsules (Sabour, Frosst & Hinton, NeurIPS 2017), which reached 0.25% test error on MNIST with a three-layer network and separated heavily overlapping digits on MultiMNIST.
What pooling throws away
A CNN detects features with convolutions and then pools them: max-pooling keeps the strongest response in a window and discards where in the window it was. Stack enough pooling and a "face" unit only needs to see an eye, a nose and a mouth somewhere in its receptive field. Their relative positions and orientations barely matter.
That is the Picasso problem: a face with the mouth above the eyes, or the nose rotated sideways, contains every part a pooled detector looks for, so it can still fire strongly. The network has evidence that the parts exist, but not that they fit together.
Hinton's point is about equivariance versus invariance. Pooling aims for invariance: the output does not change when the input moves. Capsules aim for equivariance of the pose (rotate the face and the face capsule's vector rotates with it) together with invariance of the probability (its length stays the same). Viewpoint changes become simple, linear changes in pose, which is how computer graphics describes them.
The capsule nonlinearity: squash
A capsule's raw input can have any length. To read the length as a probability, it is squashed into without changing direction:
- total input to capsule (a vector)
- output of capsule : same direction as , length
Long vectors saturate just below 1 and short ones shrink toward 0, quadratically. For , and with length . For , and the length drops to .
Votes and routing-by-agreement
Each lower capsule ("part") makes a prediction for every higher capsule ("whole") by multiplying its pose by a learned transformation matrix. encodes the part-whole relationship: if this is an eye with this pose, the face should have that pose.
- output vector of lower capsule
- learned matrix (16×8 between MNIST primary and digit capsules), trained by backprop
- prediction ("vote") of part for the pose of whole
- coupling coefficient: how much of part goes to whole ; sums to 1 over
- routing logit, reset to 0 for every input and updated by agreement
The coupling coefficients are not weights. They are computed per input, by an inner loop that rewards parts whose vote agrees with the whole's consensus:
Dynamic routing (Sabour et al. 2017, r = 3 iterations)
- 1Compute all votes once. Set , so every part starts split evenly across the wholes.
- 2: each part distributes one unit of output over the parents.
- 3, then : each whole forms a consensus pose.
- 4: a vote that points the same way as the consensus raises that coupling; one that points away lowers it.
- 5Repeat steps 2–4 times and output . Gradients flow through the whole loop, but only (and the convolutional layers below) are learned.
Agreement between several independent votes is strong evidence: in a high-dimensional pose space, random votes almost never coincide. Routing turns that into a clustering step, a little like a few iterations of soft k-means over the votes, with the whole's pose as the cluster centre.
| iteration | c(eye→face) | c(roof→face) | ‖v_face‖ | ‖v_house‖ | ‖v_face‖ scrambled |
|---|---|---|---|---|---|
| 1 | 0.500 | 0.500 | 0.186 | 0.324 | 0.252 |
| 2 | 0.588 | 0.378 | 0.535 | 0.611 | 0.278 |
| 3 | 0.788 | 0.172 | 0.829 | 0.841 | 0.307 |
Read row 1 to row 2 as one agreement update. The eye's vote for the face had dot product with the first consensus and with the house's, so its logits became and gives toward the face. Meanwhile a pose-blind summary, the mean part presence, is in both scenes: it cannot tell the face from the Picasso.
The loss: margins on lengths, plus reconstruction
The top layer has one capsule per class. Classification uses the lengths, with a separate margin loss for each class so that several classes can be present at once (as in overlapping digits):
- 1 if class is present, else 0
- a present class must be longer than 0.9, an absent one shorter than 0.1
- down-weights absent classes so early training does not shrink every capsule
With the figure's numbers, a present face at costs , the scrambled one at costs , and an absent class at length costs .
A reconstruction regularizer forces the pose to carry real information: mask out every digit capsule except the correct one, feed its 16-D vector to a three-layer decoder (512, 1024, 784 units) and add the sum of squared pixel errors, scaled by so it does not dominate. Perturbing single dimensions of the vector then changes stroke width, skew or scale in the reconstruction.
1import torch, torch.nn as nn, torch.nn.functional as F23def squash(s, dim=-1, eps=1e-8):4 n2 = (s ** 2).sum(dim, keepdim=True)5 return n2 / (1 + n2) * s / torch.sqrt(n2 + eps) # squash([3, 4]) -> [0.5769, 0.7692]67def route(u_hat, iters=3):8 # u_hat: (B, n_in, n_out, d_out), the votes W_ij u_i9 b = torch.zeros(u_hat.shape[:3], device=u_hat.device)10 for r in range(iters):11 c = F.softmax(b, dim=2) # each part splits 1 over the parents12 v = squash((c.unsqueeze(-1) * u_hat).sum(1)) # (B, n_out, d_out)13 if r < iters - 1:14 b = b + (u_hat * v.unsqueeze(1)).sum(-1) # agreement û·v15 return v1617class DigitCaps(nn.Module):18 def __init__(self, n_in=1152, d_in=8, n_out=10, d_out=16):19 super().__init__()20 self.W = nn.Parameter(0.01 * torch.randn(1, n_in, n_out, d_out, d_in))21 def forward(self, u): # u: (B, 1152, 8) primary capsules22 u_hat = (self.W @ u[:, :, None, :, None]).squeeze(-1) # (B, 1152, 10, 16)23 return route(u_hat)2425def margin_loss(v, y, m_pos=0.9, m_neg=0.1, lam=0.5):26 L, T = v.norm(dim=-1), F.one_hot(y, v.shape[1]).float()27 return (T * F.relu(m_pos - L) ** 2 + lam * (1 - T) * F.relu(L - m_neg) ** 2).sum(1).mean()EM routing and what came after
Matrix capsules with EM routing (Hinton, Sabour & Frosst, ICLR 2018) replaced the vector with a 4×4 pose matrix plus a separate logistic activation, and replaced the dot-product rule with an EM-style clustering of votes. It cut test errors on smallNORB by 45% relative to the previous best and generalised better to unseen viewpoints. Stacked Capsule Autoencoders (Kosiorek et al., 2019) later moved to unsupervised part discovery.
The honest status: capsules never scaled. The 2017 network reached only 10.6% error on CIFAR-10 even with an ensemble of 7 models, routing is sequential and memory-hungry, and on ImageNet-scale problems CNNs and later vision transformers won on accuracy per unit of compute. The ideas survive in object-centric models (slot attention also iterates a softmax assignment of inputs to slots) and in the push for equivariant architectures.
Use when
- +Keeps part–whole spatial relationships that pooling discards; robust to the Picasso-style rearrangements.
- +Pose equivariance: viewpoint changes move the pose vector instead of needing new training examples (better affNIST and smallNORB generalisation).
- +Per-class lengths with margin loss handle several overlapping objects (MultiMNIST).
Avoid when
- −Iterative routing on every forward pass: slow, sequential and memory-heavy.
- −Never matched CNNs or ViTs on large natural-image benchmarks.
- −Extra hyperparameters (iterations, capsule dimensions, margins) and training is finicky.
Check yourself
A capsule's input is s = (0.3, 0.4). What is its output after squashing?
Check yourself
You rotate an input image of a face by 30°. In an idealised capsule network, what happens to the face capsule's output?
Remember
- ✓A capsule outputs a vector: length = probability the entity is present, direction = its pose; .
- ✓Votes ; routing iterates , , (3 iterations).
- ✓Margin loss with , , , plus a reconstruction loss scaled by 0.0005.
- ✓Motivation: pooling loses spatial relationships (the Picasso problem); capsules want equivariant pose and invariant presence.
- ✓EM routing (2018) used 4×4 pose matrices; capsules stayed compute-heavy and did not scale past small benchmarks.