Capsule network

Groups of neurons output vectors whose length is the probability a part exists and whose direction is its pose; parts route to the whole their pose predictions agree on.

10 min read1032 wordsadvanced0 viewsupdated 2026-10-11

Definition

Capsule network is a neural network whose units are capsules: small groups of neurons that output a vector vjv_j instead of a scalar. The length ∥vj∥∈[0,1)\|v_j\|\in[0,1) is the probability that an entity (a digit, a face, an eye) is present, and the direction encodes its instantiation parameters, or pose (position, orientation, scale, stroke thickness …). Lower capsules send their output to higher capsules by routing-by-agreement.

Capsules were Geoffrey Hinton's long-running answer to what he saw as a flaw in convolutional networks. The version that made them famous is Dynamic Routing Between Capsules (Sabour, Frosst & Hinton, NeurIPS 2017), which reached 0.25% test error on MNIST with a three-layer network and separated heavily overlapping digits on MultiMNIST.

What pooling throws away

A CNN detects features with convolutions and then pools them: max-pooling keeps the strongest response in a window and discards where in the window it was. Stack enough pooling and a "face" unit only needs to see an eye, a nose and a mouth somewhere in its receptive field. Their relative positions and orientations barely matter.

That is the Picasso problem: a face with the mouth above the eyes, or the nose rotated sideways, contains every part a pooled detector looks for, so it can still fire strongly. The network has evidence that the parts exist, but not that they fit together.

Hinton's point is about equivariance versus invariance. Pooling aims for invariance: the output does not change when the input moves. Capsules aim for equivariance of the pose (rotate the face and the face capsule's vector rotates with it) together with invariance of the probability (its length stays the same). Viewpoint changes become simple, linear changes in pose, which is how computer graphics describes them.

The capsule nonlinearity: squash

A capsule's raw input sjs_j can have any length. To read the length as a probability, it is squashed into [0,1)[0,1) without changing direction:

vj=∥sj∥21+∥sj∥2 sj∥sj∥v_j=\frac{\|s_j\|^2}{1+\|s_j\|^2}\,\frac{s_j}{\|s_j\|}
sjs_j
total input to capsule jj (a vector)
vjv_j
output of capsule jj: same direction as sjs_j, length ∥sj∥2/(1+∥sj∥2)\|s_j\|^2/(1+\|s_j\|^2)

Long vectors saturate just below 1 and short ones shrink toward 0, quadratically. For s=(3,4)s=(3,4), ∥s∥=5\|s\|=5 and v=(0.5769, 0.7692)v=(0.5769,\,0.7692) with length 25/26=0.961525/26=0.9615. For s=(0.3,0.4)s=(0.3,0.4), ∥s∥=0.5\|s\|=0.5 and the length drops to 0.25/1.25=0.20.25/1.25=0.2.

Votes and routing-by-agreement

Each lower capsule ii ("part") makes a prediction for every higher capsule jj ("whole") by multiplying its pose by a learned transformation matrix. WijW_{ij} encodes the part-whole relationship: if this is an eye with this pose, the face should have that pose.

u^j∣i=Wij ui,sj=∑icij u^j∣i,cij=exp⁡bij∑kexp⁡bik\hat u_{j|i}=W_{ij}\,u_i,\qquad s_j=\sum_i c_{ij}\,\hat u_{j|i},\qquad c_{ij}=\frac{\exp b_{ij}}{\sum_k\exp b_{ik}}
uiu_i
output vector of lower capsule ii
WijW_{ij}
learned matrix (16×8 between MNIST primary and digit capsules), trained by backprop
u^j∣i\hat u_{j|i}
prediction ("vote") of part ii for the pose of whole jj
cijc_{ij}
coupling coefficient: how much of part ii goes to whole jj; sums to 1 over jj
bijb_{ij}
routing logit, reset to 0 for every input and updated by agreement

The coupling coefficients are not weights. They are computed per input, by an inner loop that rewards parts whose vote agrees with the whole's consensus:

Dynamic routing (Sabour et al. 2017, r = 3 iterations)

  1. 1Compute all votes u^j∣i=Wijui\hat u_{j|i}=W_{ij}u_i once. Set bij=0b_{ij}=0, so every part starts split evenly across the wholes.
  2. 2cij←softmax⁡j(bij)c_{ij}\leftarrow\operatorname{softmax}_j(b_{ij}): each part distributes one unit of output over the parents.
  3. 3sj←∑iciju^j∣is_j\leftarrow\sum_i c_{ij}\hat u_{j|i}, then vj←squash⁡(sj)v_j\leftarrow\operatorname{squash}(s_j): each whole forms a consensus pose.
  4. 4bij←bij+u^j∣i⋅vjb_{ij}\leftarrow b_{ij}+\hat u_{j|i}\cdot v_j: a vote that points the same way as the consensus raises that coupling; one that points away lowers it.
  5. 5Repeat steps 2–4 rr times and output vjv_j. Gradients flow through the whole loop, but only WijW_{ij} (and the convolutional layers below) are learned.

Agreement between several independent votes is strong evidence: in a high-dimensional pose space, random votes almost never coincide. Routing turns that into a clustering step, a little like a few iterations of soft k-means over the votes, with the whole's pose as the cluster centre.

routing iteration 1/5 · c = softmax(b) → s = Σ c·û → v = squash(s) → b += û·vparts uᵢ (lower capsules)eyenosemouthroofdoorwindowvotes û for FACE‖v_face‖ = 0.186votes û for HOUSE‖v_house‖ = 0.324dashed circle = length 1 · face parts · house parts · vote opacity = coupling cᵢⱼ · bold = output vcoupling cᵢⱼ: → face | → houseeye0.50 | 0.50nose0.50 | 0.50mouth0.50 | 0.50roof0.50 | 0.50door0.50 | 0.50window0.50 | 0.50‖v_face‖ 0.186‖v_house‖ 0.324pooled part evidence 0.900(mean ‖uᵢ‖, ignores pose) pose angle of v_face 61.9°
|
iteration 1: agreement û·v added to b next · face: 0.21 0.20 0.20 -0.13 -0.09 -0.20 · house: -0.14 -0.21 -0.22 0.37 0.32 0.34
fig.Real routing on six 2D part capsules and two parents. At iteration 1 every coupling is 0.5 and the stray votes cancel the good ones (‖v_face‖ = 0.186). Step forward: face parts commit to the face, house parts to the house, and both lengths reach 0.83–0.84 by iteration 3. Rotate the scene and the outputs rotate with it while their lengths stay put (equivariance). Then scramble the poses: the same parts are present with the same strength, but the votes no longer agree and both lengths stay near 0.3.
iterationc(eye→face)c(roof→face)‖v_face‖‖v_house‖‖v_face‖ scrambled
10.5000.5000.1860.3240.252
20.5880.3780.5350.6110.278
30.7880.1720.8290.8410.307
Routing in the figure (upright scene). c(eye→face) and c(roof→face) are coupling coefficients; the last column is the same parts with scrambled poses.

Read row 1 to row 2 as one agreement update. The eye's vote for the face had dot product 0.2100.210 with the first consensus and −0.144-0.144 with the house's, so its logits became (0.210, −0.144)(0.210,\,-0.144) and softmax⁡\operatorname{softmax} gives c=0.588c=0.588 toward the face. Meanwhile a pose-blind summary, the mean part presence, is 0.9000.900 in both scenes: it cannot tell the face from the Picasso.

The loss: margins on lengths, plus reconstruction

The top layer has one capsule per class. Classification uses the lengths, with a separate margin loss for each class so that several classes can be present at once (as in overlapping digits):

Lk=Tk max⁡(0, m+−∥vk∥)2+λ (1−Tk) max⁡(0, ∥vk∥−m−)2\mathcal{L}_k=T_k\,\max(0,\,m^+-\|v_k\|)^2+\lambda\,(1-T_k)\,\max(0,\,\|v_k\|-m^-)^2
TkT_k
1 if class kk is present, else 0
m+=0.9,  m−=0.1m^+=0.9,\;m^-=0.1
a present class must be longer than 0.9, an absent one shorter than 0.1
λ=0.5\lambda=0.5
down-weights absent classes so early training does not shrink every capsule

With the figure's numbers, a present face at ∥v∥=0.829\|v\|=0.829 costs (0.9−0.829)2=0.0050(0.9-0.829)^2=0.0050, the scrambled one at 0.3070.307 costs 0.3520.352, and an absent class at length 0.30.3 costs 0.5⋅0.22=0.020.5\cdot0.2^2=0.02.

A reconstruction regularizer forces the pose to carry real information: mask out every digit capsule except the correct one, feed its 16-D vector to a three-layer decoder (512, 1024, 784 units) and add the sum of squared pixel errors, scaled by 0.00050.0005 so it does not dominate. Perturbing single dimensions of the vector then changes stroke width, skew or scale in the reconstruction.

python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3def squash(s, dim=-1, eps=1e-8):
4 n2 = (s ** 2).sum(dim, keepdim=True)
5 return n2 / (1 + n2) * s / torch.sqrt(n2 + eps) # squash([3, 4]) -> [0.5769, 0.7692]
6
7def route(u_hat, iters=3):
8 # u_hat: (B, n_in, n_out, d_out), the votes W_ij u_i
9 b = torch.zeros(u_hat.shape[:3], device=u_hat.device)
10 for r in range(iters):
11 c = F.softmax(b, dim=2) # each part splits 1 over the parents
12 v = squash((c.unsqueeze(-1) * u_hat).sum(1)) # (B, n_out, d_out)
13 if r < iters - 1:
14 b = b + (u_hat * v.unsqueeze(1)).sum(-1) # agreement û·v
15 return v
16
17class DigitCaps(nn.Module):
18 def __init__(self, n_in=1152, d_in=8, n_out=10, d_out=16):
19 super().__init__()
20 self.W = nn.Parameter(0.01 * torch.randn(1, n_in, n_out, d_out, d_in))
21 def forward(self, u): # u: (B, 1152, 8) primary capsules
22 u_hat = (self.W @ u[:, :, None, :, None]).squeeze(-1) # (B, 1152, 10, 16)
23 return route(u_hat)
24
25def margin_loss(v, y, m_pos=0.9, m_neg=0.1, lam=0.5):
26 L, T = v.norm(dim=-1), F.one_hot(y, v.shape[1]).float()
27 return (T * F.relu(m_pos - L) ** 2 + lam * (1 - T) * F.relu(L - m_neg) ** 2).sum(1).mean()
Squash, dynamic routing and the margin loss for the MNIST DigitCaps layer (1152 primary capsules of 8-D → 10 digit capsules of 16-D)

EM routing and what came after

Matrix capsules with EM routing (Hinton, Sabour & Frosst, ICLR 2018) replaced the vector with a 4×4 pose matrix plus a separate logistic activation, and replaced the dot-product rule with an EM-style clustering of votes. It cut test errors on smallNORB by 45% relative to the previous best and generalised better to unseen viewpoints. Stacked Capsule Autoencoders (Kosiorek et al., 2019) later moved to unsupervised part discovery.

The honest status: capsules never scaled. The 2017 network reached only 10.6% error on CIFAR-10 even with an ensemble of 7 models, routing is sequential and memory-hungry, and on ImageNet-scale problems CNNs and later vision transformers won on accuracy per unit of compute. The ideas survive in object-centric models (slot attention also iterates a softmax assignment of inputs to slots) and in the push for equivariant architectures.

Use when

  • +Keeps part–whole spatial relationships that pooling discards; robust to the Picasso-style rearrangements.
  • +Pose equivariance: viewpoint changes move the pose vector instead of needing new training examples (better affNIST and smallNORB generalisation).
  • +Per-class lengths with margin loss handle several overlapping objects (MultiMNIST).

Avoid when

  • −Iterative routing on every forward pass: slow, sequential and memory-heavy.
  • −Never matched CNNs or ViTs on large natural-image benchmarks.
  • −Extra hyperparameters (iterations, capsule dimensions, margins) and training is finicky.

Check yourself

A capsule's input is s = (0.3, 0.4). What is its output after squashing?

Check yourself

You rotate an input image of a face by 30°. In an idealised capsule network, what happens to the face capsule's output?

Remember

  • ✓A capsule outputs a vector: length = probability the entity is present, direction = its pose; v=∥s∥21+∥s∥2s∥s∥v=\tfrac{\|s\|^2}{1+\|s\|^2}\tfrac{s}{\|s\|}.
  • ✓Votes u^j∣i=Wijui\hat u_{j|i}=W_{ij}u_i; routing iterates c=softmax⁡(b)c=\operatorname{softmax}(b), vj=squash⁡(∑iciju^j∣i)v_j=\operatorname{squash}(\sum_i c_{ij}\hat u_{j|i}), bij+=u^j∣i⋅vjb_{ij}\mathrel{+}=\hat u_{j|i}\cdot v_j (3 iterations).
  • ✓Margin loss with m+=0.9m^+=0.9, m−=0.1m^-=0.1, λ=0.5\lambda=0.5, plus a reconstruction loss scaled by 0.0005.
  • ✓Motivation: pooling loses spatial relationships (the Picasso problem); capsules want equivariant pose and invariant presence.
  • ✓EM routing (2018) used 4×4 pose matrices; capsules stayed compute-heavy and did not scale past small benchmarks.
← back to Deep Learning roadmap