# Capsule network

> Groups of neurons output vectors whose length is the probability a part exists and whose direction is its pose; parts route to the whole their pose predictions agree on.

Source: https://www.mlfoundry.in/learn/capsule-network  
Section: Deep Learning › Neural Network Algorithms  
Level: advanced · 10 min read · updated 2026-10-11

**Capsule network:** is a neural network whose units are **capsules**: small groups of neurons that output a vector $v_j$ instead of a scalar. The length $\|v_j\|\in[0,1)$ is the probability that an entity (a digit, a face, an eye) is present, and the direction encodes its instantiation parameters, or **pose** (position, orientation, scale, stroke thickness …). Lower capsules send their output to higher capsules by **routing-by-agreement**.

Capsules were Geoffrey Hinton's long-running answer to what he saw as a flaw in convolutional networks. The version that made them famous is *Dynamic Routing Between Capsules* (Sabour, Frosst & Hinton, NeurIPS 2017), which reached 0.25% test error on MNIST with a three-layer network and separated heavily overlapping digits on MultiMNIST.

## What pooling throws away

A CNN detects features with convolutions and then pools them: max-pooling keeps the strongest response in a window and discards *where* in the window it was. Stack enough pooling and a "face" unit only needs to see an eye, a nose and a mouth somewhere in its receptive field. Their relative positions and orientations barely matter.

That is the **Picasso problem**: a face with the mouth above the eyes, or the nose rotated sideways, contains every part a pooled detector looks for, so it can still fire strongly. The network has evidence that the parts exist, but not that they fit together.

Hinton's point is about **equivariance versus invariance**. Pooling aims for invariance: the output does not change when the input moves. Capsules aim for equivariance of the pose (rotate the face and the face capsule's vector rotates with it) together with invariance of the probability (its length stays the same). Viewpoint changes become simple, linear changes in pose, which is how computer graphics describes them.

## The capsule nonlinearity: squash

A capsule's raw input $s_j$ can have any length. To read the length as a probability, it is squashed into $[0,1)$ without changing direction:

$$
v_j=\frac{\|s_j\|^2}{1+\|s_j\|^2}\,\frac{s_j}{\|s_j\|}
$$

- $s_j$: total input to capsule $j$ (a vector)
- $v_j$: output of capsule $j$: same direction as $s_j$, length $\|s_j\|^2/(1+\|s_j\|^2)$

Long vectors saturate just below 1 and short ones shrink toward 0, quadratically. For $s=(3,4)$, $\|s\|=5$ and $v=(0.5769,\,0.7692)$ with length $25/26=0.9615$. For $s=(0.3,0.4)$, $\|s\|=0.5$ and the length drops to $0.25/1.25=0.2$.

## Votes and routing-by-agreement

Each lower capsule $i$ ("part") makes a prediction for every higher capsule $j$ ("whole") by multiplying its pose by a learned transformation matrix. $W_{ij}$ encodes the part-whole relationship: *if this is an eye with this pose, the face should have that pose.*

$$
\hat u_{j|i}=W_{ij}\,u_i,\qquad s_j=\sum_i c_{ij}\,\hat u_{j|i},\qquad c_{ij}=\frac{\exp b_{ij}}{\sum_k\exp b_{ik}}
$$

- $u_i$: output vector of lower capsule $i$
- $W_{ij}$: learned matrix (16×8 between MNIST primary and digit capsules), trained by backprop
- $\hat u_{j|i}$: prediction ("vote") of part $i$ for the pose of whole $j$
- $c_{ij}$: coupling coefficient: how much of part $i$ goes to whole $j$; sums to 1 over $j$
- $b_{ij}$: routing logit, reset to 0 for every input and updated by agreement

The coupling coefficients are not weights. They are computed per input, by an inner loop that rewards parts whose vote agrees with the whole's consensus:

**Dynamic routing (Sabour et al. 2017, r = 3 iterations)**

1. Compute all votes $\hat u_{j|i}=W_{ij}u_i$ once. Set $b_{ij}=0$, so every part starts split evenly across the wholes.
2. $c_{ij}\leftarrow\operatorname{softmax}_j(b_{ij})$: each part distributes one unit of output over the parents.
3. $s_j\leftarrow\sum_i c_{ij}\hat u_{j|i}$, then $v_j\leftarrow\operatorname{squash}(s_j)$: each whole forms a consensus pose.
4. $b_{ij}\leftarrow b_{ij}+\hat u_{j|i}\cdot v_j$: a vote that points the same way as the consensus raises that coupling; one that points away lowers it.
5. Repeat steps 2–4 $r$ times and output $v_j$. Gradients flow through the whole loop, but only $W_{ij}$ (and the convolutional layers below) are learned.

Agreement between several *independent* votes is strong evidence: in a high-dimensional pose space, random votes almost never coincide. Routing turns that into a clustering step, a little like a few iterations of soft k-means over the votes, with the whole's pose as the cluster centre.

> Figure: Real routing on six 2D part capsules and two parents. At iteration 1 every coupling is 0.5 and the stray votes cancel the good ones (‖v_face‖ = 0.186). Step forward: face parts commit to the face, house parts to the house, and both lengths reach 0.83–0.84 by iteration 3. Rotate the scene and the outputs rotate with it while their lengths stay put (equivariance). Then scramble the poses: the same parts are present with the same strength, but the votes no longer agree and both lengths stay near 0.3.

Routing in the figure (upright scene). c(eye→face) and c(roof→face) are coupling coefficients; the last column is the same parts with scrambled poses.

| iteration | c(eye→face) | c(roof→face) | ‖v_face‖ | ‖v_house‖ | ‖v_face‖ scrambled |
| --- | --- | --- | --- | --- | --- |
| 1 | 0.500 | 0.500 | 0.186 | 0.324 | 0.252 |
| 2 | 0.588 | 0.378 | 0.535 | 0.611 | 0.278 |
| 3 | 0.788 | 0.172 | 0.829 | 0.841 | 0.307 |

Read row 1 to row 2 as one agreement update. The eye's vote for the face had dot product $0.210$ with the first consensus and $-0.144$ with the house's, so its logits became $(0.210,\,-0.144)$ and $\operatorname{softmax}$ gives $c=0.588$ toward the face. Meanwhile a pose-blind summary, the mean part presence, is $0.900$ in **both** scenes: it cannot tell the face from the Picasso.

## The loss: margins on lengths, plus reconstruction

The top layer has one capsule per class. Classification uses the lengths, with a separate margin loss for each class so that several classes can be present at once (as in overlapping digits):

$$
\mathcal{L}_k=T_k\,\max(0,\,m^+-\|v_k\|)^2+\lambda\,(1-T_k)\,\max(0,\,\|v_k\|-m^-)^2
$$

- $T_k$: 1 if class $k$ is present, else 0
- $m^+=0.9,\;m^-=0.1$: a present class must be longer than 0.9, an absent one shorter than 0.1
- $\lambda=0.5$: down-weights absent classes so early training does not shrink every capsule

With the figure's numbers, a present face at $\|v\|=0.829$ costs $(0.9-0.829)^2=0.0050$, the scrambled one at $0.307$ costs $0.352$, and an absent class at length $0.3$ costs $0.5\cdot0.2^2=0.02$.

A **reconstruction** regularizer forces the pose to carry real information: mask out every digit capsule except the correct one, feed its 16-D vector to a three-layer decoder (512, 1024, 784 units) and add the sum of squared pixel errors, scaled by $0.0005$ so it does not dominate. Perturbing single dimensions of the vector then changes stroke width, skew or scale in the reconstruction.

```python
import torch, torch.nn as nn, torch.nn.functional as F

def squash(s, dim=-1, eps=1e-8):
    n2 = (s ** 2).sum(dim, keepdim=True)
    return n2 / (1 + n2) * s / torch.sqrt(n2 + eps)    # squash([3, 4]) -> [0.5769, 0.7692]

def route(u_hat, iters=3):
    # u_hat: (B, n_in, n_out, d_out), the votes W_ij u_i
    b = torch.zeros(u_hat.shape[:3], device=u_hat.device)
    for r in range(iters):
        c = F.softmax(b, dim=2)                         # each part splits 1 over the parents
        v = squash((c.unsqueeze(-1) * u_hat).sum(1))    # (B, n_out, d_out)
        if r < iters - 1:
            b = b + (u_hat * v.unsqueeze(1)).sum(-1)    # agreement û·v
    return v

class DigitCaps(nn.Module):
    def __init__(self, n_in=1152, d_in=8, n_out=10, d_out=16):
        super().__init__()
        self.W = nn.Parameter(0.01 * torch.randn(1, n_in, n_out, d_out, d_in))
    def forward(self, u):                               # u: (B, 1152, 8) primary capsules
        u_hat = (self.W @ u[:, :, None, :, None]).squeeze(-1)   # (B, 1152, 10, 16)
        return route(u_hat)

def margin_loss(v, y, m_pos=0.9, m_neg=0.1, lam=0.5):
    L, T = v.norm(dim=-1), F.one_hot(y, v.shape[1]).float()
    return (T * F.relu(m_pos - L) ** 2 + lam * (1 - T) * F.relu(L - m_neg) ** 2).sum(1).mean()
```

Squash, dynamic routing and the margin loss for the MNIST DigitCaps layer (1152 primary capsules of 8-D → 10 digit capsules of 16-D)

> **Routing is not learned, and it is not free:** The logits $b_{ij}$ restart at 0 for every example, so routing costs $r$ extra passes over every (part, whole, dimension) triple on every forward pass. In the MNIST model that is $1152\times10$ votes of 16 dimensions; for wider layers or bigger images the vote tensor dominates memory. Many implementations also detach $\hat u$ inside the intermediate iterations to stabilise training.

> **Length is presence, direction is pose:** Because $\|v\|$ only depends on how well the votes agree, not on which way they point, a rotated face gives the same length and a rotated pose vector. The Picasso face gives the same parts with long $u_i$, but short $\|v\|$, because their votes disagree.

## EM routing and what came after

*Matrix capsules with EM routing* (Hinton, Sabour & Frosst, ICLR 2018) replaced the vector with a 4×4 pose matrix plus a separate logistic activation, and replaced the dot-product rule with an EM-style clustering of votes. It cut test errors on smallNORB by 45% relative to the previous best and generalised better to unseen viewpoints. Stacked Capsule Autoencoders (Kosiorek et al., 2019) later moved to unsupervised part discovery.

The honest status: capsules never scaled. The 2017 network reached only 10.6% error on CIFAR-10 even with an ensemble of 7 models, routing is sequential and memory-hungry, and on ImageNet-scale problems CNNs and later vision transformers won on accuracy per unit of compute. The ideas survive in object-centric models (slot attention also iterates a softmax assignment of inputs to slots) and in the push for equivariant architectures.

**Strengths**

- Keeps part–whole spatial relationships that pooling discards; robust to the Picasso-style rearrangements.
- Pose equivariance: viewpoint changes move the pose vector instead of needing new training examples (better affNIST and smallNORB generalisation).
- Per-class lengths with margin loss handle several overlapping objects (MultiMNIST).

**Limitations**

- Iterative routing on every forward pass: slow, sequential and memory-heavy.
- Never matched CNNs or ViTs on large natural-image benchmarks.
- Extra hyperparameters (iterations, capsule dimensions, margins) and training is finicky.

**Question:** A capsule's input is s = (0.3, 0.4). What is its output after squashing?

- A. (0.12, 0.16), length 0.2
- B. (0.6, 0.8), length 1
- C. (0.3, 0.4), length 0.5
- D. (0.24, 0.32), length 0.4

**Answer:** A. (0.12, 0.16), length 0.2 — $\|s\|=0.5$, so the length becomes $0.25/(1+0.25)=0.2$ and the direction $(0.6,0.8)$ is kept: $v=0.2\cdot(0.6,0.8)=(0.12,0.16)$. Short inputs are shrunk quadratically toward zero.

**Question:** You rotate an input image of a face by 30°. In an idealised capsule network, what happens to the face capsule's output?

- A. Its length stays about the same and its direction (pose) changes
- B. Its length drops because the face is no longer upright
- C. Nothing changes: capsules are rotation invariant
- D. The coupling coefficients become uniform

**Answer:** A. Its length stays about the same and its direction (pose) changes — Every part's pose rotates, so every vote $W_{ij}u_i$ rotates consistently and the votes still agree. Agreement sets the length (presence, invariant), and the consensus direction moves with the face (pose, equivariant). Try the rotate slider in the figure.

**Key points**

- A capsule outputs a vector: length = probability the entity is present, direction = its pose; $v=\tfrac{\|s\|^2}{1+\|s\|^2}\tfrac{s}{\|s\|}$.
- Votes $\hat u_{j|i}=W_{ij}u_i$; routing iterates $c=\operatorname{softmax}(b)$, $v_j=\operatorname{squash}(\sum_i c_{ij}\hat u_{j|i})$, $b_{ij}\mathrel{+}=\hat u_{j|i}\cdot v_j$ (3 iterations).
- Margin loss with $m^+=0.9$, $m^-=0.1$, $\lambda=0.5$, plus a reconstruction loss scaled by 0.0005.
- Motivation: pooling loses spatial relationships (the Picasso problem); capsules want equivariant pose and invariant presence.
- EM routing (2018) used 4×4 pose matrices; capsules stayed compute-heavy and did not scale past small benchmarks.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
