# Contractive autoencoder

> Penalize the encoder's Jacobian so codes ignore small input changes, except along the data manifold where reconstruction needs them.

Source: https://www.mlfoundry.in/learn/contractive-autoencoder  
Section: Deep Learning › Neural Network Algorithms  
Level: advanced · 9 min read · updated 2026-10-11

**Contractive autoencoder (CAE):** is an autoencoder trained with the extra penalty $\lambda\,\|J_f(x)\|_F^2$, the squared Frobenius norm of the encoder's Jacobian $J_f=\partial h/\partial x$ at each training point. The penalty makes the code locally insensitive to the input. Reconstruction pushes back only in the directions the data actually varies in, so the encoder ends up contracting **across** the data manifold and staying sensitive **along** it.

A good representation should not change much when the input changes in a way that does not matter: a pixel of sensor noise, a tiny shift off the set of plausible images. It should change when the input moves to a genuinely different example. The denoising autoencoder gets this indirectly, by training on corrupted inputs. Rifai, Vincent, Muller, Glorot and Bengio (ICML 2011) asked for it directly: measure how strongly the code reacts to an infinitesimal input change and penalize it.

## The penalty

$$
\mathcal{L}_{\text{CAE}}=\sum_{x\in\mathcal D}\Big(\big\|g(f(x))-x\big\|^2+\lambda\,\big\|J_f(x)\big\|_F^2\Big),\qquad \|J_f(x)\|_F^2=\sum_{j,i}\Big(\frac{\partial h_j}{\partial x_i}\Big)^2
$$

- $J_f(x)$: $d_h\times d$ matrix of partial derivatives of the code $h=f(x)$ with respect to the input
- $\lambda$: trade-off between reconstruction and insensitivity
- $\|\cdot\|_F^2$: sum of squared entries, equal to the sum of squared singular values: $\sum_k s_k^2$

The Frobenius norm has a geometric reading. For a small perturbation $\epsilon$, $f(x+\epsilon)-f(x)\approx J_f\epsilon$, and averaged over random directions $\mathbb{E}\|J_f\epsilon\|^2=\|\epsilon\|^2\|J_f\|_F^2/d$. So the penalty is the average **squared stretch** of input perturbations. Its singular values $s_k$ give the stretch along each principal direction.

For a sigmoid encoder $h=\sigma(Wx+b)$ the Jacobian is cheap. Row $j$ is $h_j(1-h_j)\,W_j$, the slope of unit $j$ times its weight vector, so

$$
\|J_f(x)\|_F^2=\sum_{j=1}^{d_h}\big(h_j(1-h_j)\big)^2\,\|W_j\|^2
$$

- $h_j(1-h_j)$: $\sigma'$ of unit $j$ at this input: largest (0.25) at $h_j=0.5$, near 0 when the unit is saturated
- $\|W_j\|^2$: squared norm of the unit's incoming weights

This costs about as much as the reconstruction itself, with no explicit Jacobian. It shows the two ways a unit can contract: **small weights**, or **saturation** (unit firmly on or off at this $x$). Rifai et al. tie the decoder weights to the encoder, $g(h)=W^\top h+c$. Otherwise the encoder could shrink $W$ and the decoder scale back up, satisfying the penalty without learning anything.

## Why contraction is anisotropic

The penalty alone would contract every direction until the code is constant. Reconstruction forbids that, but only for directions in which training points differ. If the data lies on a low-dimensional manifold, moving **along** it changes $x$ to another training example that must be reconstructed differently, so the code must stay sensitive there. Moving **across** it reaches no training points at all, so reconstruction does not care and the penalty wins. The result: the leading singular vectors of $J_f$ line up with the manifold's **tangent space**, and the others shrink toward 0.

Rifai et al. used exactly this, reading off the top singular vectors of $J_f$ as estimated tangent directions (the "manifold tangent classifier", 2011). In the reconstruction the same effect makes $r=g\circ f$ project off-manifold inputs back onto the manifold, like a denoiser.

> Figure: A tied-weight CAE (2 inputs → 8 sigmoid units → 2) trained live by full-batch Adam on 40 points of a 1-D manifold. Dashed circles are input perturbations; mint ellipses are their images under the reconstruction Jacobian J_r. With λ = 0 on the line, the map stretches perturbations across the manifold about 5× (mean across-gain 5.08). At λ = 0.1 the ellipses flatten onto the line (across 0.098, along 0.939), and the dashed bands ±0.3 off the manifold are reconstructed almost onto it. On the sine wave, one layer of ridge units cannot follow a turning tangent everywhere, so contraction is only partial.

## Worked example: the closed form

**Two sigmoid units, W = [[1, 2], [−1, 0.5]], b = 0, x = (0.5, −0.25)**

1. Pre-activations: $a_1=0.5-0.5=0$, $a_2=-0.5-0.125=-0.625$. Activations: $h=(0.5,\;0.3486)$.
2. Slopes: $h_1(1-h_1)=0.25$, $h_2(1-h_2)=0.2271$. Weight norms: $\|W_1\|^2=5$, $\|W_2\|^2=1.25$.
3. $\|J_f\|_F^2=0.25^2\cdot5+0.2271^2\cdot1.25=0.3125+0.0645=0.3770$.
4. Check: $J_f=\begin{bmatrix}0.25&0.5\\-0.2271&0.1135\end{bmatrix}$ (rows $=$ slope $\times W_j$). Its squared entries sum to $0.3770$, as does a finite-difference Jacobian.
5. Unit 1 sits at $h=0.5$, where the slope is maximal, and has the bigger weights, so it contributes 83% of the penalty. Training will make it saturate here or shrink its weights unless reconstruction needs its sensitivity.

The figure's model after 3000 Adam steps (same seed). Gains are ‖J u‖ for the unit tangent u and the unit normal, averaged over the 40 training points.

| Manifold | λ | Recon error | mean ‖J_f‖² | J_f along / across | J_r along / across |
| --- | --- | --- | --- | --- | --- |
| line | 0 | 0.00022 | 1.394 | 0.461 / 1.086 | 0.968 / 5.082 |
| line | 0.01 | 0.00033 | 0.401 | 0.439 / 0.455 | 0.966 / 1.244 |
| line | 0.1 | 0.00056 | 0.168 | 0.403 / 0.056 | 0.939 / 0.098 |
| line | 1 | 0.02014 | 0.074 | 0.252 / 0.008 | 0.928 / 0.014 |
| sine wave | 0 | 0.00020 | 0.390 | 0.447 / 0.434 | 0.992 / 0.955 |
| sine wave | 0.1 | 0.00244 | 0.203 | 0.366 / 0.243 | 0.895 / 0.601 |
| sine wave | 1 | 0.03682 | 0.070 | 0.190 / 0.143 | 0.724 / 0.525 |

Read the line rows top to bottom. Without the penalty the reconstruction is nearly perfect on the data, but it **amplifies** off-manifold perturbations fivefold: nothing in the loss constrains those directions. A small λ removes that. At λ = 0.1 the code is 7× more sensitive along the line than across it, and the reconstruction error is still tiny. At λ = 1 even the along-direction gain of the encoder starts to fall and reconstruction suffers. The sine wave shows the limit of a one-layer encoder: each unit's sensitivity direction $W_j$ is fixed, so the code cannot align with a tangent that keeps turning.

## Relation to the denoising autoencoder

Expand a DAE's loss for small Gaussian noise. With $\tilde x=x+\sigma\varepsilon$ and $r(x+\sigma\varepsilon)\approx r(x)+\sigma J_r\varepsilon$,

$$
\mathbb{E}_\varepsilon\big\|r(x+\sigma\varepsilon)-x\big\|^2\;\approx\;\|r(x)-x\|^2+\sigma^2\,\|J_r(x)\|_F^2
$$

- $J_r=\partial r/\partial x$: Jacobian of the whole reconstruction, not just the encoder
- $\sigma^2$: plays the role of λ

So a small-noise DAE is a contractive penalty on the **reconstruction** (Alain & Bengio, 2014), while the CAE penalizes the **encoder**, which is what you keep as the representation. In the figure's sine-wave model at λ = 0.1, a Monte Carlo estimate of the DAE loss at σ = 0.05 is 0.005376, against 0.005388 from the right-hand side. The CAE penalty is analytic and deterministic, so it needs no noise sampling, but it only sees infinitesimal perturbations. Rifai et al.'s CAE+H adds a stochastic penalty on the change of $J_f$ between nearby points to reach a little further.

**Training a CAE**

1. Encoder $h=\sigma(Wx+b)$, tied decoder $\hat x=W^\top h+c$ (or a sigmoid output for data in $[0,1]$).
2. For each batch compute the reconstruction loss and $\sum_j (h_j(1-h_j))^2\|W_j\|^2$ per example; add $\lambda$ times its mean.
3. Backpropagate both terms. Autograd handles the penalty because it is an ordinary function of $h$ and $W$.
4. Tune $\lambda$ on a validation metric (downstream accuracy, or reconstruction at a target contraction). Track the singular values of $J_f$ at a few points: a sharp drop after the first few indicates a low-dimensional manifold.

```python
import torch, torch.nn as nn, torch.nn.functional as F

class CAE(nn.Module):
    def __init__(self, d=784, h=1000):
        super().__init__()
        self.W = nn.Parameter(torch.randn(h, d) * 0.01)
        self.b = nn.Parameter(torch.zeros(h))
        self.c = nn.Parameter(torch.zeros(d))
    def forward(self, x):
        h = torch.sigmoid(x @ self.W.T + self.b)
        x_hat = torch.sigmoid(h @ self.W + self.c)          # tied decoder
        return x_hat, h

def cae_loss(model, x, lam=0.1):
    x_hat, h = model(x)
    recon = F.binary_cross_entropy(x_hat, x, reduction="sum") / len(x)
    slope2 = (h * (1 - h)) ** 2                              # (B, h)
    jac = (slope2 * model.W.pow(2).sum(1)).sum(1).mean()    # Σ_j (h_j(1-h_j))² ‖W_j‖²
    return recon + lam * jac

# General encoders: estimate ||J||_F^2 with random projections, E_v ||J^T v||^2 for v ~ N(0, I),
# using torch.autograd.functional.vjp, or penalise the exact Jacobian for small inputs.
```

Contractive autoencoder with the closed-form sigmoid penalty and tied weights

> **Untied weights defeat the penalty:** $\|J_f\|_F^2$ scales with the encoder's weights. If the decoder is free, the optimizer can divide $W$ by 10 and multiply the decoder by 10. The penalty drops 100× while the reconstruction is unchanged, and nothing about the data has been learned. Tie the weights, or penalize the decoder's scale too.

> **Saturation is contraction:** The factor $(h_j(1-h_j))^2$ means a unit that is firmly on or off at $x$ costs almost nothing, whatever its weights. A CAE therefore learns units that are saturated for most inputs and sensitive only in a narrow band. The code becomes locally piecewise constant, except along the manifold directions where the reconstruction needs the units to vary.

**Strengths**

- Deterministic, analytic regularizer: no noise distribution to choose, no sampling variance.
- Its Jacobian's top singular vectors estimate the data manifold's tangent space, which is useful for tangent-propagation-style classifiers.
- Explicitly targets robustness of the features you keep (the encoder), not just of the reconstruction.

**Limitations**

- Cheap only for one-layer sigmoid encoders. For deep encoders the exact Jacobian needs one backward pass per input (or code) dimension, so it is usually estimated stochastically.
- Only constrains infinitesimal perturbations; larger corruptions need CAE+H or a DAE.
- Largely superseded in practice by denoising, masking and contrastive objectives, though Jacobian penalties remain common in robustness work.

**Question:** A sigmoid unit has weights $W_j$ with $\|W_j\|^2=4$. On one input it outputs $h_j=0.9$. What does it contribute to $\|J_f\|_F^2$ there?

- A. 0.0324
- B. 0.09
- C. 0.36
- D. 0.0081

**Answer:** A. 0.0324 — $(h_j(1-h_j))^2\|W_j\|^2=(0.9\cdot0.1)^2\cdot4=0.0081\cdot4=0.0324$. At $h_j=0.5$ the same unit would contribute $0.25^2\cdot4=0.25$, nearly 8× more: saturation is cheap.

**Question:** Data lie on a curve in 2-D. After CAE training, how should the encoder Jacobian's singular directions relate to the curve at a training point?

- A. The largest singular value's direction is roughly tangent to the curve; the normal direction has a small singular value
- B. The largest singular direction is normal to the curve
- C. Both singular values are equal, because the penalty is isotropic
- D. Both singular values are zero

**Answer:** A. The largest singular value's direction is roughly tangent to the curve; the normal direction has a small singular value — The penalty pushes every direction toward zero, but reconstruction needs the code to distinguish points along the curve. The surviving sensitivity is tangent. In the figure's line model at λ = 0.1 the mean gains are 0.403 along vs 0.056 across.

**Key points**

- CAE loss: reconstruction $+\ \lambda\|J_f(x)\|_F^2$, the encoder's squared local stretch (Rifai et al., 2011).
- Sigmoid encoder: $\|J_f\|_F^2=\sum_j(h_j(1-h_j))^2\|W_j\|^2$; units contract by shrinking weights or by saturating.
- Reconstruction keeps sensitivity along the data manifold; the penalty removes it across, so top singular vectors ≈ tangent directions.
- Tie encoder and decoder weights, or the penalty can be dodged by rescaling.
- A small-σ DAE ≈ reconstruction $+\ \sigma^2\|J_r\|_F^2$: a contractive penalty on the whole reconstruction.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
