Contractive autoencoder

Penalize the encoder's Jacobian so codes ignore small input changes, except along the data manifold where reconstruction needs them.

9 min read952 wordsadvanced0 viewsupdated 2026-10-11

Definition

Contractive autoencoder (CAE) is an autoencoder trained with the extra penalty λ ∥Jf(x)∥F2\lambda\,\|J_f(x)\|_F^2, the squared Frobenius norm of the encoder's Jacobian Jf=∂h/∂xJ_f=\partial h/\partial x at each training point. The penalty makes the code locally insensitive to the input. Reconstruction pushes back only in the directions the data actually varies in, so the encoder ends up contracting across the data manifold and staying sensitive along it.

A good representation should not change much when the input changes in a way that does not matter: a pixel of sensor noise, a tiny shift off the set of plausible images. It should change when the input moves to a genuinely different example. The denoising autoencoder gets this indirectly, by training on corrupted inputs. Rifai, Vincent, Muller, Glorot and Bengio (ICML 2011) asked for it directly: measure how strongly the code reacts to an infinitesimal input change and penalize it.

The penalty

LCAE=∑x∈D(∥g(f(x))−x∥2+λ ∥Jf(x)∥F2),∥Jf(x)∥F2=∑j,i(∂hj∂xi)2\mathcal{L}_{\text{CAE}}=\sum_{x\in\mathcal D}\Big(\big\|g(f(x))-x\big\|^2+\lambda\,\big\|J_f(x)\big\|_F^2\Big),\qquad \|J_f(x)\|_F^2=\sum_{j,i}\Big(\frac{\partial h_j}{\partial x_i}\Big)^2
Jf(x)J_f(x)
dh×dd_h\times d matrix of partial derivatives of the code h=f(x)h=f(x) with respect to the input
λ\lambda
trade-off between reconstruction and insensitivity
∥⋅∥F2\|\cdot\|_F^2
sum of squared entries, equal to the sum of squared singular values: ∑ksk2\sum_k s_k^2

The Frobenius norm has a geometric reading. For a small perturbation ϵ\epsilon, f(x+ϵ)−f(x)≈Jfϵf(x+\epsilon)-f(x)\approx J_f\epsilon, and averaged over random directions E∥Jfϵ∥2=∥ϵ∥2∥Jf∥F2/d\mathbb{E}\|J_f\epsilon\|^2=\|\epsilon\|^2\|J_f\|_F^2/d. So the penalty is the average squared stretch of input perturbations. Its singular values sks_k give the stretch along each principal direction.

For a sigmoid encoder h=σ(Wx+b)h=\sigma(Wx+b) the Jacobian is cheap. Row jj is hj(1−hj) Wjh_j(1-h_j)\,W_j, the slope of unit jj times its weight vector, so

∥Jf(x)∥F2=∑j=1dh(hj(1−hj))2 ∥Wj∥2\|J_f(x)\|_F^2=\sum_{j=1}^{d_h}\big(h_j(1-h_j)\big)^2\,\|W_j\|^2
hj(1−hj)h_j(1-h_j)
σ′\sigma' of unit jj at this input: largest (0.25) at hj=0.5h_j=0.5, near 0 when the unit is saturated
∥Wj∥2\|W_j\|^2
squared norm of the unit's incoming weights

This costs about as much as the reconstruction itself, with no explicit Jacobian. It shows the two ways a unit can contract: small weights, or saturation (unit firmly on or off at this xx). Rifai et al. tie the decoder weights to the encoder, g(h)=W⊤h+cg(h)=W^\top h+c. Otherwise the encoder could shrink WW and the decoder scale back up, satisfying the penalty without learning anything.

Why contraction is anisotropic

The penalty alone would contract every direction until the code is constant. Reconstruction forbids that, but only for directions in which training points differ. If the data lies on a low-dimensional manifold, moving along it changes xx to another training example that must be reconstructed differently, so the code must stay sensitive there. Moving across it reaches no training points at all, so reconstruction does not care and the penalty wins. The result: the leading singular vectors of JfJ_f line up with the manifold's tangent space, and the others shrink toward 0.

Rifai et al. used exactly this, reading off the top singular vectors of JfJ_f as estimated tangent directions (the "manifold tangent classifier", 2011). In the reconstruction the same effect makes r=g∘fr=g\circ f project off-manifold inputs back onto the manifold, like a denoiser.

perturbation circles → image under J_r- - inputs ±0.3 off the manifold — their reconstructionsprobe 4 at t = 0.1J_f singular values 1.181, 0.817|cos(v₁, tangent)| = 0.280‖J_f‖²_F closed form 2.0619 Σ entries² of J_f 2.0619‖J_r u‖ along / across 3.066 / 5.598mean gain over the 40 training pointsJ_f along0.687J_f across1.025J_r along2.395J_r across4.859recon error 10.58604mean ‖J_f‖²_F 1.5507loss = recon + λ‖J‖² 10.74112Adam step 0/3000ellipse: x + ε J_r u for ‖u‖ = 1cross: J_f singular directionsclick a probe circle to inspect it
manifold|
fig.A tied-weight CAE (2 inputs → 8 sigmoid units → 2) trained live by full-batch Adam on 40 points of a 1-D manifold. Dashed circles are input perturbations; mint ellipses are their images under the reconstruction Jacobian J_r. With λ = 0 on the line, the map stretches perturbations across the manifold about 5× (mean across-gain 5.08). At λ = 0.1 the ellipses flatten onto the line (across 0.098, along 0.939), and the dashed bands ±0.3 off the manifold are reconstructed almost onto it. On the sine wave, one layer of ridge units cannot follow a turning tangent everywhere, so contraction is only partial.

Worked example: the closed form

Two sigmoid units, W = [[1, 2], [−1, 0.5]], b = 0, x = (0.5, −0.25)

  1. 1Pre-activations: a1=0.5−0.5=0a_1=0.5-0.5=0, a2=−0.5−0.125=−0.625a_2=-0.5-0.125=-0.625. Activations: h=(0.5,  0.3486)h=(0.5,\;0.3486).
  2. 2Slopes: h1(1−h1)=0.25h_1(1-h_1)=0.25, h2(1−h2)=0.2271h_2(1-h_2)=0.2271. Weight norms: ∥W1∥2=5\|W_1\|^2=5, ∥W2∥2=1.25\|W_2\|^2=1.25.
  3. 3∥Jf∥F2=0.252⋅5+0.22712⋅1.25=0.3125+0.0645=0.3770\|J_f\|_F^2=0.25^2\cdot5+0.2271^2\cdot1.25=0.3125+0.0645=0.3770.
  4. 4Check: Jf=[0.250.5−0.22710.1135]J_f=\begin{bmatrix}0.25&0.5\\-0.2271&0.1135\end{bmatrix} (rows == slope ×Wj\times W_j). Its squared entries sum to 0.37700.3770, as does a finite-difference Jacobian.
  5. 5Unit 1 sits at h=0.5h=0.5, where the slope is maximal, and has the bigger weights, so it contributes 83% of the penalty. Training will make it saturate here or shrink its weights unless reconstruction needs its sensitivity.
ManifoldλRecon errormean ‖J_f‖²J_f along / acrossJ_r along / across
line00.000221.3940.461 / 1.0860.968 / 5.082
line0.010.000330.4010.439 / 0.4550.966 / 1.244
line0.10.000560.1680.403 / 0.0560.939 / 0.098
line10.020140.0740.252 / 0.0080.928 / 0.014
sine wave00.000200.3900.447 / 0.4340.992 / 0.955
sine wave0.10.002440.2030.366 / 0.2430.895 / 0.601
sine wave10.036820.0700.190 / 0.1430.724 / 0.525
The figure's model after 3000 Adam steps (same seed). Gains are ‖J u‖ for the unit tangent u and the unit normal, averaged over the 40 training points.

Read the line rows top to bottom. Without the penalty the reconstruction is nearly perfect on the data, but it amplifies off-manifold perturbations fivefold: nothing in the loss constrains those directions. A small λ removes that. At λ = 0.1 the code is 7× more sensitive along the line than across it, and the reconstruction error is still tiny. At λ = 1 even the along-direction gain of the encoder starts to fall and reconstruction suffers. The sine wave shows the limit of a one-layer encoder: each unit's sensitivity direction WjW_j is fixed, so the code cannot align with a tangent that keeps turning.

Relation to the denoising autoencoder

Expand a DAE's loss for small Gaussian noise. With x~=x+σε\tilde x=x+\sigma\varepsilon and r(x+σε)≈r(x)+σJrεr(x+\sigma\varepsilon)\approx r(x)+\sigma J_r\varepsilon,

Eε∥r(x+σε)−x∥2  ≈  ∥r(x)−x∥2+σ2 ∥Jr(x)∥F2\mathbb{E}_\varepsilon\big\|r(x+\sigma\varepsilon)-x\big\|^2\;\approx\;\|r(x)-x\|^2+\sigma^2\,\|J_r(x)\|_F^2
Jr=∂r/∂xJ_r=\partial r/\partial x
Jacobian of the whole reconstruction, not just the encoder
σ2\sigma^2
plays the role of λ

So a small-noise DAE is a contractive penalty on the reconstruction (Alain & Bengio, 2014), while the CAE penalizes the encoder, which is what you keep as the representation. In the figure's sine-wave model at λ = 0.1, a Monte Carlo estimate of the DAE loss at σ = 0.05 is 0.005376, against 0.005388 from the right-hand side. The CAE penalty is analytic and deterministic, so it needs no noise sampling, but it only sees infinitesimal perturbations. Rifai et al.'s CAE+H adds a stochastic penalty on the change of JfJ_f between nearby points to reach a little further.

Training a CAE

  1. 1Encoder h=σ(Wx+b)h=\sigma(Wx+b), tied decoder x^=W⊤h+c\hat x=W^\top h+c (or a sigmoid output for data in [0,1][0,1]).
  2. 2For each batch compute the reconstruction loss and ∑j(hj(1−hj))2∥Wj∥2\sum_j (h_j(1-h_j))^2\|W_j\|^2 per example; add λ\lambda times its mean.
  3. 3Backpropagate both terms. Autograd handles the penalty because it is an ordinary function of hh and WW.
  4. 4Tune λ\lambda on a validation metric (downstream accuracy, or reconstruction at a target contraction). Track the singular values of JfJ_f at a few points: a sharp drop after the first few indicates a low-dimensional manifold.
python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3class CAE(nn.Module):
4 def __init__(self, d=784, h=1000):
5 super().__init__()
6 self.W = nn.Parameter(torch.randn(h, d) * 0.01)
7 self.b = nn.Parameter(torch.zeros(h))
8 self.c = nn.Parameter(torch.zeros(d))
9 def forward(self, x):
10 h = torch.sigmoid(x @ self.W.T + self.b)
11 x_hat = torch.sigmoid(h @ self.W + self.c) # tied decoder
12 return x_hat, h
13
14def cae_loss(model, x, lam=0.1):
15 x_hat, h = model(x)
16 recon = F.binary_cross_entropy(x_hat, x, reduction="sum") / len(x)
17 slope2 = (h * (1 - h)) ** 2 # (B, h)
18 jac = (slope2 * model.W.pow(2).sum(1)).sum(1).mean() # Σ_j (h_j(1-h_j))² ‖W_j‖²
19 return recon + lam * jac
20
21# General encoders: estimate ||J||_F^2 with random projections, E_v ||J^T v||^2 for v ~ N(0, I),
22# using torch.autograd.functional.vjp, or penalise the exact Jacobian for small inputs.
Contractive autoencoder with the closed-form sigmoid penalty and tied weights

Use when

  • +Deterministic, analytic regularizer: no noise distribution to choose, no sampling variance.
  • +Its Jacobian's top singular vectors estimate the data manifold's tangent space, which is useful for tangent-propagation-style classifiers.
  • +Explicitly targets robustness of the features you keep (the encoder), not just of the reconstruction.

Avoid when

  • −Cheap only for one-layer sigmoid encoders. For deep encoders the exact Jacobian needs one backward pass per input (or code) dimension, so it is usually estimated stochastically.
  • −Only constrains infinitesimal perturbations; larger corruptions need CAE+H or a DAE.
  • −Largely superseded in practice by denoising, masking and contrastive objectives, though Jacobian penalties remain common in robustness work.

Check yourself

A sigmoid unit has weights WjW_j with ∥Wj∥2=4\|W_j\|^2=4. On one input it outputs hj=0.9h_j=0.9. What does it contribute to ∥Jf∥F2\|J_f\|_F^2 there?

Check yourself

Data lie on a curve in 2-D. After CAE training, how should the encoder Jacobian's singular directions relate to the curve at a training point?

Remember

  • ✓CAE loss: reconstruction + λ∥Jf(x)∥F2+\ \lambda\|J_f(x)\|_F^2, the encoder's squared local stretch (Rifai et al., 2011).
  • ✓Sigmoid encoder: ∥Jf∥F2=∑j(hj(1−hj))2∥Wj∥2\|J_f\|_F^2=\sum_j(h_j(1-h_j))^2\|W_j\|^2; units contract by shrinking weights or by saturating.
  • ✓Reconstruction keeps sensitivity along the data manifold; the penalty removes it across, so top singular vectors ≈ tangent directions.
  • ✓Tie encoder and decoder weights, or the penalty can be dodged by rescaling.
  • ✓A small-σ DAE ≈ reconstruction + σ2∥Jr∥F2+\ \sigma^2\|J_r\|_F^2: a contractive penalty on the whole reconstruction.
← back to Deep Learning roadmap