Definition
Contractive autoencoder (CAE) is an autoencoder trained with the extra penalty , the squared Frobenius norm of the encoder's Jacobian at each training point. The penalty makes the code locally insensitive to the input. Reconstruction pushes back only in the directions the data actually varies in, so the encoder ends up contracting across the data manifold and staying sensitive along it.
A good representation should not change much when the input changes in a way that does not matter: a pixel of sensor noise, a tiny shift off the set of plausible images. It should change when the input moves to a genuinely different example. The denoising autoencoder gets this indirectly, by training on corrupted inputs. Rifai, Vincent, Muller, Glorot and Bengio (ICML 2011) asked for it directly: measure how strongly the code reacts to an infinitesimal input change and penalize it.
The penalty
- matrix of partial derivatives of the code with respect to the input
- trade-off between reconstruction and insensitivity
- sum of squared entries, equal to the sum of squared singular values:
The Frobenius norm has a geometric reading. For a small perturbation , , and averaged over random directions . So the penalty is the average squared stretch of input perturbations. Its singular values give the stretch along each principal direction.
For a sigmoid encoder the Jacobian is cheap. Row is , the slope of unit times its weight vector, so
- of unit at this input: largest (0.25) at , near 0 when the unit is saturated
- squared norm of the unit's incoming weights
This costs about as much as the reconstruction itself, with no explicit Jacobian. It shows the two ways a unit can contract: small weights, or saturation (unit firmly on or off at this ). Rifai et al. tie the decoder weights to the encoder, . Otherwise the encoder could shrink and the decoder scale back up, satisfying the penalty without learning anything.
Why contraction is anisotropic
The penalty alone would contract every direction until the code is constant. Reconstruction forbids that, but only for directions in which training points differ. If the data lies on a low-dimensional manifold, moving along it changes to another training example that must be reconstructed differently, so the code must stay sensitive there. Moving across it reaches no training points at all, so reconstruction does not care and the penalty wins. The result: the leading singular vectors of line up with the manifold's tangent space, and the others shrink toward 0.
Rifai et al. used exactly this, reading off the top singular vectors of as estimated tangent directions (the "manifold tangent classifier", 2011). In the reconstruction the same effect makes project off-manifold inputs back onto the manifold, like a denoiser.
Worked example: the closed form
Two sigmoid units, W = [[1, 2], [−1, 0.5]], b = 0, x = (0.5, −0.25)
- 1Pre-activations: , . Activations: .
- 2Slopes: , . Weight norms: , .
- 3.
- 4Check: (rows slope ). Its squared entries sum to , as does a finite-difference Jacobian.
- 5Unit 1 sits at , where the slope is maximal, and has the bigger weights, so it contributes 83% of the penalty. Training will make it saturate here or shrink its weights unless reconstruction needs its sensitivity.
| Manifold | λ | Recon error | mean ‖J_f‖² | J_f along / across | J_r along / across |
|---|---|---|---|---|---|
| line | 0 | 0.00022 | 1.394 | 0.461 / 1.086 | 0.968 / 5.082 |
| line | 0.01 | 0.00033 | 0.401 | 0.439 / 0.455 | 0.966 / 1.244 |
| line | 0.1 | 0.00056 | 0.168 | 0.403 / 0.056 | 0.939 / 0.098 |
| line | 1 | 0.02014 | 0.074 | 0.252 / 0.008 | 0.928 / 0.014 |
| sine wave | 0 | 0.00020 | 0.390 | 0.447 / 0.434 | 0.992 / 0.955 |
| sine wave | 0.1 | 0.00244 | 0.203 | 0.366 / 0.243 | 0.895 / 0.601 |
| sine wave | 1 | 0.03682 | 0.070 | 0.190 / 0.143 | 0.724 / 0.525 |
Read the line rows top to bottom. Without the penalty the reconstruction is nearly perfect on the data, but it amplifies off-manifold perturbations fivefold: nothing in the loss constrains those directions. A small λ removes that. At λ = 0.1 the code is 7× more sensitive along the line than across it, and the reconstruction error is still tiny. At λ = 1 even the along-direction gain of the encoder starts to fall and reconstruction suffers. The sine wave shows the limit of a one-layer encoder: each unit's sensitivity direction is fixed, so the code cannot align with a tangent that keeps turning.
Relation to the denoising autoencoder
Expand a DAE's loss for small Gaussian noise. With and ,
- Jacobian of the whole reconstruction, not just the encoder
- plays the role of λ
So a small-noise DAE is a contractive penalty on the reconstruction (Alain & Bengio, 2014), while the CAE penalizes the encoder, which is what you keep as the representation. In the figure's sine-wave model at λ = 0.1, a Monte Carlo estimate of the DAE loss at σ = 0.05 is 0.005376, against 0.005388 from the right-hand side. The CAE penalty is analytic and deterministic, so it needs no noise sampling, but it only sees infinitesimal perturbations. Rifai et al.'s CAE+H adds a stochastic penalty on the change of between nearby points to reach a little further.
Training a CAE
- 1Encoder , tied decoder (or a sigmoid output for data in ).
- 2For each batch compute the reconstruction loss and per example; add times its mean.
- 3Backpropagate both terms. Autograd handles the penalty because it is an ordinary function of and .
- 4Tune on a validation metric (downstream accuracy, or reconstruction at a target contraction). Track the singular values of at a few points: a sharp drop after the first few indicates a low-dimensional manifold.
1import torch, torch.nn as nn, torch.nn.functional as F23class CAE(nn.Module):4 def __init__(self, d=784, h=1000):5 super().__init__()6 self.W = nn.Parameter(torch.randn(h, d) * 0.01)7 self.b = nn.Parameter(torch.zeros(h))8 self.c = nn.Parameter(torch.zeros(d))9 def forward(self, x):10 h = torch.sigmoid(x @ self.W.T + self.b)11 x_hat = torch.sigmoid(h @ self.W + self.c) # tied decoder12 return x_hat, h1314def cae_loss(model, x, lam=0.1):15 x_hat, h = model(x)16 recon = F.binary_cross_entropy(x_hat, x, reduction="sum") / len(x)17 slope2 = (h * (1 - h)) ** 2 # (B, h)18 jac = (slope2 * model.W.pow(2).sum(1)).sum(1).mean() # Σ_j (h_j(1-h_j))² ‖W_j‖²19 return recon + lam * jac2021# General encoders: estimate ||J||_F^2 with random projections, E_v ||J^T v||^2 for v ~ N(0, I),22# using torch.autograd.functional.vjp, or penalise the exact Jacobian for small inputs.Use when
- +Deterministic, analytic regularizer: no noise distribution to choose, no sampling variance.
- +Its Jacobian's top singular vectors estimate the data manifold's tangent space, which is useful for tangent-propagation-style classifiers.
- +Explicitly targets robustness of the features you keep (the encoder), not just of the reconstruction.
Avoid when
- −Cheap only for one-layer sigmoid encoders. For deep encoders the exact Jacobian needs one backward pass per input (or code) dimension, so it is usually estimated stochastically.
- −Only constrains infinitesimal perturbations; larger corruptions need CAE+H or a DAE.
- −Largely superseded in practice by denoising, masking and contrastive objectives, though Jacobian penalties remain common in robustness work.
Check yourself
A sigmoid unit has weights with . On one input it outputs . What does it contribute to there?
Check yourself
Data lie on a curve in 2-D. After CAE training, how should the encoder Jacobian's singular directions relate to the curve at a training point?
Remember
- ✓CAE loss: reconstruction , the encoder's squared local stretch (Rifai et al., 2011).
- ✓Sigmoid encoder: ; units contract by shrinking weights or by saturating.
- ✓Reconstruction keeps sensitivity along the data manifold; the penalty removes it across, so top singular vectors ≈ tangent directions.
- ✓Tie encoder and decoder weights, or the penalty can be dodged by rescaling.
- ✓A small-σ DAE ≈ reconstruction : a contractive penalty on the whole reconstruction.