Autoencoder anomaly detection

Train an autoencoder on normal data only; inputs it reconstructs badly are anomalies.

16 min read1484 wordscore0 viewsupdated 2026-10-08

Definition

Autoencoder anomaly detection trains a neural network to copy its input to its output through a narrow bottleneck, using only normal data. The network can only afford to remember the structure normal data shares, so normal inputs come back almost unchanged while unusual ones come back distorted. The reconstruction error ∥x−x^∥2\lVert x-\hat x\rVert^2 is the anomaly score, and a threshold on it gives the alarm.

At a glance

Task
Unsupervised (or semi-supervised) anomaly / novelty detection
Input → output
x∈Rdx\in\mathbb{R}^d → score s(x)=∥x−x^∥2s(x)=\lVert x-\hat x\rVert^2, and a flag s(x)>τs(x)>\tau
Learns
an encoder x↦z∈Rkx\mapsto z\in\mathbb{R}^k and a decoder z↦x^z\mapsto\hat x, with k<dk<d: in effect a kk-dimensional surface (manifold) that normal data lies near
Training
gradient descent (backprop) on the mean squared reconstruction error of normal examples; no anomaly labels needed
Prediction
one forward pass per point, then compare its error to the threshold
Hyperparameters
bottleneck size kk, depth/width, activation, regularization (weight decay, dropout, noise), epochs, the threshold τ\tau
Assumes
the training set is (almost) all normal, and normal data occupies a lower-dimensional structure that anomalies fall off

The problem

Anomalies are rare and varied. A card-fraud team might have a few hundred confirmed frauds among millions of transactions, and next month's fraud will look different from last month's. A classifier trained on labelled anomalies only learns the kinds it has already seen. What we do have in abundance is normal data. So instead of describing anomalies, we describe normality, as tightly as we can, and flag whatever does not fit.

Density models (Gaussians, elliptic envelopes) do this well in a few dimensions, but they struggle with images, spectra or hundreds of correlated sensor channels. There, normal data usually lies near a curved, low-dimensional surface inside the huge input space: neighbouring pixels move together, and sensor readings are tied together by physics. An autoencoder learns that surface directly.

The idea

Force the data through a narrow pipe. The encoder squeezes each dd-dimensional input into a code zz with only k<dk<d numbers; the decoder rebuilds the input from that code. Training minimizes the rebuilding error on normal data, so the code is pushed to keep exactly the directions in which normal points differ from each other, and to drop everything else.

An anomaly differs from normal data in the dropped directions: that is what makes it unusual. The encoder has no slot for that difference, so the decoder rebuilds the nearest normal-looking point instead. The gap between the input and that rebuilt point is the score. Geometrically, x^\hat x is a point on the learned manifold and ∥x−x^∥\lVert x-\hat x\rVert is (roughly) how far xx sits from it.

Vocabulary

Encoder
fθ:Rd→Rkf_\theta: \mathbb{R}^d\to\mathbb{R}^k, maps an input to its code
Code / latent / bottleneck
z=fθ(x)z=f_\theta(x), the kk numbers that must carry everything the decoder gets to see
Decoder
gϕ:Rk→Rdg_\phi:\mathbb{R}^k\to\mathbb{R}^d, maps a code back to input space; its image is the learned manifold
Reconstruction
x^=gϕ(fθ(x))\hat x=g_\phi(f_\theta(x))
Reconstruction error
∥x−x^∥2\lVert x-\hat x\rVert^2; averaged over the training set it is the loss, and for one point it is the anomaly score
Undercomplete
k<dk<d: the bottleneck itself prevents copying. Overcomplete autoencoders (k≥dk\ge d) need other regularization or they learn the identity
Threshold τ
the error above which a point is called an anomaly; set from the error distribution of normal data

The math

We use a single hidden (code) layer, which is all the worked example needs; deep autoencoders just stack more layers on each side and backprop through them the same way.

z=g(We x+be),x^=Wd z+bdz=g\big(W_e\,x+b_e\big),\qquad \hat x=W_d\,z+b_d
x∈Rdx\in\mathbb{R}^d
input (standardized features)
We∈Rk×d, be∈RkW_e\in\mathbb{R}^{k\times d},\ b_e\in\mathbb{R}^k
encoder weights and bias
gg
activation, applied elementwise (identity for a linear autoencoder; tanh or ReLU otherwise)
z∈Rkz\in\mathbb{R}^k
the code, with k<dk<d
Wd∈Rd×k, bd∈RdW_d\in\mathbb{R}^{d\times k},\ b_d\in\mathbb{R}^d
decoder weights and bias
x^\hat x
reconstruction of x

Training minimizes the mean squared reconstruction error over the nn normal training points:

J(θ)=1n∑i=1nei,ei=∥x^i−xi∥2=∑j=1d(x^ij−xij)2J(\theta)=\frac1n\sum_{i=1}^n e_i,\qquad e_i=\lVert \hat x_i-x_i\rVert^2=\sum_{j=1}^d(\hat x_{ij}-x_{ij})^2
θ\theta
all parameters (We,be,Wd,bd)(W_e,b_e,W_d,b_d)
eie_i
squared reconstruction error of point ii; the same quantity is the anomaly score

Some libraries divide by dd as well ("MSE per feature"). That rescales every score by the same constant, so it changes the threshold's units but never which points are flagged.

Backprop for the tiny network

For one point, write the residual r=x^−xr=\hat x-x and the encoder's pre-activation a=Wex+bea=W_ex+b_e. The chain rule, applied from the loss backwards, gives every gradient in four lines:

∂e∂x^=2r,∂e∂Wd=2r z⊤,∂e∂bd=2r\frac{\partial e}{\partial \hat x}=2r,\qquad \frac{\partial e}{\partial W_d}=2r\,z^\top,\qquad \frac{\partial e}{\partial b_d}=2r
r=x^−xr=\hat x-x
residual: where the reconstruction misses
2r z⊤2r\,z^\top
outer product, a d×kd\times k matrix like WdW_d
δ=∂e∂a=(Wd⊤ 2r)⊙g′(a),∂e∂We=δ x⊤,∂e∂be=δ\delta=\frac{\partial e}{\partial a}=\big(W_d^\top\,2r\big)\odot g'(a),\qquad \frac{\partial e}{\partial W_e}=\delta\,x^\top,\qquad \frac{\partial e}{\partial b_e}=\delta
δ\delta
error signal at the code layer: the residual sent back through the decoder weights
⊙\odot
elementwise product
g′(a)g'(a)
activation slope; equals 1 for a linear autoencoder

The batch gradient of JJ is the average of these per-point gradients, and each step updates θ←θ−η ∇θJ\theta\leftarrow\theta-\eta\,\nabla_\theta J with learning rate η\eta. Notice that the decoder is told where the output missed (rr), and the encoder is told how its code should have moved (δ\delta).

Scoring and the threshold

s(x)=∥x−x^∥2,flag x as anomalous  ⟺  s(x)>τ,τ=Qq({s(xi)}normal)s(x)=\lVert x-\hat x\rVert^2,\qquad \text{flag } x \text{ as anomalous} \iff s(x)>\tau,\qquad \tau=Q_{q}\big(\{s(x_i)\}_{\text{normal}}\big)
s(x)s(x)
anomaly score: the reconstruction error of the trained network
τ\tau
decision threshold
QqQ_q
the qq-th percentile (e.g. 95th or 99th) of the scores of normal points

The percentile is a direct dial on the false-alarm rate: at the 99th percentile about 1% of normal traffic will be flagged. Training errors are optimistic, because the network has fitted those exact points, so it is better to compute the percentile on a held-out set of normal data. If a validation set with some labelled anomalies exists, pick τ\tau to optimize what you care about: recall at a fixed false-positive rate, F1, or expected cost (see precision, recall and ROC). Rules such as mean + 3+\,3 standard deviations are common too, but error distributions are skewed, so percentiles are safer.

A linear autoencoder is PCA

Take gg = identity. Then x^=WdWex+(Wdbe+bd)\hat x=W_dW_e x+(W_db_e+b_d): the reconstruction is an affine map of rank at most kk, so all reconstructions lie in a kk-dimensional flat. Minimizing squared error over rank-kk maps is exactly the problem PCA solves, so (Baldi and Hornik, 1989) at the global minimum:

x^=xˉ+UkUk⊤(x−xˉ),min⁡θJ=n−1n∑j>kλj\hat x=\bar x+U_kU_k^\top(x-\bar x),\qquad \min_\theta J=\frac{n-1}{n}\sum_{j>k}\lambda_j
xˉ\bar x
mean of the training data; the biases learn to pass the flat through it
UkU_k
the top-kk eigenvectors of the covariance matrix (principal components)
λj\lambda_j
eigenvalues of the covariance; the minimum loss is the variance in the dropped directions

The span of WdW_d equals the principal subspace, but the weights themselves need not be orthonormal (any invertible k×kk\times k mixing Wd→WdAW_d\to W_dA, We→A−1WeW_e\to A^{-1}W_e gives the same reconstruction). The anomaly score is then the squared distance to the principal subspace, which process engineers have used for decades as the SPE or Q-statistic. So a linear autoencoder is just a slow way to compute PCA; the reason to train a network is the nonlinear gg, which lets the manifold bend.

Worked example: a 2 → 1 → 2 autoencoder

Eight normal points lie near the line x2≈x1/2x_2\approx x_1/2, a 1-D manifold in 2-D. We train a linear autoencoder with a one-number bottleneck, then score four new points. Every number below was computed with NumPy and matches the interactive figure.

Pointxzx̂error ‖x − x̂‖²
A(−3, −1.6)−3.211(−3.046, −1.517)0.0090
B(−2, −0.8)−2.022(−1.917, −0.952)0.0301
C(−1, −0.6)−1.104(−1.046, −0.517)0.0090
D(0, 0.2)0.085(0.083, 0.048)0.0301
E(0.5, 0.1)0.454(0.433, 0.223)0.0196
F(1, 0.4)1.004(0.954, 0.483)0.0090
G(2, 1.2)2.193(2.083, 1.048)0.0301
H(3, 1.4)3.111(2.954, 1.483)0.0090
Training data (normal only), and the trained model's code, reconstruction and error for each point.

Initialization (fixed, deliberately poor): We=(0.2, 0.4)W_e=(0.2,\,0.4), be=0b_e=0, Wd=(0.6, −0.2)⊤W_d=(0.6,\,-0.2)^\top, bd=(0,0)b_d=(0,0). The decoder line {bd+t Wd}\{b_d+t\,W_d\} slopes down while the data slopes up, and the mean error is J=3.247J=3.247.

QuantityFormulaValue
codez=Wex+be=0.2⋅2+0.4⋅1.2z=W_e x+b_e=0.2\cdot2+0.4\cdot1.20.88
reconstructionx^=Wdz+bd=(0.6, −0.2)⋅0.88\hat x=W_d z+b_d=(0.6,\,-0.2)\cdot0.88(0.528, −0.176)
residualr=x^−xr=\hat x-x(−1.472, −1.376)
errore=∥r∥2e=\lVert r\rVert^24.0602
decoder weight grad∂e/∂Wd=2r z\partial e/\partial W_d=2r\,z(−2.5907, −2.4218)
decoder bias grad∂e/∂bd=2r\partial e/\partial b_d=2r(−2.944, −2.752)
code error signalδ=2Wd⊤r=2 (0.6⋅(−1.472)+(−0.2)⋅(−1.376))\delta=2W_d^\top r=2\,(0.6\cdot(-1.472)+(-0.2)\cdot(-1.376))−1.216
encoder weight grad∂e/∂We=δ x\partial e/\partial W_e=\delta\,x(−2.432, −1.4592)
encoder bias grad∂e/∂be=δ\partial e/\partial b_e=\delta−1.216
after one stepθ←θ−0.05 ∇θe\theta\leftarrow\theta-0.05\,\nabla_\theta eWe=(0.3216, 0.4730)W_e=(0.3216,\,0.4730), be=0.0608b_e=0.0608, Wd=(0.7295, −0.0789)W_d=(0.7295,\,-0.0789), bd=(0.1472, 0.1376)b_d=(0.1472,\,0.1376)
new error on Gz=1.2716z=1.2716, x^=(1.0748, 0.0373)\hat x=(1.0748,\,0.0373)2.2079 (was 4.0602)
One forward and backward pass on G = (2, 1.2) with the initial weights, then one gradient step on G alone (η = 0.05).

Every gradient is negative, so every weight grows: the step stretches the decoder towards G and tilts it upward (WdW_d's second entry goes from −0.2-0.2 to −0.079-0.079). Real training averages the same gradients over all eight points per step. With full-batch gradient descent, η=0.05\eta=0.05, 1000 epochs:

EpochMean error JDecoder slopeEncoder–decoder angle
03.2472−0.33381.9°
12.4651−0.16764.5°
30.72210.12938.3°
50.11720.33024.5°
100.02050.48416.4°
1000.01920.50112.9°
10000.01820.5002.0°
Training the 2→1→2 linear autoencoder by full-batch gradient descent. The fast phase aligns the line in about 10 epochs; the slow phase turns the oblique projection into an orthogonal one.

Final weights: We=(0.828, 0.452)W_e=(0.828,\,0.452), be=−0.005b_e=-0.005, Wd=(0.949, 0.475)⊤W_d=(0.949,\,0.475)^\top, bd=(0.002, 0.007)b_d=(0.002,\,0.007). Check against PCA: the data mean is (0.0625, 0.0375)(0.0625,\,0.0375), the covariance has eigenvalues 5.0335.033 and 0.02080.0208 (the first component keeps 99.6% of the variance), and the first eigenvector has slope 0.49980.4998. The autoencoder's line has slope 0.50000.5000 and passes through the mean, and its loss 0.018240.01824 is within 0.2% of PCA's optimum 78λ2=0.01822\tfrac78\lambda_2=0.01822. The per-point errors also match PCA's to within 0.00020.0002.

Threshold. The 95th percentile of the eight training errors is τ=0.0301\tau=0.0301. (With only eight points that is essentially the maximum; with real data use thousands of held-out normal points.) Now score new points:

Pointxzx̂scorescore / τverdict
N(1.5, 0.8)1.598(1.519, 0.766)0.00150.05normal
P(0.5, 0.9)0.816(0.776, 0.394)0.33211anomaly (subtle)
Q(−1, 1.5)−0.155(−0.146, −0.067)3.184106anomaly
R(2, −1)1.199(1.140, 0.576)3.224107anomaly
Held-out points scored by the trained autoencoder, τ = 0.0301.

N lies on the manifold and reconstructs better than most training points. P, Q and R have ordinary-looking coordinates one at a time: their x1x_1 and x2x_2 values are all inside the training range. What gives them away is the combination, the component perpendicular to the line, which the one-number code cannot carry. The decoder snaps each one back onto the line, and the gap is the score.

▸ training a 2→1→2 autoencoderstep 1/27
-3-2-10123-2-1012x₁ →x₂ABCDEFGH● x ○ x̂ — error — decoder line2 → 1 → 2 network (linear)0.200.400.60-0.20x₁x₂zbe 0.000x̂₁x̂₂bd (0.000, 0.000)mean error (log) vs epoch (log)0.020.111101001000PCA optimum
epoch 0loss 3.2472slope -0.333∠(We, Wd) 81.9°η 0.05
  1. 1init We=(.2,.4) be=0 Wd=(.6,−.2) bd=0
  2. 2for epoch = 1 … 1000:
  3. 3 z = We·x + be # encode → 1 number
  4. 4 x̂ = Wd·z + bd # decode → back to 2-D
  5. 5 L = mean ‖x̂ − x‖²
  6. 6 backprop ∂L/∂Wd, ∂L/∂bd, ∂L/∂We, ∂L/∂be
  7. 7 θ ← θ − η·∇θL (η = 0.05)
  8. 8τ = 95th percentile of training errors

Fixed start. The decoder line {bd + t·Wd} points along Wd = (0.60, −0.20): the wrong way, so every reconstruction lands far from its point: the mean error is 3.247.

fig.① Training: the green decoder line swings onto the data in about 10 epochs, then the red error segments slowly turn perpendicular as the encoder direction rotates onto the decoder's. The network diagram shows the four weights, and the loss curve approaches the PCA optimum. ② Scoring: drag the test point T, move the threshold τ (log scale), and see which points fall outside the green tube where error ≤ τ. "Leak Q into training" retrains with an anomaly in the training set: the line shifts toward Q and the p95 threshold jumps from 0.030 to 1.55, so P is missed.

The algorithm, step by step

Autoencoder anomaly detection

  1. 1Collect normal data and remove known anomalies. Split it into train and a held-out normal set (plus a labelled validation set, if you have one).
  2. 2Scale each feature using statistics from the training split only (standardize, or min-max for images).
  3. 3Choose an architecture with a real bottleneck: kk well below dd. Start small; MLP for tabular data, convolutional for images, LSTM/1-D conv/transformer for windows of a time series.
  4. 4Train on normal data to minimize mean reconstruction error, with early stopping on the held-out normal loss.
  5. 5Set the threshold: a high percentile of the held-out normal scores, or the value that best trades recall against false alarms on the labelled validation set.
  6. 6Score each new input with one forward pass; flag s(x)>τs(x)>\tau. Inspect the per-feature residuals (xj−x^j)2(x_j-\hat x_j)^2 to see what is abnormal.
  7. 7Monitor: recompute the score distribution on recent normal traffic and retrain when normal behaviour drifts.

In code

python
1import numpy as np
2
3X = np.array([[-3,-1.6],[-2,-0.8],[-1,-0.6],[0,0.2],[0.5,0.1],[1,0.4],[2,1.2],[3,1.4]])
4We, be = np.array([0.2, 0.4]), 0.0 # encoder 2 -> 1 (linear)
5Wd, bd = np.array([0.6, -0.2]), np.zeros(2) # decoder 1 -> 2
6eta = 0.05
7
8for epoch in range(1000):
9 Z = X @ We + be # codes, shape (n,)
10 Xh = np.outer(Z, Wd) + bd # reconstructions, shape (n, 2)
11 R = Xh - X # residuals
12 D = 2 * R @ Wd # delta at the code layer
13 Wd -= eta * 2 * (R * Z[:, None]).mean(0)
14 bd -= eta * 2 * R.mean(0)
15 We -= eta * (D[:, None] * X).mean(0)
16 be -= eta * D.mean()
17
18score = lambda x: (((x @ We + be)[..., None] * Wd + bd - x) ** 2).sum(-1)
19tau = np.percentile(score(X), 95) # 0.0301
20test = np.array([[1.5, 0.8], [0.5, 0.9], [-1, 1.5], [2, -1]])
21print(score(test).round(4), score(test) > tau) # [0.0015 0.332 3.1841 3.2241] [F T T T]
From scratch in NumPy: the worked example above (reproduces every number in the tables).
python
1import numpy as np, torch, torch.nn as nn
2from sklearn.preprocessing import StandardScaler
3
4# X_normal: (n, d) normal rows only. X_new: rows to score.
5X_tr, X_val = X_normal[:int(.8 * len(X_normal))], X_normal[int(.8 * len(X_normal)):]
6scaler = StandardScaler().fit(X_tr) # fit on training normals only
7t = lambda a: torch.tensor(scaler.transform(a), dtype=torch.float32)
8Xtr, Xval = t(X_tr), t(X_val)
9
10d, k = Xtr.shape[1], 4
11model = nn.Sequential(
12 nn.Linear(d, 32), nn.ReLU(), nn.Linear(32, k), # encoder -> bottleneck
13 nn.Linear(k, 32), nn.ReLU(), nn.Linear(32, d), # decoder
14)
15opt = torch.optim.Adam(model.parameters(), lr=1e-3, weight_decay=1e-5)
16loader = torch.utils.data.DataLoader(Xtr, batch_size=256, shuffle=True)
17
18def scores(x):
19 model.eval()
20 with torch.no_grad():
21 return ((model(x) - x) ** 2).mean(dim=1).numpy() # per-row MSE = anomaly score
22
23best, patience = np.inf, 0
24for epoch in range(200):
25 model.train()
26 for xb in loader:
27 loss = ((model(xb) - xb) ** 2).mean()
28 opt.zero_grad(); loss.backward(); opt.step()
29 val = scores(Xval).mean() # early stopping on normal data
30 if val < best - 1e-5: best, patience, state = val, 0, model.state_dict()
31 elif (patience := patience + 1) >= 10: break
32model.load_state_dict(state)
33
34tau = np.percentile(scores(Xval), 99) # ~1% false alarms on normals
35s_new = scores(t(X_new))
36is_anomaly = s_new > tau
PyTorch: a nonlinear autoencoder for tabular data, trained on normal rows, with a held-out threshold.

Use when

  • +You have lots of normal data and few or no labelled anomalies.
  • +Data is high-dimensional and structured (images, spectra, multichannel sensors, sequences), where density estimates break down.
  • +You want an explanation: per-feature or per-pixel residuals show where the input disagrees with normal.
  • +Scoring must be fast: one forward pass, easy to deploy on GPUs or edge devices.

Avoid when

  • −Tabular data with a few dozen features: Isolation Forest or PCA are usually as good, much cheaper and need less tuning.
  • −The training data may be contaminated with anomalies: the network learns to reconstruct them too.
  • −You need a calibrated probability: the score is an error in feature units, not a likelihood.
  • −Hyperparameters (bottleneck, depth, epochs) are hard to choose without some labelled anomalies to validate on.

Where it's used

Fraud and abuse

Reconstruct transaction or session features; unusual combinations of amount, merchant, time and device reconstruct badly.

Industrial sensors

Reconstruct windows of vibration, temperature and pressure signals; a rising error warns of a failing bearing or pump before it breaks (predictive maintenance).

Visual inspection

A convolutional autoencoder trained on good parts; the per-pixel error map localizes scratches and dents (benchmarks such as MVTec AD).

Networks and IT

Intrusion detection from flow statistics; anomalous log or metric patterns in monitoring.

Medical

Flag unusual ECG beats or scans for review, trained on healthy examples.

Check yourself

An autoencoder trained on normal data is used for anomaly detection. What is the anomaly score?

Check yourself

A linear autoencoder with k code units is trained to its global minimum of squared error. What has it learned?

Check yourself

Your autoencoder reconstructs anomalies about as well as normal points. Which change is most likely to help?

Check yourself

In the worked example, why does P = (0.5, 0.9) get a score 11 times the threshold even though both of its coordinates are within the training range?

Remember

  • ✓Train x→z→x^x\to z\to\hat x through a bottleneck on normal data only; score new points by ∥x−x^∥2\lVert x-\hat x\rVert^2.
  • ✓Backprop: ∂e/∂Wd=2rz⊤\partial e/\partial W_d=2r z^\top, δ=(Wd⊤2r)⊙g′(a)\delta=(W_d^\top 2r)\odot g'(a), ∂e/∂We=δx⊤\partial e/\partial W_e=\delta x^\top.
  • ✓Set τ as a high percentile of scores on held-out normal data (it fixes the false-alarm rate), or tune it on labelled validation anomalies.
  • ✓A linear autoencoder recovers PCA's subspace and the PCA reconstruction error; nonlinear activations let the manifold curve.
  • ✓Keep the training data clean, the bottleneck real and the features scaled, or the detector will quietly learn to reconstruct anomalies.
← back to ML Algorithms roadmap