Definition
A one-class SVM is an unsupervised anomaly detector. It learns a boundary around the training data from unlabelled examples. In a kernel feature space it finds the hyperplane that pushes the data as far as possible from the origin, while letting a fraction of at most of the points fall on the wrong side. A new point on the data's side is normal; a point on the origin's side is an outlier. Schölkopf et al. introduced it in 2001.
At a glance
- Task
- Novelty / outlier detection, also called one-class classification
- Input → output
- → score ; normal if , outlier if
- Learns
- one dual weight per training point (nonzero only for support vectors) and an offset
- Objective
- maximize the margin between the data and the origin, plus a hinge-type penalty on points that fall short
- Training
- a convex quadratic program, so there is one global optimum; SMO (libsvm) takes roughly – time and kernel memory
- Prediction
- kernel evaluations per point
- Hyperparameters
- (outlier budget) and the kernel, almost always RBF with width
- Assumes
- the training set is (mostly) normal data; features are on comparable scales
The problem
A fraud team has millions of normal card transactions and a handful of known frauds. A factory has months of sensor logs from a healthy machine and none from a failing one. In both cases a classifier has nothing to train on for the second class, and the anomalies of tomorrow may not look like those of yesterday anyway.
What we can do is describe what normal looks like: draw a tight boundary around the training data and flag anything that falls outside it. The boundary should be flexible enough to follow curved or multi-modal data. It should also tolerate a few odd points in the training set, so that one bad reading does not stretch the boundary to swallow everything.
The idea
A two-class Support Vector Machine puts the widest possible street between two classes. A one-class SVM has only one class, so it invents the other: the origin of feature space plays the part of the "negative class". The algorithm looks for a hyperplane that keeps the data on one side and the origin on the other, as far from the origin as possible. The region on the data's side is the learned "normal" region.
Separating a cloud of points from the origin does not sound useful in the input space; a cloud centred near the origin cannot be separated at all. With an RBF kernel the picture changes completely. Every point is mapped to a vector of length 1 (), and every pair has a positive dot product (). All the data sits on a patch of the unit sphere, in one orthant, well away from the origin. A hyperplane slicing that patch cuts off a spherical cap. Points in dense regions have large kernel similarities to many others and land inside the cap. Isolated points land outside it. Back in input space, the cap is a closed, curved boundary around the dense parts of the data.
Vocabulary
- Feature map φ
- the (implicit) map into the kernel's feature space, with
- Hyperplane
- in feature space; the data should lie on the side
- Offset ρ
- how far the hyperplane is from the origin, in units of ; its distance is
- Slack ξᵢ
- how far point falls short of the hyperplane:
- ν (nu)
- the outlier budget: an upper bound on the fraction of training outliers and a lower bound on the fraction of support vectors
- Support vector
- a training point with : either exactly on the boundary or a training outlier
- Decision value
- ; its sign is the verdict
The math: the primal problem
Write the hyperplane in feature space as . The distance from the origin to it is . We want that distance to be large: large and small . Points that land on the origin's side are allowed, but each one pays a slack . Schölkopf's formulation balances the three goals in one objective:
- normal vector of the hyperplane in feature space
- offset; the hyperplane sits at distance ρ/‖w‖ from the origin
- slack of point i: how far it falls on the origin's side
- outlier budget in (0, 1]; small ν = expensive slack
- number of training points
- feature-space image of training point i
Read the objective term by term. is the usual SVM regularizer. rewards pushing the hyperplane away from the origin. Each unit of slack costs . Raising by gains in the objective, but every point currently on or beyond the hyperplane gains slack, which costs each. So keeps rising until about points are on the wrong side or on the boundary, and that trade is where gets its meaning.
From primal to dual
Attach multipliers to the margin constraints and to :
- multiplier of point i's margin constraint (the dual weight)
- multiplier of ξᵢ ≥ 0
Set the derivatives with respect to the primal variables to zero
- 1, so . The normal vector is a weighted sum of training points.
- 2. Since , this gives the box .
- 3, so .
- 4Substitute back. The and terms cancel, and . Replace each dot product by the kernel.
- n × n Gram matrix, Kᵢⱼ = k(xᵢ, xⱼ)
- upper bound on each αᵢ (written C below)
This is a small quadratic program: minimize a convex quadratic over a simplex cut off by a box. Note that there are no labels, and the data enters only through the kernel. The weights form a probability distribution over the training points (), and no single point may carry more than of it.
Recovering ρ. For a point with we have , so ; and makes its margin constraint tight. Such a point lies exactly on the hyperplane, which gives . In practice we average over all these "free" support vectors. Substituting into gives the decision function:
- kernel similarity between training point i and the query x
- a weighted kernel density of the support vectors at x
Three kinds of training points
The KKT conditions ( and ) sort every training point into one of three groups:
| Group | Dual weight | Position | Score |
|---|---|---|---|
| Inside | strictly on the data's side | , | |
| On the boundary (free SV) | exactly on the hyperplane | , | |
| Outlier (bounded SV) | on the origin's side (or on it) | , |
The ν-property
The box and the sum constraint together give a precise meaning, which needs only two lines of arithmetic:
Why ν bounds both counts
- 1Outliers. Every training outlier has , so and . With outliers, . Since , : at most a fraction ν of the training points are outliers.
- 2Support vectors. Every is at most , and together they must add to 1. That takes at least nonzero weights: at least a fraction ν of the points are support vectors.
- set by you; both bounds become equalities as n → ∞ (under mild conditions on the data distribution)
This is why is easy to set: it is the outlier budget. If you expect about 2% contamination in your training data, start with .
The RBF kernel and γ
- inverse squared width; the similarity falls to e⁻¹ at distance 1/√γ
- squared Euclidean distance in input space
sets how far one training point's influence reaches. With small every point is similar to every other. The density is a single smooth bump, and the boundary is a near-ellipse around the bulk of the data. With large similarities die out within a short distance. Each point becomes its own island, so even an isolated point gets a tiny bubble of "normal" around itself. The model then memorizes the training set, and almost every point becomes a support vector. scikit-learn's default gamma="scale" uses , which ties the width to the spread of the (scaled) data.
Worked example: ten sensor readings
Ten two-dimensional readings from a machine. Eight form a tight cluster around ; readings 9 and 10 are far off. We fit with and , so the box is . At most 4 points can be outliers, and at least 4 must be support vectors. Every number below was checked against scikit-learn.
| i | x₁ | x₂ |
|---|---|---|
| 1 | 2.5 | 2.5 |
| 2 | 3.0 | 3.0 |
| 3 | 3.5 | 2.5 |
| 4 | 2.5 | 3.5 |
| 5 | 3.5 | 3.5 |
| 6 | 3.0 | 2.0 |
| 7 | 2.0 | 3.0 |
| 8 | 3.0 | 4.0 |
| 9 | 0.5 | 5.0 |
| 10 | 5.5 | 1.0 |
Step 1: the kernel matrix
- 1Squared distances. Neighbours on the grid are apart squared (), axis steps of 1 give , and the far points are much further: , .
- 2Kernel values : , , , , , . Inside the cluster all similarities are 0.37–0.88; the two far points have similarities of at most 0.21 with anything.
- 3The diagonal is all 1: every point sits on the unit sphere in feature space.
Step 2: solve the dual with SMO
- 1Feasible start. Fill weight greedily in index order up to : . Then . The gradient of is , for example and .
- 2Pick a pair. To lower the objective while keeping , move weight from the point with the largest that can shrink () to the point with the smallest that can grow (). Here that is (0.9119) and (0.0922), a KKT gap of .
- 3Step size. Moving changes the objective by with . The unconstrained optimum is . That is clipped to , so point 2 hands all its weight to point 10.
- 4Repeat. The next step moves from point 1 to point 9. After that the steps shuffle weight among the cluster's edge points until the gap falls below , which takes about 50 steps.
| i | αᵢ | (Kα)ᵢ | f = (Kα)ᵢ − ρ | ξᵢ | Group |
|---|---|---|---|---|---|
| 1 | 0 | 0.4002 | +0.0162 | 0 | inside |
| 2 | 0 | 0.4316 | +0.0477 | 0 | inside |
| 3 | 0 | 0.4065 | +0.0226 | 0 | inside |
| 4 | 0 | 0.4232 | +0.0392 | 0 | inside |
| 5 | 0.0355 | 0.3840 | 0 | 0 | boundary SV |
| 6 | 0.2083 | 0.3840 | 0 | 0 | boundary SV |
| 7 | 0.0674 | 0.3840 | 0 | 0 | boundary SV |
| 8 | 0.1888 | 0.3840 | 0 | 0 | boundary SV |
| 9 | 0.25 = C | 0.3017 | −0.0823 | 0.0823 | outlier |
| 10 | 0.25 = C | 0.2921 | −0.0919 | 0.0919 | outlier |
Step 3: check the solution by hand
- 1Sum constraint: . ✓
- 2ρ from a free SV, point 5: . ✓ Points 6, 7, 8 give the same value.
- 3An outlier, point 9: , so . Most of that 0.3017 is its own weight ; its neighbours contribute almost nothing.
- 4ν-property: outliers support vectors . ✓
- 5Duality: the primal objective equals the dual value , so the gap is zero. The feature-space margin is .
| SV i | αᵢ | ‖xᵢ − x*‖² | k(xᵢ, x*) | αᵢ · k |
|---|---|---|---|---|
| 5 | 0.0355 | 1.06 | 0.7672 | 0.0272 |
| 6 | 0.2083 | 0.36 | 0.9139 | 0.1904 |
| 7 | 0.0674 | 1.16 | 0.7483 | 0.0505 |
| 8 | 0.1888 | 1.96 | 0.6126 | 0.1157 |
| 9 | 0.25 | 12.01 | 0.0497 | 0.0124 |
| 10 | 0.25 | 8.81 | 0.1105 | 0.0276 |
The same arithmetic for , between the cluster and reading 10, gives and : outlier. For a far corner such as it gives . The score falls smoothly as you move away from the dense region.
Effect of ν and γ on the example
| ν | C | SVs | Training outliers | ρ |
|---|---|---|---|---|
| 0.1 – 0.3 | 1.0 – 0.333 | 6 (5–10) | none | 0.3326 |
| 0.4 | 0.25 | 6 (5–10) | 9, 10 | 0.3840 |
| 0.5 | 0.2 | 6 (5–10) | 6, 8, 9, 10 | 0.4530 |
| 0.6 | 0.167 | 7 | 6, 7, 8, 9, 10 | 0.5045 |
| 0.8 | 0.125 | 9 | 7 points | 0.5949 |
| γ | SVs | Training outliers | Share of the 6×6 box judged normal |
|---|---|---|---|
| 0.05 | 4 | 9, 10 | 9.0% |
| 0.1 | 5 | 9, 10 | 9.5% |
| 0.25 | 6 | 9, 10 | 8.8% |
| 0.6 | 7 | none | 6.9% |
| 1 | 7 | none | 6.2% |
| 2.5 | 10 | none | 5.2% |
| 5 | 10 | none | 1.3% |
The algorithm, step by step
Train and score
- 1Prepare. Standardize the features using statistics from the (clean) training set only.
- 2Kernel. Choose ; compute (or cache on demand) .
- 3Solve the dual. Start from any feasible (sum 1, inside the box). Repeat SMO steps: pick the pair with the largest KKT gap, move weight between them by the clipped exact step, update . Stop when the gap is below the tolerance.
- 4Offset. = mean of over the free SVs .
- 5Score. For a new : . Flag , or rank points by and set your own threshold.
In code
1import numpy as np23def rbf(A, B, gamma):4 d2 = ((A[:, None, :] - B[None, :, :]) ** 2).sum(-1)5 return np.exp(-gamma * d2)67def one_class_svm(X, nu=0.4, gamma=0.25, tol=1e-9, max_iter=1000):8 """SMO on the dual: min 1/2 a^T K a s.t. sum(a) = 1, 0 <= a_i <= C = 1/(nu n)."""9 n = len(X)10 C = 1.0 / (nu * n)11 K = rbf(X, X, gamma)12 a = np.zeros(n)13 left = 1.014 for i in range(n): # feasible start: fill up to C in order15 a[i] = min(C, left)16 left -= a[i]17 g = K @ a # gradient of the objective = K a18 for _ in range(max_iter):19 up, down = a < C - 1e-12, a > 1e-1220 i = np.flatnonzero(up)[np.argmin(g[up])] # can grow, smallest K a21 j = np.flatnonzero(down)[np.argmax(g[down])] # can shrink, largest K a22 if g[j] - g[i] < tol: # KKT satisfied: done23 break24 eta = K[i, i] + K[j, j] - 2 * K[i, j]25 t = min((g[j] - g[i]) / eta, C - a[i], a[j]) # exact line search, clipped to the box26 a[i] += t27 a[j] -= t28 g += t * (K[:, i] - K[:, j])29 free = (a > 1e-8) & (a < C - 1e-8)30 rho = g[free].mean() # free SVs sit exactly on the boundary31 return a, rho3233def decision(X, a, rho, gamma, Z):34 return rbf(Z, X, gamma) @ a - rho # >= 0 normal, < 0 outlier3536X = np.array([[2.5, 2.5], [3.0, 3.0], [3.5, 2.5], [2.5, 3.5], [3.5, 3.5],37 [3.0, 2.0], [2.0, 3.0], [3.0, 4.0], [0.5, 5.0], [5.5, 1.0]])38a, rho = one_class_svm(X)39print(a.round(4), round(rho, 4)) # [0 0 0 0 0.0355 0.2083 0.0674 0.1888 0.25 0.25] 0.38440print(decision(X, a, rho, 0.25, np.array([[3.0, 2.6]])).round(4)) # [0.0398]1import numpy as np2from sklearn.svm import OneClassSVM3from sklearn.preprocessing import StandardScaler4from sklearn.pipeline import make_pipeline56X = np.array([[2.5, 2.5], [3.0, 3.0], [3.5, 2.5], [2.5, 3.5], [3.5, 3.5],7 [3.0, 2.0], [2.0, 3.0], [3.0, 4.0], [0.5, 5.0], [5.5, 1.0]])89oc = OneClassSVM(kernel="rbf", nu=0.4, gamma=0.25, tol=1e-8).fit(X)10nu_n = oc.nu * len(X) # libsvm scales the dual by nu*n11print(oc.support_ + 1) # [ 5 6 7 8 9 10]12print((oc.dual_coef_ / nu_n).round(4)) # alpha: [[0.0355 0.2083 0.0674 0.1888 0.25 0.25]]13print((-oc.intercept_ / nu_n).round(4)) # rho: [0.384]14x_star = [[3.0, 2.6]]15print((oc.decision_function(x_star) / nu_n).round(4)) # f(x*) = [0.0398] -> normal16print(oc.predict([[3.0, 2.6], [4.5, 1.5]])) # [ 1 -1]1718# On real data: scale first, train on (mostly) clean data, pick nu from the outlier budget.19model = make_pipeline(StandardScaler(), OneClassSVM(kernel="rbf", nu=0.05, gamma="scale"))20model.fit(X)Use when
- +You have plenty of normal data and few or no labelled anomalies.
- +The normal region is curved or has several clusters: the RBF boundary follows it.
- +You want a direct handle on the false-alarm rate: bounds the training outlier fraction.
- +You want a convex problem: one global optimum, no random restarts, reproducible results.
Avoid when
- − is large (beyond ~–): kernel SMO costs about memory and time, and prediction cost grows with the number of SVs.
- −Many features, or features on different scales: RBF distances become meaningless without scaling and degrade in high dimensions.
- −There are no labels to tune with: results are very sensitive to it, and unsupervised selection is guesswork.
- −You need calibrated probabilities or explanations: f(x) is a raw, uncalibrated score.
- −The training data is heavily contaminated: Isolation Forest or LOF are often more robust and faster.
Where it's used
Industrial monitoring
Train on sensor data from a healthy machine; alarm when readings leave the learned region.
Fraud and intrusion detection
Model normal transactions or network flows; flag the rare ones that look like nothing seen before.
Novelty detection for ML systems
Detect inputs far from the training distribution before trusting a downstream model.
Quality control
Learn the feature profile of good parts; reject parts outside it, including defect types never seen.
Check yourself
You fit a one-class SVM with on 2,000 points. Which statement is guaranteed?
Check yourself
At the optimum a training point has . Where is it?
Check yourself
In the worked example, , and . A new point has kernel values with SVs 5, 6, 7, 8 and 10, and with SV 9. Its verdict is…
Check yourself
You raise from 0.25 to 5 on the same data. What happens?
Remember
- ✓A one-class SVM separates the data from the origin in kernel feature space with maximum margin . With RBF, all points lie on the unit sphere, so the cut is a cap around the dense regions.
- ✓Primal: . Dual: with and .
- ✓Decision: , a thresholded kernel density of the support vectors. is at any free SV.
- ✓ν-property: outlier fraction support-vector fraction, so is the training outlier budget.
- ✓Always scale features. is the critical knob: too small ignores structure, too large memorizes points.