Support vector regression

Fit as flat a function as possible that keeps targets inside an ε-tube, penalizing only points outside it.

17 min read1530 wordsadvanced9 viewsupdated 2026-10-07

Definition

Support vector regression (SVR) fits a function f(x)=w⊤ϕ(x)+bf(x)=w^\top\phi(x)+b that is as flat (small ∥w∥\|w\|) as possible while keeping every target within ±ε\pm\varepsilon of it. Errors smaller than ε\varepsilon cost nothing; larger errors cost their excess linearly, weighted by CC. The fit depends only on the points on or outside the ε\varepsilon-tube, the support vectors.

At a glance

Task
Regression (real-valued target)
Input → output
feature vector xx → real number f(x)=∑iβi k(xi,x)+bf(x)=\sum_i \beta_i\,k(x_i,x)+b
Learns
one dual coefficient βi∈[−C,C]\beta_i\in[-C,C] per training point (most are 0) and a bias bb; for a linear kernel, equivalently a weight vector ww
Loss
ε\varepsilon-insensitive: max⁡(0, ∣y−f(x)∣−ε)\max(0,\,|y-f(x)|-\varepsilon), plus the flatness penalty 12∥w∥2\tfrac12\|w\|^2
Training
a convex quadratic program; SMO-type solvers (libsvm) take roughly O(n2)O(n^2) to O(n3)O(n^3) time and an O(n2)O(n^2) kernel cache
Prediction
O(nSV⋅d)O(n_{\text{SV}}\cdot d) — one kernel evaluation per support vector
Hyperparameters
ε\varepsilon (tube half-width, in target units), CC (cost of leaving the tube), kernel and its parameters (RBF γ\gamma, polynomial degree)
Assumes
features on comparable scales (kernels use distances / dot products); noise of roughly size ε\varepsilon is acceptable to ignore

The problem

Least squares charges every residual its square. That has two side effects. First, one bad row can dominate: a residual of 4 costs 16 times as much as a residual of 1, so the line tilts toward outliers. In the worked example below, a single data-entry typo drags the least-squares slope from 0.80 to 0.42. Second, every training point contributes to the solution, and in kernelized form (kernel ridge regression) the prediction is a sum over all nn training points.

Support vector regression keeps the parts of the support vector machine (see the Support vector machines article) that work well (a margin, a convex problem, kernels) and adapts them to real-valued targets. The loss grows only linearly beyond a dead zone of width ε\varepsilon, so no single point can pull on the fit with more than a fixed force CC, and points inside the dead zone drop out of the model entirely.

The idea

Picture a tube of half-width ε\varepsilon around the fitted curve. Points inside it are "close enough": they cost nothing and are ignored. Among all functions whose tube contains the data, SVR picks the flattest one. When the data cannot all fit inside a flat tube, points may stick out, and each one pays CC per unit it sticks out. The final fit is a compromise between flatness and total protrusion, and only the points on the tube's wall or outside it (the support vectors) take part in it.

Vocabulary

ε-tube
the band f(x)−ε≤y≤f(x)+εf(x)-\varepsilon\le y\le f(x)+\varepsilon. Residuals inside it are not penalized.
ε-insensitive loss
Lε(r)=max⁡(0,∣r∣−ε)L_\varepsilon(r)=\max(0,|r|-\varepsilon) for residual r=y−f(x)r=y-f(x): flat zero inside the tube, slope 1 outside.
Slack ξᵢ, ξᵢ*
how far point ii sits above (ξi\xi_i) or below (ξi∗\xi_i^*) the tube. At most one of the two is non-zero.
Flatness
12∥w∥2\tfrac12\|w\|^2. A small ww means the prediction changes slowly with xx; this is the regularizer, the same one ridge regression uses.
Dual coefficient βᵢ
βi=αi−αi∗∈[−C,C]\beta_i=\alpha_i-\alpha_i^*\in[-C,C]: point ii's pull on the fit. Positive if it pushes the fit up (point above the tube), negative if down.
Support vector
a training point with βi≠0\beta_i\neq0, i.e. on the tube wall or outside it. Only these appear in ff.
Free vs bounded SV
free: exactly on the wall, 0<∣βi∣<C0<|\beta_i|<C. Bounded: strictly outside, ∣βi∣=C|\beta_i|=C (the force is capped).
Kernel
k(x,x′)=ϕ(x)⊤ϕ(x′)k(x,x')=\phi(x)^\top\phi(x'); lets the same algorithm fit curves. Linear: x⊤x′x^\top x'; RBF: e−γ∥x−x′∥2e^{-\gamma\|x-x'\|^2}.

The math

The primal problem. With slack variables for the two sides of the tube:

min⁡w,b,ξ,ξ∗ 12∥w∥2+C∑i=1n(ξi+ξi∗)s.t.{yi−w⊤xi−b≤ε+ξiw⊤xi+b−yi≤ε+ξi∗ξi, ξi∗≥0\min_{w,b,\xi,\xi^*}\ \frac12\|w\|^2 + C\sum_{i=1}^n(\xi_i+\xi_i^*)\quad\text{s.t.}\quad \begin{cases} y_i-w^\top x_i-b\le\varepsilon+\xi_i\\ w^\top x_i+b-y_i\le\varepsilon+\xi_i^*\\ \xi_i,\ \xi_i^*\ge0\end{cases}
w, bw,\ b
weights and bias of the fitted function
ε\varepsilon
half-width of the tube; residuals smaller than this are free
ξi, ξi∗\xi_i,\ \xi_i^*
slack: how far point ii is above / below the tube
CC
price per unit of slack; trades flatness against fitting

At the optimum each slack is as small as the constraints allow, ξi+ξi∗=max⁡(0,∣yi−f(xi)∣−ε)\xi_i+\xi_i^*=\max(0,|y_i-f(x_i)|-\varepsilon), so the primal is just ridge-style regularization plus the ε-insensitive loss:

min⁡w,b 12∥w∥2+C∑i=1nLε(yi−f(xi)),Lε(r)=max⁡(0, ∣r∣−ε)\min_{w,b}\ \frac12\|w\|^2+C\sum_{i=1}^n L_\varepsilon\big(y_i-f(x_i)\big),\qquad L_\varepsilon(r)=\max(0,\ |r|-\varepsilon)
LεL_\varepsilon
ε-insensitive loss: 0 inside the tube, linear outside it

Compare with least squares, whose loss r2r^2 has derivative 2r2r: the pull of a point grows without limit as it moves away. The ε-insensitive loss has derivative 00 inside the tube and ±1\pm1 outside, so every point's pull on the fit is bounded by CC. That is the whole robustness story in one line.

Deriving the dual. Attach multipliers αi,αi∗≥0\alpha_i,\alpha_i^*\ge0 to the two tube constraints and ηi,ηi∗≥0\eta_i,\eta_i^*\ge0 to ξi,ξi∗≥0\xi_i,\xi_i^*\ge0:

L=12∥w∥2+C∑i(ξi+ξi∗)−∑iαi(ε+ξi−yi+w⊤xi+b)−∑iαi∗(ε+ξi∗+yi−w⊤xi−b)−∑i(ηiξi+ηi∗ξi∗)\mathcal L=\tfrac12\|w\|^2+C\sum_i(\xi_i+\xi_i^*)-\sum_i\alpha_i(\varepsilon+\xi_i-y_i+w^\top x_i+b)-\sum_i\alpha_i^*(\varepsilon+\xi_i^*+y_i-w^\top x_i-b)-\sum_i(\eta_i\xi_i+\eta_i^*\xi_i^*)
αi, αi∗\alpha_i,\ \alpha_i^*
multipliers of the upper and lower tube constraints
ηi, ηi∗\eta_i,\ \eta_i^*
multipliers of the non-negativity of the slacks

Setting the derivatives to zero

  1. 1∂L/∂w=0\partial\mathcal L/\partial w=0 gives w=∑i(αi−αi∗) xi=∑iβixiw=\sum_i(\alpha_i-\alpha_i^*)\,x_i=\sum_i\beta_i x_i. The weight vector is a combination of training points.
  2. 2∂L/∂b=0\partial\mathcal L/\partial b=0 gives ∑iβi=0\sum_i\beta_i=0: the upward and downward pulls balance.
  3. 3∂L/∂ξi=0\partial\mathcal L/\partial\xi_i=0 gives C−αi−ηi=0C-\alpha_i-\eta_i=0. Since ηi≥0\eta_i\ge0, 0≤αi≤C0\le\alpha_i\le C; the same holds for αi∗\alpha_i^*. This is the cap on each point's force.
  4. 4Substitute back. The ξ\xi terms cancel and, because a point cannot be above and below the tube at once (αiαi∗=0\alpha_i\alpha_i^*=0), αi+αi∗=∣βi∣\alpha_i+\alpha_i^*=|\beta_i|. What is left depends only on β\beta and dot products xi⊤xjx_i^\top x_j.
max⁡β −12∑i,jβiβj k(xi,xj)+∑iyiβi−ε∑i∣βi∣s.t.∑iβi=0,−C≤βi≤C\max_{\beta}\ -\frac12\sum_{i,j}\beta_i\beta_j\,k(x_i,x_j)+\sum_i y_i\beta_i-\varepsilon\sum_i|\beta_i|\quad\text{s.t.}\quad\sum_i\beta_i=0,\quad -C\le\beta_i\le C
βi\beta_i
αi−αi∗\alpha_i-\alpha_i^*, the dual coefficient of point ii
k(xi,xj)k(x_i,x_j)
kernel; xi⊤xjx_i^\top x_j for linear SVR
ε∑∣βi∣\varepsilon\sum|\beta_i|
an L1 penalty on the coefficients: this is what makes most of them exactly 0

The prediction only needs kernel values, so a non-linear kernel fits a curve at no extra conceptual cost (the kernel trick, explained in the Support vector machines article):

f(x)=∑i∈SVβi k(xi,x)+bf(x)=\sum_{i\in\text{SV}}\beta_i\,k(x_i,x)+b
SV\text{SV}
support vectors, the points with βi≠0\beta_i\neq0

The KKT conditions tell you, for every point, which of three situations it is in. They are both the optimality test and the source of bb:

Where the point isResidualDual coefficientRole
strictly inside the tube∣ri∣<ε|r_i|<\varepsilonβi=0\beta_i=0not a support vector; delete it and nothing changes
on the upper / lower wallri=±εr_i=\pm\varepsilon0<±βi<C0<\pm\beta_i<Cfree support vector; pins the fit, gives bb
outside the tube∣ri∣>ε|r_i|>\varepsilonβi=±C\beta_i=\pm Cbounded support vector; pulls with the maximum force C
Complementary slackness for SVR, with residual ri=yi−f(xi)r_i=y_i-f(x_i).

For any free support vector, yi−f(xi)=±εy_i-f(x_i)=\pm\varepsilon exactly, so b=yi−∑jβjk(xj,xi)∓εb=y_i-\sum_j\beta_j k(x_j,x_i)\mp\varepsilon; solvers average this over the free support vectors.

What each hyperparameter does, mathematically

ε
sets the dead zone of LεL_\varepsilon and appears as the L1 weight ε∑∣βi∣\varepsilon\sum|\beta_i| in the dual. Larger ε → more points inside the tube → fewer support vectors and a flatter, smoother fit. With ε above the spread of the data, w=0w=0 and ff is constant.
C
is the box ∣βi∣≤C|\beta_i|\le C: the maximum force any point can exert. Small C → a flat function that tolerates large protrusions; large C → chase every point outside the tube (overfitting with flexible kernels). As C→∞C\to\infty, SVR demands the data fit inside the tube if at all possible.
γ (RBF)
kernel width: k=e−γ∥x−x′∥2k=e^{-\gamma\|x-x'\|^2} has reach about 1/γ1/\sqrt{\gamma}. Large γ → each support vector influences only its neighbourhood → wiggly fit; small γ → nearly linear.

Worked example: a salary table with a typo

Eight employees, years of experience xx and salary yy in units of 10k dollars. Salary rises steadily with experience, except row 8: someone typed 3.0 instead of something near 8. We fit a linear SVR with ε=0.5\varepsilon=0.5 and C=1C=1.

iexperience x (yrs)salary y (10k USD)
112.0
222.6
333.9
444.1
555.3
665.6
777.0
883.0
Training data. Highlighted: row 8, the data-entry typo.

For a convex problem, any point that satisfies the KKT conditions is the global optimum. So we can propose a solution and then check it by hand. A linear fit in one dimension has two unknowns, so it is pinned by two points on the tube walls. The solver finds that these are row 1 (lower wall) and row 5 (upper wall).

Solving and verifying the fit by hand (ε = 0.5, C = 1)

  1. 1Pin the line to the walls. Row 1 on the lower wall means f(1)=y1+ε=2.5f(1)=y_1+\varepsilon=2.5; row 5 on the upper wall means f(5)=y5−ε=4.8f(5)=y_5-\varepsilon=4.8. So w=4.8−2.55−1=0.575w=\tfrac{4.8-2.5}{5-1}=0.575 and b=2.5−0.575=1.925b=2.5-0.575=1.925: f(x)=0.575x+1.925f(x)=0.575x+1.925.
  2. 2Residuals ri=yi−f(xi)r_i=y_i-f(x_i) (table below): rows 2, 3, 4, 6 have ∣ri∣<0.5|r_i|<0.5 and are inside the tube; rows 1 and 5 are exactly on the walls; row 7 (r=+1.05r=+1.05) is above the tube and row 8 (r=−3.525r=-3.525) is below it.
  3. 3Bounded coefficients. Points outside the tube pull with the full force: β7=+C=+1\beta_7=+C=+1 (pushing up) and β8=−C=−1\beta_8=-C=-1 (pushing down). Inside points have β2=β3=β4=β6=0\beta_2=\beta_3=\beta_4=\beta_6=0.
  4. 4Free coefficients from the two stationarity conditions. ∑βi=0\sum\beta_i=0: β1+β5+1−1=0\beta_1+\beta_5+1-1=0, so β1=−β5\beta_1=-\beta_5. w=∑βixiw=\sum\beta_ix_i: −β5⋅1+β5⋅5+1⋅7−1⋅8=0.575-\beta_5\cdot1+\beta_5\cdot5+1\cdot7-1\cdot8=0.575, so 4β5=1.5754\beta_5=1.575 and β5=0.39375\beta_5=0.39375, β1=−0.39375\beta_1=-0.39375.
  5. 5KKT check. Both free coefficients lie strictly inside (−C,C)(-C,C) and have the right signs (row 5 on the upper wall has β>0\beta>0, row 1 on the lower wall has β<0\beta<0). Every condition holds, so this is the optimum.
  6. 6Objective. Slack: ξ7=1.05−0.5=0.55\xi_7=1.05-0.5=0.55, ξ8∗=3.525−0.5=3.025\xi_8^*=3.525-0.5=3.025. Value =12(0.575)2+1⋅(0.55+3.025)=0.1653+3.575=3.7403=\tfrac12(0.575)^2+1\cdot(0.55+3.025)=0.1653+3.575=3.7403. For comparison, the least-squares line 0.4202x+2.29640.4202x+2.2964 scores 4.4466 on this objective.
  7. 7Predict a new employee with 9 years: f(9)=0.575⋅9+1.925=7.100f(9)=0.575\cdot9+1.925=7.100, i.e. about 71k dollars. Least squares predicts 6.079.
ixyf(x)r = y − f(x)slackstatusβᵢ
112.02.500−0.5000lower wall−0.39375
222.63.075−0.4750inside0
333.93.6500.2500inside0
444.14.225−0.1250inside0
555.34.8000.5000upper wall0.39375
665.65.3750.2250inside0
777.05.9501.0500.550above+1 (= C)
883.06.525−3.5253.025below−1 (= −C)
Residuals and dual coefficients at the optimum. Four support vectors (rows 1, 5, 7, 8); ∑βi=0\sum\beta_i=0 and ∑βixi=0.575=w\sum\beta_ix_i=0.575=w.

How much did the typo hurt? Without row 8, SVR fits 0.68x+1.740.68x+1.74 and least squares fits 0.80x+1.1570.80x+1.157. With it, SVR's slope moves to 0.575 while least squares collapses to 0.42. On the seven correct rows, SVR has MAE 0.446 and RMSE 0.529; least squares has MAE 0.738 and RMSE 0.884.

salary (10k USD) vs experience (yrs) · drag a point-202468123456789f(9) = 7.1012345678loss vs residual-4-2024squaredε-insensitivedual coefficient βᵢ+C = 1−C12345678● inside (β=0) ● on edge (0<|β|<C) ● outside (|β|=C) — SVR - - least squares
kernel|
ε0.50C1
SVR: f(x) = 0.5750·x + 1.9250 · least squares: 0.4202·x + 2.2964
objective = ½‖w‖² + C·Σξ = 0.1653 + 1·3.575 = 3.7403
support vectors: 4 of 8 (2 on edge, 2 outside) · Σβ = 0 · prediction f(9) = 7.100
fig.The worked example, solved live by the same SMO algorithm as the code below. Drag any point vertically; change ε, C or switch to an RBF kernel. Points are coloured by their KKT role: grey inside the tube (β = 0), mint on a wall (free support vector), rose outside (β = ±C, red dashes show the slack). Top right: every residual placed on the ε-insensitive loss, with squared loss dashed for comparison. Bottom right: each point's βᵢ scaled to ±C. Defaults: f(x) = 0.575x + 1.925, 4 support vectors, objective 0.1653 + 1·3.575 = 3.7403, f(9) = 7.10. Drag row 8 down to −0.5 and the line does not move. Raising ε to 1.0 flattens the line to 0.325x + 2.675.

The algorithm, step by step

Training an SVR (libsvm-style)

  1. 1Scale the features (standardize), and decide ε in the target's units, or standardize the target too.
  2. 2Kernel matrix. Compute (or cache on demand) Kij=k(xi,xj)K_{ij}=k(x_i,x_j).
  3. 3Initialize β=0\beta=0: a feasible point (∑βi=0\sum\beta_i=0, inside the box). The gradient of the smooth part is g=Kβ−yg=K\beta-y.
  4. 4Pick a pair (i,j)(i,j) that most violates the KKT conditions. Moving both at once, βi+=t\beta_i\mathrel{+}=t and βj−=t\beta_j\mathrel{-}=t, keeps ∑β=0\sum\beta=0.
  5. 5Solve the pair exactly. Along that direction the dual is a 1-D piecewise quadratic, 12ηt2+(gi−gj)t+ε(∣βi+t∣+∣βj−t∣)\tfrac12\eta t^2+(g_i-g_j)t+\varepsilon(|\beta_i+t|+|\beta_j-t|) with η=Kii+Kjj−2Kij\eta=K_{ii}+K_{jj}-2K_{ij}, clipped to the box. Minimize it in closed form and update gg with two kernel columns.
  6. 6Repeat until no pair violates KKT by more than the tolerance.
  7. 7Bias. Average yi−∑jβjKji∓εy_i-\sum_j\beta_jK_{ji}\mp\varepsilon over the free support vectors (or take the midpoint of the feasible interval if there are none).
  8. 8Keep only the support vectors {xi:βi≠0}\{x_i:\beta_i\neq0\} for prediction.

Complexity

Training time
between O(n2)O(n^2) and O(n3)O(n^3) kernel evaluations in practice; it grows with C (more bounded SVs to settle) and shrinks with ε (fewer SVs).
Memory
O(n2)O(n^2) for a full kernel matrix; libsvm uses an LRU cache (cache_size, in MB) instead.
Prediction
O(nSV⋅d)O(n_{\text{SV}}\cdot d) per query. With a linear kernel, collapse to ww for O(d)O(d).
Rule of thumb
comfortable up to roughly 10⁴–10⁵ rows; beyond that use LinearSVR or kernel approximations (Nyström, random Fourier features) feeding a linear model.

Hyperparameters, variants and tuning

εwbsupport vectorsΣ slackobjectivef(9)
00.7201.28085.7806.0397.760
0.250.7001.55064.3004.5457.850
0.50.5751.92543.5753.7407.100
0.750.4502.30042.9503.0516.350
1.00.3252.67542.3252.3785.600
1.50.1433.35731.1431.1534.643
Effect of ε on the worked example (linear kernel, C = 1). A wider tube swallows more points and lets the line go flatter.

With ε=0\varepsilon=0, every point is a support vector and SVR becomes L1 (least absolute deviation) regression with a ridge penalty. On a larger problem the sparsity effect is easier to see. Below, 80 noisy samples of sin⁡x\sin x (noise s.d. 0.2) are fit with an RBF SVR (C=10C=10, γ=1\gamma=1) and tested on 400 fresh points:

εsupport vectors (of 80)test RMSE
0800.225
0.05690.223
0.1570.213
0.2320.211
0.3150.214
0.550.298
RBF SVR on noisy sin(x): ε trades sparsity against accuracy. ε = 0.2–0.3, near the noise level, matches the best test error with 2.5–5× fewer support vectors than ε = 0; ε = 0.5 is too wide.
settingtrain RMSEtest RMSE
C = 0.01, γ = 10.5670.613
C = 0.1, γ = 10.2160.234
C = 1, γ = 10.1850.204
C = 1000, γ = 10.1800.223
C = 10, γ = 0.010.4210.434
C = 10, γ = 100.1610.241
C = 10, γ = 1000.1290.352
Same data, ε = 0.1. C and γ control the bias-variance trade-off: too small underfits, too large overfits (training error keeps falling while test error rises).

Variants

ν-SVR (NuSVR)
replaces ε with ν∈(0,1]\nu\in(0,1], an upper bound on the fraction of points outside the tube and a lower bound on the fraction of support vectors; ε is then learned. On the worked example (C=1C=1), ν=0.25\nu=0.25 gives 3 SVs and ν=0.75\nu=0.75 gives 7.
LinearSVR
liblinear solver, linear kernel only, scales to millions of rows. It also penalizes the intercept, so results differ: on the worked example it gives 0.81x + 0.83 rather than 0.575x + 1.925. Center the target or raise intercept_scaling.
Squared ε-insensitive
max⁡(0,∣r∣−ε)2\max(0,|r|-\varepsilon)^2 (loss="squared_epsilon_insensitive" in LinearSVR): smooth, but gives up bounded influence.
Kernel ridge regression
same kernels with squared loss: a closed-form solution, but dense (all nn points used) and not robust. SVR with ε→0\varepsilon\to0 and huge C behaves similarly on clean data.
Huber / quantile regression
other linear-tail losses: Huber is quadratic near 0 instead of flat; quantile (pinball) loss is asymmetric and predicts a quantile.

Practical tuning recipe

1. Scale
StandardScaler on X inside a Pipeline (the scaler must be fitted on the training folds only).
2. Set ε from the noise
start near the residual noise s.d. (or 0.1 after standardizing y). It is in target units.
3. Search C and γ on a log grid
e.g. C ∈ {0.1, 1, 10, 100}, γ ∈ {0.01, 0.1, 1, 10} (or gamma="scale"), with cross-validated RMSE.
4. Check sparsity
if nearly every point is a support vector, ε is too small (or C too small); prediction will be slow.

In code

python
1import numpy as np
2
3def kernel(A, B, kind="linear", gamma=0.5):
4 if kind == "linear":
5 return A @ B.T
6 sq = ((A[:, None, :] - B[None, :, :]) ** 2).sum(-1)
7 return np.exp(-gamma * sq) # RBF
8
9def fit_svr(X, y, C=1.0, eps=0.5, kind="linear", gamma=0.5, sweeps=500, tol=1e-12):
10 """Dual SVR solved by SMO on pairs (beta_i, beta_j).
11 minimise 1/2 b'Kb - y'b + eps*sum|b_i| s.t. -C <= b_i <= C, sum b_i = 0"""
12 n, K = len(y), kernel(X, X, kind, gamma)
13 beta = np.zeros(n)
14 for _ in range(sweeps):
15 improved = False
16 for i in range(n):
17 for j in range(n):
18 if i == j:
19 continue
20 g = K @ beta - y # gradient of the smooth part
21 eta = K[i, i] + K[j, j] - 2 * K[i, j] # curvature along the pair direction
22 lo = max(-C - beta[i], beta[j] - C) # move: beta_i += t, beta_j -= t
23 hi = min(C - beta[i], beta[j] + C) # (keeps sum beta = 0 and the box)
24 if hi - lo < 1e-15:
25 continue
26 phi = lambda t: (0.5 * eta * t * t + (g[i] - g[j]) * t
27 + eps * (abs(beta[i] + t) + abs(beta[j] - t)))
28 kinks = sorted({lo, hi, *[k for k in (-beta[i], beta[j]) if lo < k < hi]})
29 cands = list(kinks)
30 for a, c in zip(kinks[:-1], kinks[1:]): # phi is a plain quadratic between kinks
31 m = (a + c) / 2
32 s = eps * (np.sign(beta[i] + m) - np.sign(beta[j] - m))
33 if eta > 1e-12:
34 cands.append(np.clip(-(g[i] - g[j] + s) / eta, a, c))
35 t = min(cands, key=phi)
36 if phi(0) - phi(t) > tol:
37 beta[i] += t
38 beta[j] -= t
39 improved = True
40 if not improved:
41 break
42 # bias from the KKT conditions: each point bounds b from below and/or above
43 r, tiny = y - K @ beta, 1e-8 # residual before adding b
44 lo = np.r_[r[beta < C - tiny] - eps, r[beta < -tiny] + eps].max()
45 hi = np.r_[r[beta > -C + tiny] + eps, r[beta > tiny] - eps].min()
46 return beta, (lo + hi) / 2
47
48def predict(X_train, beta, b, X_new, kind="linear", gamma=0.5):
49 return kernel(X_new, X_train, kind, gamma) @ beta + b
50
51# the worked example: years of experience -> salary ($10k); the last row is a typo
52x = np.array([1, 2, 3, 4, 5, 6, 7, 8], float)
53y = np.array([2.0, 2.6, 3.9, 4.1, 5.3, 5.6, 7.0, 3.0])
54X = x[:, None]
55
56beta, b = fit_svr(X, y, C=1.0, eps=0.5)
57w = beta @ x # linear kernel: w = sum beta_i x_i
58r = y - (w * x + b)
59xi = np.maximum(0, np.abs(r) - 0.5)
60print("beta =", beta.round(5))
61print(f"w = {w:.4f} b = {b:.4f}")
62print("residuals =", r.round(3))
63print("slack =", xi.round(3), " objective = %.4f" % (0.5 * w * w + xi.sum()))
64print("f(9) = %.3f" % predict(X, beta, b, np.array([[9.0]]))[0])
65# beta = [-0.39375 0. 0. 0. 0.39375 0. 1. -1.]
66# w = 0.5750 b = 1.9250
67# residuals = [-0.5 -0.475 0.25 -0.125 0.5 0.225 1.05 -3.525]
68# slack = [0. 0. 0. 0. 0. 0. 0.55 3.025] objective = 3.7403
69# f(9) = 7.100
From scratch: the dual solved by pairwise SMO, bias from the KKT conditions. Prints the worked example exactly.
python
1import numpy as np
2from sklearn.svm import SVR
3from sklearn.linear_model import LinearRegression
4from sklearn.pipeline import make_pipeline
5from sklearn.preprocessing import StandardScaler
6from sklearn.model_selection import GridSearchCV
7from sklearn.metrics import mean_squared_error
8
9X = np.arange(1, 9, dtype=float)[:, None]
10y = np.array([2.0, 2.6, 3.9, 4.1, 5.3, 5.6, 7.0, 3.0])
11
12svr = SVR(kernel="linear", C=1.0, epsilon=0.5).fit(X, y)
13print("w =", svr.coef_.ravel().round(4), " b =", svr.intercept_.round(4))
14print("support vectors:", svr.support_ + 1, " beta:", svr.dual_coef_.ravel().round(5))
15ols = LinearRegression().fit(X, y)
16print("OLS: w = %.4f b = %.4f" % (ols.coef_[0], ols.intercept_))
17print("predict x=9 -> SVR %.3f OLS %.3f" % (svr.predict([[9.0]])[0], ols.predict([[9.0]])[0]))
18# w = [0.575] b = [1.925]
19# support vectors: [1 5 7 8] beta: [-0.39375 0.39375 1. -1.]
20# OLS: w = 0.4202 b = 2.2964
21# predict x=9 -> SVR 7.100 OLS 6.079
22
23# A realistic RBF setup: scale X, then tune C, epsilon, gamma by cross-validation
24rs = np.random.RandomState(0)
25Xs = np.sort(rs.uniform(0, 6, 80))[:, None]
26ys = np.sin(Xs[:, 0]) + rs.normal(0, 0.2, 80)
27pipe = make_pipeline(StandardScaler(), SVR(kernel="rbf"))
28grid = GridSearchCV(pipe, {"svr__C": [0.1, 1, 10, 100],
29 "svr__epsilon": [0.05, 0.1, 0.2, 0.3],
30 "svr__gamma": [0.1, 1, 10]},
31 cv=5, scoring="neg_root_mean_squared_error").fit(Xs, ys)
32print("best:", grid.best_params_, " CV RMSE = %.3f" % -grid.best_score_)
33Xt = np.linspace(0, 6, 400)[:, None]
34yt = np.sin(Xt[:, 0]) + np.random.RandomState(1).normal(0, 0.2, 400)
35print("test RMSE = %.3f" % mean_squared_error(yt, grid.predict(Xt)) ** 0.5,
36 " support vectors:", len(grid.best_estimator_[-1].support_), "of 80")
37# best: {'svr__C': 1, 'svr__epsilon': 0.1, 'svr__gamma': 1} CV RMSE = 0.199
38# test RMSE = 0.205 support vectors: 56 of 80
scikit-learn: the worked example, then a realistic RBF pipeline tuned by cross-validation.

Use when

  • +You want a regression fit that is robust to a few gross outliers or label errors in y, since each point's influence is capped at C.
  • +Small-to-medium data (up to tens of thousands of rows) with a smooth non-linear relationship: an RBF kernel gives a flexible, well-regularized curve.
  • +High-dimensional inputs with few rows (spectra, text features): the flatness penalty controls variance.
  • +Errors below some tolerance genuinely do not matter, so a dead zone of width ε matches the real cost.

Avoid when

  • −Hundreds of thousands of rows or more with a non-linear kernel: training is quadratic or worse and prediction cost grows with the number of support vectors.
  • −You need predictive uncertainty or probabilities: SVR gives only a point estimate (use Gaussian processes or Bayesian regression).
  • −Heterogeneous tabular data with categorical features and missing values: gradient-boosted trees are usually more accurate with less tuning.
  • −You cannot afford a three-way hyperparameter search (ε, C, γ) and careful scaling.

Where it's used

Chemometrics and spectroscopy

Predicting a concentration from hundreds of near-infrared absorbances with only dozens of samples, a classic SVR niche.

Financial and load forecasting

Short-term electricity load or volatility forecasting on lagged features, where heavy-tailed shocks would distort a least-squares fit.

Engineering surrogate models

Replacing an expensive simulation (CFD, FEM) with a smooth RBF-kernel model trained on a few hundred runs.

Remaining-useful-life estimation

Regressing time-to-failure of batteries or bearings from sensor features in predictive maintenance.

Check yourself

With ε = 0.5, what is the ε-insensitive loss of a point whose residual is y − f(x) = −1.3?

Check yourself

In the worked example (ε = 0.5, C = 1), the typo in row 8 is changed from y = 3.0 to y = −10. What happens to the SVR line?

Check yourself

Which training points have 0 < |βᵢ| < C at the SVR optimum?

Check yourself

In the worked example, ε is raised from 0.5 to 1.5 (C = 1). What happens?

Remember

  • ✓SVR minimizes 12∥w∥2+C∑max⁡(0,∣yi−f(xi)∣−ε)\tfrac12\|w\|^2+C\sum\max(0,|y_i-f(x_i)|-\varepsilon): ridge-style flatness plus a loss with a dead zone and linear tails.
  • ✓Dual: f(x)=∑βik(xi,x)+bf(x)=\sum\beta_ik(x_i,x)+b with ∑βi=0\sum\beta_i=0 and ∣βi∣≤C|\beta_i|\le C. Points inside the tube have βi=0\beta_i=0, so the model is sparse in the data.
  • ✓KKT roles: inside → β=0\beta=0; on the wall → 0<∣β∣<C0<|\beta|<C (gives bb); outside → ∣β∣=C|\beta|=C, a capped pull that makes SVR robust to target outliers.
  • ✓ε (in target units) controls the tube width and sparsity, C the maximum force of a point, γ the RBF kernel's reach. Tune them by cross-validation after scaling.
  • ✓Training is a convex QP solved by SMO, roughly quadratic in n; use LinearSVR or kernel approximations for large data.
← back to ML Algorithms roadmap