Definition
Support vector regression (SVR) fits a function that is as flat (small ) as possible while keeping every target within of it. Errors smaller than cost nothing; larger errors cost their excess linearly, weighted by . The fit depends only on the points on or outside the -tube, the support vectors.
At a glance
- Task
- Regression (real-valued target)
- Input → output
- feature vector → real number
- Learns
- one dual coefficient per training point (most are 0) and a bias ; for a linear kernel, equivalently a weight vector
- Loss
- -insensitive: , plus the flatness penalty
- Training
- a convex quadratic program; SMO-type solvers (libsvm) take roughly to time and an kernel cache
- Prediction
- — one kernel evaluation per support vector
- Hyperparameters
- (tube half-width, in target units), (cost of leaving the tube), kernel and its parameters (RBF , polynomial degree)
- Assumes
- features on comparable scales (kernels use distances / dot products); noise of roughly size is acceptable to ignore
The problem
Least squares charges every residual its square. That has two side effects. First, one bad row can dominate: a residual of 4 costs 16 times as much as a residual of 1, so the line tilts toward outliers. In the worked example below, a single data-entry typo drags the least-squares slope from 0.80 to 0.42. Second, every training point contributes to the solution, and in kernelized form (kernel ridge regression) the prediction is a sum over all training points.
Support vector regression keeps the parts of the support vector machine (see the Support vector machines article) that work well (a margin, a convex problem, kernels) and adapts them to real-valued targets. The loss grows only linearly beyond a dead zone of width , so no single point can pull on the fit with more than a fixed force , and points inside the dead zone drop out of the model entirely.
The idea
Picture a tube of half-width around the fitted curve. Points inside it are "close enough": they cost nothing and are ignored. Among all functions whose tube contains the data, SVR picks the flattest one. When the data cannot all fit inside a flat tube, points may stick out, and each one pays per unit it sticks out. The final fit is a compromise between flatness and total protrusion, and only the points on the tube's wall or outside it (the support vectors) take part in it.
Vocabulary
- ε-tube
- the band . Residuals inside it are not penalized.
- ε-insensitive loss
- for residual : flat zero inside the tube, slope 1 outside.
- Slack ξᵢ, ξᵢ*
- how far point sits above () or below () the tube. At most one of the two is non-zero.
- Flatness
- . A small means the prediction changes slowly with ; this is the regularizer, the same one ridge regression uses.
- Dual coefficient βᵢ
- : point 's pull on the fit. Positive if it pushes the fit up (point above the tube), negative if down.
- Support vector
- a training point with , i.e. on the tube wall or outside it. Only these appear in .
- Free vs bounded SV
- free: exactly on the wall, . Bounded: strictly outside, (the force is capped).
- Kernel
- ; lets the same algorithm fit curves. Linear: ; RBF: .
The math
The primal problem. With slack variables for the two sides of the tube:
- weights and bias of the fitted function
- half-width of the tube; residuals smaller than this are free
- slack: how far point is above / below the tube
- price per unit of slack; trades flatness against fitting
At the optimum each slack is as small as the constraints allow, , so the primal is just ridge-style regularization plus the ε-insensitive loss:
- ε-insensitive loss: 0 inside the tube, linear outside it
Compare with least squares, whose loss has derivative : the pull of a point grows without limit as it moves away. The ε-insensitive loss has derivative inside the tube and outside, so every point's pull on the fit is bounded by . That is the whole robustness story in one line.
Deriving the dual. Attach multipliers to the two tube constraints and to :
- multipliers of the upper and lower tube constraints
- multipliers of the non-negativity of the slacks
Setting the derivatives to zero
- 1 gives . The weight vector is a combination of training points.
- 2 gives : the upward and downward pulls balance.
- 3 gives . Since , ; the same holds for . This is the cap on each point's force.
- 4Substitute back. The terms cancel and, because a point cannot be above and below the tube at once (), . What is left depends only on and dot products .
- , the dual coefficient of point
- kernel; for linear SVR
- an L1 penalty on the coefficients: this is what makes most of them exactly 0
The prediction only needs kernel values, so a non-linear kernel fits a curve at no extra conceptual cost (the kernel trick, explained in the Support vector machines article):
- support vectors, the points with
The KKT conditions tell you, for every point, which of three situations it is in. They are both the optimality test and the source of :
| Where the point is | Residual | Dual coefficient | Role |
|---|---|---|---|
| strictly inside the tube | not a support vector; delete it and nothing changes | ||
| on the upper / lower wall | free support vector; pins the fit, gives | ||
| outside the tube | bounded support vector; pulls with the maximum force C |
For any free support vector, exactly, so ; solvers average this over the free support vectors.
What each hyperparameter does, mathematically
- ε
- sets the dead zone of and appears as the L1 weight in the dual. Larger ε → more points inside the tube → fewer support vectors and a flatter, smoother fit. With ε above the spread of the data, and is constant.
- C
- is the box : the maximum force any point can exert. Small C → a flat function that tolerates large protrusions; large C → chase every point outside the tube (overfitting with flexible kernels). As , SVR demands the data fit inside the tube if at all possible.
- γ (RBF)
- kernel width: has reach about . Large γ → each support vector influences only its neighbourhood → wiggly fit; small γ → nearly linear.
Worked example: a salary table with a typo
Eight employees, years of experience and salary in units of 10k dollars. Salary rises steadily with experience, except row 8: someone typed 3.0 instead of something near 8. We fit a linear SVR with and .
| i | experience x (yrs) | salary y (10k USD) |
|---|---|---|
| 1 | 1 | 2.0 |
| 2 | 2 | 2.6 |
| 3 | 3 | 3.9 |
| 4 | 4 | 4.1 |
| 5 | 5 | 5.3 |
| 6 | 6 | 5.6 |
| 7 | 7 | 7.0 |
| 8 | 8 | 3.0 |
For a convex problem, any point that satisfies the KKT conditions is the global optimum. So we can propose a solution and then check it by hand. A linear fit in one dimension has two unknowns, so it is pinned by two points on the tube walls. The solver finds that these are row 1 (lower wall) and row 5 (upper wall).
Solving and verifying the fit by hand (ε = 0.5, C = 1)
- 1Pin the line to the walls. Row 1 on the lower wall means ; row 5 on the upper wall means . So and : .
- 2Residuals (table below): rows 2, 3, 4, 6 have and are inside the tube; rows 1 and 5 are exactly on the walls; row 7 () is above the tube and row 8 () is below it.
- 3Bounded coefficients. Points outside the tube pull with the full force: (pushing up) and (pushing down). Inside points have .
- 4Free coefficients from the two stationarity conditions. : , so . : , so and , .
- 5KKT check. Both free coefficients lie strictly inside and have the right signs (row 5 on the upper wall has , row 1 on the lower wall has ). Every condition holds, so this is the optimum.
- 6Objective. Slack: , . Value . For comparison, the least-squares line scores 4.4466 on this objective.
- 7Predict a new employee with 9 years: , i.e. about 71k dollars. Least squares predicts 6.079.
| i | x | y | f(x) | r = y − f(x) | slack | status | βᵢ |
|---|---|---|---|---|---|---|---|
| 1 | 1 | 2.0 | 2.500 | −0.500 | 0 | lower wall | −0.39375 |
| 2 | 2 | 2.6 | 3.075 | −0.475 | 0 | inside | 0 |
| 3 | 3 | 3.9 | 3.650 | 0.250 | 0 | inside | 0 |
| 4 | 4 | 4.1 | 4.225 | −0.125 | 0 | inside | 0 |
| 5 | 5 | 5.3 | 4.800 | 0.500 | 0 | upper wall | 0.39375 |
| 6 | 6 | 5.6 | 5.375 | 0.225 | 0 | inside | 0 |
| 7 | 7 | 7.0 | 5.950 | 1.050 | 0.550 | above | +1 (= C) |
| 8 | 8 | 3.0 | 6.525 | −3.525 | 3.025 | below | −1 (= −C) |
How much did the typo hurt? Without row 8, SVR fits and least squares fits . With it, SVR's slope moves to 0.575 while least squares collapses to 0.42. On the seven correct rows, SVR has MAE 0.446 and RMSE 0.529; least squares has MAE 0.738 and RMSE 0.884.
The algorithm, step by step
Training an SVR (libsvm-style)
- 1Scale the features (standardize), and decide ε in the target's units, or standardize the target too.
- 2Kernel matrix. Compute (or cache on demand) .
- 3Initialize : a feasible point (, inside the box). The gradient of the smooth part is .
- 4Pick a pair that most violates the KKT conditions. Moving both at once, and , keeps .
- 5Solve the pair exactly. Along that direction the dual is a 1-D piecewise quadratic, with , clipped to the box. Minimize it in closed form and update with two kernel columns.
- 6Repeat until no pair violates KKT by more than the tolerance.
- 7Bias. Average over the free support vectors (or take the midpoint of the feasible interval if there are none).
- 8Keep only the support vectors for prediction.
Complexity
- Training time
- between and kernel evaluations in practice; it grows with C (more bounded SVs to settle) and shrinks with ε (fewer SVs).
- Memory
- for a full kernel matrix; libsvm uses an LRU cache (
cache_size, in MB) instead. - Prediction
- per query. With a linear kernel, collapse to for .
- Rule of thumb
- comfortable up to roughly 10⁴–10⁵ rows; beyond that use LinearSVR or kernel approximations (Nyström, random Fourier features) feeding a linear model.
Hyperparameters, variants and tuning
| ε | w | b | support vectors | Σ slack | objective | f(9) |
|---|---|---|---|---|---|---|
| 0 | 0.720 | 1.280 | 8 | 5.780 | 6.039 | 7.760 |
| 0.25 | 0.700 | 1.550 | 6 | 4.300 | 4.545 | 7.850 |
| 0.5 | 0.575 | 1.925 | 4 | 3.575 | 3.740 | 7.100 |
| 0.75 | 0.450 | 2.300 | 4 | 2.950 | 3.051 | 6.350 |
| 1.0 | 0.325 | 2.675 | 4 | 2.325 | 2.378 | 5.600 |
| 1.5 | 0.143 | 3.357 | 3 | 1.143 | 1.153 | 4.643 |
With , every point is a support vector and SVR becomes L1 (least absolute deviation) regression with a ridge penalty. On a larger problem the sparsity effect is easier to see. Below, 80 noisy samples of (noise s.d. 0.2) are fit with an RBF SVR (, ) and tested on 400 fresh points:
| ε | support vectors (of 80) | test RMSE |
|---|---|---|
| 0 | 80 | 0.225 |
| 0.05 | 69 | 0.223 |
| 0.1 | 57 | 0.213 |
| 0.2 | 32 | 0.211 |
| 0.3 | 15 | 0.214 |
| 0.5 | 5 | 0.298 |
| setting | train RMSE | test RMSE |
|---|---|---|
| C = 0.01, γ = 1 | 0.567 | 0.613 |
| C = 0.1, γ = 1 | 0.216 | 0.234 |
| C = 1, γ = 1 | 0.185 | 0.204 |
| C = 1000, γ = 1 | 0.180 | 0.223 |
| C = 10, γ = 0.01 | 0.421 | 0.434 |
| C = 10, γ = 10 | 0.161 | 0.241 |
| C = 10, γ = 100 | 0.129 | 0.352 |
Variants
- ν-SVR (NuSVR)
- replaces ε with , an upper bound on the fraction of points outside the tube and a lower bound on the fraction of support vectors; ε is then learned. On the worked example (), gives 3 SVs and gives 7.
- LinearSVR
- liblinear solver, linear kernel only, scales to millions of rows. It also penalizes the intercept, so results differ: on the worked example it gives 0.81x + 0.83 rather than 0.575x + 1.925. Center the target or raise
intercept_scaling. - Squared ε-insensitive
- (
loss="squared_epsilon_insensitive"in LinearSVR): smooth, but gives up bounded influence. - Kernel ridge regression
- same kernels with squared loss: a closed-form solution, but dense (all points used) and not robust. SVR with and huge C behaves similarly on clean data.
- Huber / quantile regression
- other linear-tail losses: Huber is quadratic near 0 instead of flat; quantile (pinball) loss is asymmetric and predicts a quantile.
Practical tuning recipe
- 1. Scale
- StandardScaler on X inside a Pipeline (the scaler must be fitted on the training folds only).
- 2. Set ε from the noise
- start near the residual noise s.d. (or 0.1 after standardizing y). It is in target units.
- 3. Search C and γ on a log grid
- e.g. C ∈ {0.1, 1, 10, 100}, γ ∈ {0.01, 0.1, 1, 10} (or
gamma="scale"), with cross-validated RMSE. - 4. Check sparsity
- if nearly every point is a support vector, ε is too small (or C too small); prediction will be slow.
In code
1import numpy as np23def kernel(A, B, kind="linear", gamma=0.5):4 if kind == "linear":5 return A @ B.T6 sq = ((A[:, None, :] - B[None, :, :]) ** 2).sum(-1)7 return np.exp(-gamma * sq) # RBF89def fit_svr(X, y, C=1.0, eps=0.5, kind="linear", gamma=0.5, sweeps=500, tol=1e-12):10 """Dual SVR solved by SMO on pairs (beta_i, beta_j).11 minimise 1/2 b'Kb - y'b + eps*sum|b_i| s.t. -C <= b_i <= C, sum b_i = 0"""12 n, K = len(y), kernel(X, X, kind, gamma)13 beta = np.zeros(n)14 for _ in range(sweeps):15 improved = False16 for i in range(n):17 for j in range(n):18 if i == j:19 continue20 g = K @ beta - y # gradient of the smooth part21 eta = K[i, i] + K[j, j] - 2 * K[i, j] # curvature along the pair direction22 lo = max(-C - beta[i], beta[j] - C) # move: beta_i += t, beta_j -= t23 hi = min(C - beta[i], beta[j] + C) # (keeps sum beta = 0 and the box)24 if hi - lo < 1e-15:25 continue26 phi = lambda t: (0.5 * eta * t * t + (g[i] - g[j]) * t27 + eps * (abs(beta[i] + t) + abs(beta[j] - t)))28 kinks = sorted({lo, hi, *[k for k in (-beta[i], beta[j]) if lo < k < hi]})29 cands = list(kinks)30 for a, c in zip(kinks[:-1], kinks[1:]): # phi is a plain quadratic between kinks31 m = (a + c) / 232 s = eps * (np.sign(beta[i] + m) - np.sign(beta[j] - m))33 if eta > 1e-12:34 cands.append(np.clip(-(g[i] - g[j] + s) / eta, a, c))35 t = min(cands, key=phi)36 if phi(0) - phi(t) > tol:37 beta[i] += t38 beta[j] -= t39 improved = True40 if not improved:41 break42 # bias from the KKT conditions: each point bounds b from below and/or above43 r, tiny = y - K @ beta, 1e-8 # residual before adding b44 lo = np.r_[r[beta < C - tiny] - eps, r[beta < -tiny] + eps].max()45 hi = np.r_[r[beta > -C + tiny] + eps, r[beta > tiny] - eps].min()46 return beta, (lo + hi) / 24748def predict(X_train, beta, b, X_new, kind="linear", gamma=0.5):49 return kernel(X_new, X_train, kind, gamma) @ beta + b5051# the worked example: years of experience -> salary ($10k); the last row is a typo52x = np.array([1, 2, 3, 4, 5, 6, 7, 8], float)53y = np.array([2.0, 2.6, 3.9, 4.1, 5.3, 5.6, 7.0, 3.0])54X = x[:, None]5556beta, b = fit_svr(X, y, C=1.0, eps=0.5)57w = beta @ x # linear kernel: w = sum beta_i x_i58r = y - (w * x + b)59xi = np.maximum(0, np.abs(r) - 0.5)60print("beta =", beta.round(5))61print(f"w = {w:.4f} b = {b:.4f}")62print("residuals =", r.round(3))63print("slack =", xi.round(3), " objective = %.4f" % (0.5 * w * w + xi.sum()))64print("f(9) = %.3f" % predict(X, beta, b, np.array([[9.0]]))[0])65# beta = [-0.39375 0. 0. 0. 0.39375 0. 1. -1.]66# w = 0.5750 b = 1.925067# residuals = [-0.5 -0.475 0.25 -0.125 0.5 0.225 1.05 -3.525]68# slack = [0. 0. 0. 0. 0. 0. 0.55 3.025] objective = 3.740369# f(9) = 7.1001import numpy as np2from sklearn.svm import SVR3from sklearn.linear_model import LinearRegression4from sklearn.pipeline import make_pipeline5from sklearn.preprocessing import StandardScaler6from sklearn.model_selection import GridSearchCV7from sklearn.metrics import mean_squared_error89X = np.arange(1, 9, dtype=float)[:, None]10y = np.array([2.0, 2.6, 3.9, 4.1, 5.3, 5.6, 7.0, 3.0])1112svr = SVR(kernel="linear", C=1.0, epsilon=0.5).fit(X, y)13print("w =", svr.coef_.ravel().round(4), " b =", svr.intercept_.round(4))14print("support vectors:", svr.support_ + 1, " beta:", svr.dual_coef_.ravel().round(5))15ols = LinearRegression().fit(X, y)16print("OLS: w = %.4f b = %.4f" % (ols.coef_[0], ols.intercept_))17print("predict x=9 -> SVR %.3f OLS %.3f" % (svr.predict([[9.0]])[0], ols.predict([[9.0]])[0]))18# w = [0.575] b = [1.925]19# support vectors: [1 5 7 8] beta: [-0.39375 0.39375 1. -1.]20# OLS: w = 0.4202 b = 2.296421# predict x=9 -> SVR 7.100 OLS 6.0792223# A realistic RBF setup: scale X, then tune C, epsilon, gamma by cross-validation24rs = np.random.RandomState(0)25Xs = np.sort(rs.uniform(0, 6, 80))[:, None]26ys = np.sin(Xs[:, 0]) + rs.normal(0, 0.2, 80)27pipe = make_pipeline(StandardScaler(), SVR(kernel="rbf"))28grid = GridSearchCV(pipe, {"svr__C": [0.1, 1, 10, 100],29 "svr__epsilon": [0.05, 0.1, 0.2, 0.3],30 "svr__gamma": [0.1, 1, 10]},31 cv=5, scoring="neg_root_mean_squared_error").fit(Xs, ys)32print("best:", grid.best_params_, " CV RMSE = %.3f" % -grid.best_score_)33Xt = np.linspace(0, 6, 400)[:, None]34yt = np.sin(Xt[:, 0]) + np.random.RandomState(1).normal(0, 0.2, 400)35print("test RMSE = %.3f" % mean_squared_error(yt, grid.predict(Xt)) ** 0.5,36 " support vectors:", len(grid.best_estimator_[-1].support_), "of 80")37# best: {'svr__C': 1, 'svr__epsilon': 0.1, 'svr__gamma': 1} CV RMSE = 0.19938# test RMSE = 0.205 support vectors: 56 of 80Use when
- +You want a regression fit that is robust to a few gross outliers or label errors in y, since each point's influence is capped at C.
- +Small-to-medium data (up to tens of thousands of rows) with a smooth non-linear relationship: an RBF kernel gives a flexible, well-regularized curve.
- +High-dimensional inputs with few rows (spectra, text features): the flatness penalty controls variance.
- +Errors below some tolerance genuinely do not matter, so a dead zone of width ε matches the real cost.
Avoid when
- −Hundreds of thousands of rows or more with a non-linear kernel: training is quadratic or worse and prediction cost grows with the number of support vectors.
- −You need predictive uncertainty or probabilities: SVR gives only a point estimate (use Gaussian processes or Bayesian regression).
- −Heterogeneous tabular data with categorical features and missing values: gradient-boosted trees are usually more accurate with less tuning.
- −You cannot afford a three-way hyperparameter search (ε, C, γ) and careful scaling.
Where it's used
Chemometrics and spectroscopy
Predicting a concentration from hundreds of near-infrared absorbances with only dozens of samples, a classic SVR niche.
Financial and load forecasting
Short-term electricity load or volatility forecasting on lagged features, where heavy-tailed shocks would distort a least-squares fit.
Engineering surrogate models
Replacing an expensive simulation (CFD, FEM) with a smooth RBF-kernel model trained on a few hundred runs.
Remaining-useful-life estimation
Regressing time-to-failure of batteries or bearings from sensor features in predictive maintenance.
Check yourself
With ε = 0.5, what is the ε-insensitive loss of a point whose residual is y − f(x) = −1.3?
Check yourself
In the worked example (ε = 0.5, C = 1), the typo in row 8 is changed from y = 3.0 to y = −10. What happens to the SVR line?
Check yourself
Which training points have 0 < |βᵢ| < C at the SVR optimum?
Check yourself
In the worked example, ε is raised from 0.5 to 1.5 (C = 1). What happens?
Remember
- ✓SVR minimizes : ridge-style flatness plus a loss with a dead zone and linear tails.
- ✓Dual: with and . Points inside the tube have , so the model is sparse in the data.
- ✓KKT roles: inside → ; on the wall → (gives ); outside → , a capped pull that makes SVR robust to target outliers.
- ✓ε (in target units) controls the tube width and sparsity, C the maximum force of a point, γ the RBF kernel's reach. Tune them by cross-validation after scaling.
- ✓Training is a convex QP solved by SMO, roughly quadratic in n; use LinearSVR or kernel approximations for large data.