# Support vector regression

> Fit as flat a function as possible that keeps targets inside an ε-tube, penalizing only points outside it.

Source: https://www.mlfoundry.in/learn/support-vector-regression  
Section: ML Algorithms › Supervised · Regression  
Level: advanced · 17 min read · updated 2026-10-07

**Support vector regression (SVR):** fits a function $f(x)=w^\top\phi(x)+b$ that is as **flat** (small $\|w\|$) as possible while keeping every target within $\pm\varepsilon$ of it. Errors smaller than $\varepsilon$ cost nothing; larger errors cost their excess *linearly*, weighted by $C$. The fit depends only on the points on or outside the $\varepsilon$-tube, the **support vectors**.

- **Task:** Regression (real-valued target)
- **Input → output:** feature vector $x$ → real number $f(x)=\sum_i \beta_i\,k(x_i,x)+b$
- **Learns:** one dual coefficient $\beta_i\in[-C,C]$ per training point (most are 0) and a bias $b$; for a linear kernel, equivalently a weight vector $w$
- **Loss:** $\varepsilon$-insensitive: $\max(0,\,|y-f(x)|-\varepsilon)$, plus the flatness penalty $\tfrac12\|w\|^2$
- **Training:** a convex quadratic program; SMO-type solvers (libsvm) take roughly $O(n^2)$ to $O(n^3)$ time and an $O(n^2)$ kernel cache
- **Prediction:** $O(n_{\text{SV}}\cdot d)$ — one kernel evaluation per support vector
- **Hyperparameters:** $\varepsilon$ (tube half-width, in target units), $C$ (cost of leaving the tube), kernel and its parameters (RBF $\gamma$, polynomial degree)
- **Assumes:** features on comparable scales (kernels use distances / dot products); noise of roughly size $\varepsilon$ is acceptable to ignore

## The problem

Least squares charges every residual its **square**. That has two side effects. First, one bad row can dominate: a residual of 4 costs 16 times as much as a residual of 1, so the line tilts toward outliers. In the worked example below, a single data-entry typo drags the least-squares slope from 0.80 to 0.42. Second, every training point contributes to the solution, and in kernelized form (kernel ridge regression) the prediction is a sum over **all** $n$ training points.

Support vector regression keeps the parts of the **support vector machine** (see the *Support vector machines* article) that work well (a margin, a convex problem, kernels) and adapts them to real-valued targets. The loss grows only **linearly** beyond a dead zone of width $\varepsilon$, so no single point can pull on the fit with more than a fixed force $C$, and points inside the dead zone drop out of the model entirely.

## The idea

Picture a tube of half-width $\varepsilon$ around the fitted curve. Points inside it are "close enough": they cost nothing and are ignored. Among all functions whose tube contains the data, SVR picks the **flattest** one. When the data cannot all fit inside a flat tube, points may stick out, and each one pays $C$ per unit it sticks out. The final fit is a compromise between flatness and total protrusion, and only the points on the tube's wall or outside it (the support vectors) take part in it.

**Vocabulary**

- **ε-tube:** the band $f(x)-\varepsilon\le y\le f(x)+\varepsilon$. Residuals inside it are not penalized.
- **ε-insensitive loss:** $L_\varepsilon(r)=\max(0,|r|-\varepsilon)$ for residual $r=y-f(x)$: flat zero inside the tube, slope 1 outside.
- **Slack ξᵢ, ξᵢ*:** how far point $i$ sits **above** ($\xi_i$) or **below** ($\xi_i^*$) the tube. At most one of the two is non-zero.
- **Flatness:** $\tfrac12\|w\|^2$. A small $w$ means the prediction changes slowly with $x$; this is the regularizer, the same one ridge regression uses.
- **Dual coefficient βᵢ:** $\beta_i=\alpha_i-\alpha_i^*\in[-C,C]$: point $i$'s pull on the fit. Positive if it pushes the fit up (point above the tube), negative if down.
- **Support vector:** a training point with $\beta_i\neq0$, i.e. on the tube wall or outside it. Only these appear in $f$.
- **Free vs bounded SV:** *free*: exactly on the wall, $0<|\beta_i|<C$. *Bounded*: strictly outside, $|\beta_i|=C$ (the force is capped).
- **Kernel:** $k(x,x')=\phi(x)^\top\phi(x')$; lets the same algorithm fit curves. Linear: $x^\top x'$; RBF: $e^{-\gamma\|x-x'\|^2}$.

## The math

**The primal problem.** With slack variables for the two sides of the tube:

$$
\min_{w,b,\xi,\xi^*}\ \frac12\|w\|^2 + C\sum_{i=1}^n(\xi_i+\xi_i^*)\quad\text{s.t.}\quad \begin{cases} y_i-w^\top x_i-b\le\varepsilon+\xi_i\\ w^\top x_i+b-y_i\le\varepsilon+\xi_i^*\\ \xi_i,\ \xi_i^*\ge0\end{cases}
$$

- $w,\ b$: weights and bias of the fitted function
- $\varepsilon$: half-width of the tube; residuals smaller than this are free
- $\xi_i,\ \xi_i^*$: slack: how far point $i$ is above / below the tube
- $C$: price per unit of slack; trades flatness against fitting

At the optimum each slack is as small as the constraints allow, $\xi_i+\xi_i^*=\max(0,|y_i-f(x_i)|-\varepsilon)$, so the primal is just **ridge-style regularization plus the ε-insensitive loss**:

$$
\min_{w,b}\ \frac12\|w\|^2+C\sum_{i=1}^n L_\varepsilon\big(y_i-f(x_i)\big),\qquad L_\varepsilon(r)=\max(0,\ |r|-\varepsilon)
$$

- $L_\varepsilon$: ε-insensitive loss: 0 inside the tube, linear outside it

Compare with least squares, whose loss $r^2$ has derivative $2r$: the pull of a point grows without limit as it moves away. The ε-insensitive loss has derivative $0$ inside the tube and $\pm1$ outside, so every point's pull on the fit is **bounded by $C$**. That is the whole robustness story in one line.

**Deriving the dual.** Attach multipliers $\alpha_i,\alpha_i^*\ge0$ to the two tube constraints and $\eta_i,\eta_i^*\ge0$ to $\xi_i,\xi_i^*\ge0$:

$$
\mathcal L=\tfrac12\|w\|^2+C\sum_i(\xi_i+\xi_i^*)-\sum_i\alpha_i(\varepsilon+\xi_i-y_i+w^\top x_i+b)-\sum_i\alpha_i^*(\varepsilon+\xi_i^*+y_i-w^\top x_i-b)-\sum_i(\eta_i\xi_i+\eta_i^*\xi_i^*)
$$

- $\alpha_i,\ \alpha_i^*$: multipliers of the upper and lower tube constraints
- $\eta_i,\ \eta_i^*$: multipliers of the non-negativity of the slacks

**Setting the derivatives to zero**

1. $\partial\mathcal L/\partial w=0$ gives $w=\sum_i(\alpha_i-\alpha_i^*)\,x_i=\sum_i\beta_i x_i$. The weight vector is a combination of training points.
2. $\partial\mathcal L/\partial b=0$ gives $\sum_i\beta_i=0$: the upward and downward pulls balance.
3. $\partial\mathcal L/\partial\xi_i=0$ gives $C-\alpha_i-\eta_i=0$. Since $\eta_i\ge0$, $0\le\alpha_i\le C$; the same holds for $\alpha_i^*$. This is the cap on each point's force.
4. Substitute back. The $\xi$ terms cancel and, because a point cannot be above and below the tube at once ($\alpha_i\alpha_i^*=0$), $\alpha_i+\alpha_i^*=|\beta_i|$. What is left depends only on $\beta$ and dot products $x_i^\top x_j$.

$$
\max_{\beta}\ -\frac12\sum_{i,j}\beta_i\beta_j\,k(x_i,x_j)+\sum_i y_i\beta_i-\varepsilon\sum_i|\beta_i|\quad\text{s.t.}\quad\sum_i\beta_i=0,\quad -C\le\beta_i\le C
$$

- $\beta_i$: $\alpha_i-\alpha_i^*$, the dual coefficient of point $i$
- $k(x_i,x_j)$: kernel; $x_i^\top x_j$ for linear SVR
- $\varepsilon\sum|\beta_i|$: an L1 penalty on the coefficients: this is what makes most of them exactly 0

The prediction only needs kernel values, so a non-linear kernel fits a curve at no extra conceptual cost (the kernel trick, explained in the *Support vector machines* article):

$$
f(x)=\sum_{i\in\text{SV}}\beta_i\,k(x_i,x)+b
$$

- $\text{SV}$: support vectors, the points with $\beta_i\neq0$

**The KKT conditions** tell you, for every point, which of three situations it is in. They are both the optimality test and the source of $b$:

Complementary slackness for SVR, with residual $r_i=y_i-f(x_i)$.

| Where the point is | Residual | Dual coefficient | Role |
| --- | --- | --- | --- |
| strictly inside the tube | $\|r_i\|<\varepsilon$ | $\beta_i=0$ | not a support vector; delete it and nothing changes |
| on the upper / lower wall | $r_i=\pm\varepsilon$ | $0<\pm\beta_i<C$ | free support vector; pins the fit, gives $b$ |
| outside the tube | $\|r_i\|>\varepsilon$ | $\beta_i=\pm C$ | bounded support vector; pulls with the maximum force C |

For any free support vector, $y_i-f(x_i)=\pm\varepsilon$ exactly, so $b=y_i-\sum_j\beta_j k(x_j,x_i)\mp\varepsilon$; solvers average this over the free support vectors.

**What each hyperparameter does, mathematically**

- **ε:** sets the dead zone of $L_\varepsilon$ and appears as the L1 weight $\varepsilon\sum|\beta_i|$ in the dual. Larger ε → more points inside the tube → fewer support vectors and a flatter, smoother fit. With ε above the spread of the data, $w=0$ and $f$ is constant.
- **C:** is the box $|\beta_i|\le C$: the maximum force any point can exert. Small C → a flat function that tolerates large protrusions; large C → chase every point outside the tube (overfitting with flexible kernels). As $C\to\infty$, SVR demands the data fit inside the tube if at all possible.
- **γ (RBF):** kernel width: $k=e^{-\gamma\|x-x'\|^2}$ has reach about $1/\sqrt{\gamma}$. Large γ → each support vector influences only its neighbourhood → wiggly fit; small γ → nearly linear.

## Worked example: a salary table with a typo

Eight employees, years of experience $x$ and salary $y$ in units of 10k dollars. Salary rises steadily with experience, except row 8: someone typed 3.0 instead of something near 8. We fit a **linear** SVR with $\varepsilon=0.5$ and $C=1$.

Training data. Highlighted: row 8, the data-entry typo.

| i | experience x (yrs) | salary y (10k USD) |
| --- | --- | --- |
| 1 | 1 | 2.0 |
| 2 | 2 | 2.6 |
| 3 | 3 | 3.9 |
| 4 | 4 | 4.1 |
| 5 | 5 | 5.3 |
| 6 | 6 | 5.6 |
| 7 | 7 | 7.0 |
| 8 | 8 | 3.0 |

For a convex problem, any point that satisfies the KKT conditions is the global optimum. So we can propose a solution and then **check** it by hand. A linear fit in one dimension has two unknowns, so it is pinned by two points on the tube walls. The solver finds that these are row 1 (lower wall) and row 5 (upper wall).

**Solving and verifying the fit by hand (ε = 0.5, C = 1)**

1. **Pin the line to the walls.** Row 1 on the lower wall means $f(1)=y_1+\varepsilon=2.5$; row 5 on the upper wall means $f(5)=y_5-\varepsilon=4.8$. So $w=\tfrac{4.8-2.5}{5-1}=0.575$ and $b=2.5-0.575=1.925$: $f(x)=0.575x+1.925$.
2. **Residuals** $r_i=y_i-f(x_i)$ (table below): rows 2, 3, 4, 6 have $|r_i|<0.5$ and are inside the tube; rows 1 and 5 are exactly on the walls; row 7 ($r=+1.05$) is above the tube and row 8 ($r=-3.525$) is below it.
3. **Bounded coefficients.** Points outside the tube pull with the full force: $\beta_7=+C=+1$ (pushing up) and $\beta_8=-C=-1$ (pushing down). Inside points have $\beta_2=\beta_3=\beta_4=\beta_6=0$.
4. **Free coefficients** from the two stationarity conditions. $\sum\beta_i=0$: $\beta_1+\beta_5+1-1=0$, so $\beta_1=-\beta_5$. $w=\sum\beta_ix_i$: $-\beta_5\cdot1+\beta_5\cdot5+1\cdot7-1\cdot8=0.575$, so $4\beta_5=1.575$ and $\beta_5=0.39375$, $\beta_1=-0.39375$.
5. **KKT check.** Both free coefficients lie strictly inside $(-C,C)$ and have the right signs (row 5 on the upper wall has $\beta>0$, row 1 on the lower wall has $\beta<0$). Every condition holds, so this is the optimum.
6. **Objective.** Slack: $\xi_7=1.05-0.5=0.55$, $\xi_8^*=3.525-0.5=3.025$. Value $=\tfrac12(0.575)^2+1\cdot(0.55+3.025)=0.1653+3.575=3.7403$. For comparison, the least-squares line $0.4202x+2.2964$ scores 4.4466 on this objective.
7. **Predict** a new employee with 9 years: $f(9)=0.575\cdot9+1.925=7.100$, i.e. about 71k dollars. Least squares predicts 6.079.

Residuals and dual coefficients at the optimum. Four support vectors (rows 1, 5, 7, 8); $\sum\beta_i=0$ and $\sum\beta_ix_i=0.575=w$.

| i | x | y | f(x) | r = y − f(x) | slack | status | βᵢ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 1 | 1 | 2.0 | 2.500 | −0.500 | 0 | lower wall | −0.39375 |
| 2 | 2 | 2.6 | 3.075 | −0.475 | 0 | inside | 0 |
| 3 | 3 | 3.9 | 3.650 | 0.250 | 0 | inside | 0 |
| 4 | 4 | 4.1 | 4.225 | −0.125 | 0 | inside | 0 |
| 5 | 5 | 5.3 | 4.800 | 0.500 | 0 | upper wall | 0.39375 |
| 6 | 6 | 5.6 | 5.375 | 0.225 | 0 | inside | 0 |
| 7 | 7 | 7.0 | 5.950 | 1.050 | 0.550 | above | +1 (= C) |
| 8 | 8 | 3.0 | 6.525 | −3.525 | 3.025 | below | −1 (= −C) |

**How much did the typo hurt?** Without row 8, SVR fits $0.68x+1.74$ and least squares fits $0.80x+1.157$. With it, SVR's slope moves to 0.575 while least squares collapses to 0.42. On the seven correct rows, SVR has MAE 0.446 and RMSE 0.529; least squares has MAE 0.738 and RMSE 0.884.

> **The typo's pull is capped, so how far off it is does not matter:** Row 8 is a bounded support vector: it pulls with force exactly $C$, whether it is 3.5 or 100 units below the tube. Move $y_8$ from 3.0 to $-10$ and the SVR line stays at $0.575x+1.925$ exactly, while the least-squares slope turns negative ($-0.663$). The SVR fit only changes once the point comes back to the tube; at $y_8=6.4$ it becomes $0.6333x+1.8333$.

> Figure: The worked example, solved live by the same SMO algorithm as the code below. Drag any point vertically; change ε, C or switch to an RBF kernel. Points are coloured by their KKT role: grey inside the tube (β = 0), mint on a wall (free support vector), rose outside (β = ±C, red dashes show the slack). Top right: every residual placed on the ε-insensitive loss, with squared loss dashed for comparison. Bottom right: each point's βᵢ scaled to ±C. Defaults: f(x) = 0.575x + 1.925, 4 support vectors, objective 0.1653 + 1·3.575 = 3.7403, f(9) = 7.10. Drag row 8 down to −0.5 and the line does not move. Raising ε to 1.0 flattens the line to 0.325x + 2.675.

> **Why C barely matters in the linear example:** With $\varepsilon=0.5$, every $C\ge0.1917$ gives exactly the same line; only the $\beta$ values scale with $C$. The loss is piecewise linear, so the optimum sits at a "corner" fixed by two wall points, and $\tfrac12w^2$ is too weak to move it. From step 4, $\beta_5=(w+C)/4$ must not exceed $C$, which fails for $C<0.575/3=0.1917$; below that, row 5 leaves the wall and the line flattens ($C=0.1$ gives $0.5x+2.1$). Switch the figure to RBF to see C really matter.

## The algorithm, step by step

**Training an SVR (libsvm-style)**

1. **Scale** the features (standardize), and decide ε in the target's units, or standardize the target too.
2. **Kernel matrix.** Compute (or cache on demand) $K_{ij}=k(x_i,x_j)$.
3. **Initialize** $\beta=0$: a feasible point ($\sum\beta_i=0$, inside the box). The gradient of the smooth part is $g=K\beta-y$.
4. **Pick a pair** $(i,j)$ that most violates the KKT conditions. Moving both at once, $\beta_i\mathrel{+}=t$ and $\beta_j\mathrel{-}=t$, keeps $\sum\beta=0$.
5. **Solve the pair exactly.** Along that direction the dual is a 1-D piecewise quadratic, $\tfrac12\eta t^2+(g_i-g_j)t+\varepsilon(|\beta_i+t|+|\beta_j-t|)$ with $\eta=K_{ii}+K_{jj}-2K_{ij}$, clipped to the box. Minimize it in closed form and update $g$ with two kernel columns.
6. **Repeat** until no pair violates KKT by more than the tolerance.
7. **Bias.** Average $y_i-\sum_j\beta_jK_{ji}\mp\varepsilon$ over the free support vectors (or take the midpoint of the feasible interval if there are none).
8. **Keep only the support vectors** $\{x_i:\beta_i\neq0\}$ for prediction.

**Complexity**

- **Training time:** between $O(n^2)$ and $O(n^3)$ kernel evaluations in practice; it grows with C (more bounded SVs to settle) and shrinks with ε (fewer SVs).
- **Memory:** $O(n^2)$ for a full kernel matrix; libsvm uses an LRU cache (`cache_size`, in MB) instead.
- **Prediction:** $O(n_{\text{SV}}\cdot d)$ per query. With a linear kernel, collapse to $w$ for $O(d)$.
- **Rule of thumb:** comfortable up to roughly 10⁴–10⁵ rows; beyond that use LinearSVR or kernel approximations (Nyström, random Fourier features) feeding a linear model.

## Hyperparameters, variants and tuning

Effect of ε on the worked example (linear kernel, C = 1). A wider tube swallows more points and lets the line go flatter.

| ε | w | b | support vectors | Σ slack | objective | f(9) |
| --- | --- | --- | --- | --- | --- | --- |
| 0 | 0.720 | 1.280 | 8 | 5.780 | 6.039 | 7.760 |
| 0.25 | 0.700 | 1.550 | 6 | 4.300 | 4.545 | 7.850 |
| **0.5** | **0.575** | **1.925** | **4** | **3.575** | **3.740** | **7.100** |
| 0.75 | 0.450 | 2.300 | 4 | 2.950 | 3.051 | 6.350 |
| 1.0 | 0.325 | 2.675 | 4 | 2.325 | 2.378 | 5.600 |
| 1.5 | 0.143 | 3.357 | 3 | 1.143 | 1.153 | 4.643 |

With $\varepsilon=0$, every point is a support vector and SVR becomes L1 (least absolute deviation) regression with a ridge penalty. On a larger problem the sparsity effect is easier to see. Below, 80 noisy samples of $\sin x$ (noise s.d. 0.2) are fit with an RBF SVR ($C=10$, $\gamma=1$) and tested on 400 fresh points:

RBF SVR on noisy sin(x): ε trades sparsity against accuracy. ε = 0.2–0.3, near the noise level, matches the best test error with 2.5–5× fewer support vectors than ε = 0; ε = 0.5 is too wide.

| ε | support vectors (of 80) | test RMSE |
| --- | --- | --- |
| 0 | 80 | 0.225 |
| 0.05 | 69 | 0.223 |
| 0.1 | 57 | 0.213 |
| **0.2** | **32** | **0.211** |
| 0.3 | 15 | 0.214 |
| 0.5 | 5 | 0.298 |

Same data, ε = 0.1. C and γ control the bias-variance trade-off: too small underfits, too large overfits (training error keeps falling while test error rises).

| setting | train RMSE | test RMSE |
| --- | --- | --- |
| C = 0.01, γ = 1 | 0.567 | 0.613 |
| C = 0.1, γ = 1 | 0.216 | 0.234 |
| **C = 1, γ = 1** | **0.185** | **0.204** |
| C = 1000, γ = 1 | 0.180 | 0.223 |
| C = 10, γ = 0.01 | 0.421 | 0.434 |
| C = 10, γ = 10 | 0.161 | 0.241 |
| C = 10, γ = 100 | 0.129 | 0.352 |

**Variants**

- **ν-SVR (NuSVR):** replaces ε with $\nu\in(0,1]$, an upper bound on the fraction of points outside the tube and a lower bound on the fraction of support vectors; ε is then learned. On the worked example ($C=1$), $\nu=0.25$ gives 3 SVs and $\nu=0.75$ gives 7.
- **LinearSVR:** liblinear solver, linear kernel only, scales to millions of rows. It also penalizes the intercept, so results differ: on the worked example it gives 0.81x + 0.83 rather than 0.575x + 1.925. Center the target or raise `intercept_scaling`.
- **Squared ε-insensitive:** $\max(0,|r|-\varepsilon)^2$ (`loss="squared_epsilon_insensitive"` in LinearSVR): smooth, but gives up bounded influence.
- **Kernel ridge regression:** same kernels with squared loss: a closed-form solution, but dense (all $n$ points used) and not robust. SVR with $\varepsilon\to0$ and huge C behaves similarly on clean data.
- **Huber / quantile regression:** other linear-tail losses: Huber is quadratic near 0 instead of flat; quantile (pinball) loss is asymmetric and predicts a quantile.

**Practical tuning recipe**

- **1. Scale:** StandardScaler on X inside a Pipeline (the scaler must be fitted on the training folds only).
- **2. Set ε from the noise:** start near the residual noise s.d. (or 0.1 after standardizing y). It is in **target units**.
- **3. Search C and γ on a log grid:** e.g. C ∈ {0.1, 1, 10, 100}, γ ∈ {0.01, 0.1, 1, 10} (or `gamma="scale"`), with cross-validated RMSE.
- **4. Check sparsity:** if nearly every point is a support vector, ε is too small (or C too small); prediction will be slow.

## In code

```python
import numpy as np

def kernel(A, B, kind="linear", gamma=0.5):
    if kind == "linear":
        return A @ B.T
    sq = ((A[:, None, :] - B[None, :, :]) ** 2).sum(-1)
    return np.exp(-gamma * sq)                            # RBF

def fit_svr(X, y, C=1.0, eps=0.5, kind="linear", gamma=0.5, sweeps=500, tol=1e-12):
    """Dual SVR solved by SMO on pairs (beta_i, beta_j).
    minimise 1/2 b'Kb - y'b + eps*sum|b_i|   s.t.  -C <= b_i <= C,  sum b_i = 0"""
    n, K = len(y), kernel(X, X, kind, gamma)
    beta = np.zeros(n)
    for _ in range(sweeps):
        improved = False
        for i in range(n):
            for j in range(n):
                if i == j:
                    continue
                g = K @ beta - y                          # gradient of the smooth part
                eta = K[i, i] + K[j, j] - 2 * K[i, j]     # curvature along the pair direction
                lo = max(-C - beta[i], beta[j] - C)       # move: beta_i += t, beta_j -= t
                hi = min(C - beta[i], beta[j] + C)        # (keeps sum beta = 0 and the box)
                if hi - lo < 1e-15:
                    continue
                phi = lambda t: (0.5 * eta * t * t + (g[i] - g[j]) * t
                                 + eps * (abs(beta[i] + t) + abs(beta[j] - t)))
                kinks = sorted({lo, hi, *[k for k in (-beta[i], beta[j]) if lo < k < hi]})
                cands = list(kinks)
                for a, c in zip(kinks[:-1], kinks[1:]):   # phi is a plain quadratic between kinks
                    m = (a + c) / 2
                    s = eps * (np.sign(beta[i] + m) - np.sign(beta[j] - m))
                    if eta > 1e-12:
                        cands.append(np.clip(-(g[i] - g[j] + s) / eta, a, c))
                t = min(cands, key=phi)
                if phi(0) - phi(t) > tol:
                    beta[i] += t
                    beta[j] -= t
                    improved = True
        if not improved:
            break
    # bias from the KKT conditions: each point bounds b from below and/or above
    r, tiny = y - K @ beta, 1e-8                          # residual before adding b
    lo = np.r_[r[beta < C - tiny] - eps, r[beta < -tiny] + eps].max()
    hi = np.r_[r[beta > -C + tiny] + eps, r[beta > tiny] - eps].min()
    return beta, (lo + hi) / 2

def predict(X_train, beta, b, X_new, kind="linear", gamma=0.5):
    return kernel(X_new, X_train, kind, gamma) @ beta + b

# the worked example: years of experience -> salary ($10k); the last row is a typo
x = np.array([1, 2, 3, 4, 5, 6, 7, 8], float)
y = np.array([2.0, 2.6, 3.9, 4.1, 5.3, 5.6, 7.0, 3.0])
X = x[:, None]

beta, b = fit_svr(X, y, C=1.0, eps=0.5)
w = beta @ x                                              # linear kernel: w = sum beta_i x_i
r = y - (w * x + b)
xi = np.maximum(0, np.abs(r) - 0.5)
print("beta      =", beta.round(5))
print(f"w = {w:.4f}   b = {b:.4f}")
print("residuals =", r.round(3))
print("slack     =", xi.round(3), " objective = %.4f" % (0.5 * w * w + xi.sum()))
print("f(9)      = %.3f" % predict(X, beta, b, np.array([[9.0]]))[0])
# beta      = [-0.39375  0.  0.  0.  0.39375  0.  1.  -1.]
# w = 0.5750   b = 1.9250
# residuals = [-0.5 -0.475 0.25 -0.125 0.5 0.225 1.05 -3.525]
# slack     = [0. 0. 0. 0. 0. 0. 0.55 3.025]  objective = 3.7403
# f(9)      = 7.100
```

From scratch: the dual solved by pairwise SMO, bias from the KKT conditions. Prints the worked example exactly.

```python
import numpy as np
from sklearn.svm import SVR
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import mean_squared_error

X = np.arange(1, 9, dtype=float)[:, None]
y = np.array([2.0, 2.6, 3.9, 4.1, 5.3, 5.6, 7.0, 3.0])

svr = SVR(kernel="linear", C=1.0, epsilon=0.5).fit(X, y)
print("w =", svr.coef_.ravel().round(4), " b =", svr.intercept_.round(4))
print("support vectors:", svr.support_ + 1, " beta:", svr.dual_coef_.ravel().round(5))
ols = LinearRegression().fit(X, y)
print("OLS: w = %.4f  b = %.4f" % (ols.coef_[0], ols.intercept_))
print("predict x=9 -> SVR %.3f  OLS %.3f" % (svr.predict([[9.0]])[0], ols.predict([[9.0]])[0]))
# w = [0.575]  b = [1.925]
# support vectors: [1 5 7 8]  beta: [-0.39375  0.39375  1.  -1.]
# OLS: w = 0.4202  b = 2.2964
# predict x=9 -> SVR 7.100  OLS 6.079

# A realistic RBF setup: scale X, then tune C, epsilon, gamma by cross-validation
rs = np.random.RandomState(0)
Xs = np.sort(rs.uniform(0, 6, 80))[:, None]
ys = np.sin(Xs[:, 0]) + rs.normal(0, 0.2, 80)
pipe = make_pipeline(StandardScaler(), SVR(kernel="rbf"))
grid = GridSearchCV(pipe, {"svr__C": [0.1, 1, 10, 100],
                           "svr__epsilon": [0.05, 0.1, 0.2, 0.3],
                           "svr__gamma": [0.1, 1, 10]},
                    cv=5, scoring="neg_root_mean_squared_error").fit(Xs, ys)
print("best:", grid.best_params_, " CV RMSE = %.3f" % -grid.best_score_)
Xt = np.linspace(0, 6, 400)[:, None]
yt = np.sin(Xt[:, 0]) + np.random.RandomState(1).normal(0, 0.2, 400)
print("test RMSE = %.3f" % mean_squared_error(yt, grid.predict(Xt)) ** 0.5,
      " support vectors:", len(grid.best_estimator_[-1].support_), "of 80")
# best: {'svr__C': 1, 'svr__epsilon': 0.1, 'svr__gamma': 1}  CV RMSE = 0.199
# test RMSE = 0.205  support vectors: 56 of 80
```

scikit-learn: the worked example, then a realistic RBF pipeline tuned by cross-validation.

> **Reading sklearn's attributes:** `dual_coef_` holds $\beta_i=\alpha_i-\alpha_i^*$ for the rows listed in `support_` (0-based), with positive values for points above the tube. `coef_` exists only for the linear kernel, where it equals $\sum_i\beta_ix_i$.

**Strengths**

- You want a regression fit that is robust to a few gross outliers or label errors in y, since each point's influence is capped at C.
- Small-to-medium data (up to tens of thousands of rows) with a smooth non-linear relationship: an RBF kernel gives a flexible, well-regularized curve.
- High-dimensional inputs with few rows (spectra, text features): the flatness penalty controls variance.
- Errors below some tolerance genuinely do not matter, so a dead zone of width ε matches the real cost.

**Limitations**

- Hundreds of thousands of rows or more with a non-linear kernel: training is quadratic or worse and prediction cost grows with the number of support vectors.
- You need predictive uncertainty or probabilities: SVR gives only a point estimate (use Gaussian processes or Bayesian regression).
- Heterogeneous tabular data with categorical features and missing values: gradient-boosted trees are usually more accurate with less tuning.
- You cannot afford a three-way hyperparameter search (ε, C, γ) and careful scaling.

- **Chemometrics and spectroscopy:** Predicting a concentration from hundreds of near-infrared absorbances with only dozens of samples, a classic SVR niche.
- **Financial and load forecasting:** Short-term electricity load or volatility forecasting on lagged features, where heavy-tailed shocks would distort a least-squares fit.
- **Engineering surrogate models:** Replacing an expensive simulation (CFD, FEM) with a smooth RBF-kernel model trained on a few hundred runs.
- **Remaining-useful-life estimation:** Regressing time-to-failure of batteries or bearings from sensor features in predictive maintenance.

> **ε and C are in the units of y:** With defaults (`epsilon=0.1`, `C=1`) an RBF SVR on the sin data above scores test R² = 0.927. Multiply y by 1000 (meters to millimeters) and the same call scores R² = 0.046 with all 80 points as support vectors: the slope cap C is now tiny relative to the residuals. Standardize y (e.g. `TransformedTargetRegressor`) or scale ε and C with it.

> **Unscaled features break the kernel:** The RBF kernel uses $\|x-x'\|^2$, so a feature measured in thousands drowns one measured in fractions, and a single γ cannot suit both. Put a StandardScaler inside the pipeline, fitted on training folds only.

> **Robust in y, not in x:** The cap $|\beta_i|\le C$ limits the influence of a wrong **target**. A point with an extreme **feature** value can still dominate a linear fit, because its pull acts through $w=\sum\beta_ix_i$ with a large lever arm $x_i$. Inspect or clip extreme inputs.

> **Sparsity comes from the dead zone:** The term $\varepsilon\sum|\beta_i|$ in the dual is an L1 penalty, the same mechanism that zeroes coefficients in lasso. Here it zeroes **training points** instead of features. In the worked example, refitting on only rows 1, 5, 7 and 8 gives exactly the same line, $0.575x+1.925$.

**Question:** With ε = 0.5, what is the ε-insensitive loss of a point whose residual is y − f(x) = −1.3?

- A. 0
- B. 0.8
- C. 1.3
- D. 1.69

**Answer:** B. 0.8 — $\max(0,|{-1.3}|-0.5)=0.8$. Squared loss would charge $1.69$; inside the tube ($|r|\le0.5$) the cost would be 0.

**Question:** In the worked example (ε = 0.5, C = 1), the typo in row 8 is changed from y = 3.0 to y = −10. What happens to the SVR line?

- A. It tilts down sharply, like least squares
- B. Nothing: it stays 0.575x + 1.925
- C. It becomes flat (w = 0)
- D. Row 8 stops being a support vector

**Answer:** B. Nothing: it stays 0.575x + 1.925 — Row 8 is outside the tube both before and after, so it stays a bounded support vector with $\beta_8=-C$. Its pull is the same, so the solution is unchanged (the objective rises from 3.7403 to 16.7403). Least squares goes from slope 0.42 to −0.663.

**Question:** Which training points have 0 < |βᵢ| < C at the SVR optimum?

- A. Points strictly inside the tube
- B. Points exactly on the tube's boundary
- C. Points outside the tube
- D. All points, when the kernel is RBF

**Answer:** B. Points exactly on the tube's boundary — By the KKT conditions, inside points have $\beta_i=0$, outside points have $|\beta_i|=C$, and only points exactly on a wall can have a value in between. These free support vectors are the ones used to compute $b$.

**Question:** In the worked example, ε is raised from 0.5 to 1.5 (C = 1). What happens?

- A. More support vectors, steeper line
- B. Fewer support vectors (3), flatter line (w ≈ 0.143)
- C. The same line: ε only affects the number of support vectors
- D. Every point becomes a support vector

**Answer:** B. Fewer support vectors (3), flatter line (w ≈ 0.143) — A wider tube contains more points, so fewer constraints bind and the flatness term wins. The fit becomes $0.1429x+3.3571$ with 3 support vectors (rows 1, 7, 8).

**Key points**

- SVR minimizes $\tfrac12\|w\|^2+C\sum\max(0,|y_i-f(x_i)|-\varepsilon)$: ridge-style flatness plus a loss with a dead zone and linear tails.
- Dual: $f(x)=\sum\beta_ik(x_i,x)+b$ with $\sum\beta_i=0$ and $|\beta_i|\le C$. Points inside the tube have $\beta_i=0$, so the model is sparse in the data.
- KKT roles: inside → $\beta=0$; on the wall → $0<|\beta|<C$ (gives $b$); outside → $|\beta|=C$, a capped pull that makes SVR robust to target outliers.
- ε (in target units) controls the tube width and sparsity, C the maximum force of a point, γ the RBF kernel's reach. Tune them by cross-validation after scaling.
- Training is a convex QP solved by SMO, roughly quadratic in n; use LinearSVR or kernel approximations for large data.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
