Polynomial regression

Add powers of x as features; the model stays linear in its weights, so ordinary least squares still fits it.

16 min read1459 wordscore9 viewsupdated 2026-10-07

Definition

Polynomial regression fits a curve y^=w0+w1x+w2x2+⋯+wpxp\hat y = w_0 + w_1x + w_2x^2 + \dots + w_px^p by turning one input xx into the feature vector (1,x,x2,…,xp)(1, x, x^2, \dots, x^p) and running ordinary linear least squares on those features. The curve is non-linear in xx, but the model is linear in the weights ww — which is all least squares needs.

At a glance

Task
Regression (a numeric target that bends with the input)
Input → output
xx (or a few features) → y^=∑k=0pwkxk\hat y = \sum_{k=0}^{p} w_k x^k
Learns
p+1p+1 weights for one input; one weight per monomial (x12x_1^2, x1x2x_1x_2, …) for several inputs
Training
build the n×(p+1)n\times(p+1) design matrix, then least squares: O(np2+p3)O(n p^2 + p^3)
Prediction
O(p)O(p) per point (Horner's rule)
Hyperparameters
degree pp; whether to include interaction terms; feature scaling; optional ridge/lasso penalty λ\lambda
Assumes
the target is a smooth function of xx that a low-degree polynomial can follow on the range you care about; usual least-squares noise assumptions (zero-mean, constant variance)

The problem

A straight line is often the wrong shape. The distance a car needs to stop grows with the square of its speed (braking energy is 12mv2\tfrac12mv^2); dose–response curves level off; a learning curve rises then saturates. Fit a line through such data and the residuals are not noise — they form a systematic U shape: the line overshoots in the middle and undershoots at both ends.

We could reach for a fully non-linear model (a tree, a neural net, a kernel method), but then we lose the closed-form fit, the readable coefficients, the confidence intervals and all of the tooling from linear regression. Polynomial regression is the cheapest fix: keep the linear machinery, change only the features.

The idea

Pretend x2x^2, x3x^3, … are extra columns of your spreadsheet that happen to be computed from xx. Linear regression does not know or care where its columns came from — it just finds the best weighted sum of them. A weighted sum of 1,x,x21, x, x^2 is a parabola, so "linear regression on (x,x2)(x, x^2)" is "fit the best parabola". Raise the degree and the family of curves gets more flexible; at degree n−1n-1 it can pass through every one of nn training points.

That last fact is the whole story of the method: flexibility is one integer, and turning it up always lowers training error but eventually wrecks predictions — especially near and beyond the edges of the data. Choosing the degree is the real work.

Vocabulary

Degree p
the highest power used. p=1p=1 is a line, p=2p=2 a parabola (quadratic), p=3p=3 a cubic.
Basis expansion
replacing xx by fixed functions ϕk(x)\phi_k(x) before a linear fit; polynomial regression uses ϕk(x)=xk\phi_k(x)=x^k. Splines and Fourier features are other bases.
Design matrix
the n×(p+1)n\times(p+1) matrix XX whose row ii is (1,xi,xi2,…,xip)(1, x_i, x_i^2,\dots,x_i^p). For powers of one variable it is called a Vandermonde matrix.
Monomial
one product of input powers, e.g. x12x2x_1^2x_2. Its degree is the sum of the exponents (here 3).
Interaction term
a monomial mixing two or more inputs, such as x1x2x_1x_2: the effect of x1x_1 depends on the value of x2x_2.
Interpolation
a fit with zero training error. With nn distinct xx values, degree n−1n-1 always interpolates.
Extrapolation
predicting outside the range of the training xx. Polynomials grow like ∣x∣p|x|^p there, so high degrees diverge fast.
Centring / scaling
using t=(x−c)/st=(x-c)/s instead of raw xx, so the powers stay in a sane numeric range. It changes the weights, never the fitted curve.

The math

Write the model as a dot product between a weight vector and the expanded feature vector:

y^(x)=∑k=0pwk xk=ϕ(x)⊤w,ϕ(x)=(1,x,x2,…,xp)⊤\hat y(x)=\sum_{k=0}^{p} w_k\,x^k=\boldsymbol\phi(x)^\top\mathbf w,\qquad \boldsymbol\phi(x)=(1,x,x^2,\dots,x^p)^\top
pp
the degree of the polynomial (a hyperparameter)
wkw_k
weight on the kk-th power; w0w_0 is the intercept
ϕ(x)\boldsymbol\phi(x)
the expanded feature vector of one input xx

Stack ϕ(xi)⊤\boldsymbol\phi(x_i)^\top as rows to get the design matrix XX, and fit by minimizing the sum of squared errors. The only thing that is new compared with ordinary least squares is how XX was built:

X=(1x1x12⋯x1p1x2x22⋯x2p⋮⋮1xnxn2⋯xnp),L(w)=∥y−Xw∥2=∑i=1n(yi−ϕ(xi)⊤w)2X=\begin{pmatrix}1&x_1&x_1^2&\cdots&x_1^p\\ 1&x_2&x_2^2&\cdots&x_2^p\\ \vdots&&&&\vdots\\ 1&x_n&x_n^2&\cdots&x_n^p\end{pmatrix},\qquad L(\mathbf w)=\lVert\mathbf y-X\mathbf w\rVert^2=\sum_{i=1}^n\bigl(y_i-\boldsymbol\phi(x_i)^\top\mathbf w\bigr)^2
XX
n×(p+1)n\times(p+1) Vandermonde design matrix
y\mathbf y
vector of the nn training targets
L(w)L(\mathbf w)
sum of squared residuals (SSE)

Why it is still linear. LL is a quadratic function of w\mathbf w — the powers of xx are just fixed numbers inside XX. Its gradient is ∇L=−2X⊤(y−Xw)\nabla L=-2X^\top(\mathbf y-X\mathbf w). Setting it to zero gives the normal equations, exactly as in multiple linear regression:

X⊤X w^=X⊤y⟹w^=(X⊤X)−1X⊤y,(X⊤X)jk=∑ixi j+k,    (X⊤y)j=∑ixi jyiX^\top X\,\hat{\mathbf w}=X^\top\mathbf y\quad\Longrightarrow\quad \hat{\mathbf w}=(X^\top X)^{-1}X^\top\mathbf y,\qquad (X^\top X)_{jk}=\sum_{i} x_i^{\,j+k},\;\; (X^\top\mathbf y)_j=\sum_i x_i^{\,j}y_i
X⊤XX^\top X
(p+1)×(p+1)(p+1)\times(p+1) matrix of power sums ∑ixij+k\sum_i x_i^{j+k}
X⊤yX^\top\mathbf y
moments of the target: ∑iyi, ∑ixiyi, ∑ixi2yi,…\sum_i y_i,\ \sum_i x_iy_i,\ \sum_i x_i^2y_i,\dots
w^\hat{\mathbf w}
the least-squares weights (unique when the x values have at least p+1 distinct values)

So the whole fit needs only the power sums of xx up to x2px^{2p} and the moments ∑xjy\sum x^jy. A useful trick for hand work: if the xx values are symmetric around 0 (after centring), every odd power sum vanishes, and the normal equations split into an "even" and an "odd" system.

With dd input features, a full polynomial of degree pp uses every monomial of total degree ≤p\le p. Counting them is a stars-and-bars problem:

#features=(d+pp)e.g. d=3,p=2: 1,a,b,c,a2,ab,ac,b2,bc,c2  →  10\#\text{features}=\binom{d+p}{p}\qquad\text{e.g. } d=3,p=2:\ 1,a,b,c,a^2,ab,ac,b^2,bc,c^2 \;\to\; 10
dd
number of original input features
pp
maximum total degree (bias column included in the count)

The count explodes: d=10d=10 gives 66 features at p=2p=2 and 286 at p=3p=3; d=100d=100 at p=3p=3 gives 176,851. That is why polynomial features are used with a handful of inputs, or combined with regularization.

What the degree does, mathematically

The models are nested: a degree-pp polynomial is a degree-(p+1)(p+1) polynomial with wp+1=0w_{p+1}=0. So the minimum training SSE can only go down (or stay put) as pp grows, and at p=n−1p=n-1 with distinct xix_i the Vandermonde matrix is square and invertible, so the fit interpolates: training error exactly 0. Training error is therefore useless for choosing pp.

The cost shows up in the variance of predictions. Under the usual model y=f(x)+εy=f(x)+\varepsilon with noise variance σ2\sigma^2, a prediction at a new point x0x_0 has

Var⁡[y^(x0)]=σ2 ϕ(x0)⊤(X⊤X)−1ϕ(x0)\operatorname{Var}\bigl[\hat y(x_0)\bigr]=\sigma^2\,\boldsymbol\phi(x_0)^\top(X^\top X)^{-1}\boldsymbol\phi(x_0)
σ2\sigma^2
variance of the noise in the targets
ϕ(x0)\boldsymbol\phi(x_0)
(1,x0,…,x0p)(1,x_0,\dots,x_0^p) — grows like ∣x0∣p|x_0|^p far from the centre

Each extra degree adds a column, and the quadratic form above grows — slowly in the middle of the data, explosively at the edges. In the worked example below, the prediction variance at 55 km/h (inside the data) is 0.32 σ20.32\,\sigma^2 for degree 2 and 0.77 σ20.77\,\sigma^2 for degree 6, but at 90 km/h (outside it) it is 2.43 σ22.43\,\sigma^2 versus 3431 σ23431\,\sigma^2. Low degree = more bias, high degree = more variance; see the bias–variance trade-off for the general theory.

Worked example: stopping distance

Seven test drives: speed (km/h) and the distance (m) needed to stop. The data were simulated from reaction distance (1.5 s at constant speed) plus braking distance v2/178v^2/178, with noise, then rounded to whole metres — so we know the true curve is a parabola. To keep the arithmetic small and the powers well-scaled, use the centred and scaled speed t=(x−50)/10t=(x-50)/10, which runs −3,…,3-3,\dots,3.

Speed x (km/h)t = (x−50)/10t²Distance y (m)
20−3911
30−2418
40−1125
500033
601144
702454
803969
Training data. Because tt is symmetric around 0, ∑t=∑t3=0\sum t=\sum t^3=0; the sums we need are ∑t2=28\sum t^2=28 and ∑t4=196\sum t^4=196.

Fitting the parabola ŷ = w₀ + w₁t + w₂t² by hand

  1. 1Power sums. n=7n=7, ∑t=0\sum t=0, ∑t2=28\sum t^2=28, ∑t3=0\sum t^3=0, ∑t4=196\sum t^4=196. Moments. ∑y=254\sum y=254, ∑ty=265\sum ty=265, ∑t2y=1077\sum t^2y=1077.
  2. 2Normal equations. X⊤X=(70280280280196)X^\top X=\begin{pmatrix}7&0&28\\0&28&0\\28&0&196\end{pmatrix}, X⊤y=(2542651077)X^\top\mathbf y=\begin{pmatrix}254\\265\\1077\end{pmatrix}. The zeros (odd power sums) decouple w1w_1 from the rest.
  3. 3Slope term. 28 w1=265⇒w1=9.464328\,w_1=265\Rightarrow w_1=9.4643 — identical to the straight-line slope, because tt and t2t^2 are uncorrelated on symmetric data.
  4. 4Even terms. 7w0+28w2=2547w_0+28w_2=254 and 28w0+196w2=107728w_0+196w_2=1077. Multiply the first by 4 and subtract: 84 w2=61⇒w2=0.726284\,w_2=61\Rightarrow w_2=0.7262. Then w0=(254−28⋅0.7262)/7=33.381w_0=(254-28\cdot0.7262)/7=33.381.
  5. 5Fitted curve. y^=33.381+9.4643 t+0.7262 t2\hat y=33.381+9.4643\,t+0.7262\,t^2. In raw km/h that is y^=4.2143+0.22024 x+0.0072619 x2\hat y=4.2143+0.22024\,x+0.0072619\,x^2 — the same curve, different coordinates.
  6. 6Residuals. Plugging in each tt gives the table below: SSE =3.095=3.095, training MSE =0.442=0.442, R2=0.9988R^2=0.9988. The straight line (w0=36.286w_0=36.286, w1=9.4643w_1=9.4643) has training MSE 6.770.
  7. 7Predict a new drive at 55 km/h: t=0.5t=0.5, so y^=33.381+9.4643(0.5)+0.7262(0.25)=\hat y=33.381+9.4643(0.5)+0.7262(0.25)= 38.29 m.
Speedtyŷ (degree 2)residual y − ŷ
20−31111.524−0.524
30−21817.3570.643
40−12524.6430.357
5003333.381−0.381
6014443.5710.429
7025455.214−1.214
8036968.3100.690
Fitted values and residuals of the degree-2 fit. No pattern is left in the residuals (compare the U shape a straight line leaves).

Which degree? Six more drives at 25, 35, …, 75 km/h (distances 17, 20, 29, 41, 52, 63 m) were held out as a validation set. Fitting every degree on the seven training drives:

DegreeTrain MSEValidation MSEŷ at 55 km/hŷ at 90 km/h
0365.061278.84436.2936.29
16.7706.94241.0274.14
20.4424.22138.2982.86
30.2285.25538.0185.86
40.2205.38938.1087.14
50.0824.64638.52101.29
60.0006.95638.31144.00
Training error falls monotonically to 0 at degree 6 (7 points, 7 weights). Validation error is lowest at degree 2, and the forecast at 90 km/h — outside the data — runs away as the degree grows. The generating curve gives 83.0 m at 90 km/h.
distance (m) · ● train (drag) ○ validation · - - true curveextrapolate020406080100102030405060708090speed (km/h)degree 2
train MSE 0.44validation MSE 4.22ŷ(55) = 38.29 mŷ(90) = 82.9 m (curve: 83.0)
row of X for 80 km/h (t = 3): [1, 3, 9] · ŷ = w·row
MSE (log): train · validation.01.111010010000123456best validation: degree 2weights on t…t⁶ (w₀ = 33.38)9.46t0.73t²–t³–t⁴–t⁵–t⁶bars clipped at |w| ≈ 9.6
fig.The seven training drives (filled, draggable up and down), the six validation drives (hollow) and the true generating curve (dashed). Slide the degree: residuals are the red sticks, the lower-left chart tracks train and validation MSE for every degree, and the bars show the weights on t…t⁶. With the original data, degree 2 gives train MSE 0.44 and validation MSE 4.22 (the best) and predicts 82.9 m at 90 km/h; degree 6 threads all seven points (train MSE 0.00), validation MSE rises to 6.96 and the 90 km/h forecast shoots to 144 m. Drag one point to see how a single outlier bends a high-degree fit far more than a low-degree one.

The algorithm, step by step

PolynomialRegression.fit(x, y, p)

  1. 1Scale. Compute t=(x−c)/st=(x-c)/s with cc the training mean (or midpoint) and ss the standard deviation (or half-range). Store c,sc,s for prediction.
  2. 2Expand. Build the design matrix with columns 1,t,t2,…,tp1,t,t^2,\dots,t^p (for several inputs: every monomial up to total degree pp, optionally interactions only).
  3. 3Solve. Minimize ∥y−Xw∥2\lVert\mathbf y-X\mathbf w\rVert^2 (plus λ∥w∥2\lambda\lVert\mathbf w\rVert^2 if regularized) with a QR or SVD least-squares solver — not by explicitly inverting X⊤XX^\top X.
  4. 4Choose p. Repeat for p=1,2,…p=1,2,\dots and pick the degree with the lowest validation or cross-validated error; prefer the smallest degree within noise of the best.
  5. 5Predict. Scale the new xx with the stored c,sc,s, then evaluate the polynomial with Horner's rule: y^=w0+t(w1+t(w2+⋯+t wp))\hat y=w_0+t(w_1+t(w_2+\dots+t\,w_p)).

Complexity. Building XX costs O(n m)O(n\,m) for m=(d+pp)m=\binom{d+p}{p} columns; a QR least-squares solve costs O(n m2)O(n\,m^2); prediction is O(m)O(m) per point. For one input m=p+1m=p+1 is tiny, so training is effectively linear in nn. Selecting the degree by kk-fold cross-validation multiplies the fit cost by kk times the number of degrees tried.

Variants and practical tuning

Knobs and variants

Degree p
the main capacity knob. Pick it by cross-validation; in practice 2 or 3 covers most uses, and degrees above about 4 are rarely justified for a single input.
Interactions only
PolynomialFeatures(interaction_only=True) keeps products like x1x2x_1x_2 but drops pure powers — useful when effects combine but each is linear on its own.
Ridge / lasso on top
add λ∥w∥2\lambda\lVert\mathbf w\rVert^2 (see Ridge regression) or λ∥w∥1\lambda\lVert\mathbf w\rVert_1 (see Lasso regression) to fit a generous degree and let the penalty decide how much curvature to keep. Scale features first, or the penalty hits high powers unevenly.
Orthogonal polynomials
use Legendre/Chebyshev polynomials or R's poly() instead of raw powers: the columns are uncorrelated, the fit is numerically stable, and adding a degree leaves the lower weights unchanged.
Splines
piecewise cubic polynomials joined smoothly at knots. They bend locally instead of globally, so a wiggle in one region does not propagate to the edges — usually the better choice when one low-degree polynomial is not enough.
Weighted / robust fits
weighted least squares for unequal noise; Huber loss to stop one outlier from dragging a high-degree curve.
Inputs dDegree pColumns
3210
10266
103286
1003176,851
Size of the design matrix for a full polynomial in d inputs (bias column included), computed with scikit-learn's PolynomialFeatures.

In code

python
1import numpy as np
2
3def design(x, degree):
4 """Vandermonde matrix: columns 1, x, x^2, ..., x^degree."""
5 return np.vander(x, degree + 1, increasing=True)
6
7def fit_poly(x, y, degree, lam=0.0):
8 X = design(x, degree)
9 P = lam * np.eye(degree + 1); P[0, 0] = 0 # never shrink the intercept
10 return np.linalg.solve(X.T @ X + P, X.T @ y) # normal equations
11
12def predict(w, x):
13 return design(x, len(w) - 1) @ w
14
15speed = np.array([20, 30, 40, 50, 60, 70, 80.]) # km/h
16dist = np.array([11, 18, 25, 33, 44, 54, 69.]) # stopping distance, m
17t = (speed - 50) / 10 # centre and scale first!
18
19w = fit_poly(t, dist, degree=2)
20print("weights:", w.round(4)) # [33.381 9.4643 0.7262]
21print("train MSE:", np.mean((dist - predict(w, t))**2).round(4)) # 0.4422
22print("55 km/h ->", predict(w, np.array([0.5])).round(2), "m") # [38.29] m
23
24val_speed = np.array([25, 35, 45, 55, 65, 75.])
25val_dist = np.array([17, 20, 29, 41, 52, 63.])
26tv = (val_speed - 50) / 10
27for d in range(7): # degree, train MSE, val MSE
28 wd = fit_poly(t, dist, d)
29 print(d, np.mean((dist - predict(wd, t))**2).round(3),
30 np.mean((val_dist - predict(wd, tv))**2).round(3))
31# 0 365.061 278.844 | 1 6.77 6.942 | 2 0.442 4.221 | ... | 6 0.0 6.956
From scratch with NumPy: the design matrix, the normal equations, and a degree sweep on the worked example.
python
1import numpy as np
2from sklearn.pipeline import make_pipeline
3from sklearn.preprocessing import PolynomialFeatures, StandardScaler
4from sklearn.linear_model import LinearRegression
5from sklearn.metrics import mean_squared_error
6
7speed = np.array([20, 30, 40, 50, 60, 70, 80.]).reshape(-1, 1)
8dist = np.array([11, 18, 25, 33, 44, 54, 69.])
9val_speed = np.array([25, 35, 45, 55, 65, 75.]).reshape(-1, 1)
10val_dist = np.array([17, 20, 29, 41, 52, 63.])
11
12model = make_pipeline(PolynomialFeatures(degree=2, include_bias=False), LinearRegression())
13model.fit(speed, dist)
14lr = model[-1]
15print("intercept:", round(lr.intercept_, 4), "coefs:", lr.coef_.round(5))
16# intercept: 4.2143 coefs: [0.22024 0.00726]
17print("55 km/h ->", model.predict([[55]]).round(2)) # [38.29]
18print("R^2 train:", round(model.score(speed, dist), 4)) # 0.9988
19
20# Scale BEFORE expanding, then sweep the degree
21for d in range(1, 7):
22 m = make_pipeline(StandardScaler(), PolynomialFeatures(d), LinearRegression()).fit(speed, dist)
23 print(f"degree {d}: train {mean_squared_error(dist, m.predict(speed)):7.3f}"
24 f" val {mean_squared_error(val_dist, m.predict(val_speed)):7.3f}"
25 f" 90 km/h -> {m.predict([[90]])[0]:6.1f} m")
26# degree 2: train 0.442 val 4.221 90 km/h -> 82.9 m
27# degree 6: train 0.000 val 6.956 90 km/h -> 144.0 m
scikit-learn: PolynomialFeatures + LinearRegression in a pipeline. The raw-km/h coefficients match the hand fit converted back from t.

Use when

  • +The relationship is smooth and visibly curved — a U, a hump, or accelerating growth — and you have one or a few inputs.
  • +You want a closed-form fit, readable coefficients and standard errors, with all of linear regression's tooling.
  • +Theory suggests a specific low degree (quadratic drag, braking energy, a parabolic trajectory).
  • +You need a cheap non-linear baseline before trying trees, splines or kernels.

Avoid when

  • −You must predict outside the training range — polynomials diverge.
  • −There are many inputs: the number of monomials grows as C(d+p, p).
  • −The curve has sharp local features or changes behaviour in one region: a global polynomial bends everywhere; use splines or trees.
  • −The data contain outliers and you want a high degree: one bad point bends the whole curve.

Where it's used

Physics and engineering calibration

Stopping distance, drag versus speed, sensor and thermocouple calibration curves are routinely fitted with quadratics and cubics.

Economics and pricing

Quadratic terms capture diminishing returns — e.g. wage versus experience, or demand versus price with saturation.

Dose–response and growth

Low-degree polynomials in time or dose are a standard first model for yield, growth or response curves in agriculture and pharmacology.

Feature engineering for linear models

Degree-2 interactions let a logistic or linear model capture pairwise effects in tabular data, usually with a ridge or lasso penalty on top.

Check yourself

Why can ordinary least squares fit ŷ = w₀ + w₁x + w₂x²?

Check yourself

You have 7 training points with distinct x values. What is the lowest degree that is guaranteed to give zero training error?

Check yourself

With the worked-example fit ŷ = 33.381 + 9.4643t + 0.7262t² and t = (x − 50)/10, what is the predicted stopping distance at 60 km/h?

Check yourself

How many columns does PolynomialFeatures(degree=2) produce for 3 input features (bias included)?

Remember

  • ✓Polynomial regression is linear regression on the expanded features 1, x, x², …, xᵖ: non-linear in x, linear in w, solved by the normal equations or QR.
  • ✓The degree p is the capacity knob: training error never rises with p and hits 0 at p = n − 1, so choose p by validation or cross-validation.
  • ✓Prediction variance σ2ϕ(x0)⊤(X⊤X)−1ϕ(x0)\sigma^2\boldsymbol\phi(x_0)^\top(X^\top X)^{-1}\boldsymbol\phi(x_0) explodes at the edges for high degree — never extrapolate a high-degree polynomial.
  • ✓Centre and scale x before taking powers; raw powers make the design matrix nearly singular.
  • ✓With d inputs a full degree-p expansion has C(d+p, p) columns; for many inputs or local wiggles, use interactions only, regularization, or splines.
← back to ML Algorithms roadmap