Definition
Polynomial regression fits a curve by turning one input into the feature vector and running ordinary linear least squares on those features. The curve is non-linear in , but the model is linear in the weights — which is all least squares needs.
At a glance
- Task
- Regression (a numeric target that bends with the input)
- Input → output
- (or a few features) →
- Learns
- weights for one input; one weight per monomial (, , …) for several inputs
- Training
- build the design matrix, then least squares:
- Prediction
- per point (Horner's rule)
- Hyperparameters
- degree ; whether to include interaction terms; feature scaling; optional ridge/lasso penalty
- Assumes
- the target is a smooth function of that a low-degree polynomial can follow on the range you care about; usual least-squares noise assumptions (zero-mean, constant variance)
The problem
A straight line is often the wrong shape. The distance a car needs to stop grows with the square of its speed (braking energy is ); dose–response curves level off; a learning curve rises then saturates. Fit a line through such data and the residuals are not noise — they form a systematic U shape: the line overshoots in the middle and undershoots at both ends.
We could reach for a fully non-linear model (a tree, a neural net, a kernel method), but then we lose the closed-form fit, the readable coefficients, the confidence intervals and all of the tooling from linear regression. Polynomial regression is the cheapest fix: keep the linear machinery, change only the features.
The idea
Pretend , , … are extra columns of your spreadsheet that happen to be computed from . Linear regression does not know or care where its columns came from — it just finds the best weighted sum of them. A weighted sum of is a parabola, so "linear regression on " is "fit the best parabola". Raise the degree and the family of curves gets more flexible; at degree it can pass through every one of training points.
That last fact is the whole story of the method: flexibility is one integer, and turning it up always lowers training error but eventually wrecks predictions — especially near and beyond the edges of the data. Choosing the degree is the real work.
Vocabulary
- Degree p
- the highest power used. is a line, a parabola (quadratic), a cubic.
- Basis expansion
- replacing by fixed functions before a linear fit; polynomial regression uses . Splines and Fourier features are other bases.
- Design matrix
- the matrix whose row is . For powers of one variable it is called a Vandermonde matrix.
- Monomial
- one product of input powers, e.g. . Its degree is the sum of the exponents (here 3).
- Interaction term
- a monomial mixing two or more inputs, such as : the effect of depends on the value of .
- Interpolation
- a fit with zero training error. With distinct values, degree always interpolates.
- Extrapolation
- predicting outside the range of the training . Polynomials grow like there, so high degrees diverge fast.
- Centring / scaling
- using instead of raw , so the powers stay in a sane numeric range. It changes the weights, never the fitted curve.
The math
Write the model as a dot product between a weight vector and the expanded feature vector:
- the degree of the polynomial (a hyperparameter)
- weight on the -th power; is the intercept
- the expanded feature vector of one input
Stack as rows to get the design matrix , and fit by minimizing the sum of squared errors. The only thing that is new compared with ordinary least squares is how was built:
- Vandermonde design matrix
- vector of the training targets
- sum of squared residuals (SSE)
Why it is still linear. is a quadratic function of — the powers of are just fixed numbers inside . Its gradient is . Setting it to zero gives the normal equations, exactly as in multiple linear regression:
- matrix of power sums
- moments of the target:
- the least-squares weights (unique when the x values have at least p+1 distinct values)
So the whole fit needs only the power sums of up to and the moments . A useful trick for hand work: if the values are symmetric around 0 (after centring), every odd power sum vanishes, and the normal equations split into an "even" and an "odd" system.
With input features, a full polynomial of degree uses every monomial of total degree . Counting them is a stars-and-bars problem:
- number of original input features
- maximum total degree (bias column included in the count)
The count explodes: gives 66 features at and 286 at ; at gives 176,851. That is why polynomial features are used with a handful of inputs, or combined with regularization.
What the degree does, mathematically
The models are nested: a degree- polynomial is a degree- polynomial with . So the minimum training SSE can only go down (or stay put) as grows, and at with distinct the Vandermonde matrix is square and invertible, so the fit interpolates: training error exactly 0. Training error is therefore useless for choosing .
The cost shows up in the variance of predictions. Under the usual model with noise variance , a prediction at a new point has
- variance of the noise in the targets
- — grows like far from the centre
Each extra degree adds a column, and the quadratic form above grows — slowly in the middle of the data, explosively at the edges. In the worked example below, the prediction variance at 55 km/h (inside the data) is for degree 2 and for degree 6, but at 90 km/h (outside it) it is versus . Low degree = more bias, high degree = more variance; see the bias–variance trade-off for the general theory.
Worked example: stopping distance
Seven test drives: speed (km/h) and the distance (m) needed to stop. The data were simulated from reaction distance (1.5 s at constant speed) plus braking distance , with noise, then rounded to whole metres — so we know the true curve is a parabola. To keep the arithmetic small and the powers well-scaled, use the centred and scaled speed , which runs .
| Speed x (km/h) | t = (x−50)/10 | t² | Distance y (m) |
|---|---|---|---|
| 20 | −3 | 9 | 11 |
| 30 | −2 | 4 | 18 |
| 40 | −1 | 1 | 25 |
| 50 | 0 | 0 | 33 |
| 60 | 1 | 1 | 44 |
| 70 | 2 | 4 | 54 |
| 80 | 3 | 9 | 69 |
Fitting the parabola ŷ = w₀ + w₁t + w₂t² by hand
- 1Power sums. , , , , . Moments. , , .
- 2Normal equations. , . The zeros (odd power sums) decouple from the rest.
- 3Slope term. — identical to the straight-line slope, because and are uncorrelated on symmetric data.
- 4Even terms. and . Multiply the first by 4 and subtract: . Then .
- 5Fitted curve. . In raw km/h that is — the same curve, different coordinates.
- 6Residuals. Plugging in each gives the table below: SSE , training MSE , . The straight line (, ) has training MSE 6.770.
- 7Predict a new drive at 55 km/h: , so 38.29 m.
| Speed | t | y | ŷ (degree 2) | residual y − ŷ |
|---|---|---|---|---|
| 20 | −3 | 11 | 11.524 | −0.524 |
| 30 | −2 | 18 | 17.357 | 0.643 |
| 40 | −1 | 25 | 24.643 | 0.357 |
| 50 | 0 | 33 | 33.381 | −0.381 |
| 60 | 1 | 44 | 43.571 | 0.429 |
| 70 | 2 | 54 | 55.214 | −1.214 |
| 80 | 3 | 69 | 68.310 | 0.690 |
Which degree? Six more drives at 25, 35, …, 75 km/h (distances 17, 20, 29, 41, 52, 63 m) were held out as a validation set. Fitting every degree on the seven training drives:
| Degree | Train MSE | Validation MSE | ŷ at 55 km/h | ŷ at 90 km/h |
|---|---|---|---|---|
| 0 | 365.061 | 278.844 | 36.29 | 36.29 |
| 1 | 6.770 | 6.942 | 41.02 | 74.14 |
| 2 | 0.442 | 4.221 | 38.29 | 82.86 |
| 3 | 0.228 | 5.255 | 38.01 | 85.86 |
| 4 | 0.220 | 5.389 | 38.10 | 87.14 |
| 5 | 0.082 | 4.646 | 38.52 | 101.29 |
| 6 | 0.000 | 6.956 | 38.31 | 144.00 |
The algorithm, step by step
PolynomialRegression.fit(x, y, p)
- 1Scale. Compute with the training mean (or midpoint) and the standard deviation (or half-range). Store for prediction.
- 2Expand. Build the design matrix with columns (for several inputs: every monomial up to total degree , optionally interactions only).
- 3Solve. Minimize (plus if regularized) with a QR or SVD least-squares solver — not by explicitly inverting .
- 4Choose p. Repeat for and pick the degree with the lowest validation or cross-validated error; prefer the smallest degree within noise of the best.
- 5Predict. Scale the new with the stored , then evaluate the polynomial with Horner's rule: .
Complexity. Building costs for columns; a QR least-squares solve costs ; prediction is per point. For one input is tiny, so training is effectively linear in . Selecting the degree by -fold cross-validation multiplies the fit cost by times the number of degrees tried.
Variants and practical tuning
Knobs and variants
- Degree p
- the main capacity knob. Pick it by cross-validation; in practice 2 or 3 covers most uses, and degrees above about 4 are rarely justified for a single input.
- Interactions only
PolynomialFeatures(interaction_only=True)keeps products like but drops pure powers — useful when effects combine but each is linear on its own.- Ridge / lasso on top
- add (see Ridge regression) or (see Lasso regression) to fit a generous degree and let the penalty decide how much curvature to keep. Scale features first, or the penalty hits high powers unevenly.
- Orthogonal polynomials
- use Legendre/Chebyshev polynomials or R's poly() instead of raw powers: the columns are uncorrelated, the fit is numerically stable, and adding a degree leaves the lower weights unchanged.
- Splines
- piecewise cubic polynomials joined smoothly at knots. They bend locally instead of globally, so a wiggle in one region does not propagate to the edges — usually the better choice when one low-degree polynomial is not enough.
- Weighted / robust fits
- weighted least squares for unequal noise; Huber loss to stop one outlier from dragging a high-degree curve.
| Inputs d | Degree p | Columns |
|---|---|---|
| 3 | 2 | 10 |
| 10 | 2 | 66 |
| 10 | 3 | 286 |
| 100 | 3 | 176,851 |
In code
1import numpy as np23def design(x, degree):4 """Vandermonde matrix: columns 1, x, x^2, ..., x^degree."""5 return np.vander(x, degree + 1, increasing=True)67def fit_poly(x, y, degree, lam=0.0):8 X = design(x, degree)9 P = lam * np.eye(degree + 1); P[0, 0] = 0 # never shrink the intercept10 return np.linalg.solve(X.T @ X + P, X.T @ y) # normal equations1112def predict(w, x):13 return design(x, len(w) - 1) @ w1415speed = np.array([20, 30, 40, 50, 60, 70, 80.]) # km/h16dist = np.array([11, 18, 25, 33, 44, 54, 69.]) # stopping distance, m17t = (speed - 50) / 10 # centre and scale first!1819w = fit_poly(t, dist, degree=2)20print("weights:", w.round(4)) # [33.381 9.4643 0.7262]21print("train MSE:", np.mean((dist - predict(w, t))**2).round(4)) # 0.442222print("55 km/h ->", predict(w, np.array([0.5])).round(2), "m") # [38.29] m2324val_speed = np.array([25, 35, 45, 55, 65, 75.])25val_dist = np.array([17, 20, 29, 41, 52, 63.])26tv = (val_speed - 50) / 1027for d in range(7): # degree, train MSE, val MSE28 wd = fit_poly(t, dist, d)29 print(d, np.mean((dist - predict(wd, t))**2).round(3),30 np.mean((val_dist - predict(wd, tv))**2).round(3))31# 0 365.061 278.844 | 1 6.77 6.942 | 2 0.442 4.221 | ... | 6 0.0 6.9561import numpy as np2from sklearn.pipeline import make_pipeline3from sklearn.preprocessing import PolynomialFeatures, StandardScaler4from sklearn.linear_model import LinearRegression5from sklearn.metrics import mean_squared_error67speed = np.array([20, 30, 40, 50, 60, 70, 80.]).reshape(-1, 1)8dist = np.array([11, 18, 25, 33, 44, 54, 69.])9val_speed = np.array([25, 35, 45, 55, 65, 75.]).reshape(-1, 1)10val_dist = np.array([17, 20, 29, 41, 52, 63.])1112model = make_pipeline(PolynomialFeatures(degree=2, include_bias=False), LinearRegression())13model.fit(speed, dist)14lr = model[-1]15print("intercept:", round(lr.intercept_, 4), "coefs:", lr.coef_.round(5))16# intercept: 4.2143 coefs: [0.22024 0.00726]17print("55 km/h ->", model.predict([[55]]).round(2)) # [38.29]18print("R^2 train:", round(model.score(speed, dist), 4)) # 0.99881920# Scale BEFORE expanding, then sweep the degree21for d in range(1, 7):22 m = make_pipeline(StandardScaler(), PolynomialFeatures(d), LinearRegression()).fit(speed, dist)23 print(f"degree {d}: train {mean_squared_error(dist, m.predict(speed)):7.3f}"24 f" val {mean_squared_error(val_dist, m.predict(val_speed)):7.3f}"25 f" 90 km/h -> {m.predict([[90]])[0]:6.1f} m")26# degree 2: train 0.442 val 4.221 90 km/h -> 82.9 m27# degree 6: train 0.000 val 6.956 90 km/h -> 144.0 mUse when
- +The relationship is smooth and visibly curved — a U, a hump, or accelerating growth — and you have one or a few inputs.
- +You want a closed-form fit, readable coefficients and standard errors, with all of linear regression's tooling.
- +Theory suggests a specific low degree (quadratic drag, braking energy, a parabolic trajectory).
- +You need a cheap non-linear baseline before trying trees, splines or kernels.
Avoid when
- −You must predict outside the training range — polynomials diverge.
- −There are many inputs: the number of monomials grows as C(d+p, p).
- −The curve has sharp local features or changes behaviour in one region: a global polynomial bends everywhere; use splines or trees.
- −The data contain outliers and you want a high degree: one bad point bends the whole curve.
Where it's used
Physics and engineering calibration
Stopping distance, drag versus speed, sensor and thermocouple calibration curves are routinely fitted with quadratics and cubics.
Economics and pricing
Quadratic terms capture diminishing returns — e.g. wage versus experience, or demand versus price with saturation.
Dose–response and growth
Low-degree polynomials in time or dose are a standard first model for yield, growth or response curves in agriculture and pharmacology.
Feature engineering for linear models
Degree-2 interactions let a logistic or linear model capture pairwise effects in tabular data, usually with a ridge or lasso penalty on top.
Check yourself
Why can ordinary least squares fit ŷ = w₀ + w₁x + w₂x²?
Check yourself
You have 7 training points with distinct x values. What is the lowest degree that is guaranteed to give zero training error?
Check yourself
With the worked-example fit ŷ = 33.381 + 9.4643t + 0.7262t² and t = (x − 50)/10, what is the predicted stopping distance at 60 km/h?
Check yourself
How many columns does PolynomialFeatures(degree=2) produce for 3 input features (bias included)?
Remember
- ✓Polynomial regression is linear regression on the expanded features 1, x, x², …, xᵖ: non-linear in x, linear in w, solved by the normal equations or QR.
- ✓The degree p is the capacity knob: training error never rises with p and hits 0 at p = n − 1, so choose p by validation or cross-validation.
- ✓Prediction variance explodes at the edges for high degree — never extrapolate a high-degree polynomial.
- ✓Centre and scale x before taking powers; raw powers make the design matrix nearly singular.
- ✓With d inputs a full degree-p expansion has C(d+p, p) columns; for many inputs or local wiggles, use interactions only, regularization, or splines.