# Polynomial regression

> Add powers of x as features; the model stays linear in its weights, so ordinary least squares still fits it.

Source: https://www.mlfoundry.in/learn/polynomial-regression  
Section: ML Algorithms › Supervised · Regression  
Level: core · 16 min read · updated 2026-10-07

**Polynomial regression:** fits a curve $\hat y = w_0 + w_1x + w_2x^2 + \dots + w_px^p$ by turning one input $x$ into the feature vector $(1, x, x^2, \dots, x^p)$ and running ordinary **linear** least squares on those features. The curve is non-linear in $x$, but the model is linear in the weights $w$ — which is all least squares needs.

- **Task:** Regression (a numeric target that bends with the input)
- **Input → output:** $x$ (or a few features) → $\hat y = \sum_{k=0}^{p} w_k x^k$
- **Learns:** $p+1$ weights for one input; one weight per monomial ($x_1^2$, $x_1x_2$, …) for several inputs
- **Training:** build the $n\times(p+1)$ design matrix, then least squares: $O(n p^2 + p^3)$
- **Prediction:** $O(p)$ per point (Horner's rule)
- **Hyperparameters:** degree $p$; whether to include interaction terms; feature scaling; optional ridge/lasso penalty $\lambda$
- **Assumes:** the target is a smooth function of $x$ that a low-degree polynomial can follow on the range you care about; usual least-squares noise assumptions (zero-mean, constant variance)

## The problem

A straight line is often the wrong shape. The distance a car needs to stop grows with the **square** of its speed (braking energy is $\tfrac12mv^2$); dose–response curves level off; a learning curve rises then saturates. Fit a line through such data and the residuals are not noise — they form a systematic **U shape**: the line overshoots in the middle and undershoots at both ends.

We could reach for a fully non-linear model (a tree, a neural net, a kernel method), but then we lose the closed-form fit, the readable coefficients, the confidence intervals and all of the tooling from **linear regression**. Polynomial regression is the cheapest fix: keep the linear machinery, change only the **features**.

## The idea

Pretend $x^2$, $x^3$, … are extra columns of your spreadsheet that happen to be computed from $x$. Linear regression does not know or care where its columns came from — it just finds the best weighted sum of them. A weighted sum of $1, x, x^2$ is a parabola, so "linear regression on $(x, x^2)$" *is* "fit the best parabola". Raise the degree and the family of curves gets more flexible; at degree $n-1$ it can pass through every one of $n$ training points.

That last fact is the whole story of the method: **flexibility is one integer**, and turning it up always lowers training error but eventually wrecks predictions — especially near and beyond the edges of the data. Choosing the degree is the real work.

**Vocabulary**

- **Degree p:** the highest power used. $p=1$ is a line, $p=2$ a parabola (quadratic), $p=3$ a cubic.
- **Basis expansion:** replacing $x$ by fixed functions $\phi_k(x)$ before a linear fit; polynomial regression uses $\phi_k(x)=x^k$. Splines and Fourier features are other bases.
- **Design matrix:** the $n\times(p+1)$ matrix $X$ whose row $i$ is $(1, x_i, x_i^2,\dots,x_i^p)$. For powers of one variable it is called a **Vandermonde matrix**.
- **Monomial:** one product of input powers, e.g. $x_1^2x_2$. Its degree is the sum of the exponents (here 3).
- **Interaction term:** a monomial mixing two or more inputs, such as $x_1x_2$: the effect of $x_1$ depends on the value of $x_2$.
- **Interpolation:** a fit with zero training error. With $n$ distinct $x$ values, degree $n-1$ always interpolates.
- **Extrapolation:** predicting outside the range of the training $x$. Polynomials grow like $|x|^p$ there, so high degrees diverge fast.
- **Centring / scaling:** using $t=(x-c)/s$ instead of raw $x$, so the powers stay in a sane numeric range. It changes the weights, never the fitted curve.

## The math

Write the model as a dot product between a weight vector and the expanded feature vector:

$$
\hat y(x)=\sum_{k=0}^{p} w_k\,x^k=\boldsymbol\phi(x)^\top\mathbf w,\qquad \boldsymbol\phi(x)=(1,x,x^2,\dots,x^p)^\top
$$

- $p$: the degree of the polynomial (a hyperparameter)
- $w_k$: weight on the $k$-th power; $w_0$ is the intercept
- $\boldsymbol\phi(x)$: the expanded feature vector of one input $x$

Stack $\boldsymbol\phi(x_i)^\top$ as rows to get the design matrix $X$, and fit by minimizing the sum of squared errors. The only thing that is new compared with ordinary least squares is how $X$ was built:

$$
X=\begin{pmatrix}1&x_1&x_1^2&\cdots&x_1^p\\ 1&x_2&x_2^2&\cdots&x_2^p\\ \vdots&&&&\vdots\\ 1&x_n&x_n^2&\cdots&x_n^p\end{pmatrix},\qquad L(\mathbf w)=\lVert\mathbf y-X\mathbf w\rVert^2=\sum_{i=1}^n\bigl(y_i-\boldsymbol\phi(x_i)^\top\mathbf w\bigr)^2
$$

- $X$: $n\times(p+1)$ Vandermonde design matrix
- $\mathbf y$: vector of the $n$ training targets
- $L(\mathbf w)$: sum of squared residuals (SSE)

**Why it is still linear.** $L$ is a quadratic function of $\mathbf w$ — the powers of $x$ are just fixed numbers inside $X$. Its gradient is $\nabla L=-2X^\top(\mathbf y-X\mathbf w)$. Setting it to zero gives the **normal equations**, exactly as in **multiple linear regression**:

$$
X^\top X\,\hat{\mathbf w}=X^\top\mathbf y\quad\Longrightarrow\quad \hat{\mathbf w}=(X^\top X)^{-1}X^\top\mathbf y,\qquad (X^\top X)_{jk}=\sum_{i} x_i^{\,j+k},\;\; (X^\top\mathbf y)_j=\sum_i x_i^{\,j}y_i
$$

- $X^\top X$: $(p+1)\times(p+1)$ matrix of **power sums** $\sum_i x_i^{j+k}$
- $X^\top\mathbf y$: moments of the target: $\sum_i y_i,\ \sum_i x_iy_i,\ \sum_i x_i^2y_i,\dots$
- $\hat{\mathbf w}$: the least-squares weights (unique when the x values have at least p+1 distinct values)

So the whole fit needs only the power sums of $x$ up to $x^{2p}$ and the moments $\sum x^jy$. A useful trick for hand work: if the $x$ values are **symmetric around 0** (after centring), every odd power sum vanishes, and the normal equations split into an "even" and an "odd" system.

With $d$ input features, a full polynomial of degree $p$ uses every monomial of total degree $\le p$. Counting them is a stars-and-bars problem:

$$
\#\text{features}=\binom{d+p}{p}\qquad\text{e.g. } d=3,p=2:\ 1,a,b,c,a^2,ab,ac,b^2,bc,c^2 \;\to\; 10
$$

- $d$: number of original input features
- $p$: maximum total degree (bias column included in the count)

The count explodes: $d=10$ gives 66 features at $p=2$ and 286 at $p=3$; $d=100$ at $p=3$ gives 176,851. That is why polynomial features are used with a handful of inputs, or combined with regularization.

## What the degree does, mathematically

The models are **nested**: a degree-$p$ polynomial is a degree-$(p+1)$ polynomial with $w_{p+1}=0$. So the minimum training SSE can only go down (or stay put) as $p$ grows, and at $p=n-1$ with distinct $x_i$ the Vandermonde matrix is square and invertible, so the fit interpolates: training error exactly 0. Training error is therefore useless for choosing $p$.

The cost shows up in the **variance of predictions**. Under the usual model $y=f(x)+\varepsilon$ with noise variance $\sigma^2$, a prediction at a new point $x_0$ has

$$
\operatorname{Var}\bigl[\hat y(x_0)\bigr]=\sigma^2\,\boldsymbol\phi(x_0)^\top(X^\top X)^{-1}\boldsymbol\phi(x_0)
$$

- $\sigma^2$: variance of the noise in the targets
- $\boldsymbol\phi(x_0)$: $(1,x_0,\dots,x_0^p)$ — grows like $|x_0|^p$ far from the centre

Each extra degree adds a column, and the quadratic form above grows — slowly in the middle of the data, explosively at the edges. In the worked example below, the prediction variance at 55 km/h (inside the data) is $0.32\,\sigma^2$ for degree 2 and $0.77\,\sigma^2$ for degree 6, but at 90 km/h (outside it) it is $2.43\,\sigma^2$ versus $3431\,\sigma^2$. Low degree = more **bias**, high degree = more **variance**; see **the bias–variance trade-off** for the general theory.

> **It is a basis expansion — the kernel trick's little sibling:** Nothing in least squares cares that $\phi_k(x)=x^k$. Swap in splines, Fourier terms or radial bumps and every equation above still holds. A polynomial kernel $k(x,z)=(1+x^\top z)^p$ in kernel ridge regression or an SVM computes the inner product of these same polynomial features without ever building the $\binom{d+p}{p}$ columns.

## Worked example: stopping distance

Seven test drives: speed (km/h) and the distance (m) needed to stop. The data were simulated from reaction distance (1.5 s at constant speed) plus braking distance $v^2/178$, with noise, then rounded to whole metres — so we know the true curve is a parabola. To keep the arithmetic small and the powers well-scaled, use the **centred and scaled** speed $t=(x-50)/10$, which runs $-3,\dots,3$.

Training data. Because $t$ is symmetric around 0, $\sum t=\sum t^3=0$; the sums we need are $\sum t^2=28$ and $\sum t^4=196$.

| Speed x (km/h) | t = (x−50)/10 | t² | Distance y (m) |
| --- | --- | --- | --- |
| 20 | −3 | 9 | 11 |
| 30 | −2 | 4 | 18 |
| 40 | −1 | 1 | 25 |
| 50 | 0 | 0 | 33 |
| 60 | 1 | 1 | 44 |
| 70 | 2 | 4 | 54 |
| 80 | 3 | 9 | 69 |

**Fitting the parabola ŷ = w₀ + w₁t + w₂t² by hand**

1. **Power sums.** $n=7$, $\sum t=0$, $\sum t^2=28$, $\sum t^3=0$, $\sum t^4=196$. **Moments.** $\sum y=254$, $\sum ty=265$, $\sum t^2y=1077$.
2. **Normal equations.** $X^\top X=\begin{pmatrix}7&0&28\\0&28&0\\28&0&196\end{pmatrix}$, $X^\top\mathbf y=\begin{pmatrix}254\\265\\1077\end{pmatrix}$. The zeros (odd power sums) decouple $w_1$ from the rest.
3. **Slope term.** $28\,w_1=265\Rightarrow w_1=9.4643$ — identical to the straight-line slope, because $t$ and $t^2$ are uncorrelated on symmetric data.
4. **Even terms.** $7w_0+28w_2=254$ and $28w_0+196w_2=1077$. Multiply the first by 4 and subtract: $84\,w_2=61\Rightarrow w_2=0.7262$. Then $w_0=(254-28\cdot0.7262)/7=33.381$.
5. **Fitted curve.** $\hat y=33.381+9.4643\,t+0.7262\,t^2$. In raw km/h that is $\hat y=4.2143+0.22024\,x+0.0072619\,x^2$ — the same curve, different coordinates.
6. **Residuals.** Plugging in each $t$ gives the table below: SSE $=3.095$, training MSE $=0.442$, $R^2=0.9988$. The straight line ($w_0=36.286$, $w_1=9.4643$) has training MSE 6.770.
7. **Predict** a new drive at 55 km/h: $t=0.5$, so $\hat y=33.381+9.4643(0.5)+0.7262(0.25)=$ **38.29 m**.

Fitted values and residuals of the degree-2 fit. No pattern is left in the residuals (compare the U shape a straight line leaves).

| Speed | t | y | ŷ (degree 2) | residual y − ŷ |
| --- | --- | --- | --- | --- |
| 20 | −3 | 11 | 11.524 | −0.524 |
| 30 | −2 | 18 | 17.357 | 0.643 |
| 40 | −1 | 25 | 24.643 | 0.357 |
| 50 | 0 | 33 | 33.381 | −0.381 |
| 60 | 1 | 44 | 43.571 | 0.429 |
| 70 | 2 | 54 | 55.214 | −1.214 |
| 80 | 3 | 69 | 68.310 | 0.690 |

**Which degree?** Six more drives at 25, 35, …, 75 km/h (distances 17, 20, 29, 41, 52, 63 m) were held out as a validation set. Fitting every degree on the seven training drives:

Training error falls monotonically to 0 at degree 6 (7 points, 7 weights). Validation error is lowest at degree 2, and the forecast at 90 km/h — outside the data — runs away as the degree grows. The generating curve gives 83.0 m at 90 km/h.

| Degree | Train MSE | Validation MSE | ŷ at 55 km/h | ŷ at 90 km/h |
| --- | --- | --- | --- | --- |
| 0 | 365.061 | 278.844 | 36.29 | 36.29 |
| 1 | 6.770 | 6.942 | 41.02 | 74.14 |
| **2** | **0.442** | **4.221** | **38.29** | **82.86** |
| 3 | 0.228 | 5.255 | 38.01 | 85.86 |
| 4 | 0.220 | 5.389 | 38.10 | 87.14 |
| 5 | 0.082 | 4.646 | 38.52 | 101.29 |
| 6 | 0.000 | 6.956 | 38.31 | 144.00 |

> Figure: The seven training drives (filled, draggable up and down), the six validation drives (hollow) and the true generating curve (dashed). Slide the degree: residuals are the red sticks, the lower-left chart tracks train and validation MSE for every degree, and the bars show the weights on t…t⁶. With the original data, degree 2 gives train MSE 0.44 and validation MSE 4.22 (the best) and predicts 82.9 m at 90 km/h; degree 6 threads all seven points (train MSE 0.00), validation MSE rises to 6.96 and the 90 km/h forecast shoots to 144 m. Drag one point to see how a single outlier bends a high-degree fit far more than a low-degree one.

> **Interpolation is not the same as accuracy:** Degree 6 reproduces every training distance exactly, yet it is worse on the validation drives than a plain straight line (6.956 vs 6.942) and its 90 km/h forecast is 61 m too long. It has spent its seven weights fitting the rounding noise.

## The algorithm, step by step

**PolynomialRegression.fit(x, y, p)**

1. **Scale.** Compute $t=(x-c)/s$ with $c$ the training mean (or midpoint) and $s$ the standard deviation (or half-range). Store $c,s$ for prediction.
2. **Expand.** Build the design matrix with columns $1,t,t^2,\dots,t^p$ (for several inputs: every monomial up to total degree $p$, optionally interactions only).
3. **Solve.** Minimize $\lVert\mathbf y-X\mathbf w\rVert^2$ (plus $\lambda\lVert\mathbf w\rVert^2$ if regularized) with a QR or SVD least-squares solver — not by explicitly inverting $X^\top X$.
4. **Choose p.** Repeat for $p=1,2,\dots$ and pick the degree with the lowest validation or cross-validated error; prefer the smallest degree within noise of the best.
5. **Predict.** Scale the new $x$ with the stored $c,s$, then evaluate the polynomial with Horner's rule: $\hat y=w_0+t(w_1+t(w_2+\dots+t\,w_p))$.

**Complexity.** Building $X$ costs $O(n\,m)$ for $m=\binom{d+p}{p}$ columns; a QR least-squares solve costs $O(n\,m^2)$; prediction is $O(m)$ per point. For one input $m=p+1$ is tiny, so training is effectively linear in $n$. Selecting the degree by $k$-fold cross-validation multiplies the fit cost by $k$ times the number of degrees tried.

> **Raw powers are numerically toxic:** With raw speeds $20\dots80$, the degree-6 column $x^6$ reaches $2.6\times10^{11}$ while the intercept column is 1. The condition number of that Vandermonde matrix is about $1.1\times10^{14}$ — on the edge of double precision, so solving via $X^\top X$ (which squares it) gives garbage weights. With the centred $t=(x-50)/10$ it is about 2,075. Always centre and scale before expanding, and use a QR/SVD solver (numpy's lstsq, scikit-learn's LinearRegression).

## Variants and practical tuning

**Knobs and variants**

- **Degree p:** the main capacity knob. Pick it by cross-validation; in practice 2 or 3 covers most uses, and degrees above about 4 are rarely justified for a single input.
- **Interactions only:** `PolynomialFeatures(interaction_only=True)` keeps products like $x_1x_2$ but drops pure powers — useful when effects combine but each is linear on its own.
- **Ridge / lasso on top:** add $\lambda\lVert\mathbf w\rVert^2$ (see **Ridge regression**) or $\lambda\lVert\mathbf w\rVert_1$ (see **Lasso regression**) to fit a generous degree and let the penalty decide how much curvature to keep. Scale features first, or the penalty hits high powers unevenly.
- **Orthogonal polynomials:** use Legendre/Chebyshev polynomials or R's poly() instead of raw powers: the columns are uncorrelated, the fit is numerically stable, and adding a degree leaves the lower weights unchanged.
- **Splines:** piecewise cubic polynomials joined smoothly at knots. They bend locally instead of globally, so a wiggle in one region does not propagate to the edges — usually the better choice when one low-degree polynomial is not enough.
- **Weighted / robust fits:** weighted least squares for unequal noise; Huber loss to stop one outlier from dragging a high-degree curve.

Size of the design matrix for a full polynomial in d inputs (bias column included), computed with scikit-learn's PolynomialFeatures.

| Inputs d | Degree p | Columns |
| --- | --- | --- |
| 3 | 2 | 10 |
| 10 | 2 | 66 |
| 10 | 3 | 286 |
| 100 | 3 | 176,851 |

> **Never trust a polynomial outside the data:** Every polynomial of degree $\ge1$ heads to $\pm\infty$ as $|x|$ grows, and the higher the degree the sooner it turns. In the example the degree-6 fit predicts 144 m at 90 km/h (truth 83.0 m), just 10 km/h beyond the last training point. If you must extrapolate, use the lowest degree that a physical argument supports (here: 2), or a model with sensible asymptotes.

> **Runge's phenomenon:** Fitting a high-degree polynomial through equally spaced points makes it oscillate wildly near the ends of the interval, even when the underlying function is smooth. More data at the same spacing does not fix it; lower degree, splines, or Chebyshev-spaced points do.

## In code

```python
import numpy as np

def design(x, degree):
    """Vandermonde matrix: columns 1, x, x^2, ..., x^degree."""
    return np.vander(x, degree + 1, increasing=True)

def fit_poly(x, y, degree, lam=0.0):
    X = design(x, degree)
    P = lam * np.eye(degree + 1); P[0, 0] = 0          # never shrink the intercept
    return np.linalg.solve(X.T @ X + P, X.T @ y)       # normal equations

def predict(w, x):
    return design(x, len(w) - 1) @ w

speed = np.array([20, 30, 40, 50, 60, 70, 80.])        # km/h
dist  = np.array([11, 18, 25, 33, 44, 54, 69.])        # stopping distance, m
t = (speed - 50) / 10                                  # centre and scale first!

w = fit_poly(t, dist, degree=2)
print("weights:", w.round(4))                          # [33.381   9.4643  0.7262]
print("train MSE:", np.mean((dist - predict(w, t))**2).round(4))   # 0.4422
print("55 km/h ->", predict(w, np.array([0.5])).round(2), "m")     # [38.29] m

val_speed = np.array([25, 35, 45, 55, 65, 75.])
val_dist  = np.array([17, 20, 29, 41, 52, 63.])
tv = (val_speed - 50) / 10
for d in range(7):                                     # degree, train MSE, val MSE
    wd = fit_poly(t, dist, d)
    print(d, np.mean((dist - predict(wd, t))**2).round(3),
             np.mean((val_dist - predict(wd, tv))**2).round(3))
# 0 365.061 278.844 | 1 6.77 6.942 | 2 0.442 4.221 | ... | 6 0.0 6.956
```

From scratch with NumPy: the design matrix, the normal equations, and a degree sweep on the worked example.

```python
import numpy as np
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error

speed = np.array([20, 30, 40, 50, 60, 70, 80.]).reshape(-1, 1)
dist  = np.array([11, 18, 25, 33, 44, 54, 69.])
val_speed = np.array([25, 35, 45, 55, 65, 75.]).reshape(-1, 1)
val_dist  = np.array([17, 20, 29, 41, 52, 63.])

model = make_pipeline(PolynomialFeatures(degree=2, include_bias=False), LinearRegression())
model.fit(speed, dist)
lr = model[-1]
print("intercept:", round(lr.intercept_, 4), "coefs:", lr.coef_.round(5))
# intercept: 4.2143 coefs: [0.22024 0.00726]
print("55 km/h ->", model.predict([[55]]).round(2))     # [38.29]
print("R^2 train:", round(model.score(speed, dist), 4)) # 0.9988

# Scale BEFORE expanding, then sweep the degree
for d in range(1, 7):
    m = make_pipeline(StandardScaler(), PolynomialFeatures(d), LinearRegression()).fit(speed, dist)
    print(f"degree {d}: train {mean_squared_error(dist, m.predict(speed)):7.3f}"
          f"  val {mean_squared_error(val_dist, m.predict(val_speed)):7.3f}"
          f"  90 km/h -> {m.predict([[90]])[0]:6.1f} m")
# degree 2: train   0.442  val   4.221  90 km/h ->   82.9 m
# degree 6: train   0.000  val   6.956  90 km/h ->  144.0 m
```

scikit-learn: PolynomialFeatures + LinearRegression in a pipeline. The raw-km/h coefficients match the hand fit converted back from t.

> **Put the expansion inside the pipeline:** Keeping `StandardScaler` and `PolynomialFeatures` inside the pipeline means cross-validation refits the scaling on each training fold, and `GridSearchCV` can tune `polynomialfeatures__degree` directly alongside a ridge `alpha`.

**Strengths**

- The relationship is smooth and visibly curved — a U, a hump, or accelerating growth — and you have one or a few inputs.
- You want a closed-form fit, readable coefficients and standard errors, with all of linear regression's tooling.
- Theory suggests a specific low degree (quadratic drag, braking energy, a parabolic trajectory).
- You need a cheap non-linear baseline before trying trees, splines or kernels.

**Limitations**

- You must predict outside the training range — polynomials diverge.
- There are many inputs: the number of monomials grows as C(d+p, p).
- The curve has sharp local features or changes behaviour in one region: a global polynomial bends everywhere; use splines or trees.
- The data contain outliers and you want a high degree: one bad point bends the whole curve.

- **Physics and engineering calibration:** Stopping distance, drag versus speed, sensor and thermocouple calibration curves are routinely fitted with quadratics and cubics.
- **Economics and pricing:** Quadratic terms capture diminishing returns — e.g. wage versus experience, or demand versus price with saturation.
- **Dose–response and growth:** Low-degree polynomials in time or dose are a standard first model for yield, growth or response curves in agriculture and pharmacology.
- **Feature engineering for linear models:** Degree-2 interactions let a logistic or linear model capture pairwise effects in tabular data, usually with a ridge or lasso penalty on top.

**Question:** Why can ordinary least squares fit ŷ = w₀ + w₁x + w₂x²?

- A. Because x² is approximately linear for small x
- B. Because the model is linear in the weights w; x and x² are just fixed feature columns
- C. Because least squares works for any non-linear model
- D. It cannot — you need gradient descent

**Answer:** B. Because the model is linear in the weights w; x and x² are just fixed feature columns — The loss ‖y − Xw‖² is quadratic in w no matter what numbers are inside X, so the normal equations XᵀXw = Xᵀy still give the exact minimizer.

**Question:** You have 7 training points with distinct x values. What is the lowest degree that is guaranteed to give zero training error?

- A. 3
- B. 5
- C. 6
- D. 7

**Answer:** C. 6 — Degree 6 has 7 weights, so the 7×7 Vandermonde matrix is square and invertible for distinct x: the curve passes through every point. In the worked example degree 6 has train MSE 0.000 but validation MSE 6.956.

**Question:** With the worked-example fit ŷ = 33.381 + 9.4643t + 0.7262t² and t = (x − 50)/10, what is the predicted stopping distance at 60 km/h?

- A. 42.85 m
- B. 43.57 m
- C. 44.00 m
- D. 52.40 m

**Answer:** B. 43.57 m — At 60 km/h, t = 1, so ŷ = 33.381 + 9.464 + 0.726 = 43.571 m (the observed value was 44).

**Question:** How many columns does PolynomialFeatures(degree=2) produce for 3 input features (bias included)?

- A. 6
- B. 7
- C. 9
- D. 10

**Answer:** D. 10 — C(3+2, 2) = 10: the bias 1, three linear terms a, b, c, three squares a², b², c² and three interactions ab, ac, bc.

**Key points**

- Polynomial regression is linear regression on the expanded features 1, x, x², …, xᵖ: non-linear in x, linear in w, solved by the normal equations or QR.
- The degree p is the capacity knob: training error never rises with p and hits 0 at p = n − 1, so choose p by validation or cross-validation.
- Prediction variance $\sigma^2\boldsymbol\phi(x_0)^\top(X^\top X)^{-1}\boldsymbol\phi(x_0)$ explodes at the edges for high degree — never extrapolate a high-degree polynomial.
- Centre and scale x before taking powers; raw powers make the design matrix nearly singular.
- With d inputs a full degree-p expansion has C(d+p, p) columns; for many inputs or local wiggles, use interactions only, regularization, or splines.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
