Definition
KNN regression predicts a number for a new input by finding the training points closest to and averaging their targets, either equally or with closer points counting more. It fits no parameters. The stored training set is the model, and the shape of the prediction curve comes entirely from the data.
At a glance
- Task
- Regression (numeric target)
- Input → output
- query → , a (weighted) mean of training targets
- Learns
- nothing: it stores all pairs ("lazy" or instance-based learning)
- Training cost
- to store; if you build a KD-tree or ball tree
- Prediction cost
- per query by brute force; roughly with a tree index in low dimension
- Hyperparameters
n_neighborsk,weights(uniform or distance), distance metric (p), index algorithm- Assumes
- the target changes smoothly: points close in your metric have similar . Features must be on comparable scales
The problem
Electricity demand against outdoor temperature has a U-shape. Cold days need heating, hot days need air-conditioning, and mild days need the least. A straight line (see linear regression) cannot bend, so it fits this U badly. On the 12-day dataset below, the best line explains only of the variance, and its leave-one-out error is 39.17 GWh², which is worse than always predicting the mean.
You could add and fit a parabola (polynomial regression). But that only works if you guess the right functional form. The real curve might be flat on one side, kinked at 18 °C, or bumpy. KNN regression makes no guess about the shape. It says: to predict demand at 21 °C, look at what happened on the days with temperatures nearest 21 °C.
The idea
Picture the training points as a scatter plot. Put a vertical line at the query and widen a window around it until it contains points. The prediction is the average height of those points. Slide along the axis and the window slides with it, so the prediction traces a curve that follows the data's local level, wherever it goes.
With you copy the nearest point's target exactly, which reproduces every bit of noise. With every window contains the whole dataset, so you predict the global mean everywhere. Everything useful lies between those two extremes, and choosing is the whole game.
Vocabulary
- Query point q
- the input you want a prediction for
- Neighbourhood N_k(q)
- the indices of the training points closest to
- k-th neighbour distance
- , the radius of the smallest ball around that holds training points. It shrinks where data is dense and grows where data is sparse
- Uniform weighting
- every neighbour gets weight , so is a plain mean
- Distance weighting
- neighbour gets weight , so nearer points pull harder
- Lazy learner
- all work happens at prediction time; "training" just stores the data
- Smoother
- a method whose prediction is a weighted average of training targets. KNN is a linear smoother: the weights depend on , not on
The math
Fix a distance (usually Euclidean, ). Sort the training points by distance to and keep the first . Their indices form .
- query input
- indices of the training points nearest to
- target of training point
- number of neighbours averaged
Why the mean? Ask for the single constant that best fits the neighbours in squared error, . Setting gives , so is the mean. KNN regression is therefore a local constant least-squares fit: at each query it solves a tiny regression problem using only nearby points. Under absolute error the same argument gives the neighbours' median instead, which is a robust variant.
- weight of neighbour i; larger for closer points
- distance from the query to neighbour
This is the minimiser of the weighted squared error . Repeat the derivative: . Distance weighting makes the curve continuous and lets it pass through every training point, because as the weight . If sits exactly on a training point, scikit-learn gives that point all the weight.
How k trades bias for variance. Assume , where the noise has variance , and treat the training inputs as fixed. Then the expected squared error at splits into three parts:
- the true, unknown regression function
- irreducible noise variance
- squared bias: how far the neighbours' true values sit from f(q) on average
- variance: averaging k independent noises divides their variance by k
Raising shrinks the variance term . But it also pulls in neighbours that are further away, whose differs more from , so the bias grows. When the bias is close to 0 and the variance equals . When the variance is , but the bias equals the whole gap between and the global mean. The theory says KNN is consistent (its error approaches the noise floor) if while . In practice you pick by cross-validation. The bias–variance trade-off article covers this decomposition in general.
Worked example: temperature → electricity demand
Twelve days of an illustrative regional grid: daily mean temperature (°C) and electricity demand (GWh). We want the demand on a 21 °C day, using .
| Day | Temp x (°C) | Demand y (GWh) |
|---|---|---|
| 1 | −2.5 | 41.8 |
| 2 | 0.0 | 37.9 |
| 3 | 3.0 | 32.1 |
| 4 | 5.5 | 30.4 |
| 5 | 7.5 | 27.1 |
| 6 | 10.5 | 25.4 |
| 7 | 13.0 | 22.6 |
| 8 | 17.0 | 23.7 |
| 9 | 20.0 | 23.4 |
| 10 | 24.0 | 26.9 |
| 11 | 27.5 | 28.8 |
| 12 | 30.5 | 32.6 |
Predict demand at q = 21 °C
- 1Distances. With one feature, . The nearest days are 20 °C (), 24 °C (), 17 °C (), then 27.5 °C () and 13 °C ().
- 2Keep the 3 nearest: days 9, 10 and 8, with targets 23.4, 26.9 and 23.7. The neighbourhood radius is °C, so the window is [17, 25].
- 3Uniform average: 24.67 GWh.
- 4Distance weights: , and .
- 5Weighted sum: .
- 6Distance-weighted prediction: 24.18 GWh. The 20 °C day carries of the weight, so the prediction moves toward its 23.4.
| k | Neighbours (°C) | Uniform ŷ | 1/d-weighted ŷ |
|---|---|---|---|
| 1 | 20 | 23.40 | 23.40 |
| 2 | 20, 24 | 25.15 | 24.27 |
| 3 | 20, 24, 17 | 24.67 | 24.18 |
| 5 | 20, 24, 17, 27.5, 13 | 25.08 | 24.46 |
| 12 | all | 29.39 | 25.88 |
To choose , use leave-one-out cross-validation: predict each day from the other 11 and average the squared errors. With uniform weights, gives 8.62 GWh², gives 7.08, gives 11.81, and gives 38.40. With weights the best is again , at 5.68. Both are far below the straight line's 39.17. Training error tells you nothing here: with it is exactly 0, because every point is its own nearest neighbour.
Reading the fitted curve
With one feature and uniform weights, the prediction only changes when the query's set of neighbours changes. That happens at midpoints between training points, so the fit is a step function. Unlike a regression tree, whose steps come from a few learned splits, KNN has a step at every point where its neighbour set changes. Three effects are worth seeing in the figure:
At a glance
- Boundary bias
- at the coldest day (−2.5 °C, demand 41.8), every neighbour lies to the right, at warmer and lower-demand days. With the fit there is , which is 4.5 GWh too low. The averaging window cannot centre itself on an edge point
- No extrapolation
- at 40 °C (hotter than any training day) predicts 29.43, the mean of the three hottest days, and it would predict the same at 50 °C. KNN outputs always lie inside the range of the training targets. The straight line predicts 21.95 at 40 °C, which is wrong in the other direction
- Adaptive width
- the window has a fixed count, not a fixed width. Where days are dense it is narrow; where they are sparse it widens automatically
The algorithm, step by step
fit(X, y) and predict(q)
- 1Fit: scale the features (store the means and standard deviations), then store
Xandy. Optionally build a KD-tree or ball tree overX. - 2Distances: for a query , compute for all training points. Brute force costs .
- 3Select: take the smallest distances. A partial selection (
argpartition) does this in , without a full sort. - 4Weight: (uniform) or . If some , use only those exact matches.
- 5Average: return .
- 6Tune: choose (and the weighting and metric) by cross-validated MSE or MAE, never by training error.
Complexity. Training is free, and memory is because the whole dataset is the model. A batch of queries costs by brute force. Tree indexes cut a query to roughly when is small (under about 20). In high dimension they degrade back to brute force, and approximate nearest-neighbour libraries (FAISS, HNSW, Annoy) take over, trading exactness for speed.
Variants and tuning
| Choice | Options | Effect | Starting point |
|---|---|---|---|
k (n_neighbors) | 1 … n | small k: low bias, high variance; large k: smooth fit, flattens peaks and edges | grid over 1–50 with CV; often near √n |
weights | uniform, distance, or a callable | 1/d makes the fit continuous and less sensitive to large k; it interpolates the training data exactly | try both. 1/d helps when density varies |
Metric (p) | p=2 Euclidean, p=1 Manhattan, cosine… | defines which points count as near; Manhattan is less dominated by one large difference | Euclidean on standardized features |
| Feature scaling | standardize, min-max, robust | decides how much each feature matters to distance | always, inside a Pipeline |
| Radius neighbours | RadiusNeighborsRegressor | fixed window width instead of fixed count; can find no neighbours in sparse areas | when a physical radius makes sense |
| Kernel smoothing | Nadaraya–Watson, LOESS | smooth kernel weights ; LOESS fits a local line, which removes boundary bias | when you need a smooth curve |
Index (algorithm) | brute, kd_tree, ball_tree, ANN | speed only. Results are identical, except for ANN | auto; ANN for millions of points |
In code
1import numpy as np23def knn_regress(X_train, y_train, Q, k=3, weights="uniform"):4 """Predict each row of Q as the (weighted) mean target of its k nearest rows."""5 d = np.sqrt(((Q[:, None, :] - X_train[None, :, :]) ** 2).sum(-1)) # (n_q, n_train)6 idx = np.argsort(d, axis=1, kind="stable")[:, :k] # k nearest7 dk = np.take_along_axis(d, idx, axis=1)8 yk = y_train[idx]9 if weights == "uniform":10 return yk.mean(axis=1)11 with np.errstate(divide="ignore"):12 w = 1.0 / dk13 exact = np.isinf(w) # a query sitting on a training point14 w[exact.any(axis=1)] = exact[exact.any(axis=1)] # ... takes that point's value15 return (w * yk).sum(axis=1) / w.sum(axis=1)1617def loo_mse(X, y, k, weights="uniform"):18 err = []19 for i in range(len(y)):20 m = np.arange(len(y)) != i21 err.append((y[i] - knn_regress(X[m], y[m], X[i:i + 1], k, weights)[0]) ** 2)22 return np.mean(err)2324temp = np.array([-2.5, 0, 3, 5.5, 7.5, 10.5, 13, 17, 20, 24, 27.5, 30.5])25demand = np.array([41.8, 37.9, 32.1, 30.4, 27.1, 25.4, 22.6, 23.7, 23.4, 26.9, 28.8, 32.6])26X, q = temp[:, None], np.array([[21.0]])2728for k in (1, 3, 5, 12):29 print(k, knn_regress(X, demand, q, k).round(2), knn_regress(X, demand, q, k, "distance").round(2))30# 1 [23.4] [23.4] / 3 [24.67] [24.18] / 5 [25.08] [24.46] / 12 [29.39] [25.88]31for w in ("uniform", "distance"):32 scores = [loo_mse(X, demand, k, w) for k in range(1, 12)]33 print(w, "best k =", int(np.argmin(scores)) + 1, "LOO MSE =", round(min(scores), 2))34# uniform best k = 2 LOO MSE = 7.0835# distance best k = 2 LOO MSE = 5.681import numpy as np2from sklearn.neighbors import KNeighborsRegressor3from sklearn.model_selection import GridSearchCV, LeaveOneOut4from sklearn.metrics import mean_squared_error, r2_score56temp = np.array([-2.5, 0, 3, 5.5, 7.5, 10.5, 13, 17, 20, 24, 27.5, 30.5])7demand = np.array([41.8, 37.9, 32.1, 30.4, 27.1, 25.4, 22.6, 23.7, 23.4, 26.9, 28.8, 32.6])8X = temp[:, None]910knn = KNeighborsRegressor(n_neighbors=3).fit(X, demand)11dist, idx = knn.kneighbors([[21.0]])12print("neighbours:", temp[idx[0]], "distances:", dist[0]) # [20. 24. 17.] [1. 3. 4.]13print("uniform k=3 :", knn.predict([[21.0]]).round(2)) # [24.67]14print("distance k=3:", KNeighborsRegressor(3, weights="distance").fit(X, demand).predict([[21.0]]).round(2)) # [24.18]15print("train MSE k=3:", round(mean_squared_error(demand, knn.predict(X)), 2),16 "R2:", round(r2_score(demand, knn.predict(X)), 3)) # 3.15, 0.9031718search = GridSearchCV(19 KNeighborsRegressor(),20 {"n_neighbors": range(1, 12), "weights": ["uniform", "distance"]},21 cv=LeaveOneOut(), scoring="neg_mean_squared_error",22).fit(X, demand)23print(search.best_params_, "LOO MSE:", round(-search.best_score_, 2)) # k=2, distance, 5.6824print("q=21 ->", search.predict([[21.0]]).round(2)) # [24.27]2526# With several features, always scale inside the model:27# from sklearn.pipeline import make_pipeline; from sklearn.preprocessing import StandardScaler28# model = make_pipeline(StandardScaler(), KNeighborsRegressor(n_neighbors=10, weights="distance"))Use when
- +The relationship is nonlinear and you do not know its shape: KNN follows the data with no functional form assumed.
- +You have few features (or a good learned embedding) and plenty of data that covers the input space densely.
- +New data arrives constantly: adding a point is instant, with no retraining.
- +You want predictions explained by example: "these 5 similar houses sold for…".
Avoid when
- −There are many features or many irrelevant ones: distances concentrate and neighbourhoods stop being local.
- −You must extrapolate beyond the training range, or capture a trend at the edges.
- −Latency or memory is tight: every prediction scans (or indexes) the whole training set.
- −Features have mixed units or categories that do not have a natural distance.
Where it's used
Real-estate comparables
Appraisers price a home from the sale prices of the most similar recently sold homes: KNN regression on size, location and age.
Missing-value imputation
KNNImputer fills a missing numeric field with the mean of that field over the k most similar complete rows.
Collaborative filtering
Predict a user's rating of an item as the similarity-weighted average rating from the k most similar users who rated it.
Short-term load and weather analogues
Forecast demand or rainfall by averaging what followed the k most similar past days ("analogue forecasting").
Check yourself
A query's three nearest neighbours have targets 23.4, 26.9 and 23.7 at distances 1, 3 and 4. What is the uniform k = 3 prediction?
Check yourself
With uniform weights and k = n (all 12 days), what does KNN regression predict at q = 21 °C — and at every other q?
Check yourself
On the temperature data, what does k = 3 (uniform) predict for a 40 °C day, which is hotter than any training day?
Check yourself
Under with noise variance , what is the variance term of a uniform k = 4 KNN prediction?
Remember
- ✓KNN regression predicts as the mean of the nearest targets, which is the least-squares constant fitted to the neighbourhood. With weights it is the weighted mean.
- ✓ is the bias–variance dial: the variance term is , and the bias grows as neighbours get further away. predicts the global mean everywhere. Pick by cross-validation.
- ✓Uniform KNN gives a step-function fit; distance weighting gives a continuous fit that interpolates the training points.
- ✓It cannot extrapolate, and it is biased at the edges of the data, because predictions always stay inside the range of the neighbours' targets.
- ✓Scale the features, keep the dimensionality low, and use tree or ANN indexes when n is large.