Definition
Gradient boosting regression builds a prediction one small regression tree at a time. Each new tree is fitted to the negative gradient of the loss at the current model — for squared error, simply the residuals — and only a fraction of it is added.
At a glance
- Task
- Regression (a numeric target); the same machinery with log-loss gives classification
- Input → output
- feature vector → real number , a sum of tree outputs
- Learns
- an initial constant , then shallow trees: split thresholds and one value per leaf
- Loss
- squared error (default), absolute error, Huber, quantile — any differentiable loss
- Training
- sequential: trees, each about with sorting, or with histograms
- Prediction
- comparisons
- Hyperparameters
n_estimators,learning_rate,max_depth,min_samples_leaf,subsample,loss- Assumes
- nothing about the shape of the function; features need no scaling. The model is piecewise constant and cannot extrapolate
The problem
A single regression tree (see decision-tree-regression) is a step function. A shallow tree is too coarse and underfits — high bias. A deep tree fits every bump in the training data, noise included — high variance. Random-forest-regression fixes the variance side: it averages many deep trees grown independently on bootstrap samples. But averaging cannot fix bias. If every tree misses the same pattern, so does the average.
Boosting attacks the bias. It starts from a model that is far too simple and adds small trees one after another. Each new tree is aimed at what the current model still gets wrong. AdaBoost (see adaboost-regression) does this by reweighting the training points, which works but is hard to extend to other losses. Gradient boosting (Friedman, 2001) gives a cleaner recipe that works for any differentiable loss: treat the model's predictions as parameters, and do gradient descent on them.
The idea
Start by predicting the average rent for every flat. Look at the errors: the small flats are overpriced and the big ones underpriced. Fit a tiny tree that predicts those errors from the features, and add half of what it says to the model. The errors shrink, so look at them again and fit another tiny tree. Each tree is a weak, local correction. A hundred of them, each added cautiously, add up to an accurate model.
Why call it gradient? For squared error, the residual is exactly the direction in which moving the prediction lowers the loss fastest — the negative gradient. A tree fitted to residuals is therefore an approximate gradient step. Unlike a raw step, though, it is a function defined at every , so the correction also applies to new inputs.
- 1F₀(x) ← mean(y)
- 2for m = 1 … M:
- 3 rᵢ ← yᵢ − F(xᵢ)
- 4 hₘ ← stump fit to (xᵢ, rᵢ)
- 5 F ← F + ν·hₘ
- 6predict: F_M(x)
Start with a constant: F₀(x) = mean(y) = 0.420 for every x. Train MSE 0.0682.
Vocabulary
- Weak learner / base learner
- the small model added each round — here a regression tree with 1–8 levels. A depth-1 tree (one split, two leaves) is a stump
- Initial model F₀
- the best constant: the mean of for squared loss, the median for absolute loss
- Residual
- , what the current model still gets wrong on point
- Pseudo-residual
- the negative gradient at the current prediction. It equals the residual for squared loss and for absolute loss. Trees are fitted to these
- Leaf value γ
- the constant a leaf adds, chosen by a line search: the value that minimizes the loss of the points in that leaf
- Learning rate / shrinkage ν
- fraction of each tree that is added,
- Stage / round / iteration
- one tree added; is the model after stages
- Staged prediction
- the predictions for every , used to pick the best number of trees
The math
We want the function that minimizes the total training loss, and we restrict to sums of trees:
- loss for one point, e.g.
- features and target of training point ; points in total
- initial constant prediction
- regression tree added at stage
- learning rate (shrinkage), 0 < ν ≤ 1
- number of trees (stages)
Step 0 — the best constant. Before any tree, use the single number that minimizes the loss:
- a candidate constant prediction
- mean of the training targets
For squared loss, set the derivative to zero and you get . For absolute loss, the derivative is zero when half the points lie above and half below — the median.
Step 1 — the gradient. Think of the predictions as free parameters. Gradient descent would move each one along the negative partial derivative of the loss:
- pseudo-residual of point at stage
- the model after trees
- ordinary residual: target minus current prediction
The factor is only there to cancel the 2 from differentiating. So for squared error, "fit the residuals" and "follow the negative gradient" are the same instruction. A raw gradient step only says how to change the predictions at the training points, though. It says nothing about new inputs.
Step 2 — project the gradient onto a tree. Fit a regression tree to the pairs by least squares. This gives a function defined everywhere that points roughly in the gradient direction. The tree splits the input space into leaf regions:
- maximum tree depth (
max_depth) - region of input space covered by leaf of tree
- number of leaves of tree (at most )
The tree is grown greedily, exactly like a single regression tree: at each node it picks the split that most reduces the squared error of the pseudo-residuals.
Step 3 — line search in each leaf. The fitted tree fixes the regions. We then re-choose each leaf's value to minimize the actual loss rather than the squared error on gradients:
- value of leaf in tree
- the training points that fall in that leaf
For squared loss, differentiate: , so , the mean residual in the leaf. That is what the tree predicted anyway, so for squared loss Step 3 changes nothing. For absolute loss the tree is fitted to signs , but the line search sets to the median residual in the leaf, which is in the target's units.
Step 4 — take a shrunken step.
- 1 if the condition holds, else 0 — so exactly one leaf's γ is added
- learning rate: the step size of this function-space gradient descent
What does, exactly. Suppose a group of points keeps landing in the same leaf. With squared loss, each round adds times their mean residual, so their mean residual is multiplied by every round. After rounds it is of the original. With that is after 4 rounds. With it takes 22 rounds to get under 10% (). A small means more trees, but each tree's mistakes get diluted by the later ones, and that is why small usually generalizes better (Friedman's empirical finding). is a plain greedy fit that jumps straight to the leaf means.
The other knobs. max_depth sets the interaction order. A stump uses one feature, so the sum of stumps is additive: . A depth- tree can model interactions between up to features. More depth also means more leaves (), so each tree fits more of the residual. min_samples_leaf keeps every an average over several points, so a single noisy point cannot get its own correction. subsample fits each tree on a random fraction of the rows (stochastic gradient boosting). Like minibatch SGD, this adds noise that regularizes and speeds training.
| Loss | L(y, F) | Pseudo-residual g | F₀ | Leaf value γ | Use when |
|---|---|---|---|---|---|
| Squared | mean | mean of r in the leaf | default; you want the conditional mean | ||
| Absolute | median | median of r in the leaf | heavy-tailed noise, outliers | ||
| Huber | if , else | median | median + mean of clipped deviations | mostly Gaussian noise with some outliers | |
| Quantile | if , else | or | -quantile | -quantile of r in the leaf | prediction intervals |
Worked example: predicting rent from floor area
Ten flats, one feature. We boost stumps (max_depth = 1) with squared loss and , and predict the rent of a new 75 m² flat. Rent is in hundreds of dollars per month.
| Flat | Area x (m²) | Rent y (hundreds of dollars) |
|---|---|---|
| 1 | 30 | 8 |
| 2 | 40 | 9 |
| 3 | 50 | 11 |
| 4 | 60 | 10 |
| 5 | 70 | 15 |
| 6 | 80 | 16 |
| 7 | 90 | 18 |
| 8 | 100 | 17 |
| 9 | 110 | 24 |
| 10 | 120 | 25 |
Two rounds by hand
- 1Initialize. for every flat. Training MSE (total squared error 320.1).
- 2Round 1 — residuals. : . They sum to zero, as residuals from the mean always do.
- 3Round 1 — fit a stump. Try all 9 midpoint thresholds (table below). The best is : left leaf (flats 1–4) mean residual , right leaf (flats 5–10) , leaving squared error 95.833 of the 320.1 — the stump explains 70.1% of it.
- 4Round 1 — shrunken update. for and for . Training MSE .
- 5Round 2 — new residuals. : . Only half the first correction was applied, so the two groups are still off in the same direction. The big flats are also now clearly the worst-fitted.
- 6Round 2 — fit a stump. The best split is now (squared error 19.889): left leaf (flats 1–8) , right leaf (flats 9–10) . Note that the new tree picks a different threshold — it attacks the largest remaining error.
- 7Round 2 — update. : for , for , and for . Training MSE .
- 8Predict . It goes right in tree 1 () and left in tree 2 (): , i.e. about 1,633 dollars per month. A third stump (split at ) would give 15.524 and training MSE 2.399.
| Threshold | Left mean | Right mean | Squared error |
|---|---|---|---|
| x ≤ 35 | −7.300 | +0.811 | 260.889 |
| x ≤ 45 | −6.800 | +1.700 | 204.500 |
| x ≤ 55 | −5.967 | +2.557 | 167.524 |
| x ≤ 65 | −5.800 | +3.867 | 95.833 |
| x ≤ 75 | −4.700 | +4.700 | 99.200 |
| x ≤ 85 | −3.800 | +5.700 | 103.500 |
| x ≤ 95 | −2.871 | +6.700 | 127.714 |
| x ≤ 105 | −2.300 | +9.200 | 108.500 |
| x ≤ 115 | −1.078 | +9.700 | 215.556 |
| x | y | r₁ | h₁ | F₁ | r₂ | h₂ | F₂ |
|---|---|---|---|---|---|---|---|
| 30 | 8 | −7.3 | −5.800 | 12.400 | −4.400 | −1.817 | 11.492 |
| 40 | 9 | −6.3 | −5.800 | 12.400 | −3.400 | −1.817 | 11.492 |
| 50 | 11 | −4.3 | −5.800 | 12.400 | −1.400 | −1.817 | 11.492 |
| 60 | 10 | −5.3 | −5.800 | 12.400 | −2.400 | −1.817 | 11.492 |
| 70 | 15 | −0.3 | +3.867 | 17.233 | −2.233 | −1.817 | 16.325 |
| 80 | 16 | +0.7 | +3.867 | 17.233 | −1.233 | −1.817 | 16.325 |
| 90 | 18 | +2.7 | +3.867 | 17.233 | +0.767 | −1.817 | 16.325 |
| 100 | 17 | +1.7 | +3.867 | 17.233 | −0.233 | −1.817 | 16.325 |
| 110 | 24 | +8.7 | +3.867 | 17.233 | +6.767 | +7.267 | 20.867 |
| 120 | 25 | +9.7 | +3.867 | 17.233 | +7.767 | +7.267 | 20.867 |
The algorithm, step by step
GradientBoost(X, y, L, M, ν, D)
- 1Initialize — the mean for squared loss, the median for absolute or Huber loss.
- 2For :
- 3 Pseudo-residuals. at for every training point (with
subsample< 1, only for a random subset of rows). - 4 Fit a tree. Grow a regression tree of depth on with the squared-error split criterion, respecting
min_samples_leaf. - 5 Line search. For each leaf , set .
- 6 Update. , where is the leaf falls into.
- 7 Monitor. Record validation loss; stop if it has not improved for a set number of rounds (early stopping).
- 8Predict — or at the best validation round .
Complexity
- Training time
- with exact sorted splits; with histogram binning (
HistGradientBoostingRegressor, LightGBM). The rounds are sequential — only the split search inside one tree parallelizes. - Prediction time
- comparisons per input; a model with 500 depth-3 trees does 1,500 comparisons.
- Memory
- trees with at most leaves each, plus the current predictions during training.
Learning rate, depth and number of trees
These three interact. A smaller needs proportionally more trees. Deeper trees fit faster but each one overfits more. On the 10 test flats from the figure (squared loss, test MSE of the constant is 31.445), the best test MSE over 50 rounds was:
| Learning rate | depth 1 (stumps) | depth 2 | depth 3 |
|---|---|---|---|
| ν = 0.1 | 1.922 @ round 39 | 1.650 @ round 23 | 1.397 @ round 18 |
| ν = 0.5 | 2.079 @ round 5 | 1.764 @ round 4 | 1.423 @ round 3 |
| ν = 1.0 | 2.154 @ round 8 | 2.566 @ round 4 | 2.400 @ round 1 |
Practical tuning
- learning_rate ν
- Fix it small (0.05–0.1) and let early stopping choose the number of trees. Lower ν rarely hurts accuracy, only training time.
- n_estimators M
- Do not tune it by grid search — set it large and use
n_iter_no_changewithvalidation_fraction, or pick the best round fromstaged_predicton a validation set. - max_depth
- 3–8. Depth 1 gives an additive model with no interactions; deeper trees capture interactions but overfit faster. Alternatively cap
max_leaf_nodes. - min_samples_leaf
- 10–100 on real data. It stops a leaf from being spent on one outlier and smooths each γ.
- subsample
- 0.5–0.8 (stochastic gradient boosting): faster, and often more accurate through added randomness.
- max_features
- a random subset of features per split, as in random forests — decorrelates trees and speeds up wide data.
- loss
squared_errorfor the mean,absolute_errororhuber(withalphasetting δ as a quantile of |r|) for outliers,quantilefor intervals.
In code
1import numpy as np23def fit_tree(X, g, depth, min_leaf=1):4 """Least-squares regression tree on pseudo-residuals g. Returns a nested dict."""5 if depth == 0 or len(g) < 2 * min_leaf:6 return {"rows": np.arange(len(g))}7 best = (np.inf, None, None)8 for j in range(X.shape[1]):9 order = np.argsort(X[:, j], kind="stable")10 xs, gs = X[order, j], g[order]11 for k in range(min_leaf, len(g) - min_leaf + 1):12 if xs[k - 1] == xs[k]:13 continue14 L, R = gs[:k], gs[k:]15 sse = ((L - L.mean()) ** 2).sum() + ((R - R.mean()) ** 2).sum()16 if sse < best[0] - 1e-12:17 best = (sse, j, (xs[k - 1] + xs[k]) / 2)18 if best[1] is None:19 return {"rows": np.arange(len(g))}20 _, j, t = best21 left = X[:, j] <= t22 lt, rt = fit_tree(X[left], g[left], depth - 1, min_leaf), fit_tree(X[~left], g[~left], depth - 1, min_leaf)23 return {"j": j, "t": t, "left": lt, "right": rt, "lidx": np.where(left)[0], "ridx": np.where(~left)[0]}2425def set_leaf_values(node, r, idx, loss):26 """Line search per leaf: the constant that minimises the loss of the raw residuals r in that leaf."""27 if "j" not in node:28 rl = r[idx]29 node["value"] = rl.mean() if loss == "squared" else np.median(rl)30 return31 set_leaf_values(node["left"], r, idx[node["lidx"]], loss)32 set_leaf_values(node["right"], r, idx[node["ridx"]], loss)3334def predict_tree(node, X):35 if "j" not in node:36 return np.full(len(X), node["value"])37 out, left = np.empty(len(X)), X[:, node["j"]] <= node["t"]38 out[left], out[~left] = predict_tree(node["left"], X[left]), predict_tree(node["right"], X[~left])39 return out4041class GradientBoostingRegressor:42 def __init__(self, n_estimators=100, learning_rate=0.1, max_depth=3, loss="squared"):43 self.M, self.nu, self.depth, self.loss = n_estimators, learning_rate, max_depth, loss4445 def fit(self, X, y):46 self.f0 = y.mean() if self.loss == "squared" else np.median(y) # argmin_c sum L(y, c)47 F, self.trees = np.full(len(y), self.f0), []48 for _ in range(self.M):49 r = y - F # raw residuals50 g = r if self.loss == "squared" else np.sign(r) # negative gradient of the loss51 tree = fit_tree(X, g, self.depth) # tree on pseudo-residuals52 set_leaf_values(tree, r, np.arange(len(y)), self.loss)53 F = F + self.nu * predict_tree(tree, X) # shrunken step54 self.trees.append(tree)55 return self5657 def predict(self, X):58 return self.f0 + self.nu * sum(predict_tree(t, X) for t in self.trees)5960X = np.array([30, 40, 50, 60, 70, 80, 90, 100, 110, 120.])[:, None]61y = np.array([8, 9, 11, 10, 15, 16, 18, 17, 24, 25.])62for M in (1, 2, 3):63 gbr = GradientBoostingRegressor(n_estimators=M, learning_rate=0.5, max_depth=1).fit(X, y)64 print(M, "trees: F(75) =", round(gbr.predict(np.array([[75.]]))[0], 3),65 " train MSE =", round(((gbr.predict(X) - y) ** 2).mean(), 3))66# 1 trees: F(75) = 17.233 train MSE = 15.1967# 2 trees: F(75) = 16.325 train MSE = 5.28968# 3 trees: F(75) = 15.524 train MSE = 2.3991import numpy as np2from sklearn.ensemble import GradientBoostingRegressor3from sklearn.metrics import mean_squared_error45X = np.array([30, 40, 50, 60, 70, 80, 90, 100, 110, 120.])[:, None] # area, m^26y = np.array([8, 9, 11, 10, 15, 16, 18, 17, 24, 25.]) # rent, $100s7Xt = np.array([33, 44, 57, 63, 76, 87, 94, 107, 116, 125.])[:, None] # held-out flats8yt = np.array([8.5, 10, 10.5, 12, 15.5, 16.5, 17, 20, 24, 25.5])910gbr = GradientBoostingRegressor(loss="squared_error", n_estimators=2,11 learning_rate=0.5, max_depth=1).fit(X, y)12print("F0 =", gbr.init_.constant_.ravel()[0])13for m, est in enumerate(gbr.estimators_[:, 0], 1):14 t = est.tree_15 print(f"tree {m}: x <= {t.threshold[0]:.0f} -> {t.value[1,0,0]:+.3f}, else {t.value[2,0,0]:+.3f}")16print("F(75) =", round(gbr.predict([[75]])[0], 3))1718# Learning curve on held-out data: staged_predict gives F_m for every m19big = GradientBoostingRegressor(n_estimators=50, learning_rate=0.5, max_depth=1).fit(X, y)20test = [mean_squared_error(yt, p) for p in big.staged_predict(Xt)]21print("best round:", int(np.argmin(test)) + 1, " test MSE:", round(min(test), 3),22 " after 50:", round(test[-1], 3))2324# F0 = 15.325# tree 1: x <= 65 -> -5.800, else +3.86726# tree 2: x <= 105 -> -1.817, else +7.26727# F(75) = 16.32528# best round: 5 test MSE: 2.079 after 50: 2.425Use when
- +Tabular data with nonlinear effects and interactions — boosted trees are usually the most accurate off-the-shelf model there.
- +Mixed feature types and scales, with no normalization needed; monotonic transforms of features change nothing.
- +You need a non-standard regression loss: absolute, Huber, quantile (prediction intervals), Poisson or Tweedie.
- +You can afford a validation set and some tuning of ν, depth and the number of trees.
Avoid when
- −Images, audio or text, where neural networks learn the features far better.
- −Predictions outside the training range of the target or features (trends, extrapolation).
- −Very small or very noisy data with no budget for validation — it overfits easily without early stopping.
- −Training must be parallel across trees or updated online — the rounds are inherently sequential.
- −A smooth or exactly interpretable model is required; a sum of hundreds of trees is a black box (use SHAP for explanations).
Where it's used
Demand and sales forecasting
Store-item sales from calendar, price and promotion features — boosted trees dominate retail forecasting competitions such as M5.
Real-estate and price estimation
House values, insurance premiums and used-car prices from dozens of mixed tabular features.
Travel-time and logistics ETA
Delivery and ride-hailing arrival times from distance, traffic, time of day and region.
Quantile risk models
With quantile loss, separate models for the 5th and 95th percentile give prediction intervals for energy load or claim size.
Check yourself
With squared loss, F₀ = 15.3 and a first stump whose right leaf holds +3.867. With ν = 0.5, what is F₁ for a 100 m² flat?
Check yourself
A leaf contains the residuals 2, 3 and 7. What value γ does the line search give it under squared loss and under absolute loss?
Check yourself
For absolute loss L = |y − F|, the residuals of three points are −7.3, 0.7 and 9.7. What pseudo-residuals is the next tree fitted to?
Check yourself
You lower the learning rate from 0.5 to 0.1 and keep everything else. What should you expect?
Remember
- ✓Model: , a constant plus many shrunken shallow trees, added one at a time.
- ✓Each tree is fitted to the pseudo-residuals . For squared loss these are ordinary residuals , and each leaf predicts their mean.
- ✓Other losses only change the pseudo-residuals and the leaf line search: sign and median for absolute loss, clipping for Huber, quantiles for intervals.
- ✓Learning rate × number of trees is the main trade-off: small ν with many trees generalizes better; choose the stopping round on validation data.
- ✓Depth sets the interaction order (stumps = additive model); min_samples_leaf and subsample regularize.
- ✓Training error always falls to zero and the model never extrapolates — use early stopping, and watch the target range.