Gradient boosting regression

Fit each new tree to the residuals of the current model and add a shrunken copy — gradient descent in function space.

17 min read1750 wordscore18 viewsupdated 2026-10-07

Definition

Gradient boosting regression builds a prediction FM(x)=F0+ν∑m=1Mhm(x)F_M(x)=F_0+\nu\sum_{m=1}^{M}h_m(x) one small regression tree at a time. Each new tree hmh_m is fitted to the negative gradient of the loss at the current model — for squared error, simply the residuals yi−Fm−1(xi)y_i-F_{m-1}(x_i) — and only a fraction ν\nu of it is added.

At a glance

Task
Regression (a numeric target); the same machinery with log-loss gives classification
Input → output
feature vector xx → real number FM(x)F_M(x), a sum of MM tree outputs
Learns
an initial constant F0F_0, then MM shallow trees: split thresholds and one value γ\gamma per leaf
Loss
squared error (default), absolute error, Huber, quantile — any differentiable loss
Training
sequential: MM trees, each about O(d nlog⁡n)O(d\,n\log n) with sorting, or O(d n)O(d\,n) with histograms
Prediction
O(M⋅depth)O(M\cdot\text{depth}) comparisons
Hyperparameters
n_estimators MM, learning_rate ν\nu, max_depth, min_samples_leaf, subsample, loss
Assumes
nothing about the shape of the function; features need no scaling. The model is piecewise constant and cannot extrapolate

The problem

A single regression tree (see decision-tree-regression) is a step function. A shallow tree is too coarse and underfits — high bias. A deep tree fits every bump in the training data, noise included — high variance. Random-forest-regression fixes the variance side: it averages many deep trees grown independently on bootstrap samples. But averaging cannot fix bias. If every tree misses the same pattern, so does the average.

Boosting attacks the bias. It starts from a model that is far too simple and adds small trees one after another. Each new tree is aimed at what the current model still gets wrong. AdaBoost (see adaboost-regression) does this by reweighting the training points, which works but is hard to extend to other losses. Gradient boosting (Friedman, 2001) gives a cleaner recipe that works for any differentiable loss: treat the model's predictions as parameters, and do gradient descent on them.

The idea

Start by predicting the average rent for every flat. Look at the errors: the small flats are overpriced and the big ones underpriced. Fit a tiny tree that predicts those errors from the features, and add half of what it says to the model. The errors shrink, so look at them again and fit another tiny tree. Each tree is a weak, local correction. A hundred of them, each added cautiously, add up to an accurate model.

Why call it gradient? For squared error, the residual yi−F(xi)y_i-F(x_i) is exactly the direction in which moving the prediction F(xi)F(x_i) lowers the loss fastest — the negative gradient. A tree fitted to residuals is therefore an approximate gradient step. Unlike a raw step, though, it is a function defined at every xx, so the correction also applies to new inputs.

▸ gradient boosting · stumpsstep 1/18
data and model Fresiduals r = y − F
round 0/40ν = 0.3train MSE 0.0682
  1. 1F₀(x) ← mean(y)
  2. 2for m = 1 … M:
  3. 3 rᵢ ← yᵢ − F(xᵢ)
  4. 4 hₘ ← stump fit to (xᵢ, rᵢ)
  5. 5 F ← F + ν·hₘ
  6. 6predict: F_M(x)

Start with a constant: F₀(x) = mean(y) = 0.420 for every x. Train MSE 0.0682.

fig.Gradient boosting with stumps on 30 noisy points of a sine wave (ν = 0.3, 40 rounds). Top: data and the current model F. Bottom: the residuals and the stump fitted to them. Every round adds 0.3 of that stump, and the training MSE (sparkline) falls toward the noise level and then below it.

Vocabulary

Weak learner / base learner
the small model added each round — here a regression tree with 1–8 levels. A depth-1 tree (one split, two leaves) is a stump
Initial model F₀
the best constant: the mean of yy for squared loss, the median for absolute loss
Residual
ri=yi−Fm−1(xi)r_i=y_i-F_{m-1}(x_i), what the current model still gets wrong on point ii
Pseudo-residual
the negative gradient gi=−∂L/∂Fg_i=-\partial L/\partial F at the current prediction. It equals the residual for squared loss and sign⁡(ri)\operatorname{sign}(r_i) for absolute loss. Trees are fitted to these
Leaf value γ
the constant a leaf adds, chosen by a line search: the value that minimizes the loss of the points in that leaf
Learning rate / shrinkage ν
fraction of each tree that is added, 0<ν≤10<\nu\le 1
Stage / round / iteration
one tree added; FmF_m is the model after mm stages
Staged prediction
the predictions F0,F1,…,FMF_0, F_1,\dots,F_M for every mm, used to pick the best number of trees

The math

We want the function FF that minimizes the total training loss, and we restrict FF to sums of trees:

min⁡F ∑i=1nL(yi,F(xi)),F(x)=F0+ν∑m=1Mhm(x)\min_{F}\ \sum_{i=1}^{n}L\big(y_i,F(x_i)\big),\qquad F(x)=F_0+\nu\sum_{m=1}^{M}h_m(x)
L(y,F)L(y,F)
loss for one point, e.g. 12(y−F)2\tfrac12(y-F)^2
xi, yix_i,\ y_i
features and target of training point ii; nn points in total
F0F_0
initial constant prediction
hmh_m
regression tree added at stage mm
ν\nu
learning rate (shrinkage), 0 < ν ≤ 1
MM
number of trees (stages)

Step 0 — the best constant. Before any tree, use the single number that minimizes the loss:

F0=arg⁡min⁡c∑iL(yi,c)  =  {yˉsquared lossmedian⁡(y)absolute lossF_0=\arg\min_{c}\sum_i L(y_i,c)\;=\;\begin{cases}\bar y & \text{squared loss}\\ \operatorname{median}(y) & \text{absolute loss}\end{cases}
cc
a candidate constant prediction
yˉ\bar y
mean of the training targets

For squared loss, set the derivative ∑i−(yi−c)\sum_i -(y_i-c) to zero and you get c=yˉc=\bar y. For absolute loss, the derivative ∑i−sign⁡(yi−c)\sum_i -\operatorname{sign}(y_i-c) is zero when half the points lie above cc and half below — the median.

Step 1 — the gradient. Think of the nn predictions F(x1),…,F(xn)F(x_1),\dots,F(x_n) as nn free parameters. Gradient descent would move each one along the negative partial derivative of the loss:

gim=−[∂L(yi,F(xi))∂F(xi)]F=Fm−1for L=12(y−F)2:gim=yi−Fm−1(xi)=rimg_{im}=-\left[\frac{\partial L(y_i,F(x_i))}{\partial F(x_i)}\right]_{F=F_{m-1}}\qquad\text{for }L=\tfrac12(y-F)^2:\quad g_{im}=y_i-F_{m-1}(x_i)=r_{im}
gimg_{im}
pseudo-residual of point ii at stage mm
Fm−1F_{m-1}
the model after m−1m-1 trees
rimr_{im}
ordinary residual: target minus current prediction

The factor 12\tfrac12 is only there to cancel the 2 from differentiating. So for squared error, "fit the residuals" and "follow the negative gradient" are the same instruction. A raw gradient step only says how to change the predictions at the nn training points, though. It says nothing about new inputs.

Step 2 — project the gradient onto a tree. Fit a regression tree to the pairs (xi,gim)(x_i, g_{im}) by least squares. This gives a function defined everywhere that points roughly in the gradient direction. The tree splits the input space into leaf regions:

hm=arg⁡min⁡h∈trees of depth≤D ∑i(gim−h(xi))2 ⇒ leaf regions R1m,…,RJmmh_m=\arg\min_{h\in\text{trees of depth}\le D}\ \sum_i\big(g_{im}-h(x_i)\big)^2\ \Rightarrow\ \text{leaf regions } R_{1m},\dots,R_{J_m m}
DD
maximum tree depth (max_depth)
RjmR_{jm}
region of input space covered by leaf jj of tree mm
JmJ_m
number of leaves of tree mm (at most 2D2^D)

The tree is grown greedily, exactly like a single regression tree: at each node it picks the split that most reduces the squared error of the pseudo-residuals.

Step 3 — line search in each leaf. The fitted tree fixes the regions. We then re-choose each leaf's value to minimize the actual loss rather than the squared error on gradients:

γjm=arg⁡min⁡γ∑xi∈RjmL(yi, Fm−1(xi)+γ)\gamma_{jm}=\arg\min_{\gamma}\sum_{x_i\in R_{jm}}L\big(y_i,\,F_{m-1}(x_i)+\gamma\big)
γjm\gamma_{jm}
value of leaf jj in tree mm
xi∈Rjmx_i\in R_{jm}
the training points that fall in that leaf

For squared loss, differentiate: −∑i∈Rjm(rim−γ)=0-\sum_{i\in R_{jm}}\big(r_{im}-\gamma\big)=0, so γjm=1∣Rjm∣∑i∈Rjmrim\gamma_{jm}=\tfrac{1}{|R_{jm}|}\sum_{i\in R_{jm}} r_{im}, the mean residual in the leaf. That is what the tree predicted anyway, so for squared loss Step 3 changes nothing. For absolute loss the tree is fitted to signs ±1\pm1, but the line search sets γ\gamma to the median residual in the leaf, which is in the target's units.

Step 4 — take a shrunken step.

Fm(x)=Fm−1(x)+ν∑j=1Jmγjm 1 ⁣[x∈Rjm]F_m(x)=F_{m-1}(x)+\nu\sum_{j=1}^{J_m}\gamma_{jm}\,\mathbf 1\!\left[x\in R_{jm}\right]
1[⋅]\mathbf 1[\cdot]
1 if the condition holds, else 0 — so exactly one leaf's γ is added
ν\nu
learning rate: the step size of this function-space gradient descent

What ν\nu does, exactly. Suppose a group of points keeps landing in the same leaf. With squared loss, each round adds ν\nu times their mean residual, so their mean residual is multiplied by (1−ν)(1-\nu) every round. After kk rounds it is (1−ν)k(1-\nu)^k of the original. With ν=0.5\nu=0.5 that is 0.54=0.06250.5^4=0.0625 after 4 rounds. With ν=0.1\nu=0.1 it takes 22 rounds to get under 10% (0.922=0.0980.9^{22}=0.098). A small ν\nu means more trees, but each tree's mistakes get diluted by the later ones, and that is why small ν\nu usually generalizes better (Friedman's empirical finding). ν=1\nu=1 is a plain greedy fit that jumps straight to the leaf means.

The other knobs. max_depth DD sets the interaction order. A stump uses one feature, so the sum of stumps is additive: F(x)=∑jfj(xj)F(x)=\sum_j f_j(x_j). A depth-DD tree can model interactions between up to DD features. More depth also means more leaves (≤2D\le 2^D), so each tree fits more of the residual. min_samples_leaf keeps every γ\gamma an average over several points, so a single noisy point cannot get its own correction. subsample <1<1 fits each tree on a random fraction of the rows (stochastic gradient boosting). Like minibatch SGD, this adds noise that regularizes and speeds training.

LossL(y, F)Pseudo-residual gF₀Leaf value γUse when
Squared12(y−F)2\tfrac12(y-F)^2rrmeanmean of r in the leafdefault; you want the conditional mean
Absolute∣y−F∣|y-F|sign⁡(r)\operatorname{sign}(r)medianmedian of r in the leafheavy-tailed noise, outliers
Huber12r2\tfrac12 r^2 if ∣r∣≤δ|r|\le\delta, else δ(∣r∣−δ2)\delta(|r|-\tfrac\delta2)clip⁡(r,−δ,δ)\operatorname{clip}(r,-\delta,\delta)medianmedian + mean of clipped deviationsmostly Gaussian noise with some outliers
Quantileαr\alpha r if r>0r>0, else (α−1)r(\alpha-1)rα\alpha or α−1\alpha-1α\alpha-quantileα\alpha-quantile of r in the leafprediction intervals
Regression losses and what the boosting loop uses for each. δ\delta is the Huber threshold; α\alpha is the target quantile.

Worked example: predicting rent from floor area

Ten flats, one feature. We boost stumps (max_depth = 1) with squared loss and ν=0.5\nu=0.5, and predict the rent of a new 75 m² flat. Rent is in hundreds of dollars per month.

FlatArea x (m²)Rent y (hundreds of dollars)
1308
2409
35011
46010
57015
68016
79018
810017
911024
1012025
Training data. Mean rent 15.3; the variance (MSE of predicting the mean) is 32.01.

Two rounds by hand

  1. 1Initialize. F0=yˉ=153/10=15.3F_0=\bar y=153/10=15.3 for every flat. Training MSE =32.01=32.01 (total squared error 320.1).
  2. 2Round 1 — residuals. ri1=yi−15.3r_{i1}=y_i-15.3: −7.3, −6.3, −4.3, −5.3, −0.3, 0.7, 2.7, 1.7, 8.7, 9.7-7.3,\ -6.3,\ -4.3,\ -5.3,\ -0.3,\ 0.7,\ 2.7,\ 1.7,\ 8.7,\ 9.7. They sum to zero, as residuals from the mean always do.
  3. 3Round 1 — fit a stump. Try all 9 midpoint thresholds (table below). The best is x≤65x\le 65: left leaf (flats 1–4) mean residual −23.2/4=−5.8-23.2/4=-5.8, right leaf (flats 5–10) 23.2/6=3.86723.2/6=3.867, leaving squared error 95.833 of the 320.1 — the stump explains 70.1% of it.
  4. 4Round 1 — shrunken update. F1=15.3+0.5(−5.8)=12.4F_1=15.3+0.5(-5.8)=12.4 for x≤65x\le65 and 15.3+0.5(3.867)=17.23315.3+0.5(3.867)=17.233 for x>65x>65. Training MSE 32.01→15.1932.01\to15.19.
  5. 5Round 2 — new residuals. ri2=yi−F1(xi)r_{i2}=y_i-F_1(x_i): −4.4, −3.4, −1.4, −2.4, −2.233, −1.233, 0.767, −0.233, 6.767, 7.767-4.4,\ -3.4,\ -1.4,\ -2.4,\ -2.233,\ -1.233,\ 0.767,\ -0.233,\ 6.767,\ 7.767. Only half the first correction was applied, so the two groups are still off in the same direction. The big flats are also now clearly the worst-fitted.
  6. 6Round 2 — fit a stump. The best split is now x≤105x\le105 (squared error 19.889): left leaf (flats 1–8) −1.817-1.817, right leaf (flats 9–10) +7.267+7.267. Note that the new tree picks a different threshold — it attacks the largest remaining error.
  7. 7Round 2 — update. F2=F1+0.5 h2F_2=F_1+0.5\,h_2: 12.4−0.908=11.49212.4-0.908=11.492 for x≤60x\le60, 17.233−0.908=16.32517.233-0.908=16.325 for 70≤x≤10070\le x\le100, and 17.233+3.633=20.86717.233+3.633=20.867 for x≥110x\ge110. Training MSE 15.19→5.28915.19\to5.289.
  8. 8Predict x=75x=75. It goes right in tree 1 (75>6575>65) and left in tree 2 (75≤10575\le105): F2(75)=15.3+0.5(3.867)+0.5(−1.817)=16.325F_2(75)=15.3+0.5(3.867)+0.5(-1.817)=\mathbf{16.325}, i.e. about 1,633 dollars per month. A third stump (split at x≤85x\le85) would give 15.524 and training MSE 2.399.
ThresholdLeft meanRight meanSquared error
x ≤ 35−7.300+0.811260.889
x ≤ 45−6.800+1.700204.500
x ≤ 55−5.967+2.557167.524
x ≤ 65−5.800+3.86795.833
x ≤ 75−4.700+4.70099.200
x ≤ 85−3.800+5.700103.500
x ≤ 95−2.871+6.700127.714
x ≤ 105−2.300+9.200108.500
x ≤ 115−1.078+9.700215.556
Round 1: every candidate stump scored on the residuals. Squared error = sum of squared deviations from the leaf means.
xyr₁h₁F₁r₂h₂F₂
308−7.3−5.80012.400−4.400−1.81711.492
409−6.3−5.80012.400−3.400−1.81711.492
5011−4.3−5.80012.400−1.400−1.81711.492
6010−5.3−5.80012.400−2.400−1.81711.492
7015−0.3+3.86717.233−2.233−1.81716.325
8016+0.7+3.86717.233−1.233−1.81716.325
9018+2.7+3.86717.233+0.767−1.81716.325
10017+1.7+3.86717.233−0.233−1.81716.325
11024+8.7+3.86717.233+6.767+7.26720.867
12025+9.7+3.86717.233+7.767+7.26720.867
The two rounds row by row (ν = 0.5). h is the leaf value the stump gives that row; F₁ = 15.3 + 0.5·h₁ and F₂ = F₁ + 0.5·h₂.
lossνdepth
010203030405060708090100110120rent ($100s) vs area (m²) · ● train (drag ↕) ○ test · — F2(x)F(75) = 16.325round 2: residuals y − F1 — tree h2 (leaf γ)γ = -1.817γ = +7.2670.1110MSE per round (log) · train testbest test: round 501020304050
round 2
train MSE 5.289test MSE 4.562 (best 2.079 at round 5)F₀ = 15.30 (mean)
fig.The worked example, live. Top: the 10 flats (drag any point up or down), 10 held-out test flats, and the model F_m (orange; faded = F_{m−1}). Middle: round m's residuals and the tree fitted to them, with its leaf values γ. Bottom: train and test MSE for every round. The defaults (squared loss, ν = 0.5, depth 1, round 2) reproduce the hand calculation: F₂(75) = 16.325, train MSE 5.289. Press ▶ to run to 50 rounds: train MSE falls to 0.019, while test MSE is lowest at round 5 (2.079) and ends at 2.425. Try ν = 0.1 (best test 1.922 at round 39), or drag the 60 m² flat up to 30 and compare squared with absolute loss.

The algorithm, step by step

GradientBoost(X, y, L, M, ν, D)

  1. 1Initialize F0(x)=arg⁡min⁡c∑iL(yi,c)F_0(x)=\arg\min_c\sum_i L(y_i,c) — the mean for squared loss, the median for absolute or Huber loss.
  2. 2For m=1,…,Mm=1,\dots,M:
  3. 3 Pseudo-residuals. gim=−∂L(yi,F)/∂Fg_{im}=-\partial L(y_i,F)/\partial F at F=Fm−1(xi)F=F_{m-1}(x_i) for every training point (with subsample < 1, only for a random subset of rows).
  4. 4 Fit a tree. Grow a regression tree of depth ≤D\le D on (xi,gim)(x_i,g_{im}) with the squared-error split criterion, respecting min_samples_leaf.
  5. 5 Line search. For each leaf jj, set γjm=arg⁡min⁡γ∑i∈RjmL(yi,Fm−1(xi)+γ)\gamma_{jm}=\arg\min_\gamma\sum_{i\in R_{jm}}L(y_i,F_{m-1}(x_i)+\gamma).
  6. 6 Update. Fm(x)=Fm−1(x)+ν γj(x)mF_m(x)=F_{m-1}(x)+\nu\,\gamma_{j(x)m}, where j(x)j(x) is the leaf xx falls into.
  7. 7 Monitor. Record validation loss; stop if it has not improved for a set number of rounds (early stopping).
  8. 8Predict FM(x)F_M(x) — or Fm∗(x)F_{m^*}(x) at the best validation round m∗m^*.

Complexity

Training time
O(M⋅d⋅nlog⁡n)O(M\cdot d\cdot n\log n) with exact sorted splits; O(M⋅d⋅n)O(M\cdot d\cdot n) with histogram binning (HistGradientBoostingRegressor, LightGBM). The MM rounds are sequential — only the split search inside one tree parallelizes.
Prediction time
O(M⋅D)O(M\cdot D) comparisons per input; a model with 500 depth-3 trees does 1,500 comparisons.
Memory
MM trees with at most 2D2^D leaves each, plus the nn current predictions during training.

Learning rate, depth and number of trees

These three interact. A smaller ν\nu needs proportionally more trees. Deeper trees fit faster but each one overfits more. On the 10 test flats from the figure (squared loss, test MSE of the constant F0F_0 is 31.445), the best test MSE over 50 rounds was:

Learning ratedepth 1 (stumps)depth 2depth 3
ν = 0.11.922 @ round 391.650 @ round 231.397 @ round 18
ν = 0.52.079 @ round 51.764 @ round 41.423 @ round 3
ν = 1.02.154 @ round 82.566 @ round 42.400 @ round 1
Best held-out MSE over 50 rounds and the round where it occurs, for every ν × depth combination (squared loss). By round 50 every setting has driven training MSE below 0.2, and test MSE has risen to 1.97–2.63.

Practical tuning

learning_rate ν
Fix it small (0.05–0.1) and let early stopping choose the number of trees. Lower ν rarely hurts accuracy, only training time.
n_estimators M
Do not tune it by grid search — set it large and use n_iter_no_change with validation_fraction, or pick the best round from staged_predict on a validation set.
max_depth
3–8. Depth 1 gives an additive model with no interactions; deeper trees capture interactions but overfit faster. Alternatively cap max_leaf_nodes.
min_samples_leaf
10–100 on real data. It stops a leaf from being spent on one outlier and smooths each γ.
subsample
0.5–0.8 (stochastic gradient boosting): faster, and often more accurate through added randomness.
max_features
a random subset of features per split, as in random forests — decorrelates trees and speeds up wide data.
loss
squared_error for the mean, absolute_error or huber (with alpha setting δ as a quantile of |r|) for outliers, quantile for intervals.

In code

python
1import numpy as np
2
3def fit_tree(X, g, depth, min_leaf=1):
4 """Least-squares regression tree on pseudo-residuals g. Returns a nested dict."""
5 if depth == 0 or len(g) < 2 * min_leaf:
6 return {"rows": np.arange(len(g))}
7 best = (np.inf, None, None)
8 for j in range(X.shape[1]):
9 order = np.argsort(X[:, j], kind="stable")
10 xs, gs = X[order, j], g[order]
11 for k in range(min_leaf, len(g) - min_leaf + 1):
12 if xs[k - 1] == xs[k]:
13 continue
14 L, R = gs[:k], gs[k:]
15 sse = ((L - L.mean()) ** 2).sum() + ((R - R.mean()) ** 2).sum()
16 if sse < best[0] - 1e-12:
17 best = (sse, j, (xs[k - 1] + xs[k]) / 2)
18 if best[1] is None:
19 return {"rows": np.arange(len(g))}
20 _, j, t = best
21 left = X[:, j] <= t
22 lt, rt = fit_tree(X[left], g[left], depth - 1, min_leaf), fit_tree(X[~left], g[~left], depth - 1, min_leaf)
23 return {"j": j, "t": t, "left": lt, "right": rt, "lidx": np.where(left)[0], "ridx": np.where(~left)[0]}
24
25def set_leaf_values(node, r, idx, loss):
26 """Line search per leaf: the constant that minimises the loss of the raw residuals r in that leaf."""
27 if "j" not in node:
28 rl = r[idx]
29 node["value"] = rl.mean() if loss == "squared" else np.median(rl)
30 return
31 set_leaf_values(node["left"], r, idx[node["lidx"]], loss)
32 set_leaf_values(node["right"], r, idx[node["ridx"]], loss)
33
34def predict_tree(node, X):
35 if "j" not in node:
36 return np.full(len(X), node["value"])
37 out, left = np.empty(len(X)), X[:, node["j"]] <= node["t"]
38 out[left], out[~left] = predict_tree(node["left"], X[left]), predict_tree(node["right"], X[~left])
39 return out
40
41class GradientBoostingRegressor:
42 def __init__(self, n_estimators=100, learning_rate=0.1, max_depth=3, loss="squared"):
43 self.M, self.nu, self.depth, self.loss = n_estimators, learning_rate, max_depth, loss
44
45 def fit(self, X, y):
46 self.f0 = y.mean() if self.loss == "squared" else np.median(y) # argmin_c sum L(y, c)
47 F, self.trees = np.full(len(y), self.f0), []
48 for _ in range(self.M):
49 r = y - F # raw residuals
50 g = r if self.loss == "squared" else np.sign(r) # negative gradient of the loss
51 tree = fit_tree(X, g, self.depth) # tree on pseudo-residuals
52 set_leaf_values(tree, r, np.arange(len(y)), self.loss)
53 F = F + self.nu * predict_tree(tree, X) # shrunken step
54 self.trees.append(tree)
55 return self
56
57 def predict(self, X):
58 return self.f0 + self.nu * sum(predict_tree(t, X) for t in self.trees)
59
60X = np.array([30, 40, 50, 60, 70, 80, 90, 100, 110, 120.])[:, None]
61y = np.array([8, 9, 11, 10, 15, 16, 18, 17, 24, 25.])
62for M in (1, 2, 3):
63 gbr = GradientBoostingRegressor(n_estimators=M, learning_rate=0.5, max_depth=1).fit(X, y)
64 print(M, "trees: F(75) =", round(gbr.predict(np.array([[75.]]))[0], 3),
65 " train MSE =", round(((gbr.predict(X) - y) ** 2).mean(), 3))
66# 1 trees: F(75) = 17.233 train MSE = 15.19
67# 2 trees: F(75) = 16.325 train MSE = 5.289
68# 3 trees: F(75) = 15.524 train MSE = 2.399
From scratch in NumPy: squared or absolute loss, trees on pseudo-residuals, line search per leaf. Prints 17.233 / 16.325 / 15.524 for F(75) — the hand calculation.
python
1import numpy as np
2from sklearn.ensemble import GradientBoostingRegressor
3from sklearn.metrics import mean_squared_error
4
5X = np.array([30, 40, 50, 60, 70, 80, 90, 100, 110, 120.])[:, None] # area, m^2
6y = np.array([8, 9, 11, 10, 15, 16, 18, 17, 24, 25.]) # rent, $100s
7Xt = np.array([33, 44, 57, 63, 76, 87, 94, 107, 116, 125.])[:, None] # held-out flats
8yt = np.array([8.5, 10, 10.5, 12, 15.5, 16.5, 17, 20, 24, 25.5])
9
10gbr = GradientBoostingRegressor(loss="squared_error", n_estimators=2,
11 learning_rate=0.5, max_depth=1).fit(X, y)
12print("F0 =", gbr.init_.constant_.ravel()[0])
13for m, est in enumerate(gbr.estimators_[:, 0], 1):
14 t = est.tree_
15 print(f"tree {m}: x <= {t.threshold[0]:.0f} -> {t.value[1,0,0]:+.3f}, else {t.value[2,0,0]:+.3f}")
16print("F(75) =", round(gbr.predict([[75]])[0], 3))
17
18# Learning curve on held-out data: staged_predict gives F_m for every m
19big = GradientBoostingRegressor(n_estimators=50, learning_rate=0.5, max_depth=1).fit(X, y)
20test = [mean_squared_error(yt, p) for p in big.staged_predict(Xt)]
21print("best round:", int(np.argmin(test)) + 1, " test MSE:", round(min(test), 3),
22 " after 50:", round(test[-1], 3))
23
24# F0 = 15.3
25# tree 1: x <= 65 -> -5.800, else +3.867
26# tree 2: x <= 105 -> -1.817, else +7.267
27# F(75) = 16.325
28# best round: 5 test MSE: 2.079 after 50: 2.425
scikit-learn on the same data: the fitted stumps match the hand calculation, and staged_predict gives the test-MSE curve used to pick the number of trees.

Use when

  • +Tabular data with nonlinear effects and interactions — boosted trees are usually the most accurate off-the-shelf model there.
  • +Mixed feature types and scales, with no normalization needed; monotonic transforms of features change nothing.
  • +You need a non-standard regression loss: absolute, Huber, quantile (prediction intervals), Poisson or Tweedie.
  • +You can afford a validation set and some tuning of ν, depth and the number of trees.

Avoid when

  • −Images, audio or text, where neural networks learn the features far better.
  • −Predictions outside the training range of the target or features (trends, extrapolation).
  • −Very small or very noisy data with no budget for validation — it overfits easily without early stopping.
  • −Training must be parallel across trees or updated online — the rounds are inherently sequential.
  • −A smooth or exactly interpretable model is required; a sum of hundreds of trees is a black box (use SHAP for explanations).

Where it's used

Demand and sales forecasting

Store-item sales from calendar, price and promotion features — boosted trees dominate retail forecasting competitions such as M5.

Real-estate and price estimation

House values, insurance premiums and used-car prices from dozens of mixed tabular features.

Travel-time and logistics ETA

Delivery and ride-hailing arrival times from distance, traffic, time of day and region.

Quantile risk models

With quantile loss, separate models for the 5th and 95th percentile give prediction intervals for energy load or claim size.

Check yourself

With squared loss, F₀ = 15.3 and a first stump whose right leaf holds +3.867. With ν = 0.5, what is F₁ for a 100 m² flat?

Check yourself

A leaf contains the residuals 2, 3 and 7. What value γ does the line search give it under squared loss and under absolute loss?

Check yourself

For absolute loss L = |y − F|, the residuals of three points are −7.3, 0.7 and 9.7. What pseudo-residuals is the next tree fitted to?

Check yourself

You lower the learning rate from 0.5 to 0.1 and keep everything else. What should you expect?

Remember

  • ✓Model: FM=F0+ν∑mhmF_M=F_0+\nu\sum_m h_m, a constant plus many shrunken shallow trees, added one at a time.
  • ✓Each tree is fitted to the pseudo-residuals −∂L/∂F-\partial L/\partial F. For squared loss these are ordinary residuals y−Fy-F, and each leaf predicts their mean.
  • ✓Other losses only change the pseudo-residuals and the leaf line search: sign and median for absolute loss, clipping for Huber, quantiles for intervals.
  • ✓Learning rate × number of trees is the main trade-off: small ν with many trees generalizes better; choose the stopping round on validation data.
  • ✓Depth sets the interaction order (stumps = additive model); min_samples_leaf and subsample regularize.
  • ✓Training error always falls to zero and the model never extrapolates — use early stopping, and watch the target range.
← back to ML Algorithms roadmap