# Gradient boosting regression

> Fit each new tree to the residuals of the current model and add a shrunken copy — gradient descent in function space.

Source: https://www.mlfoundry.in/learn/gradient-boosting-regression  
Section: ML Algorithms › Supervised · Regression  
Level: core · 17 min read · updated 2026-10-07

**Gradient boosting regression:** builds a prediction $F_M(x)=F_0+\nu\sum_{m=1}^{M}h_m(x)$ one small regression tree at a time. Each new tree $h_m$ is fitted to the **negative gradient of the loss** at the current model — for squared error, simply the residuals $y_i-F_{m-1}(x_i)$ — and only a fraction $\nu$ of it is added.

- **Task:** Regression (a numeric target); the same machinery with log-loss gives classification
- **Input → output:** feature vector $x$ → real number $F_M(x)$, a sum of $M$ tree outputs
- **Learns:** an initial constant $F_0$, then $M$ shallow trees: split thresholds and one value $\gamma$ per leaf
- **Loss:** squared error (default), absolute error, Huber, quantile — any differentiable loss
- **Training:** sequential: $M$ trees, each about $O(d\,n\log n)$ with sorting, or $O(d\,n)$ with histograms
- **Prediction:** $O(M\cdot\text{depth})$ comparisons
- **Hyperparameters:** `n_estimators` $M$, `learning_rate` $\nu$, `max_depth`, `min_samples_leaf`, `subsample`, `loss`
- **Assumes:** nothing about the shape of the function; features need no scaling. The model is piecewise constant and cannot extrapolate

## The problem

A single regression tree (see **decision-tree-regression**) is a step function. A shallow tree is too coarse and underfits — *high bias*. A deep tree fits every bump in the training data, noise included — *high variance*. **Random-forest-regression** fixes the variance side: it averages many deep trees grown independently on bootstrap samples. But averaging cannot fix bias. If every tree misses the same pattern, so does the average.

Boosting attacks the **bias**. It starts from a model that is far too simple and adds small trees **one after another**. Each new tree is aimed at what the current model still gets wrong. AdaBoost (see **adaboost-regression**) does this by reweighting the training points, which works but is hard to extend to other losses. Gradient boosting (Friedman, 2001) gives a cleaner recipe that works for *any* differentiable loss: treat the model's predictions as parameters, and do gradient descent on them.

## The idea

Start by predicting the average rent for every flat. Look at the errors: the small flats are overpriced and the big ones underpriced. Fit a tiny tree that predicts those errors from the features, and add **half** of what it says to the model. The errors shrink, so look at them again and fit another tiny tree. Each tree is a weak, local correction. A hundred of them, each added cautiously, add up to an accurate model.

Why call it *gradient*? For squared error, the residual $y_i-F(x_i)$ is exactly the direction in which moving the prediction $F(x_i)$ lowers the loss fastest — the negative gradient. A tree fitted to residuals is therefore an approximate gradient step. Unlike a raw step, though, it is a *function* defined at every $x$, so the correction also applies to new inputs.

> Figure: Gradient boosting with stumps on 30 noisy points of a sine wave (ν = 0.3, 40 rounds). Top: data and the current model F. Bottom: the residuals and the stump fitted to them. Every round adds 0.3 of that stump, and the training MSE (sparkline) falls toward the noise level and then below it.

**Vocabulary**

- **Weak learner / base learner:** the small model added each round — here a regression tree with 1–8 levels. A depth-1 tree (one split, two leaves) is a **stump**
- **Initial model F₀:** the best constant: the mean of $y$ for squared loss, the median for absolute loss
- **Residual:** $r_i=y_i-F_{m-1}(x_i)$, what the current model still gets wrong on point $i$
- **Pseudo-residual:** the negative gradient $g_i=-\partial L/\partial F$ at the current prediction. It equals the residual for squared loss and $\operatorname{sign}(r_i)$ for absolute loss. Trees are fitted to these
- **Leaf value γ:** the constant a leaf adds, chosen by a **line search**: the value that minimizes the loss of the points in that leaf
- **Learning rate / shrinkage ν:** fraction of each tree that is added, $0<\nu\le 1$
- **Stage / round / iteration:** one tree added; $F_m$ is the model after $m$ stages
- **Staged prediction:** the predictions $F_0, F_1,\dots,F_M$ for every $m$, used to pick the best number of trees

## The math

We want the function $F$ that minimizes the total training loss, and we restrict $F$ to sums of trees:

$$
\min_{F}\ \sum_{i=1}^{n}L\big(y_i,F(x_i)\big),\qquad F(x)=F_0+\nu\sum_{m=1}^{M}h_m(x)
$$

- $L(y,F)$: loss for one point, e.g. $\tfrac12(y-F)^2$
- $x_i,\ y_i$: features and target of training point $i$; $n$ points in total
- $F_0$: initial constant prediction
- $h_m$: regression tree added at stage $m$
- $\nu$: learning rate (shrinkage), 0 < ν ≤ 1
- $M$: number of trees (stages)

**Step 0 — the best constant.** Before any tree, use the single number that minimizes the loss:

$$
F_0=\arg\min_{c}\sum_i L(y_i,c)\;=\;\begin{cases}\bar y & \text{squared loss}\\ \operatorname{median}(y) & \text{absolute loss}\end{cases}
$$

- $c$: a candidate constant prediction
- $\bar y$: mean of the training targets

For squared loss, set the derivative $\sum_i -(y_i-c)$ to zero and you get $c=\bar y$. For absolute loss, the derivative $\sum_i -\operatorname{sign}(y_i-c)$ is zero when half the points lie above $c$ and half below — the median.

**Step 1 — the gradient.** Think of the $n$ predictions $F(x_1),\dots,F(x_n)$ as $n$ free parameters. Gradient descent would move each one along the negative partial derivative of the loss:

$$
g_{im}=-\left[\frac{\partial L(y_i,F(x_i))}{\partial F(x_i)}\right]_{F=F_{m-1}}\qquad\text{for }L=\tfrac12(y-F)^2:\quad g_{im}=y_i-F_{m-1}(x_i)=r_{im}
$$

- $g_{im}$: pseudo-residual of point $i$ at stage $m$
- $F_{m-1}$: the model after $m-1$ trees
- $r_{im}$: ordinary residual: target minus current prediction

The factor $\tfrac12$ is only there to cancel the 2 from differentiating. So for squared error, **"fit the residuals" and "follow the negative gradient" are the same instruction**. A raw gradient step only says how to change the predictions at the $n$ training points, though. It says nothing about new inputs.

**Step 2 — project the gradient onto a tree.** Fit a regression tree to the pairs $(x_i, g_{im})$ by least squares. This gives a function defined everywhere that points roughly in the gradient direction. The tree splits the input space into leaf regions:

$$
h_m=\arg\min_{h\in\text{trees of depth}\le D}\ \sum_i\big(g_{im}-h(x_i)\big)^2\ \Rightarrow\ \text{leaf regions } R_{1m},\dots,R_{J_m m}
$$

- $D$: maximum tree depth (`max_depth`)
- $R_{jm}$: region of input space covered by leaf $j$ of tree $m$
- $J_m$: number of leaves of tree $m$ (at most $2^D$)

The tree is grown greedily, exactly like a single regression tree: at each node it picks the split that most reduces the squared error of the pseudo-residuals.

**Step 3 — line search in each leaf.** The fitted tree fixes the regions. We then re-choose each leaf's value to minimize the *actual* loss rather than the squared error on gradients:

$$
\gamma_{jm}=\arg\min_{\gamma}\sum_{x_i\in R_{jm}}L\big(y_i,\,F_{m-1}(x_i)+\gamma\big)
$$

- $\gamma_{jm}$: value of leaf $j$ in tree $m$
- $x_i\in R_{jm}$: the training points that fall in that leaf

For squared loss, differentiate: $-\sum_{i\in R_{jm}}\big(r_{im}-\gamma\big)=0$, so $\gamma_{jm}=\tfrac{1}{|R_{jm}|}\sum_{i\in R_{jm}} r_{im}$, the **mean residual in the leaf**. That is what the tree predicted anyway, so for squared loss Step 3 changes nothing. For absolute loss the tree is fitted to signs $\pm1$, but the line search sets $\gamma$ to the **median residual** in the leaf, which is in the target's units.

**Step 4 — take a shrunken step.**

$$
F_m(x)=F_{m-1}(x)+\nu\sum_{j=1}^{J_m}\gamma_{jm}\,\mathbf 1\!\left[x\in R_{jm}\right]
$$

- $\mathbf 1[\cdot]$: 1 if the condition holds, else 0 — so exactly one leaf's γ is added
- $\nu$: learning rate: the step size of this function-space gradient descent

**What $\nu$ does, exactly.** Suppose a group of points keeps landing in the same leaf. With squared loss, each round adds $\nu$ times their mean residual, so their mean residual is multiplied by $(1-\nu)$ every round. After $k$ rounds it is $(1-\nu)^k$ of the original. With $\nu=0.5$ that is $0.5^4=0.0625$ after 4 rounds. With $\nu=0.1$ it takes 22 rounds to get under 10% ($0.9^{22}=0.098$). A small $\nu$ means more trees, but each tree's mistakes get diluted by the later ones, and that is why small $\nu$ usually generalizes better (Friedman's empirical finding). $\nu=1$ is a plain greedy fit that jumps straight to the leaf means.

**The other knobs.** `max_depth` $D$ sets the **interaction order**. A stump uses one feature, so the sum of stumps is *additive*: $F(x)=\sum_j f_j(x_j)$. A depth-$D$ tree can model interactions between up to $D$ features. More depth also means more leaves ($\le 2^D$), so each tree fits more of the residual. `min_samples_leaf` keeps every $\gamma$ an average over several points, so a single noisy point cannot get its own correction. `subsample` $<1$ fits each tree on a random fraction of the rows (*stochastic* gradient boosting). Like minibatch SGD, this adds noise that regularizes and speeds training.

Regression losses and what the boosting loop uses for each. $\delta$ is the Huber threshold; $\alpha$ is the target quantile.

| Loss | L(y, F) | Pseudo-residual g | F₀ | Leaf value γ | Use when |
| --- | --- | --- | --- | --- | --- |
| Squared | $\tfrac12(y-F)^2$ | $r$ | mean | mean of r in the leaf | default; you want the conditional mean |
| Absolute | $\|y-F\|$ | $\operatorname{sign}(r)$ | median | median of r in the leaf | heavy-tailed noise, outliers |
| Huber | $\tfrac12 r^2$ if $\|r\|\le\delta$, else $\delta(\|r\|-\tfrac\delta2)$ | $\operatorname{clip}(r,-\delta,\delta)$ | median | median + mean of clipped deviations | mostly Gaussian noise with some outliers |
| Quantile | $\alpha r$ if $r>0$, else $(\alpha-1)r$ | $\alpha$ or $\alpha-1$ | $\alpha$-quantile | $\alpha$-quantile of r in the leaf | prediction intervals |

## Worked example: predicting rent from floor area

Ten flats, one feature. We boost **stumps** (`max_depth = 1`) with squared loss and $\nu=0.5$, and predict the rent of a new **75 m²** flat. Rent is in hundreds of dollars per month.

Training data. Mean rent 15.3; the variance (MSE of predicting the mean) is 32.01.

| Flat | Area x (m²) | Rent y (hundreds of dollars) |
| --- | --- | --- |
| 1 | 30 | 8 |
| 2 | 40 | 9 |
| 3 | 50 | 11 |
| 4 | 60 | 10 |
| 5 | 70 | 15 |
| 6 | 80 | 16 |
| 7 | 90 | 18 |
| 8 | 100 | 17 |
| 9 | 110 | 24 |
| 10 | 120 | 25 |

**Two rounds by hand**

1. **Initialize.** $F_0=\bar y=153/10=15.3$ for every flat. Training MSE $=32.01$ (total squared error 320.1).
2. **Round 1 — residuals.** $r_{i1}=y_i-15.3$: $-7.3,\ -6.3,\ -4.3,\ -5.3,\ -0.3,\ 0.7,\ 2.7,\ 1.7,\ 8.7,\ 9.7$. They sum to zero, as residuals from the mean always do.
3. **Round 1 — fit a stump.** Try all 9 midpoint thresholds (table below). The best is $x\le 65$: left leaf (flats 1–4) mean residual $-23.2/4=-5.8$, right leaf (flats 5–10) $23.2/6=3.867$, leaving squared error 95.833 of the 320.1 — the stump explains 70.1% of it.
4. **Round 1 — shrunken update.** $F_1=15.3+0.5(-5.8)=12.4$ for $x\le65$ and $15.3+0.5(3.867)=17.233$ for $x>65$. Training MSE $32.01\to15.19$.
5. **Round 2 — new residuals.** $r_{i2}=y_i-F_1(x_i)$: $-4.4,\ -3.4,\ -1.4,\ -2.4,\ -2.233,\ -1.233,\ 0.767,\ -0.233,\ 6.767,\ 7.767$. Only half the first correction was applied, so the two groups are still off in the same direction. The big flats are also now clearly the worst-fitted.
6. **Round 2 — fit a stump.** The best split is now $x\le105$ (squared error 19.889): left leaf (flats 1–8) $-1.817$, right leaf (flats 9–10) $+7.267$. Note that the new tree picks a *different* threshold — it attacks the largest remaining error.
7. **Round 2 — update.** $F_2=F_1+0.5\,h_2$: $12.4-0.908=11.492$ for $x\le60$, $17.233-0.908=16.325$ for $70\le x\le100$, and $17.233+3.633=20.867$ for $x\ge110$. Training MSE $15.19\to5.289$.
8. **Predict $x=75$.** It goes right in tree 1 ($75>65$) and left in tree 2 ($75\le105$): $F_2(75)=15.3+0.5(3.867)+0.5(-1.817)=\mathbf{16.325}$, i.e. about **1,633 dollars per month**. A third stump (split at $x\le85$) would give 15.524 and training MSE 2.399.

Round 1: every candidate stump scored on the residuals. Squared error = sum of squared deviations from the leaf means.

| Threshold | Left mean | Right mean | Squared error |
| --- | --- | --- | --- |
| x ≤ 35 | −7.300 | +0.811 | 260.889 |
| x ≤ 45 | −6.800 | +1.700 | 204.500 |
| x ≤ 55 | −5.967 | +2.557 | 167.524 |
| **x ≤ 65** | **−5.800** | **+3.867** | **95.833** |
| x ≤ 75 | −4.700 | +4.700 | 99.200 |
| x ≤ 85 | −3.800 | +5.700 | 103.500 |
| x ≤ 95 | −2.871 | +6.700 | 127.714 |
| x ≤ 105 | −2.300 | +9.200 | 108.500 |
| x ≤ 115 | −1.078 | +9.700 | 215.556 |

The two rounds row by row (ν = 0.5). h is the leaf value the stump gives that row; F₁ = 15.3 + 0.5·h₁ and F₂ = F₁ + 0.5·h₂.

| x | y | r₁ | h₁ | F₁ | r₂ | h₂ | F₂ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 30 | 8 | −7.3 | −5.800 | 12.400 | −4.400 | −1.817 | 11.492 |
| 40 | 9 | −6.3 | −5.800 | 12.400 | −3.400 | −1.817 | 11.492 |
| 50 | 11 | −4.3 | −5.800 | 12.400 | −1.400 | −1.817 | 11.492 |
| 60 | 10 | −5.3 | −5.800 | 12.400 | −2.400 | −1.817 | 11.492 |
| 70 | 15 | −0.3 | +3.867 | 17.233 | −2.233 | −1.817 | 16.325 |
| 80 | 16 | +0.7 | +3.867 | 17.233 | −1.233 | −1.817 | 16.325 |
| 90 | 18 | +2.7 | +3.867 | 17.233 | +0.767 | −1.817 | 16.325 |
| 100 | 17 | +1.7 | +3.867 | 17.233 | −0.233 | −1.817 | 16.325 |
| 110 | 24 | +8.7 | +3.867 | 17.233 | +6.767 | +7.267 | 20.867 |
| 120 | 25 | +9.7 | +3.867 | 17.233 | +7.767 | +7.267 | 20.867 |

> Figure: The worked example, live. Top: the 10 flats (drag any point up or down), 10 held-out test flats, and the model F_m (orange; faded = F_{m−1}). Middle: round m's residuals and the tree fitted to them, with its leaf values γ. Bottom: train and test MSE for every round. The defaults (squared loss, ν = 0.5, depth 1, round 2) reproduce the hand calculation: F₂(75) = 16.325, train MSE 5.289. Press ▶ to run to 50 rounds: train MSE falls to 0.019, while test MSE is lowest at round 5 (2.079) and ends at 2.425. Try ν = 0.1 (best test 1.922 at round 39), or drag the 60 m² flat up to 30 and compare squared with absolute loss.

> **Each tree attacks what is left:** Tree 1 split at 65 m² (small vs large flats). Tree 2 did not repeat that split. It moved to 105 m², because after round 1 the two biggest flats had the largest residuals. Boosting is **adaptive**: every tree sees a different target. Random-forest trees all fit the same $y$ and differ only through random sampling.

## The algorithm, step by step

**GradientBoost(X, y, L, M, ν, D)**

1. **Initialize** $F_0(x)=\arg\min_c\sum_i L(y_i,c)$ — the mean for squared loss, the median for absolute or Huber loss.
2. **For** $m=1,\dots,M$:
3.   **Pseudo-residuals.** $g_{im}=-\partial L(y_i,F)/\partial F$ at $F=F_{m-1}(x_i)$ for every training point (with `subsample` < 1, only for a random subset of rows).
4.   **Fit a tree.** Grow a regression tree of depth $\le D$ on $(x_i,g_{im})$ with the squared-error split criterion, respecting `min_samples_leaf`.
5.   **Line search.** For each leaf $j$, set $\gamma_{jm}=\arg\min_\gamma\sum_{i\in R_{jm}}L(y_i,F_{m-1}(x_i)+\gamma)$.
6.   **Update.** $F_m(x)=F_{m-1}(x)+\nu\,\gamma_{j(x)m}$, where $j(x)$ is the leaf $x$ falls into.
7.   **Monitor.** Record validation loss; stop if it has not improved for a set number of rounds (early stopping).
8. **Predict** $F_M(x)$ — or $F_{m^*}(x)$ at the best validation round $m^*$.

**Complexity**

- **Training time:** $O(M\cdot d\cdot n\log n)$ with exact sorted splits; $O(M\cdot d\cdot n)$ with histogram binning (`HistGradientBoostingRegressor`, LightGBM). The $M$ rounds are **sequential** — only the split search inside one tree parallelizes.
- **Prediction time:** $O(M\cdot D)$ comparisons per input; a model with 500 depth-3 trees does 1,500 comparisons.
- **Memory:** $M$ trees with at most $2^D$ leaves each, plus the $n$ current predictions during training.

## Learning rate, depth and number of trees

These three interact. A smaller $\nu$ needs proportionally more trees. Deeper trees fit faster but each one overfits more. On the 10 test flats from the figure (squared loss, test MSE of the constant $F_0$ is 31.445), the best test MSE over 50 rounds was:

Best held-out MSE over 50 rounds and the round where it occurs, for every ν × depth combination (squared loss). By round 50 every setting has driven training MSE below 0.2, and test MSE has risen to 1.97–2.63.

| Learning rate | depth 1 (stumps) | depth 2 | depth 3 |
| --- | --- | --- | --- |
| ν = 0.1 | 1.922 @ round 39 | 1.650 @ round 23 | 1.397 @ round 18 |
| ν = 0.5 | 2.079 @ round 5 | 1.764 @ round 4 | 1.423 @ round 3 |
| ν = 1.0 | 2.154 @ round 8 | 2.566 @ round 4 | 2.400 @ round 1 |

**Practical tuning**

- **learning_rate ν:** Fix it small (0.05–0.1) and let early stopping choose the number of trees. Lower ν rarely hurts accuracy, only training time.
- **n_estimators M:** Do not tune it by grid search — set it large and use `n_iter_no_change` with `validation_fraction`, or pick the best round from `staged_predict` on a validation set.
- **max_depth:** 3–8. Depth 1 gives an additive model with no interactions; deeper trees capture interactions but overfit faster. Alternatively cap `max_leaf_nodes`.
- **min_samples_leaf:** 10–100 on real data. It stops a leaf from being spent on one outlier and smooths each γ.
- **subsample:** 0.5–0.8 (stochastic gradient boosting): faster, and often more accurate through added randomness.
- **max_features:** a random subset of features per split, as in random forests — decorrelates trees and speeds up wide data.
- **loss:** `squared_error` for the mean, `absolute_error` or `huber` (with `alpha` setting δ as a quantile of |r|) for outliers, `quantile` for intervals.

> **Where the famous libraries differ:** **XGBoost**, **LightGBM** and **CatBoost** run this same loop with three upgrades. They take a second-order (Newton) step using the Hessian. They add explicit L1/L2 penalties on leaf values. And they find splits with histograms or special category handling. See **xgboost-regression**, **lightgbm-regression** and **catboost-regression**; for classification with log-loss, see **gradient-boosting**.

## In code

```python
import numpy as np

def fit_tree(X, g, depth, min_leaf=1):
  """Least-squares regression tree on pseudo-residuals g. Returns a nested dict."""
  if depth == 0 or len(g) < 2 * min_leaf:
      return {"rows": np.arange(len(g))}
  best = (np.inf, None, None)
  for j in range(X.shape[1]):
      order = np.argsort(X[:, j], kind="stable")
      xs, gs = X[order, j], g[order]
      for k in range(min_leaf, len(g) - min_leaf + 1):
          if xs[k - 1] == xs[k]:
              continue
          L, R = gs[:k], gs[k:]
          sse = ((L - L.mean()) ** 2).sum() + ((R - R.mean()) ** 2).sum()
          if sse < best[0] - 1e-12:
              best = (sse, j, (xs[k - 1] + xs[k]) / 2)
  if best[1] is None:
      return {"rows": np.arange(len(g))}
  _, j, t = best
  left = X[:, j] <= t
  lt, rt = fit_tree(X[left], g[left], depth - 1, min_leaf), fit_tree(X[~left], g[~left], depth - 1, min_leaf)
  return {"j": j, "t": t, "left": lt, "right": rt, "lidx": np.where(left)[0], "ridx": np.where(~left)[0]}

def set_leaf_values(node, r, idx, loss):
  """Line search per leaf: the constant that minimises the loss of the raw residuals r in that leaf."""
  if "j" not in node:
      rl = r[idx]
      node["value"] = rl.mean() if loss == "squared" else np.median(rl)
      return
  set_leaf_values(node["left"], r, idx[node["lidx"]], loss)
  set_leaf_values(node["right"], r, idx[node["ridx"]], loss)

def predict_tree(node, X):
  if "j" not in node:
      return np.full(len(X), node["value"])
  out, left = np.empty(len(X)), X[:, node["j"]] <= node["t"]
  out[left], out[~left] = predict_tree(node["left"], X[left]), predict_tree(node["right"], X[~left])
  return out

class GradientBoostingRegressor:
  def __init__(self, n_estimators=100, learning_rate=0.1, max_depth=3, loss="squared"):
      self.M, self.nu, self.depth, self.loss = n_estimators, learning_rate, max_depth, loss

  def fit(self, X, y):
      self.f0 = y.mean() if self.loss == "squared" else np.median(y)   # argmin_c sum L(y, c)
      F, self.trees = np.full(len(y), self.f0), []
      for _ in range(self.M):
          r = y - F                                            # raw residuals
          g = r if self.loss == "squared" else np.sign(r)      # negative gradient of the loss
          tree = fit_tree(X, g, self.depth)                    # tree on pseudo-residuals
          set_leaf_values(tree, r, np.arange(len(y)), self.loss)
          F = F + self.nu * predict_tree(tree, X)              # shrunken step
          self.trees.append(tree)
      return self

  def predict(self, X):
      return self.f0 + self.nu * sum(predict_tree(t, X) for t in self.trees)

X = np.array([30, 40, 50, 60, 70, 80, 90, 100, 110, 120.])[:, None]
y = np.array([8, 9, 11, 10, 15, 16, 18, 17, 24, 25.])
for M in (1, 2, 3):
  gbr = GradientBoostingRegressor(n_estimators=M, learning_rate=0.5, max_depth=1).fit(X, y)
  print(M, "trees: F(75) =", round(gbr.predict(np.array([[75.]]))[0], 3),
        " train MSE =", round(((gbr.predict(X) - y) ** 2).mean(), 3))
# 1 trees: F(75) = 17.233  train MSE = 15.19
# 2 trees: F(75) = 16.325  train MSE = 5.289
# 3 trees: F(75) = 15.524  train MSE = 2.399
```

From scratch in NumPy: squared or absolute loss, trees on pseudo-residuals, line search per leaf. Prints 17.233 / 16.325 / 15.524 for F(75) — the hand calculation.

```python
import numpy as np
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.metrics import mean_squared_error

X = np.array([30, 40, 50, 60, 70, 80, 90, 100, 110, 120.])[:, None]   # area, m^2
y = np.array([8, 9, 11, 10, 15, 16, 18, 17, 24, 25.])                 # rent, $100s
Xt = np.array([33, 44, 57, 63, 76, 87, 94, 107, 116, 125.])[:, None]   # held-out flats
yt = np.array([8.5, 10, 10.5, 12, 15.5, 16.5, 17, 20, 24, 25.5])

gbr = GradientBoostingRegressor(loss="squared_error", n_estimators=2,
                              learning_rate=0.5, max_depth=1).fit(X, y)
print("F0 =", gbr.init_.constant_.ravel()[0])
for m, est in enumerate(gbr.estimators_[:, 0], 1):
  t = est.tree_
  print(f"tree {m}: x <= {t.threshold[0]:.0f} -> {t.value[1,0,0]:+.3f}, else {t.value[2,0,0]:+.3f}")
print("F(75) =", round(gbr.predict([[75]])[0], 3))

# Learning curve on held-out data: staged_predict gives F_m for every m
big = GradientBoostingRegressor(n_estimators=50, learning_rate=0.5, max_depth=1).fit(X, y)
test = [mean_squared_error(yt, p) for p in big.staged_predict(Xt)]
print("best round:", int(np.argmin(test)) + 1, " test MSE:", round(min(test), 3),
    " after 50:", round(test[-1], 3))

# F0 = 15.3
# tree 1: x <= 65 -> -5.800, else +3.867
# tree 2: x <= 105 -> -1.817, else +7.267
# F(75) = 16.325
# best round: 5  test MSE: 2.079  after 50: 2.425
```

scikit-learn on the same data: the fitted stumps match the hand calculation, and staged_predict gives the test-MSE curve used to pick the number of trees.

> **On real data:** For more than ~10,000 rows use `HistGradientBoostingRegressor`: it bins features into 255 histogram buckets, handles missing values natively, and early-stops by default. Its `max_iter` plays the role of `n_estimators`.

> **Training error always goes to zero:** Every round lowers the training loss, so the training curve never tells you when to stop. In the example, 50 stumps at $\nu=0.5$ push train MSE to 0.019. Test MSE was best at round 5 (2.079) and is 2.425 by round 50. Always select the number of trees on held-out data (early stopping or `staged_predict`), never on training error.

> **Squared loss chases outliers:** Change one rent in the example from 10 to 30 (the 60 m² flat) and run 50 rounds ($\nu=0.5$, stumps). With squared loss the model predicts 28.765 for that flat and drags its neighbours with it: test MSE 63.514. With absolute loss the pseudo-residual of the outlier is just $+1$ and leaf values are medians. The model predicts 16.250 there, and test MSE is 8.024. Huber with $\delta=2$ lands in between: 22.468 and 27.076. Huber still overfits eventually, because once a leaf isolates one point its line-search value is that point's full residual. Use a robust loss **and** `min_samples_leaf`.

> **No extrapolation:** Every tree is constant beyond the last training threshold, and so is their sum. A 200 m² flat gets exactly the same prediction as the 120 m² flat. If the target trends beyond the training range (prices over time, dose-response), model the trend first (e.g. a linear model) and boost its residuals, or use a method that extrapolates.

**Strengths**

- Tabular data with nonlinear effects and interactions — boosted trees are usually the most accurate off-the-shelf model there.
- Mixed feature types and scales, with no normalization needed; monotonic transforms of features change nothing.
- You need a non-standard regression loss: absolute, Huber, quantile (prediction intervals), Poisson or Tweedie.
- You can afford a validation set and some tuning of ν, depth and the number of trees.

**Limitations**

- Images, audio or text, where neural networks learn the features far better.
- Predictions outside the training range of the target or features (trends, extrapolation).
- Very small or very noisy data with no budget for validation — it overfits easily without early stopping.
- Training must be parallel across trees or updated online — the rounds are inherently sequential.
- A smooth or exactly interpretable model is required; a sum of hundreds of trees is a black box (use SHAP for explanations).

- **Demand and sales forecasting:** Store-item sales from calendar, price and promotion features — boosted trees dominate retail forecasting competitions such as M5.
- **Real-estate and price estimation:** House values, insurance premiums and used-car prices from dozens of mixed tabular features.
- **Travel-time and logistics ETA:** Delivery and ride-hailing arrival times from distance, traffic, time of day and region.
- **Quantile risk models:** With quantile loss, separate models for the 5th and 95th percentile give prediction intervals for energy load or claim size.

**Question:** With squared loss, F₀ = 15.3 and a first stump whose right leaf holds +3.867. With ν = 0.5, what is F₁ for a 100 m² flat?

- A. 15.300
- B. 17.233
- C. 19.167
- D. 13.367

**Answer:** B. 17.233 — Since $100>65$, the flat lands in the right leaf: $F_1=15.3+0.5\times3.867=17.233$. With $\nu=1$ it would be $19.167$.

**Question:** A leaf contains the residuals 2, 3 and 7. What value γ does the line search give it under squared loss and under absolute loss?

- A. 4 and 4
- B. 3 and 4
- C. 4 and 3
- D. 7 and 2

**Answer:** C. 4 and 3 — Squared loss: $\gamma=\text{mean}=(2+3+7)/3=4$. Absolute loss: the median, 3 — the 7 pulls the mean up but not the median.

**Question:** For absolute loss L = |y − F|, the residuals of three points are −7.3, 0.7 and 9.7. What pseudo-residuals is the next tree fitted to?

- A. −7.3, 0.7, 9.7
- B. −1, 1, 1
- C. 53.29, 0.49, 94.09
- D. 0, 0, 0

**Answer:** B. −1, 1, 1 — $-\partial|y-F|/\partial F=\operatorname{sign}(y-F)$, so every point counts as $\pm1$ regardless of how large its error is — which is why absolute loss is robust to outliers.

**Question:** You lower the learning rate from 0.5 to 0.1 and keep everything else. What should you expect?

- A. The same best model with the same number of trees
- B. More trees needed to reach the best validation error, which is often slightly lower
- C. Fewer trees needed, because each step is more accurate
- D. Training MSE can no longer reach zero

**Answer:** B. More trees needed to reach the best validation error, which is often slightly lower — Each tree contributes a fifth as much, so roughly five times as many are needed. In the example (stumps) the best test MSE moved from 2.079 at round 5 to 1.922 at round 39.

**Key points**

- Model: $F_M=F_0+\nu\sum_m h_m$, a constant plus many shrunken shallow trees, added one at a time.
- Each tree is fitted to the pseudo-residuals $-\partial L/\partial F$. For squared loss these are ordinary residuals $y-F$, and each leaf predicts their mean.
- Other losses only change the pseudo-residuals and the leaf line search: sign and median for absolute loss, clipping for Huber, quantiles for intervals.
- Learning rate × number of trees is the main trade-off: small ν with many trees generalizes better; choose the stopping round on validation data.
- Depth sets the interaction order (stumps = additive model); min_samples_leaf and subsample regularize.
- Training error always falls to zero and the model never extrapolates — use early stopping, and watch the target range.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
