Gradient Boosting

Add trees one at a time, each fit to the negative gradient of the current ensemble's loss.

17 min read1399 wordscore9 viewsupdated 2026-10-07

Definition

Gradient boosting (for classification) builds a log-odds score as a sum of many small regression trees, FM(x)=F0+ν∑m=1Mhm(x)F_M(x)=F_0+\nu\sum_{m=1}^{M}h_m(x), and turns it into a probability with the sigmoid, p(x)=σ(FM(x))p(x)=\sigma(F_M(x)). Each new tree hmh_m is fit to the negative gradient of the log-loss of the model built so far — the pseudo-residuals yi−piy_i-p_i — so every round takes one gradient-descent step in the space of functions.

At a glance

Task
Binary and multiclass classification (the same machinery handles regression and ranking with other losses)
Input → output
xx → log-odds FM(x)F_M(x) → probability σ(FM(x))\sigma(F_M(x)) → class (threshold ½ or argmax of softmax)
Learns
MM shallow regression trees (depth 1–8) whose leaves hold log-odds corrections γ\gamma
Training
strictly sequential: MM tree fits, each about O(d nlog⁡n)O(d\,n\log n) (or O(d n)O(d\,n) with histogram binning)
Prediction
O(M⋅depth)O(M\cdot\text{depth}) comparisons, then one sigmoid
Hyperparameters
learning rate ν\nu, number of trees MM, tree depth / leaves, row subsample, min samples per leaf, L2 on leaf values
Assumes
nothing distributional; a differentiable loss; features need no scaling (trees only look at order)

The problem

A single decision tree has to choose between two failures. Shallow, it is too blunt to capture the real boundary (high bias); deep, it memorises noise (high variance). Random forests and Extra Trees fix the variance side: grow many deep, decorrelated trees independently and average them. But averaging cannot remove bias — if every tree misses the same structure, so does their mean.

Boosting attacks bias. Start with a model that is clearly too simple, and keep adding small trees exactly where the current model is wrong. AdaBoost did this first by re-weighting misclassified points, but its recipe is tied to one loss (the exponential loss) and to hard class votes. Gradient boosting generalises it: pick any differentiable loss — for classification, the log-loss that calibrated probabilities need — and let its gradient say what "wrong" means.

The idea

Keep a running score F(x)F(x) in log-odds units, starting from the base rate. For every training point, ask: in which direction, and how hard, should my score move to lower the loss? For log-loss the answer is simply yi−piy_i-p_i: a churner predicted at 0.3 wants +0.7+0.7; a stayer predicted at 0.3 wants −0.3-0.3. Those numbers are not labels any more — they are continuous targets — so we fit a small regression tree to them. The tree groups points that want similar pushes, each leaf decides one push size, and we add a fraction ν\nu of it to the score. Repeat.

Two consequences follow. First, points that are already confidently right (p≈yp\approx y) have residuals near zero, so later trees focus on the hard, misclassified or uncertain points — the same intuition as AdaBoost, derived instead of designed. Second, because the score is in log-odds, many small additive corrections compose cleanly and the final sigmoid always lands in (0,1)(0,1).

Vocabulary

Weak learner
a model only slightly better than a constant — here a shallow regression tree; a depth-1 tree is a stump
Raw score / margin $F(x)$
the ensemble's output before the sigmoid, in log-odds: F=log⁡p1−pF=\log\frac{p}{1-p}
Initial model $F_0$
the best constant score: the log-odds of the training base rate
Pseudo-residual $r_i$
the negative gradient of the loss with respect to F(xi)F(x_i); for log-loss ri=yi−pir_i=y_i-p_i
Hessian $h_i$
the second derivative of the loss, pi(1−pi)p_i(1-p_i); it measures how curved the loss is at that point
Leaf value $gamma$
the constant a leaf adds to FF; chosen by a Newton step, ∑r/∑h\sum r/\sum h over the leaf
Shrinkage / learning rate $ u$
the fraction of each tree actually added; small ν\nu = small, cautious steps
Stage / round / iteration
one cycle of compute residuals → fit tree → update; the model after mm rounds is FmF_m
Early stopping
choosing MM as the round where validation loss is lowest

The math

Labels are yi∈{0,1}y_i\in\{0,1\}, the model outputs a score F(x)F(x), and the predicted probability is its sigmoid. The loss for one point is the binary cross-entropy (log-loss):

ℓ(y,F)=−[ylog⁡p+(1−y)log⁡(1−p)],p=σ(F)=11+e−F\ell(y,F)=-\big[y\log p+(1-y)\log(1-p)\big],\qquad p=\sigma(F)=\frac{1}{1+e^{-F}}
yy
true label, 0 or 1
FF
the model's raw score (log-odds) for this point
pp
predicted probability of class 1
σ\sigma
the logistic sigmoid

Training minimises L(F)=∑iℓ(yi,F(xi))\mathcal L(F)=\sum_i \ell(y_i,F(x_i)). Substituting p=σ(F)p=\sigma(F) gives ℓ=log⁡(1+eF)−yF\ell=\log(1+e^{F})-yF, whose derivatives with respect to the score are remarkably simple (use σ′=σ(1−σ)\sigma'=\sigma(1-\sigma)):

gi=∂ℓ∂F(xi)=pi−yi,hi=∂2ℓ∂F(xi)2=pi(1−pi)g_i=\frac{\partial \ell}{\partial F(x_i)}=p_i-y_i,\qquad h_i=\frac{\partial^2 \ell}{\partial F(x_i)^2}=p_i(1-p_i)
gig_i
gradient of the loss at point i
hih_i
second derivative (Hessian) at point i; largest at p = ½, tiny when p is near 0 or 1

Gradient descent in function space. Think of the vector of predictions (F(x1),…,F(xn))(F(x_1),\dots,F(x_n)) as the parameters. Steepest descent would move each one by −gi=yi−pi-g_i=y_i-p_i. But a raw vector of nn numbers says nothing about new inputs, so we approximate the negative gradient with a tree: fit a least-squares regression tree to the targets ri=yi−pir_i=y_i-p_i. The tree's structure (its splits) defines regions R1,…,RJR_{1},\dots,R_{J} that generalise to unseen xx.

rim=−[∂ℓ(yi,F)∂F]F=Fm−1(xi)=yi−pi,{Rjm}=tree fit to (xi,rim) by squared errorr_{im}=-\left[\frac{\partial\ell(y_i,F)}{\partial F}\right]_{F=F_{m-1}(x_i)}=y_i-p_i,\qquad \{R_{jm}\}=\text{tree fit to }(x_i,r_{im})\text{ by squared error}
rimr_{im}
pseudo-residual of point i in round m
RjmR_{jm}
the region covered by leaf j of tree m
Fm−1F_{m-1}
the model after m − 1 rounds

Leaf values by a Newton step. The tree's own leaf means would be gradient-step sizes in the wrong units (probability, not log-odds). Instead, for each leaf we choose the constant γ\gamma that minimises the real loss over the points in it, using a second-order Taylor expansion around the current scores:

∑i∈Rjℓ(yi,Fm−1(xi)+γ)≈∑i∈Rj[ℓi+giγ+12hiγ2]  ⟹  γjm=−∑i∈Rjgi∑i∈Rjhi=∑i∈Rj(yi−pi)∑i∈Rjpi(1−pi)\sum_{i\in R_j}\ell\big(y_i,F_{m-1}(x_i)+\gamma\big)\approx\sum_{i\in R_j}\Big[\ell_i+g_i\gamma+\tfrac12h_i\gamma^2\Big]\;\Longrightarrow\;\gamma_{jm}=-\frac{\sum_{i\in R_j}g_i}{\sum_{i\in R_j}h_i}=\frac{\sum_{i\in R_j}(y_i-p_i)}{\sum_{i\in R_j}p_i(1-p_i)}
γjm\gamma_{jm}
value of leaf j in tree m, in log-odds units
ℓi\ell_i
current loss of point i (a constant in the minimisation)

Setting the derivative ∑gi+γ∑hi\sum g_i+\gamma\sum h_i to zero gives the formula. This is exactly what scikit-learn's GradientBoostingClassifier does (Friedman's one Newton step), and XGBoost and LightGBM go one step further by also using gg and hh to choose the splits (see the XGBoost article).

Fm(x)=Fm−1(x)+ν∑j=1Jγjm 1[x∈Rjm],F0=log⁡yˉ1−yˉF_m(x)=F_{m-1}(x)+\nu\sum_{j=1}^{J}\gamma_{jm}\,\mathbb 1[x\in R_{jm}],\qquad F_0=\log\frac{\bar y}{1-\bar y}
ν\nu
learning rate (shrinkage), 0 < ν ≤ 1
JJ
number of leaves per tree (2 for a stump)
yˉ\bar y
fraction of positives in the training set

F0F_0 is the best constant: minimising ∑iℓ(yi,c)\sum_i\ell(y_i,c) means ∑i(σ(c)−yi)=0\sum_i(\sigma(c)-y_i)=0, so σ(c)=yˉ\sigma(c)=\bar y and cc is the log-odds of the base rate. Multiclass with KK classes uses the softmax pik=eFk(xi)/∑k′eFk′(xi)p_{ik}=e^{F_k(x_i)}/\sum_{k'}e^{F_{k'}(x_i)} and grows KK trees per round, one per class, each fit to rik=yik−pikr_{ik}=y_{ik}-p_{ik} — so a 10-class problem with 300 rounds trains 3,000 trees.

What each hyperparameter does to the math

Learning rate ν
scales every step. With ν=1\nu=1 each Newton step is taken in full and the model reaches the training data in a few rounds; with small ν\nu it moves a fraction of the way, so later trees still get a say and the noise of any one tree is averaged out. Needed rounds scale roughly like 1/ν1/\nu.
Number of trees M
number of descent steps. The training loss decreases monotonically in MM (to 0 on separable data, by sending ∣F∣→∞|F|\to\infty); validation loss is U-shaped. MM is the main regulariser — choose it by early stopping.
Depth / leaves J
a depth-dd tree can express interactions between at most dd features; stumps give an additive model F(x)=∑jfj(xj)F(x)=\sum_j f_j(x_j). More leaves = bigger steps in more directions per round.
Subsample
fit each tree on a random fraction of rows (stochastic gradient boosting): noisier gradient estimates, decorrelated trees, less overfitting, faster.
Min samples / Hessian per leaf
keeps ∑hi\sum h_i in the Newton denominator away from 0. Without it, a tiny leaf of already-confident points (p≈0p\approx0 or 1, h≈0h\approx0) produces an enormous γ\gamma.
L2 on leaves λ
(XGBoost / LightGBM / HistGB) shrinks the leaf value to ∑ri/(∑hi+λ)\sum r_i/(\sum h_i+\lambda) — the same fix, made explicit.

Worked example: who will churn?

Ten customers, one feature — how many support tickets they filed last quarter — and whether they churned. 4 of 10 churned. The labels are deliberately noisy: customer 4 churned with few tickets and customer 9 stayed with many. We boost stumps with learning rate ν=0.5\nu=0.5 for 3 rounds.

CustomerTickets xChurned y
110
220
330
441
550
660
771
881
990
10101
Training data. Highlighted: the four churners.

Three rounds by hand (ν = 0.5)

  1. 1Initialise. yˉ=0.4\bar y=0.4, so F0=log⁡(0.4/0.6)=−0.405F_0=\log(0.4/0.6)=-0.405 and every pi=0.4p_i=0.4. Training log-loss 0.6730.673; predicting "stay" for all gets 6/10 right.
  2. 2Round 1 — residuals. ri=yi−0.4r_i=y_i-0.4: churners get +0.6+0.6, stayers −0.4-0.4. Every hi=0.4⋅0.6=0.24h_i=0.4\cdot0.6=0.24.
  3. 3Round 1 — split. Try the 9 midpoints; the squared error of a stump on the residuals is smallest at x≤6.5x\le6.5: 1.5831.583, down from 2.4002.400 with no split (the next best, x≤3.5x\le3.5, leaves 1.7141.714).
  4. 4Round 1 — leaves. Left (customers 1–6): ∑r=−1.4\sum r=-1.4, ∑h=1.44\sum h=1.44, γL=−0.972\gamma_L=-0.972. Right (7–10): ∑r=+1.4\sum r=+1.4, ∑h=0.96\sum h=0.96, γR=+1.458\gamma_R=+1.458. Note the right leaf moves more: fewer points share the same total push.
  5. 5Round 1 — update. F←F+0.5γF\leftarrow F+0.5\gamma: left −0.405−0.486=−0.892-0.405-0.486=-0.892 (p=0.291p=0.291), right −0.405+0.729=0.324-0.405+0.729=0.324 (p=0.580p=0.580). Log-loss 0.5450.545; 8/10 correct (only customers 4 and 9 are wrong).
  6. 6Round 2. New residuals: −0.291-0.291 for customers 1–3, 5, 6; +0.709+0.709 for customer 4; +0.420+0.420 for 7, 8, 10; −0.580-0.580 for 9. Best split x≤3.5x\le3.5 (SSE 1.4441.444 vs 1.7911.791). Leaves: γL=−0.872/0.619=−1.410\gamma_L=-0.872/0.619=-1.410, γR=0.807/1.593=+0.506\gamma_R=0.807/1.593=+0.506. Log-loss 0.4820.482.
  7. 7Round 3. Best split x≤9.5x\le9.5 (SSE 1.3971.397 vs 1.5491.549), isolating customer 10. γL=−0.463/1.790=−0.259\gamma_L=-0.463/1.790=-0.259, γR=0.360/0.230=+1.562\gamma_R=0.360/0.230=+1.562 — a single point with small hh gets a large step. Log-loss 0.4560.456.
  8. 8Predict a new customer with 8 tickets: they fall right of 6.5, right of 3.5, left of 9.5, so F=−0.405+0.5(1.458+0.506−0.259)=0.4475F=-0.405+0.5(1.458+0.506-0.259)=0.4475 and p=σ(0.4475)=0.610p=\sigma(0.4475)=0.610 → predict churn. A customer with 4 tickets gets F=−0.768F=-0.768, p=0.317p=0.317 → stay.
Ticketsyp₀F₁p₁F₂p₂F₃p₃
1–300.400−0.8920.291−1.5970.169−1.7260.151
410.400−0.8920.291−0.6380.346−0.7680.317
5–600.400−0.8920.291−0.6380.346−0.7680.317
7–810.4000.3240.5800.5770.6400.4480.610
900.4000.3240.5800.5770.6400.4480.610
1010.4000.3240.5800.5770.6401.3580.795
Scores and probabilities after each round (ν = 0.5). The decision boundary sits at 6.5 tickets throughout; only the confidence changes.
▸ gradient boosting · log-loss · stumpsstep 1/20
learning rate ν
p(churn) = σ(F(x)) · ● churned ● stayed · shade = predicted class00.510.400.400.400.400.400.400.400.400.400.40ppseudo-residuals rᵢ = yᵢ − pᵢ appear in round 1-10112345678910training log-loss over rounds 0 … 100 (ν = 0.5)m = 0: 0.673
trees 0ν 0.5log-loss 0.673correct 6/10
  1. 1F₀ ← log(p̄ / (1 − p̄))
  2. 2for m = 1 … M:
  3. 3 pᵢ ← σ(F(xᵢ)); rᵢ ← yᵢ − pᵢ
  4. 4 split ← argmin Σ(rᵢ − r̄_leaf)²
  5. 5 γ_leaf ← Σrᵢ / Σpᵢ(1 − pᵢ)
  6. 6 F(x) ← F(x) + ν·γ_leaf(x)
  7. 7predict: churn if σ(F_M(x)) > ½

4 of 10 customers churned, so the best constant is the log-odds F₀ = log(0.4/0.6) = -0.405: every customer gets p = 0.400 (log-loss 0.673).

fig.The ten customers, boosted with stumps on log-loss. Top: σ(F) per cell, shaded by predicted class. Middle: pseudo-residuals yᵢ − pᵢ, the stump fit to them (mint) and its Newton leaf values γ. Bottom: squared error of every candidate split while a stump is fit, otherwise the training log-loss. With ν = 0.5 round 1 splits at x ≤ 6.5 with γ = −0.972 / +1.458 and 3 rounds reach log-loss 0.456 (8/10 correct); by round 20 all 10 are correct and by round 100 the loss is 0.008 with p = 0.992 at x = 8 — memorised noise. Switch ν: at 0.1, 100 rounds only reach 0.237; at 1.0, 5 rounds already fit every point.

The algorithm, step by step

GradientBoostingClassifier (binary log-loss)

  1. 1Initialise F0=log⁡yˉ1−yˉF_0=\log\frac{\bar y}{1-\bar y} for every point.
  2. 2For m=1,…,Mm=1,\dots,M:
  3. 3 Probabilities and gradients. pi=σ(Fm−1(xi))p_i=\sigma(F_{m-1}(x_i)), ri=yi−pir_i=y_i-p_i, hi=pi(1−pi)h_i=p_i(1-p_i). (With subsampling, draw the rows used this round.)
  4. 4 Fit the tree structure. Grow a regression tree of the chosen depth on (xi,ri)(x_i,r_i) using squared error (scikit-learn: Friedman MSE), respecting min samples per leaf.
  5. 5 Set leaf values. For each leaf jj: γj=∑i∈Rjri / ∑i∈Rjhi\gamma_j=\sum_{i\in R_j}r_i\,/\,\sum_{i\in R_j}h_i.
  6. 6 Update. Fm(x)=Fm−1(x)+ν γj(x)F_m(x)=F_{m-1}(x)+\nu\,\gamma_{j(x)}. Evaluate training and validation log-loss; stop if validation loss has not improved for kk rounds.
  7. 7Predict p(x)=σ(FM(x))p(x)=\sigma(F_M(x)); class 1 if p>12p>\tfrac12 (or a threshold tuned to your costs).

Complexity

Training
O(M⋅d⋅nlog⁡n)O(M\cdot d\cdot n\log n) with exact sorted splits per depth level; histogram methods bin each feature into ~256 buckets once and then cost O(M⋅d⋅n)O(M\cdot d\cdot n). Multiclass multiplies by KK.
Parallelism
rounds are inherently sequential; only the split search inside one tree can be parallelised (over features or histogram bins).
Prediction
O(M⋅depth)O(M\cdot\text{depth}) per input (×K for multiclass) — fast, but slower than a single tree; hundreds to thousands of trees is typical.
Memory
the nn-vector of scores FF, plus the trees: O(M⋅2depth)O(M\cdot 2^{\text{depth}}) nodes.

Tuning: ν, M and early stopping

Learning rate and number of trees trade against each other. The table boosts depth-2 trees on a noisy two-moons problem (300 training, 300 validation points) for 1,000 rounds and records the round with the lowest validation log-loss.

νBest MVal log-loss @ best MVal accuracy @ best MVal log-loss @ 1000Val accuracy @ 1000
1.030.3970.8502.2880.840
0.3110.3560.8471.6090.847
0.1360.3560.8530.7320.850
0.031060.3560.8470.4430.857
make_moons(600, noise 0.35), depth 2. Best M grows roughly like 1/ν; the best achievable loss barely changes, but running all 1,000 rounds is catastrophic for large ν. Accuracy hides it: about 85% throughout.

Practical recipe

1. Fix ν
0.05–0.1 for the final model; 0.3 for quick experiments.
2. Pick M by early stopping
set a large cap (1,000–10,000) and stop when validation log-loss stalls for 20–50 rounds. On the moons data with ν = 0.1 and n_iter_no_change=10, scikit-learn stopped at 34 trees.
3. Depth
3–6 levels (or 15–63 leaves) for most tabular data. On the moons, best loss at ν = 0.1: depth 1 → 0.355 (140 trees), depth 3 → 0.369 (28), depth 6 → 0.358 (17): deeper trees need far fewer rounds and overfit faster.
4. Subsample
0.5–0.8 of rows per tree; with 0.5 on the moons the best validation loss was 0.355 at 33 trees, with a gentler rise afterwards (0.751 at 1,000).
5. Leaf regularisation
min samples per leaf 20+ on large data; L2 λ = 1 in the libraries that have it.
6. Imbalanced classes
F0F_0 already absorbs the base rate; use class weights or tune the decision threshold rather than resampling.
ImplementationSplit searchLeaf valuesDistinctive features
sklearn GradientBoostingClassifierexact, squared error on rNewton step Σr / Σhreference implementation; slow on large n
sklearn HistGradientBoostingClassifierhistograms, gradient + Hessian gain−Σg / (Σh + λ)native missing values and categoricals, built-in early stopping
XGBoostsecond-order gain, exact or histogram−Σg / (Σh + λ)L1/L2 and γ split penalty, sparsity-aware splits
LightGBMhistograms, leaf-wise growth−Σg / (Σh + λ)GOSS row sampling, feature bundling; very fast
CatBoostsymmetric (oblivious) treesNewton or gradientordered boosting and target statistics for categoricals
Gradient-boosting implementations for classification. The algorithm above is the common core; see the XGBoost, LightGBM and CatBoost articles for their specifics.

In code

python
1import numpy as np
2
3def sigmoid(f):
4 return 1 / (1 + np.exp(-f))
5
6def build(X, r, h, depth, min_leaf=1):
7 """Least-squares tree on residuals r; Newton value sum(r)/sum(h) in each leaf."""
8 best = None
9 if depth > 0:
10 for j in range(X.shape[1]):
11 v = np.unique(X[:, j])
12 for t in (v[:-1] + v[1:]) / 2:
13 L = X[:, j] <= t
14 if L.sum() < min_leaf or (~L).sum() < min_leaf:
15 continue
16 sse = ((r[L] - r[L].mean()) ** 2).sum() + ((r[~L] - r[~L].mean()) ** 2).sum()
17 if best is None or sse < best[0]:
18 best = (sse, j, t)
19 if best is None:
20 return {"value": r.sum() / max(h.sum(), 1e-12)}
21 _, j, t = best
22 L = X[:, j] <= t
23 return {"j": j, "t": t,
24 "left": build(X[L], r[L], h[L], depth - 1, min_leaf),
25 "right": build(X[~L], r[~L], h[~L], depth - 1, min_leaf)}
26
27def tree_predict(node, X):
28 if "value" in node:
29 return np.full(len(X), node["value"])
30 L = X[:, node["j"]] <= node["t"]
31 out = np.empty(len(X))
32 out[L] = tree_predict(node["left"], X[L])
33 out[~L] = tree_predict(node["right"], X[~L])
34 return out
35
36def fit_gb(X, y, M=100, lr=0.1, depth=1):
37 F0 = np.log(y.mean() / (1 - y.mean())) # best constant: log-odds of the base rate
38 F, trees = np.full(len(y), F0), []
39 for _ in range(M):
40 p = sigmoid(F)
41 r = y - p # negative gradient of log-loss
42 h = p * (1 - p) # second derivative (Hessian)
43 tree = build(X, r, h, depth)
44 F += lr * tree_predict(tree, X)
45 trees.append(tree)
46 return F0, trees
47
48def predict_proba(F0, trees, X, lr=0.1):
49 return sigmoid(F0 + lr * sum(tree_predict(t, X) for t in trees))
50
51x = np.arange(1, 11, dtype=float).reshape(-1, 1) # support tickets
52y = np.array([0, 0, 0, 1, 0, 0, 1, 1, 0, 1], float) # churned?
53F0, trees = fit_gb(x, y, M=3, lr=0.5, depth=1)
54print("F0 =", round(F0, 4))
55for t in trees:
56 print(f"split x <= {t['t']}: left {t['left']['value']:.4f}, right {t['right']['value']:.4f}")
57print("p(train) =", predict_proba(F0, trees, x, lr=0.5).round(4))
58print("p(x=8) =", predict_proba(F0, trees, np.array([[8.0]]), lr=0.5).round(4))
From scratch with NumPy: log-loss gradient boosting with Newton leaves, run on the worked example. Prints F0 = -0.4055, the three splits (6.5, 3.5, 9.5) with leaf values -0.9722/1.4583, -1.4100/0.5064, -0.2589/1.5616, and p(x=8) = 0.61.
python
1import numpy as np
2from sklearn.ensemble import GradientBoostingClassifier
3from sklearn.metrics import log_loss
4
5x = np.arange(1, 11).reshape(-1, 1) # support tickets
6y = np.array([0, 0, 0, 1, 0, 0, 1, 1, 0, 1]) # churned?
7
8gb = GradientBoostingClassifier(n_estimators=3, learning_rate=0.5, max_depth=1).fit(x, y)
9for m, p in enumerate(gb.staged_predict_proba(x), start=1):
10 print(f"round {m}: log-loss {log_loss(y, p):.4f}")
11print("p(train) =", gb.predict_proba(x)[:, 1].round(4))
12print("p(x=8) =", gb.predict_proba([[8]])[:, 1].round(4))
13print("F(x=8) =", gb.decision_function([[8]]).round(4))
scikit-learn on the same data — identical probabilities. Prints round losses 0.5454, 0.4824, 0.4562; p(train) = 0.1511 ×3, 0.317 ×3, 0.61 ×3, 0.7954; p(x=8) = 0.61 and F(x=8) = 0.4475.
python
1from sklearn.ensemble import HistGradientBoostingClassifier
2
3gb = HistGradientBoostingClassifier(
4 learning_rate=0.1, max_iter=2000, max_leaf_nodes=31, l2_regularization=1.0,
5 early_stopping=True, validation_fraction=0.15, n_iter_no_change=30,
6 scoring="loss", # monitor log-loss, not accuracy
7).fit(X_train, y_train)
8print(gb.n_iter_) # trees actually kept
9proba = gb.predict_proba(X_test)[:, 1]
On real-sized data: histogram boosting with early stopping on log-loss.

Use when

  • +Tabular data where accuracy and well-ranked probabilities matter — gradient-boosted trees are the default winner on structured data.
  • +Mixed numeric and categorical features, missing values, unscaled and skewed inputs, non-linear interactions.
  • +You need a custom or non-standard objective: any differentiable loss with a gradient (and ideally a Hessian) plugs in.
  • +You can afford a validation set for early stopping and a modest tuning budget.

Avoid when

  • −Images, audio, raw text — neural networks learn much better representations.
  • −Very noisy labels or tiny datasets: later rounds chase mislabeled points, as customers 4 and 9 show above.
  • −You need extrapolation or smooth boundaries: the score is piecewise constant and flat beyond the training range.
  • −Training must be massively parallel or online: rounds are sequential, and the model must be refit to absorb new data.

Where it's used

Credit risk & fraud

Default-probability and transaction-fraud scoring on hundreds of tabular features, where calibrated probabilities feed directly into pricing and review queues.

Customer churn & conversion

Predicting which subscribers will cancel or which visitors will buy, from usage, billing and support history — the worked example in miniature.

Click-through & ranking

Ad CTR prediction and learning-to-rank (LambdaMART is gradient boosting on a ranking loss) in search engines.

Physics & medicine

Signal-vs-background event selection at particle colliders and risk scores from electronic health records.

Check yourself

A churner (y = 1) currently has predicted probability p = 0.3. What is its pseudo-residual for log-loss, and which way will the next tree push its score?

Check yourself

A leaf holds four points, all currently at p = 0.4, with labels 1, 1, 0, 1. What is its Newton leaf value γ?

Check yourself

Training data has 4 positives out of 10. What is the initial score F₀ of a log-loss gradient-boosting classifier?

Check yourself

With ν = 1 on the moons data, validation accuracy at 1,000 trees (0.840) is close to its value at the best round (0.850), yet validation log-loss is 2.288 instead of 0.397. What happened?

Remember

  • ✓The model is a log-odds score F=F0+ν∑mhmF=F_0+\nu\sum_m h_m, turned into a probability by the sigmoid; F0F_0 is the log-odds of the base rate.
  • ✓Each round fits a regression tree to the pseudo-residuals yi−piy_i-p_i — the negative gradient of log-loss — so boosting is gradient descent in function space.
  • ✓Leaf values are Newton steps γ=∑ri/∑pi(1−pi)\gamma=\sum r_i/\sum p_i(1-p_i), in log-odds units; tiny confident leaves need min-leaf or L2 limits.
  • ✓ν\nu and MM trade off (best M∝1/νM\propto1/\nu); choose MM by early stopping on validation log-loss, because training loss always goes to zero.
  • ✓Depth controls interaction order; subsampling and leaf regularisation add robustness. XGBoost, LightGBM and CatBoost are engineered versions of this same loop.
← back to ML Algorithms roadmap