# Gradient Boosting

> Add trees one at a time, each fit to the negative gradient of the current ensemble's loss.

Source: https://www.mlfoundry.in/learn/gradient-boosting  
Section: ML Algorithms › Supervised · Classification  
Level: core · 17 min read · updated 2026-10-07

**Gradient boosting (for classification):** builds a **log-odds score** as a sum of many small regression trees, $F_M(x)=F_0+\nu\sum_{m=1}^{M}h_m(x)$, and turns it into a probability with the sigmoid, $p(x)=\sigma(F_M(x))$. Each new tree $h_m$ is fit to the **negative gradient of the log-loss** of the model built so far — the pseudo-residuals $y_i-p_i$ — so every round takes one gradient-descent step in the space of functions.

- **Task:** Binary and multiclass classification (the same machinery handles regression and ranking with other losses)
- **Input → output:** $x$ → log-odds $F_M(x)$ → probability $\sigma(F_M(x))$ → class (threshold ½ or argmax of softmax)
- **Learns:** $M$ shallow **regression** trees (depth 1–8) whose leaves hold log-odds corrections $\gamma$
- **Training:** strictly sequential: $M$ tree fits, each about $O(d\,n\log n)$ (or $O(d\,n)$ with histogram binning)
- **Prediction:** $O(M\cdot\text{depth})$ comparisons, then one sigmoid
- **Hyperparameters:** learning rate $\nu$, number of trees $M$, tree depth / leaves, row subsample, min samples per leaf, L2 on leaf values
- **Assumes:** nothing distributional; a differentiable loss; features need no scaling (trees only look at order)

## The problem

A single decision tree has to choose between two failures. Shallow, it is too blunt to capture the real boundary (**high bias**); deep, it memorises noise (**high variance**). **Random forests** and **Extra Trees** fix the variance side: grow many deep, decorrelated trees *independently* and average them. But averaging cannot remove bias — if every tree misses the same structure, so does their mean.

Boosting attacks bias. Start with a model that is clearly too simple, and keep adding small trees **exactly where the current model is wrong**. **AdaBoost** did this first by re-weighting misclassified points, but its recipe is tied to one loss (the exponential loss) and to hard class votes. Gradient boosting generalises it: pick any differentiable loss — for classification, the **log-loss** that calibrated probabilities need — and let its gradient say what "wrong" means.

## The idea

Keep a running score $F(x)$ in **log-odds** units, starting from the base rate. For every training point, ask: *in which direction, and how hard, should my score move to lower the loss?* For log-loss the answer is simply $y_i-p_i$: a churner predicted at 0.3 wants $+0.7$; a stayer predicted at 0.3 wants $-0.3$. Those numbers are not labels any more — they are continuous targets — so we fit a small **regression** tree to them. The tree groups points that want similar pushes, each leaf decides one push size, and we add a fraction $\nu$ of it to the score. Repeat.

Two consequences follow. First, points that are already confidently right ($p\approx y$) have residuals near zero, so later trees focus on the hard, misclassified or uncertain points — the same intuition as AdaBoost, derived instead of designed. Second, because the score is in log-odds, many small additive corrections compose cleanly and the final sigmoid always lands in $(0,1)$.

**Vocabulary**

- **Weak learner:** a model only slightly better than a constant — here a shallow regression tree; a depth-1 tree is a **stump**
- **Raw score / margin $F(x)$:** the ensemble's output before the sigmoid, in log-odds: $F=\log\frac{p}{1-p}$
- **Initial model $F_0$:** the best constant score: the log-odds of the training base rate
- **Pseudo-residual $r_i$:** the negative gradient of the loss with respect to $F(x_i)$; for log-loss $r_i=y_i-p_i$
- **Hessian $h_i$:** the second derivative of the loss, $p_i(1-p_i)$; it measures how curved the loss is at that point
- **Leaf value $gamma$:** the constant a leaf adds to $F$; chosen by a Newton step, $\sum r/\sum h$ over the leaf
- **Shrinkage / learning rate $
u$:** the fraction of each tree actually added; small $\nu$ = small, cautious steps
- **Stage / round / iteration:** one cycle of compute residuals → fit tree → update; the model after $m$ rounds is $F_m$
- **Early stopping:** choosing $M$ as the round where validation loss is lowest

## The math

Labels are $y_i\in\{0,1\}$, the model outputs a score $F(x)$, and the predicted probability is its sigmoid. The loss for one point is the **binary cross-entropy** (log-loss):

$$
\ell(y,F)=-\big[y\log p+(1-y)\log(1-p)\big],\qquad p=\sigma(F)=\frac{1}{1+e^{-F}}
$$

- $y$: true label, 0 or 1
- $F$: the model's raw score (log-odds) for this point
- $p$: predicted probability of class 1
- $\sigma$: the logistic sigmoid

Training minimises $\mathcal L(F)=\sum_i \ell(y_i,F(x_i))$. Substituting $p=\sigma(F)$ gives $\ell=\log(1+e^{F})-yF$, whose derivatives with respect to the score are remarkably simple (use $\sigma'=\sigma(1-\sigma)$):

$$
g_i=\frac{\partial \ell}{\partial F(x_i)}=p_i-y_i,\qquad h_i=\frac{\partial^2 \ell}{\partial F(x_i)^2}=p_i(1-p_i)
$$

- $g_i$: gradient of the loss at point i
- $h_i$: second derivative (Hessian) at point i; largest at p = ½, tiny when p is near 0 or 1

**Gradient descent in function space.** Think of the vector of predictions $(F(x_1),\dots,F(x_n))$ as the parameters. Steepest descent would move each one by $-g_i=y_i-p_i$. But a raw vector of $n$ numbers says nothing about new inputs, so we **approximate the negative gradient with a tree**: fit a least-squares regression tree to the targets $r_i=y_i-p_i$. The tree's structure (its splits) defines regions $R_{1},\dots,R_{J}$ that generalise to unseen $x$.

$$
r_{im}=-\left[\frac{\partial\ell(y_i,F)}{\partial F}\right]_{F=F_{m-1}(x_i)}=y_i-p_i,\qquad \{R_{jm}\}=\text{tree fit to }(x_i,r_{im})\text{ by squared error}
$$

- $r_{im}$: pseudo-residual of point i in round m
- $R_{jm}$: the region covered by leaf j of tree m
- $F_{m-1}$: the model after m − 1 rounds

**Leaf values by a Newton step.** The tree's own leaf means would be gradient-step sizes in the wrong units (probability, not log-odds). Instead, for each leaf we choose the constant $\gamma$ that minimises the real loss over the points in it, using a second-order Taylor expansion around the current scores:

$$
\sum_{i\in R_j}\ell\big(y_i,F_{m-1}(x_i)+\gamma\big)\approx\sum_{i\in R_j}\Big[\ell_i+g_i\gamma+\tfrac12h_i\gamma^2\Big]\;\Longrightarrow\;\gamma_{jm}=-\frac{\sum_{i\in R_j}g_i}{\sum_{i\in R_j}h_i}=\frac{\sum_{i\in R_j}(y_i-p_i)}{\sum_{i\in R_j}p_i(1-p_i)}
$$

- $\gamma_{jm}$: value of leaf j in tree m, in log-odds units
- $\ell_i$: current loss of point i (a constant in the minimisation)

Setting the derivative $\sum g_i+\gamma\sum h_i$ to zero gives the formula. This is exactly what scikit-learn's `GradientBoostingClassifier` does (Friedman's *one Newton step*), and XGBoost and LightGBM go one step further by also using $g$ and $h$ to *choose the splits* (see the **XGBoost** article).

$$
F_m(x)=F_{m-1}(x)+\nu\sum_{j=1}^{J}\gamma_{jm}\,\mathbb 1[x\in R_{jm}],\qquad F_0=\log\frac{\bar y}{1-\bar y}
$$

- $\nu$: learning rate (shrinkage), 0 < ν ≤ 1
- $J$: number of leaves per tree (2 for a stump)
- $\bar y$: fraction of positives in the training set

$F_0$ is the best constant: minimising $\sum_i\ell(y_i,c)$ means $\sum_i(\sigma(c)-y_i)=0$, so $\sigma(c)=\bar y$ and $c$ is the log-odds of the base rate. **Multiclass** with $K$ classes uses the softmax $p_{ik}=e^{F_k(x_i)}/\sum_{k'}e^{F_{k'}(x_i)}$ and grows **$K$ trees per round**, one per class, each fit to $r_{ik}=y_{ik}-p_{ik}$ — so a 10-class problem with 300 rounds trains 3,000 trees.

**What each hyperparameter does to the math**

- **Learning rate ν:** scales every step. With $\nu=1$ each Newton step is taken in full and the model reaches the training data in a few rounds; with small $\nu$ it moves a fraction of the way, so later trees still get a say and the noise of any one tree is averaged out. Needed rounds scale roughly like $1/\nu$.
- **Number of trees M:** number of descent steps. The training loss decreases monotonically in $M$ (to 0 on separable data, by sending $|F|\to\infty$); validation loss is U-shaped. $M$ is the main regulariser — choose it by early stopping.
- **Depth / leaves J:** a depth-$d$ tree can express interactions between at most $d$ features; stumps give an **additive** model $F(x)=\sum_j f_j(x_j)$. More leaves = bigger steps in more directions per round.
- **Subsample:** fit each tree on a random fraction of rows (*stochastic gradient boosting*): noisier gradient estimates, decorrelated trees, less overfitting, faster.
- **Min samples / Hessian per leaf:** keeps $\sum h_i$ in the Newton denominator away from 0. Without it, a tiny leaf of already-confident points ($p\approx0$ or 1, $h\approx0$) produces an enormous $\gamma$.
- **L2 on leaves λ:** (XGBoost / LightGBM / HistGB) shrinks the leaf value to $\sum r_i/(\sum h_i+\lambda)$ — the same fix, made explicit.

## Worked example: who will churn?

Ten customers, one feature — how many support tickets they filed last quarter — and whether they churned. 4 of 10 churned. The labels are deliberately noisy: customer 4 churned with few tickets and customer 9 stayed with many. We boost **stumps** with learning rate $\nu=0.5$ for 3 rounds.

Training data. Highlighted: the four churners.

| Customer | Tickets x | Churned y |
| --- | --- | --- |
| 1 | 1 | 0 |
| 2 | 2 | 0 |
| 3 | 3 | 0 |
| 4 | 4 | 1 |
| 5 | 5 | 0 |
| 6 | 6 | 0 |
| 7 | 7 | 1 |
| 8 | 8 | 1 |
| 9 | 9 | 0 |
| 10 | 10 | 1 |

**Three rounds by hand (ν = 0.5)**

1. **Initialise.** $\bar y=0.4$, so $F_0=\log(0.4/0.6)=-0.405$ and every $p_i=0.4$. Training log-loss $0.673$; predicting "stay" for all gets 6/10 right.
2. **Round 1 — residuals.** $r_i=y_i-0.4$: churners get $+0.6$, stayers $-0.4$. Every $h_i=0.4\cdot0.6=0.24$.
3. **Round 1 — split.** Try the 9 midpoints; the squared error of a stump on the residuals is smallest at $x\le6.5$: $1.583$, down from $2.400$ with no split (the next best, $x\le3.5$, leaves $1.714$).
4. **Round 1 — leaves.** Left (customers 1–6): $\sum r=-1.4$, $\sum h=1.44$, $\gamma_L=-0.972$. Right (7–10): $\sum r=+1.4$, $\sum h=0.96$, $\gamma_R=+1.458$. Note the right leaf moves *more*: fewer points share the same total push.
5. **Round 1 — update.** $F\leftarrow F+0.5\gamma$: left $-0.405-0.486=-0.892$ ($p=0.291$), right $-0.405+0.729=0.324$ ($p=0.580$). Log-loss $0.545$; 8/10 correct (only customers 4 and 9 are wrong).
6. **Round 2.** New residuals: $-0.291$ for customers 1–3, 5, 6; $+0.709$ for customer 4; $+0.420$ for 7, 8, 10; $-0.580$ for 9. Best split $x\le3.5$ (SSE $1.444$ vs $1.791$). Leaves: $\gamma_L=-0.872/0.619=-1.410$, $\gamma_R=0.807/1.593=+0.506$. Log-loss $0.482$.
7. **Round 3.** Best split $x\le9.5$ (SSE $1.397$ vs $1.549$), isolating customer 10. $\gamma_L=-0.463/1.790=-0.259$, $\gamma_R=0.360/0.230=+1.562$ — a single point with small $h$ gets a large step. Log-loss $0.456$.
8. **Predict** a new customer with 8 tickets: they fall right of 6.5, right of 3.5, left of 9.5, so $F=-0.405+0.5(1.458+0.506-0.259)=0.4475$ and $p=\sigma(0.4475)=0.610$ → **predict churn**. A customer with 4 tickets gets $F=-0.768$, $p=0.317$ → stay.

Scores and probabilities after each round (ν = 0.5). The decision boundary sits at 6.5 tickets throughout; only the confidence changes.

| Tickets | y | p₀ | F₁ | p₁ | F₂ | p₂ | F₃ | p₃ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 1–3 | 0 | 0.400 | −0.892 | 0.291 | −1.597 | 0.169 | −1.726 | 0.151 |
| 4 | 1 | 0.400 | −0.892 | 0.291 | −0.638 | 0.346 | −0.768 | 0.317 |
| 5–6 | 0 | 0.400 | −0.892 | 0.291 | −0.638 | 0.346 | −0.768 | 0.317 |
| 7–8 | 1 | 0.400 | 0.324 | 0.580 | 0.577 | 0.640 | 0.448 | 0.610 |
| 9 | 0 | 0.400 | 0.324 | 0.580 | 0.577 | 0.640 | 0.448 | 0.610 |
| 10 | 1 | 0.400 | 0.324 | 0.580 | 0.577 | 0.640 | 1.358 | 0.795 |

> **The split is chosen on residuals, the step on the loss:** In round 1 the left leaf's mean residual is $-0.233$, but its Newton value is $-0.233/0.24=-0.972$: dividing by the curvature converts "how far off in probability" into "how far to move in log-odds". When $p$ is near ½ that factor is about 4; when $p$ is near 0 or 1 it becomes huge — which is why leaves holding only confident points need a minimum size or an L2 penalty.

> Figure: The ten customers, boosted with stumps on log-loss. Top: σ(F) per cell, shaded by predicted class. Middle: pseudo-residuals yᵢ − pᵢ, the stump fit to them (mint) and its Newton leaf values γ. Bottom: squared error of every candidate split while a stump is fit, otherwise the training log-loss. With ν = 0.5 round 1 splits at x ≤ 6.5 with γ = −0.972 / +1.458 and 3 rounds reach log-loss 0.456 (8/10 correct); by round 20 all 10 are correct and by round 100 the loss is 0.008 with p = 0.992 at x = 8 — memorised noise. Switch ν: at 0.1, 100 rounds only reach 0.237; at 1.0, 5 rounds already fit every point.

## The algorithm, step by step

**GradientBoostingClassifier (binary log-loss)**

1. **Initialise** $F_0=\log\frac{\bar y}{1-\bar y}$ for every point.
2. **For** $m=1,\dots,M$:
3.   **Probabilities and gradients.** $p_i=\sigma(F_{m-1}(x_i))$, $r_i=y_i-p_i$, $h_i=p_i(1-p_i)$. (With subsampling, draw the rows used this round.)
4.   **Fit the tree structure.** Grow a regression tree of the chosen depth on $(x_i,r_i)$ using squared error (scikit-learn: Friedman MSE), respecting min samples per leaf.
5.   **Set leaf values.** For each leaf $j$: $\gamma_j=\sum_{i\in R_j}r_i\,/\,\sum_{i\in R_j}h_i$.
6.   **Update.** $F_m(x)=F_{m-1}(x)+\nu\,\gamma_{j(x)}$. Evaluate training and validation log-loss; stop if validation loss has not improved for $k$ rounds.
7. **Predict** $p(x)=\sigma(F_M(x))$; class 1 if $p>\tfrac12$ (or a threshold tuned to your costs).

**Complexity**

- **Training:** $O(M\cdot d\cdot n\log n)$ with exact sorted splits per depth level; histogram methods bin each feature into ~256 buckets once and then cost $O(M\cdot d\cdot n)$. Multiclass multiplies by $K$.
- **Parallelism:** rounds are inherently sequential; only the split search inside one tree can be parallelised (over features or histogram bins).
- **Prediction:** $O(M\cdot\text{depth})$ per input (×K for multiclass) — fast, but slower than a single tree; hundreds to thousands of trees is typical.
- **Memory:** the $n$-vector of scores $F$, plus the trees: $O(M\cdot 2^{\text{depth}})$ nodes.

## Tuning: ν, M and early stopping

Learning rate and number of trees trade against each other. The table boosts depth-2 trees on a noisy two-moons problem (300 training, 300 validation points) for 1,000 rounds and records the round with the lowest **validation log-loss**.

make_moons(600, noise 0.35), depth 2. Best M grows roughly like 1/ν; the best achievable loss barely changes, but running all 1,000 rounds is catastrophic for large ν. Accuracy hides it: about 85% throughout.

| ν | Best M | Val log-loss @ best M | Val accuracy @ best M | Val log-loss @ 1000 | Val accuracy @ 1000 |
| --- | --- | --- | --- | --- | --- |
| 1.0 | 3 | 0.397 | 0.850 | 2.288 | 0.840 |
| 0.3 | 11 | 0.356 | 0.847 | 1.609 | 0.847 |
| 0.1 | 36 | 0.356 | 0.853 | 0.732 | 0.850 |
| 0.03 | 106 | 0.356 | 0.847 | 0.443 | 0.857 |

**Practical recipe**

- **1. Fix ν:** 0.05–0.1 for the final model; 0.3 for quick experiments.
- **2. Pick M by early stopping:** set a large cap (1,000–10,000) and stop when validation log-loss stalls for 20–50 rounds. On the moons data with ν = 0.1 and `n_iter_no_change=10`, scikit-learn stopped at 34 trees.
- **3. Depth:** 3–6 levels (or 15–63 leaves) for most tabular data. On the moons, best loss at ν = 0.1: depth 1 → 0.355 (140 trees), depth 3 → 0.369 (28), depth 6 → 0.358 (17): deeper trees need far fewer rounds and overfit faster.
- **4. Subsample:** 0.5–0.8 of rows per tree; with 0.5 on the moons the best validation loss was 0.355 at 33 trees, with a gentler rise afterwards (0.751 at 1,000).
- **5. Leaf regularisation:** min samples per leaf 20+ on large data; L2 λ = 1 in the libraries that have it.
- **6. Imbalanced classes:** $F_0$ already absorbs the base rate; use class weights or tune the decision threshold rather than resampling.

Gradient-boosting implementations for classification. The algorithm above is the common core; see the XGBoost, LightGBM and CatBoost articles for their specifics.

| Implementation | Split search | Leaf values | Distinctive features |
| --- | --- | --- | --- |
| sklearn GradientBoostingClassifier | exact, squared error on r | Newton step Σr / Σh | reference implementation; slow on large n |
| sklearn HistGradientBoostingClassifier | histograms, gradient + Hessian gain | −Σg / (Σh + λ) | native missing values and categoricals, built-in early stopping |
| XGBoost | second-order gain, exact or histogram | −Σg / (Σh + λ) | L1/L2 and γ split penalty, sparsity-aware splits |
| LightGBM | histograms, leaf-wise growth | −Σg / (Σh + λ) | GOSS row sampling, feature bundling; very fast |
| CatBoost | symmetric (oblivious) trees | Newton or gradient | ordered boosting and target statistics for categoricals |

## In code

```python
import numpy as np

def sigmoid(f):
    return 1 / (1 + np.exp(-f))

def build(X, r, h, depth, min_leaf=1):
    """Least-squares tree on residuals r; Newton value sum(r)/sum(h) in each leaf."""
    best = None
    if depth > 0:
        for j in range(X.shape[1]):
            v = np.unique(X[:, j])
            for t in (v[:-1] + v[1:]) / 2:
                L = X[:, j] <= t
                if L.sum() < min_leaf or (~L).sum() < min_leaf:
                    continue
                sse = ((r[L] - r[L].mean()) ** 2).sum() + ((r[~L] - r[~L].mean()) ** 2).sum()
                if best is None or sse < best[0]:
                    best = (sse, j, t)
    if best is None:
        return {"value": r.sum() / max(h.sum(), 1e-12)}
    _, j, t = best
    L = X[:, j] <= t
    return {"j": j, "t": t,
            "left": build(X[L], r[L], h[L], depth - 1, min_leaf),
            "right": build(X[~L], r[~L], h[~L], depth - 1, min_leaf)}

def tree_predict(node, X):
    if "value" in node:
        return np.full(len(X), node["value"])
    L = X[:, node["j"]] <= node["t"]
    out = np.empty(len(X))
    out[L] = tree_predict(node["left"], X[L])
    out[~L] = tree_predict(node["right"], X[~L])
    return out

def fit_gb(X, y, M=100, lr=0.1, depth=1):
    F0 = np.log(y.mean() / (1 - y.mean()))     # best constant: log-odds of the base rate
    F, trees = np.full(len(y), F0), []
    for _ in range(M):
        p = sigmoid(F)
        r = y - p                              # negative gradient of log-loss
        h = p * (1 - p)                        # second derivative (Hessian)
        tree = build(X, r, h, depth)
        F += lr * tree_predict(tree, X)
        trees.append(tree)
    return F0, trees

def predict_proba(F0, trees, X, lr=0.1):
    return sigmoid(F0 + lr * sum(tree_predict(t, X) for t in trees))

x = np.arange(1, 11, dtype=float).reshape(-1, 1)    # support tickets
y = np.array([0, 0, 0, 1, 0, 0, 1, 1, 0, 1], float) # churned?
F0, trees = fit_gb(x, y, M=3, lr=0.5, depth=1)
print("F0 =", round(F0, 4))
for t in trees:
    print(f"split x <= {t['t']}: left {t['left']['value']:.4f}, right {t['right']['value']:.4f}")
print("p(train) =", predict_proba(F0, trees, x, lr=0.5).round(4))
print("p(x=8)   =", predict_proba(F0, trees, np.array([[8.0]]), lr=0.5).round(4))
```

From scratch with NumPy: log-loss gradient boosting with Newton leaves, run on the worked example. Prints F0 = -0.4055, the three splits (6.5, 3.5, 9.5) with leaf values -0.9722/1.4583, -1.4100/0.5064, -0.2589/1.5616, and p(x=8) = 0.61.

```python
import numpy as np
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.metrics import log_loss

x = np.arange(1, 11).reshape(-1, 1)            # support tickets
y = np.array([0, 0, 0, 1, 0, 0, 1, 1, 0, 1])   # churned?

gb = GradientBoostingClassifier(n_estimators=3, learning_rate=0.5, max_depth=1).fit(x, y)
for m, p in enumerate(gb.staged_predict_proba(x), start=1):
    print(f"round {m}: log-loss {log_loss(y, p):.4f}")
print("p(train) =", gb.predict_proba(x)[:, 1].round(4))
print("p(x=8)   =", gb.predict_proba([[8]])[:, 1].round(4))
print("F(x=8)   =", gb.decision_function([[8]]).round(4))
```

scikit-learn on the same data — identical probabilities. Prints round losses 0.5454, 0.4824, 0.4562; p(train) = 0.1511 ×3, 0.317 ×3, 0.61 ×3, 0.7954; p(x=8) = 0.61 and F(x=8) = 0.4475.

```python
from sklearn.ensemble import HistGradientBoostingClassifier

gb = HistGradientBoostingClassifier(
    learning_rate=0.1, max_iter=2000, max_leaf_nodes=31, l2_regularization=1.0,
    early_stopping=True, validation_fraction=0.15, n_iter_no_change=30,
    scoring="loss",                      # monitor log-loss, not accuracy
).fit(X_train, y_train)
print(gb.n_iter_)                        # trees actually kept
proba = gb.predict_proba(X_test)[:, 1]
```

On real-sized data: histogram boosting with early stopping on log-loss.

**Strengths**

- Tabular data where accuracy and well-ranked probabilities matter — gradient-boosted trees are the default winner on structured data.
- Mixed numeric and categorical features, missing values, unscaled and skewed inputs, non-linear interactions.
- You need a custom or non-standard objective: any differentiable loss with a gradient (and ideally a Hessian) plugs in.
- You can afford a validation set for early stopping and a modest tuning budget.

**Limitations**

- Images, audio, raw text — neural networks learn much better representations.
- Very noisy labels or tiny datasets: later rounds chase mislabeled points, as customers 4 and 9 show above.
- You need extrapolation or smooth boundaries: the score is piecewise constant and flat beyond the training range.
- Training must be massively parallel or online: rounds are sequential, and the model must be refit to absorb new data.

- **Credit risk & fraud:** Default-probability and transaction-fraud scoring on hundreds of tabular features, where calibrated probabilities feed directly into pricing and review queues.
- **Customer churn & conversion:** Predicting which subscribers will cancel or which visitors will buy, from usage, billing and support history — the worked example in miniature.
- **Click-through & ranking:** Ad CTR prediction and learning-to-rank (LambdaMART is gradient boosting on a ranking loss) in search engines.
- **Physics & medicine:** Signal-vs-background event selection at particle colliders and risk scores from electronic health records.

> **Watch log-loss, not accuracy:** In the tuning table, validation accuracy stays near 85% whether you stop at the best round or run all 1,000 — but log-loss rises from 0.356 to 2.288 at ν = 1. The overfit model is just as often right and **wildly overconfident** when wrong. Early-stop and compare models on log-loss (or Brier score) whenever the probabilities are used.

> **Training loss always reaches zero:** With enough rounds the ensemble carves a private region around every noisy point: in the figure at ν = 0.5, customers 4 and 9 get their own cells by round 20 and $p(8)$ climbs to 0.992 by round 100. The number of trees is a regulariser; choose it on held-out data and report that iteration, not the cap you allowed.

> **ν and M are coupled:** Halving $\nu$ roughly doubles the trees you need (best M: 3, 11, 36, 106 for ν = 1, 0.3, 0.1, 0.03). Comparing learning rates at a fixed $M$ is meaningless — always let early stopping pick $M$ for each $\nu$.

> **Relation to AdaBoost:** Run gradient boosting with the exponential loss $e^{-\tilde yF}$ ($\tilde y=\pm1$) and you recover AdaBoost's reweighting: the gradient weights are exactly AdaBoost's sample weights. Log-loss grows only linearly for badly misclassified points instead of exponentially, which makes gradient boosting much more robust to label noise. Details in **AdaBoost**.

**Question:** A churner (y = 1) currently has predicted probability p = 0.3. What is its pseudo-residual for log-loss, and which way will the next tree push its score?

- A. −0.3, down
- B. +0.7, up
- C. +0.21, up
- D. 0.7 × 0.3 = 0.21, down

**Answer:** B. +0.7, up — The negative gradient of log-loss with respect to the score is $y-p=1-0.3=0.7$, so the next tree tries to raise $F$ for this point. 0.21 is $p(1-p)$, the Hessian, which sets the step size, not the direction.

**Question:** A leaf holds four points, all currently at p = 0.4, with labels 1, 1, 0, 1. What is its Newton leaf value γ?

- A. 0.35
- B. 0.6
- C. 1.458
- D. −0.972

**Answer:** C. 1.458 — $\sum r=0.6+0.6-0.4+0.6=1.4$ and $\sum h=4\times0.4\times0.6=0.96$, so $\gamma=1.4/0.96=1.458$ log-odds — round 1's right leaf in the worked example. 0.35 is the mean residual, the leaf value a plain regression tree would store.

**Question:** Training data has 4 positives out of 10. What is the initial score F₀ of a log-loss gradient-boosting classifier?

- A. 0.4
- B. 0
- C. log(0.4/0.6) ≈ −0.405
- D. log(0.4) ≈ −0.916

**Answer:** C. log(0.4/0.6) ≈ −0.405 — The best constant score satisfies $\sigma(F_0)=\bar y=0.4$, i.e. $F_0$ is the log-odds of the base rate, $\log(0.4/0.6)=-0.405$.

**Question:** With ν = 1 on the moons data, validation accuracy at 1,000 trees (0.840) is close to its value at the best round (0.850), yet validation log-loss is 2.288 instead of 0.397. What happened?

- A. The model underfits
- B. The probabilities became extremely overconfident on the points it gets wrong
- C. Log-loss is not defined for boosting
- D. The learning rate was too small

**Answer:** B. The probabilities became extremely overconfident on the points it gets wrong — Extra rounds keep pushing |F| up to fit every training point. The predicted class rarely changes, but wrong predictions now come with p near 0 or 1, and log-loss punishes those heavily. Early stopping on log-loss prevents this.

**Key points**

- The model is a log-odds score $F=F_0+\nu\sum_m h_m$, turned into a probability by the sigmoid; $F_0$ is the log-odds of the base rate.
- Each round fits a **regression** tree to the pseudo-residuals $y_i-p_i$ — the negative gradient of log-loss — so boosting is gradient descent in function space.
- Leaf values are Newton steps $\gamma=\sum r_i/\sum p_i(1-p_i)$, in log-odds units; tiny confident leaves need min-leaf or L2 limits.
- $\nu$ and $M$ trade off (best $M\propto1/\nu$); choose $M$ by early stopping on validation **log-loss**, because training loss always goes to zero.
- Depth controls interaction order; subsampling and leaf regularisation add robustness. XGBoost, LightGBM and CatBoost are engineered versions of this same loop.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
