# XGBoost regression

> Gradient boosting with second-order (Newton) steps and regularized leaf weights w* = −G / (H + λ).

Source: https://www.mlfoundry.in/learn/xgboost-regression  
Section: ML Algorithms › Supervised · Regression  
Level: advanced · 17 min read · updated 2026-10-07

**XGBoost regression:** predicts a number as a constant plus a sum of many small regression trees, $\hat y = F_0 + \eta\sum_m f_m(x)$. Each new tree is grown on the **gradients** $g_i$ and **hessians** $h_i$ of the loss at the current predictions, its splits are scored by a regularized gain, and each leaf outputs the closed-form weight $w^*=-G/(H+\lambda)$ — a Newton step on the loss, shrunk toward zero.

- **Task:** Regression (squared error by default; also absolute, Huber, quantile, Poisson, Gamma, Tweedie losses)
- **Input → output:** feature vector $x$ → real number $\hat y = F_0 + \eta\sum_{m=1}^{M} f_m(x)$
- **Learns:** $M$ trees: a split (feature, threshold, default direction for missing values) per internal node and a weight $w_j$ per leaf
- **Training:** exact greedy: $O(M\cdot d\cdot n\log n)$; histogram mode: $O(M\cdot d\cdot n)$ plus binning — usually seconds for $10^5$–$10^6$ rows
- **Prediction:** $O(M\cdot\text{depth})$ comparisons
- **Key hyperparameters:** `n_estimators`, `learning_rate` (η), `max_depth`, `reg_lambda` (λ), `gamma` (γ), `min_child_weight`, `subsample`, `colsample_bytree`
- **Assumes:** a twice-differentiable loss (or a special leaf rule for absolute/quantile loss); no distributional assumptions on features, no scaling needed; axis-aligned, piecewise-constant fit

## The problem

Plain gradient boosting (see **Gradient boosting regression**) fits each new tree to the residuals by least squares and adds a shrunken copy. It works, but it has three loose ends. (1) The tree is grown with one criterion (variance reduction of the residuals) and its leaves are then filled with a value from another — the two only agree for squared error. (2) Nothing in the objective penalizes a tree for being big or its leaf values for being large; complexity is controlled only from the outside, by depth limits and early stopping. (3) Leaves with very few points get extreme values: a leaf with one outlier predicts the whole outlier.

XGBoost (Chen & Guestrin, 2016) closes all three by writing down **one regularized objective** and deriving everything — which split to make, whether to make it, and what each leaf predicts — from a second-order Taylor expansion of that objective. The same derivation then works unchanged for any twice-differentiable loss.

## The idea

Treat every training row as a little quadratic bowl: at the current prediction it tells you the slope $g_i$ (which way to move) and the curvature $h_i$ (how sure that step is). A leaf groups some rows, so their bowls add up to one bowl with slope $G=\sum g_i$ and curvature $H=\sum h_i$. The best constant for that leaf is the bottom of the bowl, $-G/H$ — a **Newton step** — except that XGBoost adds $\lambda$ to the curvature, which pulls the step toward 0. A split is worth making if the two smaller bowls have lower bottoms than the merged one by more than the price $\gamma$ of an extra leaf.

For squared error the bowls are exact: $g_i = \hat y_i - y_i$ is minus the residual and $h_i = 1$ for every row. So in regression, $H$ is just the number of rows in a leaf, and $-G/(H+\lambda)$ is the **sum of residuals divided by (count + λ)** — a mean residual shrunk as if $\lambda$ extra rows with residual 0 had joined the leaf.

**Vocabulary**

- **Base score F₀:** the starting prediction for every row; for squared error XGBoost sets it to the mean of $y$ (`base_score` overrides it)
- **Gradient gᵢ:** $\partial l(y_i,\hat y_i)/\partial \hat y_i$ at the current prediction. Squared error: $g_i=\hat y_i-y_i$ (negative residual)
- **Hessian hᵢ:** $\partial^2 l/\partial \hat y_i^2$. Squared error: $h_i=1$; it is also called the row's **cover** or weight
- **G, H:** sums of $g_i$ and $h_i$ over the rows in a node; $H$ = row count for squared error
- **Leaf weight w:** the value a leaf adds (before shrinkage); optimum $w^*=-G/(H+\lambda)$
- **Gain:** how much a split lowers the regularized objective; must exceed γ to survive pruning
- **Shrinkage η:** `learning_rate`: every tree's output is multiplied by η before it is added
- **Cover:** $H$ of a node; `min_child_weight` is a lower bound on it

## The math

$$
\hat y_i^{(m)} = \hat y_i^{(m-1)} + \eta\, f_m(x_i),\qquad f_m(x)=w_{q(x)}
$$

- $\hat y_i^{(m)}$: prediction for row $i$ after $m$ trees; $\hat y_i^{(0)}=F_0$
- $f_m$: the $m$-th regression tree
- $q(x)$: the leaf that $x$ falls into, $q(x)\in\{1,\dots,T\}$
- $w_j$: weight (output) of leaf $j$
- $\eta$: learning rate / shrinkage, 0 < η ≤ 1 (default 0.3)

$$
\mathcal{L}^{(m)}=\sum_{i=1}^{n} l\big(y_i,\ \hat y_i^{(m-1)}+f_m(x_i)\big)+\gamma T+\tfrac12\lambda\sum_{j=1}^{T}w_j^2
$$

- $l$: the loss; for regression by default $l(y,\hat y)=\tfrac12(y-\hat y)^2$
- $T$: number of leaves in the new tree
- $\gamma$: price per leaf (`gamma` / `min_split_loss`)
- $\lambda$: L2 penalty on leaf weights (`reg_lambda`, default 1)

Only the new tree $f_m$ is unknown. Expand the loss to second order around the current prediction $\hat y_i^{(m-1)}$:

$$
l\big(y_i,\hat y_i+f\big)\approx l(y_i,\hat y_i)+g_i f+\tfrac12 h_i f^2,\qquad g_i=\frac{\partial l}{\partial \hat y_i},\ \ h_i=\frac{\partial^2 l}{\partial \hat y_i^2}
$$

- $g_i$: gradient; squared error: $g_i=\hat y_i-y_i$
- $h_i$: hessian; squared error: $h_i=1$ (so this expansion is exact, not an approximation)

Drop the constant $l(y_i,\hat y_i)$ and group rows by leaf: every row in leaf $j$ receives the same $f=w_j$. With $G_j=\sum_{i\in j} g_i$ and $H_j=\sum_{i\in j} h_i$ the objective separates into one independent quadratic per leaf:

$$
\tilde{\mathcal{L}}=\sum_{j=1}^{T}\Big[G_j w_j+\tfrac12\big(H_j+\lambda\big)w_j^2\Big]+\gamma T
$$

Set $\partial\tilde{\mathcal L}/\partial w_j = G_j + (H_j+\lambda)w_j = 0$:

$$
w_j^{*}=-\frac{G_j}{H_j+\lambda}\qquad\Longrightarrow\qquad \tilde{\mathcal{L}}^{*}=-\frac12\sum_{j=1}^{T}\frac{G_j^2}{H_j+\lambda}+\gamma T
$$

- $w_j^*$: optimal leaf weight: a Newton step −G/H with λ added to the curvature
- $\tilde{\mathcal L}^*$: the best objective this tree structure can reach — a score for the structure

For squared error, $-G_j=\sum_{i\in j}(y_i-\hat y_i)$ is the residual sum and $H_j=n_j$, so $w_j^*=\frac{n_j}{n_j+\lambda}\,\bar r_j$: the mean residual $\bar r_j$, shrunk by the factor $n_j/(n_j+\lambda)$. With $\lambda=0$ it is exactly the least-squares leaf of plain gradient boosting; with $\lambda=1$ a 1-row leaf moves only half-way to its residual while a 100-row leaf moves 99% of the way.

Now score a split of node $S$ into $L$ and $R$. The structure score before is $-\tfrac12 G^2/(H+\lambda)+\gamma$; after, $-\tfrac12\big[G_L^2/(H_L+\lambda)+G_R^2/(H_R+\lambda)\big]+2\gamma$. The decrease is the **gain**:

$$
\text{Gain}=\frac12\left[\frac{G_L^2}{H_L+\lambda}+\frac{G_R^2}{H_R+\lambda}-\frac{(G_L+G_R)^2}{H_L+H_R+\lambda}\right]-\gamma
$$

- $G_L,H_L$: gradient and hessian sums of the rows sent left
- $G_R,H_R$: the same for the rows sent right
- $\gamma$: the cost of the one extra leaf the split creates

> **What the library actually reports:** XGBoost's code computes the bracket **without** the ½ — that is the "gain" printed in a tree dump — and prunes a split when that bracket is below $\gamma$. We checked: on the worked example below, a split with bracket 10.75 survives `gamma=6` (the paper's ½·10.75 = 5.375 would not). This article reports gains the way the library does, so the numbers match `get_dump()`.

**What each hyperparameter does to the equations**

- **λ (reg_lambda):** adds to every $H$: leaf weights shrink by $n_j/(n_j+\lambda)$, and the gain of splits that isolate few rows drops. Unlike variance reduction, a gain can become **negative** — the split is then never made.
- **γ (gamma):** subtracted from every gain: a split must lower the (un-halved) objective bracket by more than $\gamma$ or it is pruned. Applied bottom-up after the tree is grown, so a weak split above a strong one is kept.
- **η (learning_rate):** multiplies $w^*$ before adding: $\hat y\leftarrow\hat y+\eta w^*$. Smaller η needs proportionally more trees but averages over more of them, which usually generalizes better.
- **min_child_weight:** a split is only allowed if $H_L$ and $H_R$ are both at least this. Since $h_i=1$ for squared error, it is simply a **minimum number of rows per leaf**.
- **alpha (reg_alpha):** L1 penalty $\alpha\sum|w_j|$: the leaf rule becomes soft-thresholding, $w^*=-\operatorname{sign}(G)\max(|G|-\alpha,0)/(H+\lambda)$, so leaves with small $|G|$ output exactly 0.
- **max_delta_step:** caps $|w^*|$; rarely needed in regression, useful when $H$ can be tiny (Poisson, Tweedie).

## Worked example: rent from flat size

Eight flats, one feature (size in m²), target = monthly rent in units of 100 €. Settings: squared error, `max_depth=2`, $\eta=0.3$, $\lambda=1$, $\gamma=4$, two rounds, exact split search. The base score is the mean rent, $F_0=104/8=13$, so the first gradients are $g_i=13-y_i$.

Training data with the round-1 gradients g = F₀ − y (h = 1 for every row). The four highlighted rows end up on the right of the root split.

| Flat | Size x (m²) | Rent y | F₀ | g = F₀ − y |
| --- | --- | --- | --- | --- |
| 1 | 30 | 5 | 13 | 8 |
| 2 | 40 | 7 | 13 | 6 |
| 3 | 50 | 8 | 13 | 5 |
| 4 | 60 | 12 | 13 | 1 |
| 5 | 70 | 15 | 13 | −2 |
| 6 | 80 | 16 | 13 | −3 |
| 7 | 90 | 20 | 13 | −7 |
| 8 | 100 | 21 | 13 | −8 |

Round 1, root: every midpoint threshold. $G=0$, $H=8$, so the parent term $G^2/(H+\lambda)$ is 0 and the gain is $G_L^2/(H_L+1)+G_R^2/(H_R+1)$.

| Split | G_L / H_L | G_R / H_R | Gain |
| --- | --- | --- | --- |
| x < 35 | 8 / 1 | −8 / 7 | 40.00 |
| x < 45 | 14 / 2 | −14 / 6 | 93.33 |
| x < 55 | 19 / 3 | −19 / 5 | 150.42 |
| **x < 65** | **20 / 4** | **−20 / 4** | **160.00** |
| x < 75 | 18 / 5 | −18 / 3 | 135.00 |
| x < 85 | 15 / 6 | −15 / 2 | 107.14 |
| x < 95 | 8 / 7 | −8 / 1 | 40.00 |

**Round 1 by hand**

1. **Root.** Best split $x<65$: $\tfrac{20^2}{4+1}+\tfrac{(-20)^2}{4+1}-0 = 80+80 = 160$.
2. **Left child** (flats 1–4, $G=20$, $H=4$, parent term $400/5=80$). Candidates: $x<35$: $\tfrac{64}{2}+\tfrac{144}{4}-80=-12$; $x<45$: $\tfrac{196}{3}+\tfrac{36}{3}-80=-2.67$; $x<55$: $\tfrac{361}{4}+\tfrac{1}{2}-80=10.75$. Best: $x<55$ with gain 10.75. Note the **negative** gains — $\lambda$ makes splitting off small groups unattractive.
3. **Right child** (flats 5–8, $G=-20$, $H=4$). $x<75$: $\tfrac{4}{2}+\tfrac{324}{4}-80=3$; $x<85$: $\tfrac{25}{3}+\tfrac{225}{3}-80=3.33$; $x<95$: $-12$. Best: $x<85$ with gain 3.33.
4. **Prune with γ = 4.** Left split: $10.75>4$, kept. Right split: $3.33<4$, removed — the right child becomes a single leaf. Tree 1 has 3 leaves.
5. **Leaf weights** $w^*=-G/(H+1)$: flats 1–3: $-19/4=-4.75$; flat 4: $-1/2=-0.5$; flats 5–8: $20/5=4$. (With $\lambda=0$ they would be the mean residuals $-6.33$, $-1$, $+5$.)
6. **Shrink and add.** $\eta w^*$: $-1.425$, $-0.15$, $+1.2$. New predictions $F_1$: 11.575 (flats 1–3), 12.85 (flat 4), 14.2 (flats 5–8). Training RMSE falls from 5.612 to 4.492.

Round 2 starts from F₁. The new gradients are smaller but keep their sign pattern; their sum is G = 0.375.

| Flat | x | y | F₁ | g = F₁ − y | Tree-2 leaf η·w | F₂ |
| --- | --- | --- | --- | --- | --- | --- |
| 1 | 30 | 5 | 11.575 | 6.575 | −1.104 | 10.471 |
| 2 | 40 | 7 | 11.575 | 4.575 | −1.104 | 10.471 |
| 3 | 50 | 8 | 11.575 | 3.575 | −1.104 | 10.471 |
| 4 | 60 | 12 | 12.850 | 0.850 | −0.128 | 12.723 |
| 5 | 70 | 15 | 14.200 | −0.800 | +0.260 | 14.460 |
| 6 | 80 | 16 | 14.200 | −1.800 | +0.260 | 14.460 |
| 7 | 90 | 20 | 14.200 | −5.800 | +1.260 | 15.460 |
| 8 | 100 | 21 | 14.200 | −6.800 | +1.260 | 15.460 |

**Round 2 and a prediction**

1. **Tree 2** picks the same root $x<65$ (gain 94.71) and the same left split $x<55$ (gain 6.05).
2. On the right, $x<85$ now has gain **8.97** ($G_L=-2.6$, $G_R=-12.6$, two rows each) — above $\gamma=4$, so this time it survives: the remaining error is concentrated in flats 7–8.
3. Leaves: $-14.725/4=-3.681$, $-0.85/2=-0.425$, $2.6/3=0.867$, $12.6/3=4.2$; times $\eta$: $-1.104$, $-0.1275$, $+0.26$, $+1.26$. Training RMSE 4.492 → 3.581.
4. **Predict a new 75 m² flat.** Tree 1: $75\ge65$ → $+1.2$. Tree 2: $75\ge65$, $75<85$ → $+0.26$. So $\hat y = 13 + 1.2 + 0.26 = 14.46$, i.e. about 1,446 € per month — exactly what `XGBRegressor.predict` returns below.

> Figure: The worked example, live. Top: the eight flats, the model before (dashed, F₀ = 13) and after (orange) the current round, and each flat's gradient g = F − y as a stick coloured by the leaf it falls in. Bottom: the tree XGBoost grows on those gradients, with G, H, gain and w = −G/(H+λ) in every node. At the defaults (λ = 1, γ = 4, η = 0.3, depth 2) round 1 splits at x < 65 (gain 160) and x < 55 (gain 10.75), and the right split x < 85 (gain 3.33) is drawn as pruned; step to round 2 and it survives with gain 8.97, and F₂(75 m²) = 14.460. Drag λ up to watch every leaf shrink and small-group gains turn negative, γ up to prune, η to change the step size — or drag any flat's rent.

> **λ acts like phantom rows with zero residual:** $w^*=\frac{\sum r_i}{n+\lambda}$ is what you would get from a plain mean if $\lambda$ extra rows with residual 0 joined every leaf. That is why λ hurts small leaves far more than large ones, and why the gains of splits that peel off one or two rows go negative in the example. In the figure, raise λ to 10: the root's right leaf drops from $20/5=4$ to $20/14=1.43$.

## The algorithm, step by step

**XGBoost regression training**

1. **Initialize** $\hat y_i=F_0$ (the mean of $y$ for squared error).
2. **For** $m=1\dots M$: compute $g_i=\partial l/\partial\hat y_i$ and $h_i=\partial^2 l/\partial\hat y_i^2$ for every row (squared error: $g_i=\hat y_i-y_i$, $h_i=1$). Optionally subsample rows (`subsample`) and columns (`colsample_bytree`).
3. **Grow a tree** depth-first to `max_depth`: at each node, for every feature, scan the rows in sorted order (or the histogram bins), keep running sums $G_L,H_L$, and evaluate the gain for every candidate threshold with $H_L,H_R\ge$ `min_child_weight`. Rows with a missing value are tried on both sides; the better side becomes the node's default direction.
4. **Prune** bottom-up: any split whose children are leaves and whose gain is below $\gamma$ is collapsed into a leaf.
5. **Set leaves** $w_j=-G_j/(H_j+\lambda)$ and **update** $\hat y_i\leftarrow\hat y_i+\eta\,w_{q(x_i)}$.
6. **Stop** after $M$ trees or when the validation metric (RMSE, MAE…) has not improved for `early_stopping_rounds` rounds. Predict with $F_0+etasum_m f_m(x)$.

**Cost**

- **Exact greedy:** sort once per feature, then each level is one linear scan: about $O(d\,n)$ per level, $O(M\cdot \text{depth}\cdot d\,n)$ overall after an $O(d\,n\log n)$ pre-sort
- **Histogram (tree_method="hist", the default):** bucket each feature into at most `max_bin` (256) bins once; a node then costs $O(d\cdot\text{bins})$ after accumulating $G,H$ per bin. Children's histograms come from parent minus sibling
- **Memory:** $O(n\,d)$ for the (binned) data plus $O(n)$ for $g,h$ and predictions
- **Prediction:** $O(M\cdot\text{depth})$ per row; trees are independent, so it parallelizes trivially

## Regression objectives

Every loss only has to supply $g_i$ and $h_i$; the tree-growing code is identical. Here $r=y-\hat y$; pseudo-Huber loss is $\delta^2(s-1)$ with $s=\sqrt{1+(r/\delta)^2}$ ($\delta$ = huber_slope); for log-link losses $\hat y$ is the log of the mean. reg:gamma and reg:tweedie work like Poisson, for positive skewed amounts.

| objective | g | h | predicts |
| --- | --- | --- | --- |
| `reg:squarederror` | $\hat y-y$ | $1$ | conditional mean (default) |
| `reg:pseudohubererror` | $-r/s$ | $s^{-3}$ | mean, robust to outliers |
| `reg:absoluteerror` | $\operatorname{sign}(\hat y-y)$ | 0 → leaf = residual median | conditional median |
| `reg:quantileerror` | $\mathbb 1[y<\hat y]-\alpha$ | 0 → leaf = residual quantile | α-quantile |
| `count:poisson` | $e^{\hat y}-y$ | $e^{\hat y}$ | mean count (log link) |

> **Siblings:** LightGBM and CatBoost build on the same G/H leaf formula and gain. **LightGBM regression** changes *how* trees are grown (histograms, leaf-wise growth, GOSS); **CatBoost regression** changes *what* they see (symmetric trees, ordered target statistics for categories). Read those pages for the differences; for log-loss and probabilities see **XGBoost** (classification).

## Tuning in practice

To see how the knobs matter, we trained on scikit-learn's Friedman #1 benchmark (3,000 rows, 10 features of which 5 are noise, Gaussian noise σ = 1, so the best possible test RMSE is about 1.0; target standard deviation 5.16). 1,800 rows to train, 450 for early stopping, 750 for test.

One run (tree_method="hist", random_state=0); ES = early stopping after 50 rounds without validation improvement, 'sub' = subsample 0.8 and colsample_bytree 0.8. Low η with shallow trees and ES wins; deep trees memorize (train RMSE ≈ 0) without helping test error.

| setting | trees | train RMSE | test RMSE | test MAE | test R² |
| --- | --- | --- | --- | --- | --- |
| η 0.3, depth 6 (defaults) | 100 | 0.129 | 1.531 | 1.204 | 0.912 |
| η 0.3, depth 6 | 1000 | 0.001 | 1.531 | 1.202 | 0.912 |
| **η 0.05, depth 3, ES** | 949 | 0.617 | 1.221 | 0.970 | 0.944 |
| η 0.05, depth 3, λ 10, ES | 1130 | 0.654 | 1.257 | 1.000 | 0.941 |
| **η 0.05, depth 3, sub, ES** | 767 | 0.646 | **1.207** | **0.962** | **0.945** |
| η 0.05, depth 8, ES | 604 | 0.017 | 1.612 | 1.268 | 0.902 |

**A tuning recipe**

- **1. Fix η:** start at 0.05–0.1 and set `n_estimators` large (2,000+) with `early_stopping_rounds` ≈ 50 on a validation set; the number of trees then chooses itself
- **2. Tree size:** `max_depth` 3–8 (or `max_leaves` with `grow_policy="lossguide"`) and `min_child_weight` (= min rows per leaf for squared error): the most important pair
- **3. Randomness:** `subsample` and `colsample_bytree` in 0.6–0.9 decorrelate trees and usually help a little
- **4. Penalties:** `reg_lambda` (1–10), `gamma`, `reg_alpha`: fine adjustments — on this data λ = 10 did not help
- **5. Finish:** lower η by 2–5× and re-fit with proportionally more trees for a last small gain

> **γ is in units of the loss — so it depends on the scale of y:** Gains are sums of squared gradient sums, so they scale with $y^2$ and with $n$. A `gamma=4` that prunes sensibly when rent is in hundreds of euros prunes nothing when rent is in euros (all gains are 10,000× larger). Tune γ on the target's actual scale, or standardize $y$ first.

> **Trees cannot extrapolate:** Every prediction is $F_0$ plus a sum of leaf values learned from the training range. A 300 m² flat gets the same prediction as the 100 m² one (15.46 in the example). For targets with a trend (prices over time, growth curves), model the trend separately or predict a ratio/difference, and let XGBoost model the rest.

## In code

```python
import numpy as np

def grad_hess(y, pred):                      # squared error  l = (y - pred)^2 / 2
    return pred - y, np.ones_like(y)

def build(X, g, h, depth, max_depth=2, lam=1.0, gamma=0.0, min_child_weight=1.0):
    G, H = g.sum(), h.sum()
    leaf = {"w": -G / (H + lam), "G": G, "H": H}
    if depth == max_depth:
        return leaf
    best = (1e-6, None, None)                # (gain, feature, threshold)
    for j in range(X.shape[1]):
        order = np.argsort(X[:, j]); xs = X[order, j]
        GL, HL = np.cumsum(g[order])[:-1], np.cumsum(h[order])[:-1]
        GR, HR = G - GL, H - HL
        gain = GL**2/(HL+lam) + GR**2/(HR+lam) - G**2/(H+lam)
        ok = (xs[1:] > xs[:-1]) & (HL >= min_child_weight) & (HR >= min_child_weight)
        for k in np.where(ok)[0]:
            if gain[k] > best[0]:
                best = (gain[k], j, (xs[k] + xs[k+1]) / 2)
    gain, j, t = best
    if j is None:
        return leaf
    L = X[:, j] < t
    node = {"j": j, "t": t, "gain": gain, "G": G, "H": H,
            "left":  build(X[L],  g[L],  h[L],  depth+1, max_depth, lam, gamma, min_child_weight),
            "right": build(X[~L], g[~L], h[~L], depth+1, max_depth, lam, gamma, min_child_weight)}
    if "w" in node["left"] and "w" in node["right"] and gain < gamma:
        return leaf                          # prune: split doesn't pay for its extra leaf
    return node

def predict_tree(node, x):
    while "w" not in node:
        node = node["left"] if x[node["j"]] < node["t"] else node["right"]
    return node["w"]

def fit(X, y, n_rounds=2, eta=0.3, **kw):
    base = y.mean()
    pred, trees = np.full(len(y), base), []
    for _ in range(n_rounds):
        g, h = grad_hess(y, pred)
        tree = build(X, g, h, 0, **kw)
        trees.append(tree)
        pred = pred + eta * np.array([predict_tree(tree, x) for x in X])
    return base, trees

def predict(model, X, eta=0.3):
    base, trees = model
    return np.array([base + eta * sum(predict_tree(t, x) for t in trees) for x in X])

X = np.array([[30], [40], [50], [60], [70], [80], [90], [100.]])
y = np.array([5, 7, 8, 12, 15, 16, 20, 21.])
model = fit(X, y, n_rounds=2, eta=0.3, max_depth=2, lam=1.0, gamma=4.0)
for t in model[1]:
    print(t)
print("train preds:", predict(model, X).round(4))
print("75 m^2 ->", predict(model, np.array([[75.]]))[0].round(4))
```

From scratch in NumPy: g/h, exact greedy split search with the XGBoost gain, γ-pruning, w = −G/(H+λ). Reproduces the worked example.

```python
import numpy as np, xgboost as xgb
X = np.array([[30], [40], [50], [60], [70], [80], [90], [100.]])   # flat size, m^2
y = np.array([5, 7, 8, 12, 15, 16, 20, 21.])                        # rent, x100 EUR / month

model = xgb.XGBRegressor(
    objective="reg:squarederror", n_estimators=2, max_depth=2,
    learning_rate=0.3, reg_lambda=1.0, gamma=4.0, tree_method="exact")
model.fit(X, y)
for i, tree in enumerate(model.get_booster().get_dump(with_stats=True)):
    print(f"--- tree {i} ---\n{tree}")
print("train preds:", model.predict(X).round(4))
print("75 m^2 ->", model.predict(np.array([[75.]])).round(4))
```

The same model with the xgboost library (3.x). Its dump shows the same splits, gains (160, 10.75; 94.71, 6.05, 8.97), η-scaled leaves, and 14.46 for the 75 m² flat.

```text
--- tree 0 ---
0:[f0<65] yes=1,no=2,missing=1,gain=160,cover=8
	1:[f0<55] yes=3,no=4,missing=3,gain=10.75,cover=4
		3:leaf=-1.42500007,cover=3
		4:leaf=-0.150000006,cover=1
	2:leaf=1.20000005,cover=4

--- tree 1 ---
0:[f0<65] yes=1,no=2,missing=1,gain=94.7084961,cover=8
	1:[f0<55] yes=3,no=4,missing=3,gain=6.05153275,cover=4
		3:leaf=-1.104375,cover=3
		4:leaf=-0.127500057,cover=1
	2:[f0<85] yes=5,no=6,missing=5,gain=8.96533203,cover=4
		5:leaf=0.26000005,cover=2
		6:leaf=1.26000011,cover=2

train preds: [10.4706 10.4706 10.4706 12.7225 14.46   14.46   15.46   15.46  ]
75 m^2 -> [14.46]
```

Output of the library script (both scripts print the same predictions).

**Strengths**

- Tabular regression with up to millions of rows, mixed feature types, nonlinear effects and interactions — XGBoost is a default first choice.
- Features have missing values: each split learns a default direction, no imputation needed.
- You need a non-squared loss (Huber, quantile, Poisson, Tweedie, Gamma): only g and h change.
- You want strong accuracy with modest tuning (η, depth, early stopping) and fast, parallel prediction.

**Limitations**

- The target trends outside the training range (time series levels, physical extrapolation): the fit is piecewise constant.
- Very small data (tens of rows): a regularized linear model or Gaussian process is more stable.
- Inputs are images, audio or raw text: neural networks learn the representation; boosting needs good features.
- You need a smooth, monotone or interpretable formula — though monotone_constraints and SHAP values help.

- **Demand and sales forecasting:** Store-item sales from calendar, price and promotion features — gradient-boosted trees dominated the M5 and Rossmann forecasting competitions.
- **Real estate and pricing:** House-price and rent models (the Zillow Prize, Kaggle House Prices) mix location, size and categorical features with many missing values.
- **Insurance and risk:** Claim frequency and severity with `count:poisson`, `reg:gamma` or `reg:tweedie` objectives, often with monotone constraints for regulators.
- **Delivery and travel-time estimation:** ETA, latency and duration prediction from trip, route and time features, frequently with quantile objectives for upper bounds.

**Question:** A leaf holds three rows with residuals $y-\hat y = 4, 5, 6$ (squared error). With $\lambda=2$, what is its weight $w^*$ before shrinkage?

- A. 5
- B. 3
- C. −3
- D. 0.3

**Answer:** B. 3 — $g_i=\hat y-y=-4,-5,-6$, so $G=-15$, $H=3$ and $w^*=-G/(H+\lambda)=15/5=3$. The plain mean residual would be 5; λ shrinks it. With η = 0.1 the leaf would add 0.3.

**Question:** A candidate split has $G_L=-6, H_L=2$ and $G_R=6, H_R=2$, with $\lambda=1$ and `gamma=30`. Does XGBoost keep it?

- A. Yes, its gain is 24
- B. No — its gain 24 is below γ = 30
- C. Yes, gain is 36
- D. It cannot be evaluated because G = 0

**Answer:** B. No — its gain 24 is below γ = 30 — Gain (as the library computes it) $=\tfrac{36}{3}+\tfrac{36}{3}-\tfrac{0}{5}=24$. A parent sum $G=0$ is fine — it just means the parent leaf would predict 0. Since $24<30$ the split is pruned.

**Question:** For `reg:squarederror`, what does `min_child_weight=5` mean?

- A. Leaf weights must be at least 5
- B. Each child must contain at least 5 rows
- C. Each split must have gain ≥ 5
- D. Each tree must have at least 5 leaves

**Answer:** B. Each child must contain at least 5 rows — It bounds the hessian sum $H$ of each child. With squared error $h_i=1$, so $H$ is the row count.

**Question:** In the worked example the root's right leaf has $G=-20$, $H=4$. If $\lambda$ is raised from 1 to 4, its weight $w^*$ becomes…

- A. 4
- B. 5
- C. 2.5
- D. 1.43

**Answer:** C. 2.5 — $w^*=20/(4+4)=2.5$. With λ = 0 it would be the mean residual 5, with λ = 1 it is 4, with λ = 10 it is 1.43.

**Key points**

- XGBoost fits each tree to a second-order expansion of the loss: row gradients $g_i$ and hessians $h_i$. For squared error $g_i=\hat y_i-y_i$, $h_i=1$, and the expansion is exact.
- Optimal leaf: $w^*=-G/(H+\lambda)$ — in regression, the residual sum over (count + λ), a shrunken mean residual.
- Split gain $=G_L^2/(H_L+\lambda)+G_R^2/(H_R+\lambda)-G^2/(H+\lambda)$ (the library drops the paper's ½); splits with gain below γ are pruned bottom-up.
- η scales every tree; small η with many trees and early stopping on RMSE is the most reliable recipe. min_child_weight is the minimum leaf size for squared error.
- Only g and h depend on the loss, so the same machinery fits Huber, quantile, Poisson, Gamma and Tweedie regression — but trees still cannot extrapolate beyond the training targets.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
