XGBoost regression

Gradient boosting with second-order (Newton) steps and regularized leaf weights w* = −G / (H + λ).

17 min read1435 wordsadvanced21 viewsupdated 2026-10-07

Definition

XGBoost regression predicts a number as a constant plus a sum of many small regression trees, y^=F0+η∑mfm(x)\hat y = F_0 + \eta\sum_m f_m(x). Each new tree is grown on the gradients gig_i and hessians hih_i of the loss at the current predictions, its splits are scored by a regularized gain, and each leaf outputs the closed-form weight w∗=−G/(H+λ)w^*=-G/(H+\lambda) — a Newton step on the loss, shrunk toward zero.

At a glance

Task
Regression (squared error by default; also absolute, Huber, quantile, Poisson, Gamma, Tweedie losses)
Input → output
feature vector xx → real number y^=F0+η∑m=1Mfm(x)\hat y = F_0 + \eta\sum_{m=1}^{M} f_m(x)
Learns
MM trees: a split (feature, threshold, default direction for missing values) per internal node and a weight wjw_j per leaf
Training
exact greedy: O(M⋅d⋅nlog⁡n)O(M\cdot d\cdot n\log n); histogram mode: O(M⋅d⋅n)O(M\cdot d\cdot n) plus binning — usually seconds for 10510^5–10610^6 rows
Prediction
O(M⋅depth)O(M\cdot\text{depth}) comparisons
Key hyperparameters
n_estimators, learning_rate (η), max_depth, reg_lambda (λ), gamma (γ), min_child_weight, subsample, colsample_bytree
Assumes
a twice-differentiable loss (or a special leaf rule for absolute/quantile loss); no distributional assumptions on features, no scaling needed; axis-aligned, piecewise-constant fit

The problem

Plain gradient boosting (see Gradient boosting regression) fits each new tree to the residuals by least squares and adds a shrunken copy. It works, but it has three loose ends. (1) The tree is grown with one criterion (variance reduction of the residuals) and its leaves are then filled with a value from another — the two only agree for squared error. (2) Nothing in the objective penalizes a tree for being big or its leaf values for being large; complexity is controlled only from the outside, by depth limits and early stopping. (3) Leaves with very few points get extreme values: a leaf with one outlier predicts the whole outlier.

XGBoost (Chen & Guestrin, 2016) closes all three by writing down one regularized objective and deriving everything — which split to make, whether to make it, and what each leaf predicts — from a second-order Taylor expansion of that objective. The same derivation then works unchanged for any twice-differentiable loss.

The idea

Treat every training row as a little quadratic bowl: at the current prediction it tells you the slope gig_i (which way to move) and the curvature hih_i (how sure that step is). A leaf groups some rows, so their bowls add up to one bowl with slope G=∑giG=\sum g_i and curvature H=∑hiH=\sum h_i. The best constant for that leaf is the bottom of the bowl, −G/H-G/H — a Newton step — except that XGBoost adds λ\lambda to the curvature, which pulls the step toward 0. A split is worth making if the two smaller bowls have lower bottoms than the merged one by more than the price γ\gamma of an extra leaf.

For squared error the bowls are exact: gi=y^i−yig_i = \hat y_i - y_i is minus the residual and hi=1h_i = 1 for every row. So in regression, HH is just the number of rows in a leaf, and −G/(H+λ)-G/(H+\lambda) is the sum of residuals divided by (count + λ) — a mean residual shrunk as if λ\lambda extra rows with residual 0 had joined the leaf.

Vocabulary

Base score F₀
the starting prediction for every row; for squared error XGBoost sets it to the mean of yy (base_score overrides it)
Gradient gᵢ
∂l(yi,y^i)/∂y^i\partial l(y_i,\hat y_i)/\partial \hat y_i at the current prediction. Squared error: gi=y^i−yig_i=\hat y_i-y_i (negative residual)
Hessian hᵢ
∂2l/∂y^i2\partial^2 l/\partial \hat y_i^2. Squared error: hi=1h_i=1; it is also called the row's cover or weight
G, H
sums of gig_i and hih_i over the rows in a node; HH = row count for squared error
Leaf weight w
the value a leaf adds (before shrinkage); optimum w∗=−G/(H+λ)w^*=-G/(H+\lambda)
Gain
how much a split lowers the regularized objective; must exceed γ to survive pruning
Shrinkage η
learning_rate: every tree's output is multiplied by η before it is added
Cover
HH of a node; min_child_weight is a lower bound on it

The math

y^i(m)=y^i(m−1)+η fm(xi),fm(x)=wq(x)\hat y_i^{(m)} = \hat y_i^{(m-1)} + \eta\, f_m(x_i),\qquad f_m(x)=w_{q(x)}
y^i(m)\hat y_i^{(m)}
prediction for row ii after mm trees; y^i(0)=F0\hat y_i^{(0)}=F_0
fmf_m
the mm-th regression tree
q(x)q(x)
the leaf that xx falls into, q(x)∈{1,…,T}q(x)\in\{1,\dots,T\}
wjw_j
weight (output) of leaf jj
η\eta
learning rate / shrinkage, 0 < η ≤ 1 (default 0.3)
L(m)=∑i=1nl(yi, y^i(m−1)+fm(xi))+γT+12λ∑j=1Twj2\mathcal{L}^{(m)}=\sum_{i=1}^{n} l\big(y_i,\ \hat y_i^{(m-1)}+f_m(x_i)\big)+\gamma T+\tfrac12\lambda\sum_{j=1}^{T}w_j^2
ll
the loss; for regression by default l(y,y^)=12(y−y^)2l(y,\hat y)=\tfrac12(y-\hat y)^2
TT
number of leaves in the new tree
γ\gamma
price per leaf (gamma / min_split_loss)
λ\lambda
L2 penalty on leaf weights (reg_lambda, default 1)

Only the new tree fmf_m is unknown. Expand the loss to second order around the current prediction y^i(m−1)\hat y_i^{(m-1)}:

l(yi,y^i+f)≈l(yi,y^i)+gif+12hif2,gi=∂l∂y^i,  hi=∂2l∂y^i2l\big(y_i,\hat y_i+f\big)\approx l(y_i,\hat y_i)+g_i f+\tfrac12 h_i f^2,\qquad g_i=\frac{\partial l}{\partial \hat y_i},\ \ h_i=\frac{\partial^2 l}{\partial \hat y_i^2}
gig_i
gradient; squared error: gi=y^i−yig_i=\hat y_i-y_i
hih_i
hessian; squared error: hi=1h_i=1 (so this expansion is exact, not an approximation)

Drop the constant l(yi,y^i)l(y_i,\hat y_i) and group rows by leaf: every row in leaf jj receives the same f=wjf=w_j. With Gj=∑i∈jgiG_j=\sum_{i\in j} g_i and Hj=∑i∈jhiH_j=\sum_{i\in j} h_i the objective separates into one independent quadratic per leaf:

L~=∑j=1T[Gjwj+12(Hj+λ)wj2]+γT\tilde{\mathcal{L}}=\sum_{j=1}^{T}\Big[G_j w_j+\tfrac12\big(H_j+\lambda\big)w_j^2\Big]+\gamma T

Set ∂L~/∂wj=Gj+(Hj+λ)wj=0\partial\tilde{\mathcal L}/\partial w_j = G_j + (H_j+\lambda)w_j = 0:

wj∗=−GjHj+λ⟹L~∗=−12∑j=1TGj2Hj+λ+γTw_j^{*}=-\frac{G_j}{H_j+\lambda}\qquad\Longrightarrow\qquad \tilde{\mathcal{L}}^{*}=-\frac12\sum_{j=1}^{T}\frac{G_j^2}{H_j+\lambda}+\gamma T
wj∗w_j^*
optimal leaf weight: a Newton step −G/H with λ added to the curvature
L~∗\tilde{\mathcal L}^*
the best objective this tree structure can reach — a score for the structure

For squared error, −Gj=∑i∈j(yi−y^i)-G_j=\sum_{i\in j}(y_i-\hat y_i) is the residual sum and Hj=njH_j=n_j, so wj∗=njnj+λ rˉjw_j^*=\frac{n_j}{n_j+\lambda}\,\bar r_j: the mean residual rˉj\bar r_j, shrunk by the factor nj/(nj+λ)n_j/(n_j+\lambda). With λ=0\lambda=0 it is exactly the least-squares leaf of plain gradient boosting; with λ=1\lambda=1 a 1-row leaf moves only half-way to its residual while a 100-row leaf moves 99% of the way.

Now score a split of node SS into LL and RR. The structure score before is −12G2/(H+λ)+γ-\tfrac12 G^2/(H+\lambda)+\gamma; after, −12[GL2/(HL+λ)+GR2/(HR+λ)]+2γ-\tfrac12\big[G_L^2/(H_L+\lambda)+G_R^2/(H_R+\lambda)\big]+2\gamma. The decrease is the gain:

Gain=12[GL2HL+λ+GR2HR+λ−(GL+GR)2HL+HR+λ]−γ\text{Gain}=\frac12\left[\frac{G_L^2}{H_L+\lambda}+\frac{G_R^2}{H_R+\lambda}-\frac{(G_L+G_R)^2}{H_L+H_R+\lambda}\right]-\gamma
GL,HLG_L,H_L
gradient and hessian sums of the rows sent left
GR,HRG_R,H_R
the same for the rows sent right
γ\gamma
the cost of the one extra leaf the split creates

What each hyperparameter does to the equations

λ (reg_lambda)
adds to every HH: leaf weights shrink by nj/(nj+λ)n_j/(n_j+\lambda), and the gain of splits that isolate few rows drops. Unlike variance reduction, a gain can become negative — the split is then never made.
γ (gamma)
subtracted from every gain: a split must lower the (un-halved) objective bracket by more than γ\gamma or it is pruned. Applied bottom-up after the tree is grown, so a weak split above a strong one is kept.
η (learning_rate)
multiplies w∗w^* before adding: y^←y^+ηw∗\hat y\leftarrow\hat y+\eta w^*. Smaller η needs proportionally more trees but averages over more of them, which usually generalizes better.
min_child_weight
a split is only allowed if HLH_L and HRH_R are both at least this. Since hi=1h_i=1 for squared error, it is simply a minimum number of rows per leaf.
alpha (reg_alpha)
L1 penalty α∑∣wj∣\alpha\sum|w_j|: the leaf rule becomes soft-thresholding, w∗=−sign⁡(G)max⁡(∣G∣−α,0)/(H+λ)w^*=-\operatorname{sign}(G)\max(|G|-\alpha,0)/(H+\lambda), so leaves with small ∣G∣|G| output exactly 0.
max_delta_step
caps ∣w∗∣|w^*|; rarely needed in regression, useful when HH can be tiny (Poisson, Tweedie).

Worked example: rent from flat size

Eight flats, one feature (size in m²), target = monthly rent in units of 100 €. Settings: squared error, max_depth=2, η=0.3\eta=0.3, λ=1\lambda=1, γ=4\gamma=4, two rounds, exact split search. The base score is the mean rent, F0=104/8=13F_0=104/8=13, so the first gradients are gi=13−yig_i=13-y_i.

FlatSize x (m²)Rent yF₀g = F₀ − y
1305138
2407136
3508135
46012131
5701513−2
6801613−3
7902013−7
81002113−8
Training data with the round-1 gradients g = F₀ − y (h = 1 for every row). The four highlighted rows end up on the right of the root split.
SplitG_L / H_LG_R / H_RGain
x < 358 / 1−8 / 740.00
x < 4514 / 2−14 / 693.33
x < 5519 / 3−19 / 5150.42
x < 6520 / 4−20 / 4160.00
x < 7518 / 5−18 / 3135.00
x < 8515 / 6−15 / 2107.14
x < 958 / 7−8 / 140.00
Round 1, root: every midpoint threshold. G=0G=0, H=8H=8, so the parent term G2/(H+λ)G^2/(H+\lambda) is 0 and the gain is GL2/(HL+1)+GR2/(HR+1)G_L^2/(H_L+1)+G_R^2/(H_R+1).

Round 1 by hand

  1. 1Root. Best split x<65x<65: 2024+1+(−20)24+1−0=80+80=160\tfrac{20^2}{4+1}+\tfrac{(-20)^2}{4+1}-0 = 80+80 = 160.
  2. 2Left child (flats 1–4, G=20G=20, H=4H=4, parent term 400/5=80400/5=80). Candidates: x<35x<35: 642+1444−80=−12\tfrac{64}{2}+\tfrac{144}{4}-80=-12; x<45x<45: 1963+363−80=−2.67\tfrac{196}{3}+\tfrac{36}{3}-80=-2.67; x<55x<55: 3614+12−80=10.75\tfrac{361}{4}+\tfrac{1}{2}-80=10.75. Best: x<55x<55 with gain 10.75. Note the negative gains — λ\lambda makes splitting off small groups unattractive.
  3. 3Right child (flats 5–8, G=−20G=-20, H=4H=4). x<75x<75: 42+3244−80=3\tfrac{4}{2}+\tfrac{324}{4}-80=3; x<85x<85: 253+2253−80=3.33\tfrac{25}{3}+\tfrac{225}{3}-80=3.33; x<95x<95: −12-12. Best: x<85x<85 with gain 3.33.
  4. 4Prune with γ = 4. Left split: 10.75>410.75>4, kept. Right split: 3.33<43.33<4, removed — the right child becomes a single leaf. Tree 1 has 3 leaves.
  5. 5Leaf weights w∗=−G/(H+1)w^*=-G/(H+1): flats 1–3: −19/4=−4.75-19/4=-4.75; flat 4: −1/2=−0.5-1/2=-0.5; flats 5–8: 20/5=420/5=4. (With λ=0\lambda=0 they would be the mean residuals −6.33-6.33, −1-1, +5+5.)
  6. 6Shrink and add. ηw∗\eta w^*: −1.425-1.425, −0.15-0.15, +1.2+1.2. New predictions F1F_1: 11.575 (flats 1–3), 12.85 (flat 4), 14.2 (flats 5–8). Training RMSE falls from 5.612 to 4.492.
FlatxyF₁g = F₁ − yTree-2 leaf η·wF₂
130511.5756.575−1.10410.471
240711.5754.575−1.10410.471
350811.5753.575−1.10410.471
4601212.8500.850−0.12812.723
5701514.200−0.800+0.26014.460
6801614.200−1.800+0.26014.460
7902014.200−5.800+1.26015.460
81002114.200−6.800+1.26015.460
Round 2 starts from F₁. The new gradients are smaller but keep their sign pattern; their sum is G = 0.375.

Round 2 and a prediction

  1. 1Tree 2 picks the same root x<65x<65 (gain 94.71) and the same left split x<55x<55 (gain 6.05).
  2. 2On the right, x<85x<85 now has gain 8.97 (GL=−2.6G_L=-2.6, GR=−12.6G_R=-12.6, two rows each) — above γ=4\gamma=4, so this time it survives: the remaining error is concentrated in flats 7–8.
  3. 3Leaves: −14.725/4=−3.681-14.725/4=-3.681, −0.85/2=−0.425-0.85/2=-0.425, 2.6/3=0.8672.6/3=0.867, 12.6/3=4.212.6/3=4.2; times η\eta: −1.104-1.104, −0.1275-0.1275, +0.26+0.26, +1.26+1.26. Training RMSE 4.492 → 3.581.
  4. 4Predict a new 75 m² flat. Tree 1: 75≥6575\ge65 → +1.2+1.2. Tree 2: 75≥6575\ge65, 75<8575<85 → +0.26+0.26. So y^=13+1.2+0.26=14.46\hat y = 13 + 1.2 + 0.26 = 14.46, i.e. about 1,446 € per month — exactly what XGBRegressor.predict returns below.
rent (×100 €) vs size (m²) · drag a flat · – – F₀ — F₁η·w = -1.425η·w = -0.150η·w = +1.200051015202530405060708090100new: 75 m²8651-2-3-7-8tree 1, fit to gᵢ = F₀(xᵢ) − yᵢ and hᵢ = 1x < 65gain 160.00G 0.00 · H 8x < 55gain 10.75G 20.00 · H 4G 19.00 · H 3w -4.75η·w -1.425G 1.00 · H 1w -0.50η·w -0.150G -20.00 · H 4w +4.00η·w +1.200x<85 cut: 3.33 < γgain = G²L/(HL+λ) + G²R/(HR+λ) − G²/(H+λ) · leaf w = −G/(H+λ)dashed red leaf = its split was pruned because gain < γ
round 1/12depth
F₀ = mean(y) = 13.000train RMSE F₀ → F₁: 5.612 → 4.492F₁(75 m²) = 14.200leaves: 3
fig.The worked example, live. Top: the eight flats, the model before (dashed, F₀ = 13) and after (orange) the current round, and each flat's gradient g = F − y as a stick coloured by the leaf it falls in. Bottom: the tree XGBoost grows on those gradients, with G, H, gain and w = −G/(H+λ) in every node. At the defaults (λ = 1, γ = 4, η = 0.3, depth 2) round 1 splits at x < 65 (gain 160) and x < 55 (gain 10.75), and the right split x < 85 (gain 3.33) is drawn as pruned; step to round 2 and it survives with gain 8.97, and F₂(75 m²) = 14.460. Drag λ up to watch every leaf shrink and small-group gains turn negative, γ up to prune, η to change the step size — or drag any flat's rent.

The algorithm, step by step

XGBoost regression training

  1. 1Initialize y^i=F0\hat y_i=F_0 (the mean of yy for squared error).
  2. 2For m=1…Mm=1\dots M: compute gi=∂l/∂y^ig_i=\partial l/\partial\hat y_i and hi=∂2l/∂y^i2h_i=\partial^2 l/\partial\hat y_i^2 for every row (squared error: gi=y^i−yig_i=\hat y_i-y_i, hi=1h_i=1). Optionally subsample rows (subsample) and columns (colsample_bytree).
  3. 3Grow a tree depth-first to max_depth: at each node, for every feature, scan the rows in sorted order (or the histogram bins), keep running sums GL,HLG_L,H_L, and evaluate the gain for every candidate threshold with HL,HR≥H_L,H_R\ge min_child_weight. Rows with a missing value are tried on both sides; the better side becomes the node's default direction.
  4. 4Prune bottom-up: any split whose children are leaves and whose gain is below γ\gamma is collapsed into a leaf.
  5. 5Set leaves wj=−Gj/(Hj+λ)w_j=-G_j/(H_j+\lambda) and update y^i←y^i+η wq(xi)\hat y_i\leftarrow\hat y_i+\eta\,w_{q(x_i)}.
  6. 6Stop after MM trees or when the validation metric (RMSE, MAE…) has not improved for early_stopping_rounds rounds. Predict with F0+etasummfm(x)F_0+etasum_m f_m(x).

Cost

Exact greedy
sort once per feature, then each level is one linear scan: about O(d n)O(d\,n) per level, O(M⋅depth⋅d n)O(M\cdot \text{depth}\cdot d\,n) overall after an O(d nlog⁡n)O(d\,n\log n) pre-sort
Histogram (tree_method="hist", the default)
bucket each feature into at most max_bin (256) bins once; a node then costs O(d⋅bins)O(d\cdot\text{bins}) after accumulating G,HG,H per bin. Children's histograms come from parent minus sibling
Memory
O(n d)O(n\,d) for the (binned) data plus O(n)O(n) for g,hg,h and predictions
Prediction
O(M⋅depth)O(M\cdot\text{depth}) per row; trees are independent, so it parallelizes trivially

Regression objectives

objectiveghpredicts
reg:squarederrory^−y\hat y-y11conditional mean (default)
reg:pseudohubererror−r/s-r/ss−3s^{-3}mean, robust to outliers
reg:absoluteerrorsign⁡(y^−y)\operatorname{sign}(\hat y-y)0 → leaf = residual medianconditional median
reg:quantileerror1[y<y^]−α\mathbb 1[y<\hat y]-\alpha0 → leaf = residual quantileα-quantile
count:poissoney^−ye^{\hat y}-yey^e^{\hat y}mean count (log link)
Every loss only has to supply gig_i and hih_i; the tree-growing code is identical. Here r=y−y^r=y-\hat y; pseudo-Huber loss is δ2(s−1)\delta^2(s-1) with s=1+(r/δ)2s=\sqrt{1+(r/\delta)^2} (δ\delta = huber_slope); for log-link losses y^\hat y is the log of the mean. reg:gamma and reg:tweedie work like Poisson, for positive skewed amounts.

Tuning in practice

To see how the knobs matter, we trained on scikit-learn's Friedman #1 benchmark (3,000 rows, 10 features of which 5 are noise, Gaussian noise σ = 1, so the best possible test RMSE is about 1.0; target standard deviation 5.16). 1,800 rows to train, 450 for early stopping, 750 for test.

settingtreestrain RMSEtest RMSEtest MAEtest R²
η 0.3, depth 6 (defaults)1000.1291.5311.2040.912
η 0.3, depth 610000.0011.5311.2020.912
η 0.05, depth 3, ES9490.6171.2210.9700.944
η 0.05, depth 3, λ 10, ES11300.6541.2571.0000.941
η 0.05, depth 3, sub, ES7670.6461.2070.9620.945
η 0.05, depth 8, ES6040.0171.6121.2680.902
One run (tree_method="hist", random_state=0); ES = early stopping after 50 rounds without validation improvement, 'sub' = subsample 0.8 and colsample_bytree 0.8. Low η with shallow trees and ES wins; deep trees memorize (train RMSE ≈ 0) without helping test error.

A tuning recipe

1. Fix η
start at 0.05–0.1 and set n_estimators large (2,000+) with early_stopping_rounds ≈ 50 on a validation set; the number of trees then chooses itself
2. Tree size
max_depth 3–8 (or max_leaves with grow_policy="lossguide") and min_child_weight (= min rows per leaf for squared error): the most important pair
3. Randomness
subsample and colsample_bytree in 0.6–0.9 decorrelate trees and usually help a little
4. Penalties
reg_lambda (1–10), gamma, reg_alpha: fine adjustments — on this data λ = 10 did not help
5. Finish
lower η by 2–5× and re-fit with proportionally more trees for a last small gain

In code

python
1import numpy as np
2
3def grad_hess(y, pred): # squared error l = (y - pred)^2 / 2
4 return pred - y, np.ones_like(y)
5
6def build(X, g, h, depth, max_depth=2, lam=1.0, gamma=0.0, min_child_weight=1.0):
7 G, H = g.sum(), h.sum()
8 leaf = {"w": -G / (H + lam), "G": G, "H": H}
9 if depth == max_depth:
10 return leaf
11 best = (1e-6, None, None) # (gain, feature, threshold)
12 for j in range(X.shape[1]):
13 order = np.argsort(X[:, j]); xs = X[order, j]
14 GL, HL = np.cumsum(g[order])[:-1], np.cumsum(h[order])[:-1]
15 GR, HR = G - GL, H - HL
16 gain = GL**2/(HL+lam) + GR**2/(HR+lam) - G**2/(H+lam)
17 ok = (xs[1:] > xs[:-1]) & (HL >= min_child_weight) & (HR >= min_child_weight)
18 for k in np.where(ok)[0]:
19 if gain[k] > best[0]:
20 best = (gain[k], j, (xs[k] + xs[k+1]) / 2)
21 gain, j, t = best
22 if j is None:
23 return leaf
24 L = X[:, j] < t
25 node = {"j": j, "t": t, "gain": gain, "G": G, "H": H,
26 "left": build(X[L], g[L], h[L], depth+1, max_depth, lam, gamma, min_child_weight),
27 "right": build(X[~L], g[~L], h[~L], depth+1, max_depth, lam, gamma, min_child_weight)}
28 if "w" in node["left"] and "w" in node["right"] and gain < gamma:
29 return leaf # prune: split doesn't pay for its extra leaf
30 return node
31
32def predict_tree(node, x):
33 while "w" not in node:
34 node = node["left"] if x[node["j"]] < node["t"] else node["right"]
35 return node["w"]
36
37def fit(X, y, n_rounds=2, eta=0.3, **kw):
38 base = y.mean()
39 pred, trees = np.full(len(y), base), []
40 for _ in range(n_rounds):
41 g, h = grad_hess(y, pred)
42 tree = build(X, g, h, 0, **kw)
43 trees.append(tree)
44 pred = pred + eta * np.array([predict_tree(tree, x) for x in X])
45 return base, trees
46
47def predict(model, X, eta=0.3):
48 base, trees = model
49 return np.array([base + eta * sum(predict_tree(t, x) for t in trees) for x in X])
50
51X = np.array([[30], [40], [50], [60], [70], [80], [90], [100.]])
52y = np.array([5, 7, 8, 12, 15, 16, 20, 21.])
53model = fit(X, y, n_rounds=2, eta=0.3, max_depth=2, lam=1.0, gamma=4.0)
54for t in model[1]:
55 print(t)
56print("train preds:", predict(model, X).round(4))
57print("75 m^2 ->", predict(model, np.array([[75.]]))[0].round(4))
From scratch in NumPy: g/h, exact greedy split search with the XGBoost gain, γ-pruning, w = −G/(H+λ). Reproduces the worked example.
python
1import numpy as np, xgboost as xgb
2X = np.array([[30], [40], [50], [60], [70], [80], [90], [100.]]) # flat size, m^2
3y = np.array([5, 7, 8, 12, 15, 16, 20, 21.]) # rent, x100 EUR / month
4
5model = xgb.XGBRegressor(
6 objective="reg:squarederror", n_estimators=2, max_depth=2,
7 learning_rate=0.3, reg_lambda=1.0, gamma=4.0, tree_method="exact")
8model.fit(X, y)
9for i, tree in enumerate(model.get_booster().get_dump(with_stats=True)):
10 print(f"--- tree {i} ---\n{tree}")
11print("train preds:", model.predict(X).round(4))
12print("75 m^2 ->", model.predict(np.array([[75.]])).round(4))
The same model with the xgboost library (3.x). Its dump shows the same splits, gains (160, 10.75; 94.71, 6.05, 8.97), η-scaled leaves, and 14.46 for the 75 m² flat.
text
1--- tree 0 ---
20:[f0<65] yes=1,no=2,missing=1,gain=160,cover=8
3 1:[f0<55] yes=3,no=4,missing=3,gain=10.75,cover=4
4 3:leaf=-1.42500007,cover=3
5 4:leaf=-0.150000006,cover=1
6 2:leaf=1.20000005,cover=4
7
8--- tree 1 ---
90:[f0<65] yes=1,no=2,missing=1,gain=94.7084961,cover=8
10 1:[f0<55] yes=3,no=4,missing=3,gain=6.05153275,cover=4
11 3:leaf=-1.104375,cover=3
12 4:leaf=-0.127500057,cover=1
13 2:[f0<85] yes=5,no=6,missing=5,gain=8.96533203,cover=4
14 5:leaf=0.26000005,cover=2
15 6:leaf=1.26000011,cover=2
16
17train preds: [10.4706 10.4706 10.4706 12.7225 14.46 14.46 15.46 15.46 ]
1875 m^2 -> [14.46]
Output of the library script (both scripts print the same predictions).

Use when

  • +Tabular regression with up to millions of rows, mixed feature types, nonlinear effects and interactions — XGBoost is a default first choice.
  • +Features have missing values: each split learns a default direction, no imputation needed.
  • +You need a non-squared loss (Huber, quantile, Poisson, Tweedie, Gamma): only g and h change.
  • +You want strong accuracy with modest tuning (η, depth, early stopping) and fast, parallel prediction.

Avoid when

  • −The target trends outside the training range (time series levels, physical extrapolation): the fit is piecewise constant.
  • −Very small data (tens of rows): a regularized linear model or Gaussian process is more stable.
  • −Inputs are images, audio or raw text: neural networks learn the representation; boosting needs good features.
  • −You need a smooth, monotone or interpretable formula — though monotone_constraints and SHAP values help.

Where it's used

Demand and sales forecasting

Store-item sales from calendar, price and promotion features — gradient-boosted trees dominated the M5 and Rossmann forecasting competitions.

Real estate and pricing

House-price and rent models (the Zillow Prize, Kaggle House Prices) mix location, size and categorical features with many missing values.

Insurance and risk

Claim frequency and severity with count:poisson, reg:gamma or reg:tweedie objectives, often with monotone constraints for regulators.

Delivery and travel-time estimation

ETA, latency and duration prediction from trip, route and time features, frequently with quantile objectives for upper bounds.

Check yourself

A leaf holds three rows with residuals y−y^=4,5,6y-\hat y = 4, 5, 6 (squared error). With λ=2\lambda=2, what is its weight w∗w^* before shrinkage?

Check yourself

A candidate split has GL=−6,HL=2G_L=-6, H_L=2 and GR=6,HR=2G_R=6, H_R=2, with λ=1\lambda=1 and gamma=30. Does XGBoost keep it?

Check yourself

For reg:squarederror, what does min_child_weight=5 mean?

Check yourself

In the worked example the root's right leaf has G=−20G=-20, H=4H=4. If λ\lambda is raised from 1 to 4, its weight w∗w^* becomes…

Remember

  • ✓XGBoost fits each tree to a second-order expansion of the loss: row gradients gig_i and hessians hih_i. For squared error gi=y^i−yig_i=\hat y_i-y_i, hi=1h_i=1, and the expansion is exact.
  • ✓Optimal leaf: w∗=−G/(H+λ)w^*=-G/(H+\lambda) — in regression, the residual sum over (count + λ), a shrunken mean residual.
  • ✓Split gain =GL2/(HL+λ)+GR2/(HR+λ)−G2/(H+λ)=G_L^2/(H_L+\lambda)+G_R^2/(H_R+\lambda)-G^2/(H+\lambda) (the library drops the paper's ½); splits with gain below γ are pruned bottom-up.
  • ✓η scales every tree; small η with many trees and early stopping on RMSE is the most reliable recipe. min_child_weight is the minimum leaf size for squared error.
  • ✓Only g and h depend on the loss, so the same machinery fits Huber, quantile, Poisson, Gamma and Tweedie regression — but trees still cannot extrapolate beyond the training targets.
← back to ML Algorithms roadmap