Definition
XGBoost regression predicts a number as a constant plus a sum of many small regression trees, . Each new tree is grown on the gradients and hessians of the loss at the current predictions, its splits are scored by a regularized gain, and each leaf outputs the closed-form weight — a Newton step on the loss, shrunk toward zero.
At a glance
- Task
- Regression (squared error by default; also absolute, Huber, quantile, Poisson, Gamma, Tweedie losses)
- Input → output
- feature vector → real number
- Learns
- trees: a split (feature, threshold, default direction for missing values) per internal node and a weight per leaf
- Training
- exact greedy: ; histogram mode: plus binning — usually seconds for – rows
- Prediction
- comparisons
- Key hyperparameters
n_estimators,learning_rate(η),max_depth,reg_lambda(λ),gamma(γ),min_child_weight,subsample,colsample_bytree- Assumes
- a twice-differentiable loss (or a special leaf rule for absolute/quantile loss); no distributional assumptions on features, no scaling needed; axis-aligned, piecewise-constant fit
The problem
Plain gradient boosting (see Gradient boosting regression) fits each new tree to the residuals by least squares and adds a shrunken copy. It works, but it has three loose ends. (1) The tree is grown with one criterion (variance reduction of the residuals) and its leaves are then filled with a value from another — the two only agree for squared error. (2) Nothing in the objective penalizes a tree for being big or its leaf values for being large; complexity is controlled only from the outside, by depth limits and early stopping. (3) Leaves with very few points get extreme values: a leaf with one outlier predicts the whole outlier.
XGBoost (Chen & Guestrin, 2016) closes all three by writing down one regularized objective and deriving everything — which split to make, whether to make it, and what each leaf predicts — from a second-order Taylor expansion of that objective. The same derivation then works unchanged for any twice-differentiable loss.
The idea
Treat every training row as a little quadratic bowl: at the current prediction it tells you the slope (which way to move) and the curvature (how sure that step is). A leaf groups some rows, so their bowls add up to one bowl with slope and curvature . The best constant for that leaf is the bottom of the bowl, — a Newton step — except that XGBoost adds to the curvature, which pulls the step toward 0. A split is worth making if the two smaller bowls have lower bottoms than the merged one by more than the price of an extra leaf.
For squared error the bowls are exact: is minus the residual and for every row. So in regression, is just the number of rows in a leaf, and is the sum of residuals divided by (count + λ) — a mean residual shrunk as if extra rows with residual 0 had joined the leaf.
Vocabulary
- Base score F₀
- the starting prediction for every row; for squared error XGBoost sets it to the mean of (
base_scoreoverrides it) - Gradient gᵢ
- at the current prediction. Squared error: (negative residual)
- Hessian hᵢ
- . Squared error: ; it is also called the row's cover or weight
- G, H
- sums of and over the rows in a node; = row count for squared error
- Leaf weight w
- the value a leaf adds (before shrinkage); optimum
- Gain
- how much a split lowers the regularized objective; must exceed γ to survive pruning
- Shrinkage η
learning_rate: every tree's output is multiplied by η before it is added- Cover
- of a node;
min_child_weightis a lower bound on it
The math
- prediction for row after trees;
- the -th regression tree
- the leaf that falls into,
- weight (output) of leaf
- learning rate / shrinkage, 0 < η ≤ 1 (default 0.3)
- the loss; for regression by default
- number of leaves in the new tree
- price per leaf (
gamma/min_split_loss) - L2 penalty on leaf weights (
reg_lambda, default 1)
Only the new tree is unknown. Expand the loss to second order around the current prediction :
- gradient; squared error:
- hessian; squared error: (so this expansion is exact, not an approximation)
Drop the constant and group rows by leaf: every row in leaf receives the same . With and the objective separates into one independent quadratic per leaf:
Set :
- optimal leaf weight: a Newton step −G/H with λ added to the curvature
- the best objective this tree structure can reach — a score for the structure
For squared error, is the residual sum and , so : the mean residual , shrunk by the factor . With it is exactly the least-squares leaf of plain gradient boosting; with a 1-row leaf moves only half-way to its residual while a 100-row leaf moves 99% of the way.
Now score a split of node into and . The structure score before is ; after, . The decrease is the gain:
- gradient and hessian sums of the rows sent left
- the same for the rows sent right
- the cost of the one extra leaf the split creates
What each hyperparameter does to the equations
- λ (reg_lambda)
- adds to every : leaf weights shrink by , and the gain of splits that isolate few rows drops. Unlike variance reduction, a gain can become negative — the split is then never made.
- γ (gamma)
- subtracted from every gain: a split must lower the (un-halved) objective bracket by more than or it is pruned. Applied bottom-up after the tree is grown, so a weak split above a strong one is kept.
- η (learning_rate)
- multiplies before adding: . Smaller η needs proportionally more trees but averages over more of them, which usually generalizes better.
- min_child_weight
- a split is only allowed if and are both at least this. Since for squared error, it is simply a minimum number of rows per leaf.
- alpha (reg_alpha)
- L1 penalty : the leaf rule becomes soft-thresholding, , so leaves with small output exactly 0.
- max_delta_step
- caps ; rarely needed in regression, useful when can be tiny (Poisson, Tweedie).
Worked example: rent from flat size
Eight flats, one feature (size in m²), target = monthly rent in units of 100 €. Settings: squared error, max_depth=2, , , , two rounds, exact split search. The base score is the mean rent, , so the first gradients are .
| Flat | Size x (m²) | Rent y | F₀ | g = F₀ − y |
|---|---|---|---|---|
| 1 | 30 | 5 | 13 | 8 |
| 2 | 40 | 7 | 13 | 6 |
| 3 | 50 | 8 | 13 | 5 |
| 4 | 60 | 12 | 13 | 1 |
| 5 | 70 | 15 | 13 | −2 |
| 6 | 80 | 16 | 13 | −3 |
| 7 | 90 | 20 | 13 | −7 |
| 8 | 100 | 21 | 13 | −8 |
| Split | G_L / H_L | G_R / H_R | Gain |
|---|---|---|---|
| x < 35 | 8 / 1 | −8 / 7 | 40.00 |
| x < 45 | 14 / 2 | −14 / 6 | 93.33 |
| x < 55 | 19 / 3 | −19 / 5 | 150.42 |
| x < 65 | 20 / 4 | −20 / 4 | 160.00 |
| x < 75 | 18 / 5 | −18 / 3 | 135.00 |
| x < 85 | 15 / 6 | −15 / 2 | 107.14 |
| x < 95 | 8 / 7 | −8 / 1 | 40.00 |
Round 1 by hand
- 1Root. Best split : .
- 2Left child (flats 1–4, , , parent term ). Candidates: : ; : ; : . Best: with gain 10.75. Note the negative gains — makes splitting off small groups unattractive.
- 3Right child (flats 5–8, , ). : ; : ; : . Best: with gain 3.33.
- 4Prune with γ = 4. Left split: , kept. Right split: , removed — the right child becomes a single leaf. Tree 1 has 3 leaves.
- 5Leaf weights : flats 1–3: ; flat 4: ; flats 5–8: . (With they would be the mean residuals , , .)
- 6Shrink and add. : , , . New predictions : 11.575 (flats 1–3), 12.85 (flat 4), 14.2 (flats 5–8). Training RMSE falls from 5.612 to 4.492.
| Flat | x | y | F₁ | g = F₁ − y | Tree-2 leaf η·w | F₂ |
|---|---|---|---|---|---|---|
| 1 | 30 | 5 | 11.575 | 6.575 | −1.104 | 10.471 |
| 2 | 40 | 7 | 11.575 | 4.575 | −1.104 | 10.471 |
| 3 | 50 | 8 | 11.575 | 3.575 | −1.104 | 10.471 |
| 4 | 60 | 12 | 12.850 | 0.850 | −0.128 | 12.723 |
| 5 | 70 | 15 | 14.200 | −0.800 | +0.260 | 14.460 |
| 6 | 80 | 16 | 14.200 | −1.800 | +0.260 | 14.460 |
| 7 | 90 | 20 | 14.200 | −5.800 | +1.260 | 15.460 |
| 8 | 100 | 21 | 14.200 | −6.800 | +1.260 | 15.460 |
Round 2 and a prediction
- 1Tree 2 picks the same root (gain 94.71) and the same left split (gain 6.05).
- 2On the right, now has gain 8.97 (, , two rows each) — above , so this time it survives: the remaining error is concentrated in flats 7–8.
- 3Leaves: , , , ; times : , , , . Training RMSE 4.492 → 3.581.
- 4Predict a new 75 m² flat. Tree 1: → . Tree 2: , → . So , i.e. about 1,446 € per month — exactly what
XGBRegressor.predictreturns below.
The algorithm, step by step
XGBoost regression training
- 1Initialize (the mean of for squared error).
- 2For : compute and for every row (squared error: , ). Optionally subsample rows (
subsample) and columns (colsample_bytree). - 3Grow a tree depth-first to
max_depth: at each node, for every feature, scan the rows in sorted order (or the histogram bins), keep running sums , and evaluate the gain for every candidate threshold withmin_child_weight. Rows with a missing value are tried on both sides; the better side becomes the node's default direction. - 4Prune bottom-up: any split whose children are leaves and whose gain is below is collapsed into a leaf.
- 5Set leaves and update .
- 6Stop after trees or when the validation metric (RMSE, MAE…) has not improved for
early_stopping_roundsrounds. Predict with .
Cost
- Exact greedy
- sort once per feature, then each level is one linear scan: about per level, overall after an pre-sort
- Histogram (tree_method="hist", the default)
- bucket each feature into at most
max_bin(256) bins once; a node then costs after accumulating per bin. Children's histograms come from parent minus sibling - Memory
- for the (binned) data plus for and predictions
- Prediction
- per row; trees are independent, so it parallelizes trivially
Regression objectives
| objective | g | h | predicts |
|---|---|---|---|
reg:squarederror | conditional mean (default) | ||
reg:pseudohubererror | mean, robust to outliers | ||
reg:absoluteerror | 0 → leaf = residual median | conditional median | |
reg:quantileerror | 0 → leaf = residual quantile | α-quantile | |
count:poisson | mean count (log link) |
Tuning in practice
To see how the knobs matter, we trained on scikit-learn's Friedman #1 benchmark (3,000 rows, 10 features of which 5 are noise, Gaussian noise σ = 1, so the best possible test RMSE is about 1.0; target standard deviation 5.16). 1,800 rows to train, 450 for early stopping, 750 for test.
| setting | trees | train RMSE | test RMSE | test MAE | test R² |
|---|---|---|---|---|---|
| η 0.3, depth 6 (defaults) | 100 | 0.129 | 1.531 | 1.204 | 0.912 |
| η 0.3, depth 6 | 1000 | 0.001 | 1.531 | 1.202 | 0.912 |
| η 0.05, depth 3, ES | 949 | 0.617 | 1.221 | 0.970 | 0.944 |
| η 0.05, depth 3, λ 10, ES | 1130 | 0.654 | 1.257 | 1.000 | 0.941 |
| η 0.05, depth 3, sub, ES | 767 | 0.646 | 1.207 | 0.962 | 0.945 |
| η 0.05, depth 8, ES | 604 | 0.017 | 1.612 | 1.268 | 0.902 |
A tuning recipe
- 1. Fix η
- start at 0.05–0.1 and set
n_estimatorslarge (2,000+) withearly_stopping_rounds≈ 50 on a validation set; the number of trees then chooses itself - 2. Tree size
max_depth3–8 (ormax_leaveswithgrow_policy="lossguide") andmin_child_weight(= min rows per leaf for squared error): the most important pair- 3. Randomness
subsampleandcolsample_bytreein 0.6–0.9 decorrelate trees and usually help a little- 4. Penalties
reg_lambda(1–10),gamma,reg_alpha: fine adjustments — on this data λ = 10 did not help- 5. Finish
- lower η by 2–5× and re-fit with proportionally more trees for a last small gain
In code
1import numpy as np23def grad_hess(y, pred): # squared error l = (y - pred)^2 / 24 return pred - y, np.ones_like(y)56def build(X, g, h, depth, max_depth=2, lam=1.0, gamma=0.0, min_child_weight=1.0):7 G, H = g.sum(), h.sum()8 leaf = {"w": -G / (H + lam), "G": G, "H": H}9 if depth == max_depth:10 return leaf11 best = (1e-6, None, None) # (gain, feature, threshold)12 for j in range(X.shape[1]):13 order = np.argsort(X[:, j]); xs = X[order, j]14 GL, HL = np.cumsum(g[order])[:-1], np.cumsum(h[order])[:-1]15 GR, HR = G - GL, H - HL16 gain = GL**2/(HL+lam) + GR**2/(HR+lam) - G**2/(H+lam)17 ok = (xs[1:] > xs[:-1]) & (HL >= min_child_weight) & (HR >= min_child_weight)18 for k in np.where(ok)[0]:19 if gain[k] > best[0]:20 best = (gain[k], j, (xs[k] + xs[k+1]) / 2)21 gain, j, t = best22 if j is None:23 return leaf24 L = X[:, j] < t25 node = {"j": j, "t": t, "gain": gain, "G": G, "H": H,26 "left": build(X[L], g[L], h[L], depth+1, max_depth, lam, gamma, min_child_weight),27 "right": build(X[~L], g[~L], h[~L], depth+1, max_depth, lam, gamma, min_child_weight)}28 if "w" in node["left"] and "w" in node["right"] and gain < gamma:29 return leaf # prune: split doesn't pay for its extra leaf30 return node3132def predict_tree(node, x):33 while "w" not in node:34 node = node["left"] if x[node["j"]] < node["t"] else node["right"]35 return node["w"]3637def fit(X, y, n_rounds=2, eta=0.3, **kw):38 base = y.mean()39 pred, trees = np.full(len(y), base), []40 for _ in range(n_rounds):41 g, h = grad_hess(y, pred)42 tree = build(X, g, h, 0, **kw)43 trees.append(tree)44 pred = pred + eta * np.array([predict_tree(tree, x) for x in X])45 return base, trees4647def predict(model, X, eta=0.3):48 base, trees = model49 return np.array([base + eta * sum(predict_tree(t, x) for t in trees) for x in X])5051X = np.array([[30], [40], [50], [60], [70], [80], [90], [100.]])52y = np.array([5, 7, 8, 12, 15, 16, 20, 21.])53model = fit(X, y, n_rounds=2, eta=0.3, max_depth=2, lam=1.0, gamma=4.0)54for t in model[1]:55 print(t)56print("train preds:", predict(model, X).round(4))57print("75 m^2 ->", predict(model, np.array([[75.]]))[0].round(4))1import numpy as np, xgboost as xgb2X = np.array([[30], [40], [50], [60], [70], [80], [90], [100.]]) # flat size, m^23y = np.array([5, 7, 8, 12, 15, 16, 20, 21.]) # rent, x100 EUR / month45model = xgb.XGBRegressor(6 objective="reg:squarederror", n_estimators=2, max_depth=2,7 learning_rate=0.3, reg_lambda=1.0, gamma=4.0, tree_method="exact")8model.fit(X, y)9for i, tree in enumerate(model.get_booster().get_dump(with_stats=True)):10 print(f"--- tree {i} ---\n{tree}")11print("train preds:", model.predict(X).round(4))12print("75 m^2 ->", model.predict(np.array([[75.]])).round(4))1--- tree 0 ---20:[f0<65] yes=1,no=2,missing=1,gain=160,cover=83 1:[f0<55] yes=3,no=4,missing=3,gain=10.75,cover=44 3:leaf=-1.42500007,cover=35 4:leaf=-0.150000006,cover=16 2:leaf=1.20000005,cover=478--- tree 1 ---90:[f0<65] yes=1,no=2,missing=1,gain=94.7084961,cover=810 1:[f0<55] yes=3,no=4,missing=3,gain=6.05153275,cover=411 3:leaf=-1.104375,cover=312 4:leaf=-0.127500057,cover=113 2:[f0<85] yes=5,no=6,missing=5,gain=8.96533203,cover=414 5:leaf=0.26000005,cover=215 6:leaf=1.26000011,cover=21617train preds: [10.4706 10.4706 10.4706 12.7225 14.46 14.46 15.46 15.46 ]1875 m^2 -> [14.46]Use when
- +Tabular regression with up to millions of rows, mixed feature types, nonlinear effects and interactions — XGBoost is a default first choice.
- +Features have missing values: each split learns a default direction, no imputation needed.
- +You need a non-squared loss (Huber, quantile, Poisson, Tweedie, Gamma): only g and h change.
- +You want strong accuracy with modest tuning (η, depth, early stopping) and fast, parallel prediction.
Avoid when
- −The target trends outside the training range (time series levels, physical extrapolation): the fit is piecewise constant.
- −Very small data (tens of rows): a regularized linear model or Gaussian process is more stable.
- −Inputs are images, audio or raw text: neural networks learn the representation; boosting needs good features.
- −You need a smooth, monotone or interpretable formula — though monotone_constraints and SHAP values help.
Where it's used
Demand and sales forecasting
Store-item sales from calendar, price and promotion features — gradient-boosted trees dominated the M5 and Rossmann forecasting competitions.
Real estate and pricing
House-price and rent models (the Zillow Prize, Kaggle House Prices) mix location, size and categorical features with many missing values.
Insurance and risk
Claim frequency and severity with count:poisson, reg:gamma or reg:tweedie objectives, often with monotone constraints for regulators.
Delivery and travel-time estimation
ETA, latency and duration prediction from trip, route and time features, frequently with quantile objectives for upper bounds.
Check yourself
A leaf holds three rows with residuals (squared error). With , what is its weight before shrinkage?
Check yourself
A candidate split has and , with and gamma=30. Does XGBoost keep it?
Check yourself
For reg:squarederror, what does min_child_weight=5 mean?
Check yourself
In the worked example the root's right leaf has , . If is raised from 1 to 4, its weight becomes…
Remember
- ✓XGBoost fits each tree to a second-order expansion of the loss: row gradients and hessians . For squared error , , and the expansion is exact.
- ✓Optimal leaf: — in regression, the residual sum over (count + λ), a shrunken mean residual.
- ✓Split gain (the library drops the paper's ½); splits with gain below γ are pruned bottom-up.
- ✓η scales every tree; small η with many trees and early stopping on RMSE is the most reliable recipe. min_child_weight is the minimum leaf size for squared error.
- ✓Only g and h depend on the loss, so the same machinery fits Huber, quantile, Poisson, Gamma and Tweedie regression — but trees still cannot extrapolate beyond the training targets.