CatBoost regression

Gradient boosting with symmetric trees and ordered boosting, which stops categorical encodings from leaking the target.

17 min read2064 wordsadvanced13 viewsupdated 2026-10-07

Definition

CatBoost regression predicts a number as a constant plus a sum of small trees, y^=F0+η∑mfm(x)\hat y = F_0+\eta\sum_m f_m(x), like any gradient-boosted model. What is special is what the trees see and how they are shaped: every categorical feature is replaced by an ordered target statistic — the average target of the earlier rows with the same category in a random permutation — and every tree is symmetric (oblivious): one question per level, shared by all nodes on that level.

At a glance

Task
Regression: RMSE (squared error) by default; also MAE, Quantile, Huber, MAPE, Poisson, Tweedie, RMSEWithUncertainty
Input → output
numeric and raw categorical features xx → real number y^=F0+η∑m=1Mfm(x)\hat y=F_0+\eta\sum_{m=1}^M f_m(x)
Learns
MM oblivious trees: dd (feature, border) pairs and 2d2^d leaf values each, plus per-category target statistics stored with the model
Training
features quantized into ≤254\le 254 borders; per tree about O(d⋅n⋅p)O(d\cdot n\cdot p) for pp features — slower per tree than LightGBM on numeric data, competitive once categories are involved
Prediction
O(M⋅d)O(M\cdot d): each tree is dd comparisons that form a dd-bit leaf index — branch-free and very fast
Key hyperparameters
iterations, learning_rate (η), depth (default 6), l2_leaf_reg (λ, default 3), random_strength, bagging_temperature / subsample, one_hot_max_size, boosting_type (Plain / Ordered)
Assumes
a differentiable loss; no scaling; categories need no manual encoding; the fit is piecewise constant and axis-aligned, so no extrapolation beyond the training targets

The problem

Tabular regression data is full of categories: district, product id, store, device, postcode. Trees need numbers. One-hot encoding works for a handful of levels but explodes for thousands, and each indicator column carries almost no signal on its own. The popular alternative is target encoding: replace each category by the mean target of its rows. For a rent model, "district = West" becomes "average West rent".

Target encoding computed on the training rows leaks the target. If row ii's own rent is part of the average that replaces its district, the encoded feature contains a piece of yiy_i. For a category with a single row the encoding is a rescaled yiy_i. The trees happily split on it, the training error drops, and the model fails on new data, where the encoding has no such piece. The CatBoost paper (Prokhorenkova et al., 2018) calls this prediction shift: the distribution of a feature at training time differs from its distribution at test time.

Boosting itself has the same flaw in a quieter form. The residuals that tree mm is fitted to are computed by a model that was trained on those same rows, so they are systematically smaller than the residuals the model will make on fresh rows. CatBoost's answer to both problems is the same trick: order the data randomly and let each row use only the rows before it.

ModelTrain RMSETest RMSEcorr(encoding, y) on train
GBM on xx only0.8011.041—
GBM + greedy target encoding0.6561.1920.621
GBM + ordered target statistic0.7741.0530.052
CatBoost, Plain (raw category)0.9191.012—
CatBoost, Ordered (raw category)0.9271.008—
A synthetic check (script-verified): 1000 training rows, y=3x+εy=3x+\varepsilon with ε∼N(0,1)\varepsilon\sim N(0,1), plus a 400-level category that carries no signal. Each scikit-learn GBM gets xx and one encoding of the category (200 trees, depth 3); test rows use full-training statistics. The irreducible error is 1.0.

The greedy encoding looks like the best model on the training set and is the worst on the test set: a pure-noise category correlates 0.621 with the target once each row's own target is inside its encoding. The ordered statistic is honest (0.052), and CatBoost's own pipeline does best of all.

The idea

Imagine the training rows arriving one at a time, in a random order, like a stream. When row ii arrives, you may summarise only what you have already seen. Its district encoding is the average rent of the earlier rows from the same district, smoothed toward a prior. Row ii's own rent was never used, so the feature is exactly as informative as it will be for a brand-new flat. The first rows of each district get mostly the prior (noisy but unbiased); later rows get sharper estimates.

The trees are deliberately simple: a symmetric tree of depth dd asks the same question of every node on a level, so it is really a list of dd yes/no tests and a table of 2d2^d leaf values. That is a strong regularizer (one split must be good for every branch at once), and it turns prediction into computing a dd-bit binary number.

Vocabulary

Target statistic (TS / CTR)
a number that replaces a category value, computed from the targets of rows with that value — e.g. a smoothed mean rent. CatBoost's docs call them CTRs.
Prior p, weight a
the value a TS falls back to when a category has few rows, and how many "virtual rows" it is worth. Here pp is the mean target and a=1a=1.
Greedy TS
a TS computed from all training rows of the category, including the row being encoded — leaks the target
Leave-one-out TS
all rows of the category except the row itself — still leaks, in reverse
Permutation σ
a random ordering of the training rows; σ(i)\sigma(i) is row ii's position. CatBoost draws several.
Ordered TS
the TS of row ii computed only from rows that come before it in σ\sigma
Prediction shift
the gap between a feature's (or residual's) distribution on training rows and on new rows, caused by reusing a row's own target
Ordered boosting
computing row ii's residual with a model that never saw row ii (only rows before it in σ\sigma) — boosting_type="Ordered"
Oblivious (symmetric) tree
a tree in which every node at depth kk uses the same feature and border; dd tests, 2d2^d leaves
Border
a candidate threshold. Features are quantized into at most 254 borders by default (border_count).
Feature combination
a new categorical feature formed by joining categories (e.g. district × building type), encoded with its own TS

The math

Target statistics. For a categorical feature with value cic_i in row ii, the three candidate encodings differ only in which rows they average:

x^igreedy=∑j=1n[cj=ci] yj+a p∑j=1n[cj=ci]+ax^iordered=∑j: σ(j)<σ(i)[cj=ci] yj+a p∑j: σ(j)<σ(i)[cj=ci]+a\hat x_i^{\text{greedy}}=\frac{\sum_{j=1}^{n}[c_j=c_i]\,y_j+a\,p}{\sum_{j=1}^{n}[c_j=c_i]+a}\qquad \hat x_i^{\text{ordered}}=\frac{\sum_{j:\,\sigma(j)<\sigma(i)}[c_j=c_i]\,y_j+a\,p}{\sum_{j:\,\sigma(j)<\sigma(i)}[c_j=c_i]+a}
x^i\hat x_i
the number that replaces row ii's category
[cj=ci][c_j=c_i]
1 if row jj has the same category as row ii, else 0
yjy_j
target (here: rent) of row j
pp
prior: the value an unseen category gets (here the mean target)
a>0a>0
prior weight: how many pseudo-rows the prior is worth
σ(j)<σ(i)\sigma(j)<\sigma(i)
row jj comes before row ii in the random permutation

Why greedy leaks. If category cic_i has ncn_c rows, the greedy TS contains yiy_i with weight 1/(nc+a)1/(n_c+a). For a category seen once, x^i=(yi+ap)/(1+a)\hat x_i=(y_i+ap)/(1+a) — a straight line in yiy_i, so a tree can recover the target exactly. On a new row of that category the encoding is a fixed number with no link to the new target. Why leave-one-out leaks. Removing the row gives x^i=(Sc−yi+ap)/(nc−1+a)\hat x_i=(S_c-y_i+ap)/(n_c-1+a) with ScS_c the category's target sum. Within one category this is a decreasing linear function of yiy_i: the row with the highest target always gets the lowest encoding, and one split on x^\hat x separates them perfectly. Why ordered does not leak. x^i\hat x_i is a function of yjy_j for rows before ii only, so E[x^i∣yi]\mathbb E[\hat x_i\mid y_i] does not depend on yiy_i — the same property a test row has.

The boosting step for squared error. Start from F0=yˉF_0=\bar y. At round mm the loss and its derivatives for row ii are

ℓi=12(yi−Fm−1(xi))2,gi=∂ℓi∂F=−(yi−Fm−1(xi))=−ri,hi=∂2ℓi∂F2=1\ell_i=\tfrac12\big(y_i-F_{m-1}(x_i)\big)^2,\qquad g_i=\frac{\partial \ell_i}{\partial F}=-(y_i-F_{m-1}(x_i))=-r_i,\qquad h_i=\frac{\partial^2 \ell_i}{\partial F^2}=1
Fm−1(xi)F_{m-1}(x_i)
current prediction for row i
rir_i
residual: what the next tree should add
gi, hig_i,\ h_i
gradient and hessian; for squared error the hessian is 1, so Newton and gradient leaf steps coincide

A tree assigns every row in leaf jj the same value wjw_j. Adding it to the predictions and penalising large values with λ2wj2\tfrac{\lambda}{2}w_j^2, the objective for leaf jj is

Lj(w)=∑i∈j12(ri−w)2+λ2w2  ⇒  dLjdw=−∑i∈jri+(nj+λ)w=0  ⇒  wj∗=Gjnj+λ,Gj=∑i∈jri\mathcal L_j(w)=\sum_{i\in j}\tfrac12(r_i-w)^2+\tfrac{\lambda}{2}w^2\;\Rightarrow\;\frac{d\mathcal L_j}{dw}=-\sum_{i\in j}r_i+(n_j+\lambda)w=0\;\Rightarrow\; w_j^*=\frac{G_j}{n_j+\lambda},\quad G_j=\sum_{i\in j}r_i
njn_j
number of training rows in leaf j
GjG_j
sum of residuals in leaf j (= minus the gradient sum)
λ\lambda
l2_leaf_reg (default 3): shrinks small leaves toward 0

With λ=0\lambda=0 the leaf value is the mean residual; with λ>0\lambda>0 a leaf of 1 row is shrunk to a quarter of its residual (at λ=3\lambda=3), a leaf of 100 rows hardly at all. Plugging w∗w^* back in, the loss saved by leaf jj is 12Gj2/(nj+λ)\tfrac12 G_j^2/(n_j+\lambda), so a tree structure is scored by

score(tree)=∑j=12dGj2nj+λ,leaf(x)=∑k=1d2 d−k [xfk>bk]\text{score}(\text{tree})=\sum_{j=1}^{2^d}\frac{G_j^2}{n_j+\lambda},\qquad \text{leaf}(x)=\sum_{k=1}^{d}2^{\,d-k}\,\big[x_{f_k}>b_k\big]
(fk,bk)(f_k,b_k)
feature and border of level kk — one pair per level, shared by all 2k−12^{k-1} nodes on it
[xfk>bk][x_{f_k}>b_k]
bit kk of the leaf index: 1 = "no, go right"
2d2^d
number of leaves of a depth-d symmetric tree

Growing a symmetric tree is greedy over levels: at level kk, try every (feature, border) pair, apply it to all current nodes at once, and keep the pair with the highest score. A normal tree picks the best split per node; the oblivious tree picks the split that is best summed over every node of the level, even if it is useless for some of them. This is also why λ\lambda changes the structure, not just the leaf values: the denominator nj+λn_j+\lambda makes splits that isolate one or two rows score less. (CatBoost's default CPU score is Cosine, a normalised variant of this L2 score; score_function="L2" gives exactly the formula above.)

Ordered boosting. For the residuals, the same no-peeking rule applies: ri=yi−Mσ(i)−1(xi)r_i=y_i-M_{\sigma(i)-1}(x_i), where MkM_{k} is a model trained on the first kk rows of σ\sigma only. Keeping nn models would be absurd, so CatBoost keeps supporting models for prefixes of exponentially growing length (1, 2, 4, 8, …) and shares the tree structure between them; only leaf values differ. The final model's leaf values are computed normally from all rows. In Plain mode — the default in current versions — residuals are ordinary, but categorical features still get ordered target statistics.

How each hyperparameter enters the math

learning_rate η
scales every tree: Fm=Fm−1+ηfmF_m=F_{m-1}+\eta f_m. Smaller η needs more iterations but averages more trees, which generalises better. Chosen automatically from data size if not set.
depth d
2d2^d leaves and interactions of up to dd features per tree. Default 6 (64 leaves); 4–10 is the useful range.
l2_leaf_reg λ
denominator nj+λn_j+\lambda in both the leaf value and the split score; larger → smaller steps, fewer splits that isolate tiny groups.
random_strength
adds random noise (scaled by this value and shrinking over iterations) to split scores, so near-tied splits are chosen randomly — a regularizer against greedy over-commitment.
a, p (CTR priors)
pull rare categories toward the prior; a larger aa means a category needs more rows before its TS departs from pp.
one_hot_max_size
categories with at most this many levels are one-hot encoded instead of target-encoded

Worked example: rent by district and size

Twelve flats, one categorical feature (district) and one numeric feature (size in m²). Rent is in hundreds of euros per month. Assume the random permutation drawn is simply the row order shown. Mean rent p=yˉ=168/12=14p=\bar y=168/12=14, and the prior weight is a=1a=1.

#DistrictSize (m²)Rent (×100 €)
1North459
2South6014
3North7015
4East5010
5South4011
6West8022
7North5512
8East6513
9South7518
10West6019
11East357
12North8518
Training data, in permutation order. A new flat, South and 70 m², will be priced at the end.

Step 1 — ordered target statistics

  1. 1Row 1 (North) is the first North row: no history, so x^1=(0+1⋅14)/(0+1)=14\hat x_1=(0+1\cdot14)/(0+1)=14. The same holds for the first South (row 2), East (row 4) and West (row 6) rows: all get the prior, 14.
  2. 2Row 3 (North) has one earlier North row, rent 9: x^3=(9+14)/(1+1)=11.5\hat x_3=(9+14)/(1+1)=11.5.
  3. 3Row 7 (North) sees rows 1 and 3: x^7=(9+15+14)/(2+1)=12.667\hat x_7=(9+15+14)/(2+1)=12.667. Row 12 sees rows 1, 3 and 7: (9+15+12+14)/4=12.5(9+15+12+14)/4=12.5.
  4. 4East: row 8 sees row 4 → (10+14)/2=12(10+14)/2=12; row 11 sees rows 4 and 8 → (10+13+14)/3=12.333(10+13+14)/3=12.333. South: row 5 → (14+14)/2=14(14+14)/2=14; row 9 → (14+11+14)/3=13(14+11+14)/3=13. West: row 10 → (22+14)/2=18(22+14)/2=18.
  5. 5For comparison, the greedy TS gives every North row (54+14)/5=13.6(54+14)/5=13.6 and both West rows (41+14)/3=18.333(41+14)/3=18.333 — each including its own rent.
#DistrictEarlier same-district rentsOrdered TSGreedy TSLOO TSResidual r
1North—14.00013.60014.750−5
2South—14.00014.25014.3330
3North911.50013.60013.250+1
4East—14.00011.00011.333−4
5South1414.00014.25015.333−3
6West—14.00018.33316.500+8
7North9, 1512.66713.60014.000−2
8East1012.00011.00010.333−1
9South14, 1113.00014.25013.000+4
10West2218.00018.33318.000+5
11East10, 1312.33311.00012.333−7
12North9, 15, 1212.50013.60012.500+4
The three encodings and the first residuals ri=yi−14r_i=y_i-14. Correlation with rent over the 12 rows: greedy 0.769, leave-one-out 0.441, ordered 0.284. Inside North, leave-one-out orders the rows exactly backwards (rents 9, 15, 12, 18 → TS 14.75, 13.25, 14.00, 12.50).

Step 2 — one symmetric tree of depth 2 on (size, ordered TS), λ = 3, η = 0.3

  1. 1Start. F0=14F_0=14; the residuals sum to 0, so the root's score is 02/(12+3)=00^2/(12+3)=0. RMSE of F0F_0: 4.340.
  2. 2Level 1. Try every border on both features. Best: size ≤ 57.5 — left rows {1, 4, 5, 7, 11} with G=−21G=-21, n=5n=5; right rows {2, 3, 6, 8, 9, 10, 12} with G=+21G=+21, n=7n=7. Score 4415+3+4417+3=55.125+44.1=99.225\tfrac{441}{5+3}+\tfrac{441}{7+3}=55.125+44.1=99.225 (runner-up: size ≤ 52.5, 84.39).
  3. 3Level 2. Now each candidate is applied to both halves. Best: TS ≤ 12.17. On the left half it separates nothing (all five left rows have TS above 12.17, leaf 00 is empty); on the right it isolates rows 3 and 8 (G=+1−1=0G=+1-1=0) from rows {2, 6, 9, 10, 12} (G=21G=21). Score 0+55.125+0+4415+3=110.250+55.125+0+\tfrac{441}{5+3}=110.25. Next best: size ≤ 72.5 at 101.36.
  4. 4λ changed the choice. TS ≤ 12.42 would also move row 11 (TS 12.333, r=−7r=-7) into its own leaf, which is what plain least squares wants (λ=0\lambda=0). With λ=3\lambda=3 that one-row leaf is worth only 49/4=12.2549/4=12.25, and the total falls to 95.375 — below 110.25.
  5. 5Leaf values w=G/(n+λ)w=G/(n+\lambda): leaf 00: empty → 0; leaf 01: −21/8=−2.625-21/8=-2.625; leaf 10: 0/5=00/5=0; leaf 11: 21/8=2.62521/8=2.625.
  6. 6Update F1=F0+0.3 wF_1=F_0+0.3\,w: rows {1, 4, 5, 7, 11} → 13.2125, rows {3, 8} → 14, rows {2, 6, 9, 10, 12} → 14.7875. Training RMSE 4.340 → 3.720.
  7. 7New flat (South, 70 m²). At prediction time the TS uses all training rows of South: (14+11+18+1⋅14)/(3+1)=14.25(14+11+18+1\cdot14)/(3+1)=14.25. Size 70 > 57.5 → bit 1; TS 14.25 > 12.17 → bit 1; leaf 112=311_2=3. Prediction 14+0.3⋅2.625=14+0.3\cdot2.625= 14.7875, i.e. about 1,479 € per month after one tree.

Running 100 such rounds (same TS, depth 2) drives the training RMSE to 0.337 and prices the new flat at 17.87 — that is badly overfitted with only 12 rows, which is why real CatBoost redraws permutations, adds score noise and is normally stopped early on a validation set.

prior p = 14 · a = 1
rows in permutation order · bar = rent y · white tick = TS fed to the trees06121824prior p = 14N#1S#2N#3E#4S#5W#6N#7E#8S#9W#10E#11N#12TS (x) against the row's own rent (y) · correlation ρ = 0.2841012141618205101520target statistic →
row #7 · North · rent 12 · uses rows #1, #3
TS = (9 + 15 + 1·14) / (2 + 1) = 12.667 ← only rows earlier in the permutation
permutation
fig.The worked example, live. Tab 1: each column is a flat in permutation order; the bar is its rent and the white tick the target statistic the trees will see. Press play to stream the rows: in ordered mode later rows of the same district are greyed out and unused. Switch to greedy or leave-one-out and watch the ticks jump and the correlation with rent rise from 0.284 to 0.769 or 0.441; reshuffle the permutation or change the prior weight a. Tab 2: the depth-2 symmetric tree on (size, TS) with λ = 3 — size ≤ 57.5, then TS ≤ 12.17 drawn across the whole plane, leaves −2.625 / 0 / 2.625, RMSE 4.340 → 3.720, new flat 14.7875. Drag λ to 0 and the second split moves to TS ≤ 12.42 to isolate row 11.

The algorithm, step by step

CatBoost regression (Plain mode, with ordered TS)

  1. 1Quantize every numeric feature into at most border_count (254) borders. Categories with at most one_hot_max_size levels are one-hot encoded; the rest are left for target statistics.
  2. 2Draw permutations σ1,…,σs\sigma_1,\dots,\sigma_s of the training rows. Set F0=yˉF_0=\bar y (boost_from_average).
  3. 3For each round m=1…Mm=1\dots M: pick a permutation; compute the ordered TS of every categorical feature (and of feature combinations built from splits already used) under it; compute residuals ri=yi−Fm−1(xi)r_i=y_i-F_{m-1}(x_i) — or, in Ordered mode, with the supporting model that has not seen row ii.
  4. 4Grow a symmetric tree level by level: for each (feature, border) compute ∑jGj2/(nj+λ)\sum_j G_j^2/(n_j+\lambda) over all 2k2^k nodes, perturb the scores with random_strength noise, keep the best pair.
  5. 5Leaf values wj=Gj/(nj+λ)w_j=G_j/(n_j+\lambda) (for losses other than RMSE: a Newton step with the hessians). Update Fm=Fm−1+ηfmF_m=F_{m-1}+\eta f_m.
  6. 6Stop after iterations, or earlier when the validation metric (RMSE, MAE, R²) has not improved for early_stopping_rounds.
  7. 7Predict: encode each category with statistics over all training rows, compute each tree's dd-bit leaf index, sum the leaf values.

Complexity

Per level
one histogram pass over nn rows for each of pp features, then O(2k⋅p⋅B)O(2^k\cdot p\cdot B) to score BB borders
Per tree
about O(d⋅n⋅p)O(d\cdot n\cdot p); ordered TS add O(n)O(n) per categorical feature per permutation
Ordered boosting
a constant-factor slowdown (supporting models on growing prefixes), worth it mainly for small and medium datasets
Prediction
O(M⋅d)O(M\cdot d) comparisons with no branching; vectorizes well on CPU and GPU

Variants and tuning

loss_functionLeaf value aims atUse when
RMSE (default)mean of the residuals, shrunk by λerrors are roughly symmetric; you report RMSE or R²
MAE, Quantile:alpha=qmedian / q-quantile of the residualsoutliers or prediction intervals (fit q = 0.1 and 0.9)
Huber:delta=δmean near zero, median for large residualsmostly Gaussian noise with a few gross errors
Poisson, Tweedie:variance_power=ρlog-mean of counts / non-negative amountscounts, insurance claims, sales with many zeros
MAPErelative errortargets of very different scales where % error matters
RMSEWithUncertaintya mean and a variance per rowyou need a predictive standard deviation
Regression losses CatBoost supports out of the box, and when to choose them.

Practical tuning order

1. Pass categories raw
list them in cat_features; do not one-hot or target-encode yourself — that throws away the ordered statistics
2. Early stopping
set a large iterations, an eval_set, and early_stopping_rounds (e.g. 100); the auto learning rate is usually a decent start
3. depth
try 4, 6, 8. Each extra level doubles the leaves
4. l2_leaf_reg
try 1, 3, 10, 30 — more for small or noisy data
5. Randomness
random_strength, bagging_temperature (Bayesian bootstrap) or subsample (MVS / Bernoulli)
6. boosting_type
Ordered for small data (thousands of rows), Plain for large; compare on validation

In code

python
1import numpy as np
2
3def ordered_ts(cat, y, perm, a=1.0, prior=None):
4 """Ordered target statistic: each row sees only the rows before it in perm."""
5 prior = y.mean() if prior is None else prior
6 s, k, ts = {}, {}, np.empty(len(y))
7 for i in perm:
8 c = cat[i]
9 ts[i] = (s.get(c, 0.0) + a * prior) / (k.get(c, 0) + a)
10 s[c] = s.get(c, 0.0) + y[i]
11 k[c] = k.get(c, 0) + 1
12 return ts, s, k, prior
13
14def full_ts(cat, s, k, prior, a=1.0):
15 """For new rows: statistics over the whole training set."""
16 return np.array([(s.get(c, 0.0) + a * prior) / (k.get(c, 0) + a) for c in cat])
17
18def leaf_index(X, splits):
19 """Oblivious tree: the leaf is a binary number, one bit per level."""
20 idx = np.zeros(len(X), dtype=int)
21 for j, t in splits:
22 idx = 2 * idx + (X[:, j] > t)
23 return idx
24
25def fit_oblivious(X, r, depth, lam):
26 splits = []
27 for _ in range(depth):
28 best = (-np.inf, None)
29 for j in range(X.shape[1]):
30 v = np.unique(X[:, j])
31 for t in (v[:-1] + v[1:]) / 2:
32 idx = leaf_index(X, splits + [(j, t)])
33 G = np.bincount(idx, r, minlength=2 ** (len(splits) + 1))
34 n = np.bincount(idx, minlength=2 ** (len(splits) + 1))
35 score = (G ** 2 / np.maximum(n + lam, 1e-9)).sum() # same split for every node
36 if score > best[0]:
37 best = (score, (j, t))
38 splits.append(best[1])
39 idx = leaf_index(X, splits)
40 G = np.bincount(idx, r, minlength=2 ** depth)
41 n = np.bincount(idx, minlength=2 ** depth)
42 return splits, G / np.maximum(n + lam, 1e-9) # leaf values (0 if empty)
43
44def fit(X, y, rounds=100, depth=2, eta=0.3, lam=3.0):
45 F0 = y.mean()
46 F = np.full(len(y), F0)
47 trees = []
48 for _ in range(rounds):
49 splits, w = fit_oblivious(X, y - F, depth, lam) # residual = -gradient of squared error
50 F += eta * w[leaf_index(X, splits)]
51 trees.append((splits, w))
52 return F0, trees
53
54def predict(model, X, eta=0.3):
55 F0, trees = model
56 return F0 + eta * sum(w[leaf_index(X, s)] for s, w in trees)
57
58district = np.array("North South North East South West North East South West East North".split())
59size = np.array([45, 60, 70, 50, 40, 80, 55, 65, 75, 60, 35, 85], float)
60rent = np.array([9, 14, 15, 10, 11, 22, 12, 13, 18, 19, 7, 18], float)
61
62ts, s, k, prior = ordered_ts(district, rent, perm=np.arange(12))
63print("ordered TS:", np.round(ts, 3))
64X = np.c_[size, ts]
65Xq = np.c_[[70.0], full_ts(np.array(["South"]), s, k, prior)]
66
67one = fit(X, rent, rounds=1)
68print("tree 1 splits:", [(int(j), round(float(t), 3)) for j, t in one[1][0][0]])
69print("tree 1 leaves:", np.round(one[1][0][1], 4))
70print("after 1 tree, South 70 m2 ->", round(float(predict(one, Xq)[0]), 4))
71
72model = fit(X, rent, rounds=100)
73rmse = np.sqrt(np.mean((predict(model, X) - rent) ** 2))
74print("100 trees: train RMSE", round(float(rmse), 3), "| South 70 m2 ->", round(float(predict(model, Xq)[0]), 3))
From scratch: ordered target statistics + symmetric trees fitted to squared-error residuals. Prints the hand-computed tree and prediction.
text
1ordered TS: [14. 14. 11.5 14. 14. 14. 12.667 12. 13. 18.
2 12.333 12.5 ]
3tree 1 splits: [(0, 57.5), (1, 12.167)]
4tree 1 leaves: [ 0. -2.625 0. 2.625]
5after 1 tree, South 70 m2 -> 14.7875
6100 trees: train RMSE 0.337 | South 70 m2 -> 17.87
Output of the script above.
python
1import numpy as np, pandas as pd
2from catboost import CatBoostRegressor
3
4df = pd.DataFrame({
5 "district": "North South North East South West North East South West East North".split(),
6 "size": [45, 60, 70, 50, 40, 80, 55, 65, 75, 60, 35, 85],
7 "rent": [9, 14, 15, 10, 11, 22, 12, 13, 18, 19, 7, 18],
8})
9new = pd.DataFrame({"district": ["South"], "size": [70]})
10
11# 1) Reproduce the hand-computed tree: feed the ordered TS as a number, one tree, L2 score.
12ts = [14, 14, 11.5, 14, 14, 14, 38 / 3, 12, 13, 18, 37 / 3, 12.5]
13X = pd.DataFrame({"size": df["size"], "ts": ts})
14one = CatBoostRegressor(iterations=1, depth=2, learning_rate=0.3, l2_leaf_reg=3,
15 score_function="L2", random_strength=0, bootstrap_type="No", verbose=0)
16one.fit(X, df.rent)
17print("1 tree, train predictions:", np.round(one.predict(X), 4))
18print("1 tree, South 70 m2:", round(float(one.predict(pd.DataFrame({"size": [70], "ts": [14.25]}))[0]), 4))
19
20# 2) The normal way: hand CatBoost the raw category and let it build its own statistics.
21for mode in ["Plain", "Ordered"]:
22 m = CatBoostRegressor(iterations=300, depth=2, learning_rate=0.1, loss_function="RMSE",
23 boosting_type=mode, random_seed=0, verbose=0)
24 m.fit(df[["district", "size"]], df.rent, cat_features=["district"])
25 rmse = np.sqrt(np.mean((m.predict(df[["district", "size"]]) - df.rent) ** 2))
26 print(mode, "| train RMSE", round(float(rmse), 3), "| South 70 m2 ->", round(float(m.predict(new)[0]), 2),
27 "| importances", {k: round(float(v), 1) for k, v in zip(["district", "size"], m.get_feature_importance())})
28print("default regression CTR:", m.get_all_params()["simple_ctr"][0].split(":")[0])
With the catboost library: first reproducing the hand-built tree exactly, then the normal workflow with the raw category.
text
11 tree, train predictions: [13.2125 14.7875 14. 13.2125 13.2125 14.7875 13.2125 14. 14.7875
2 14.7875 13.2125 14.7875]
31 tree, South 70 m2: 14.7875
4Plain | train RMSE 0.411 | South 70 m2 -> 15.38 | importances {'district': 27.5, 'size': 72.5}
5Ordered | train RMSE 0.641 | South 70 m2 -> 15.65 | importances {'district': 23.8, 'size': 76.2}
6default regression CTR: Borders
Output (catboost 1.2.10). The library's own CTRs (target borders + counts) and random permutations give different numbers from our hand-built TS.

Use when

  • +The data has categorical columns, especially high-cardinality ones (ids, postcodes, SKUs), and you want them used without leakage.
  • +You want strong results with default settings and little tuning — symmetric trees and ordered TS are robust regularizers.
  • +Prediction latency matters: oblivious trees evaluate as branch-free bit operations.
  • +The dataset is small or medium, where prediction shift hurts most and Ordered boosting helps.

Avoid when

  • −All features are numeric and the data is huge: LightGBM is usually faster to train at similar accuracy.
  • −The target depends smoothly or linearly on the inputs, or you must extrapolate beyond the training range — tree sums are flat outside it.
  • −You need a single readable model: hundreds of trees plus stored category statistics are not interpretable rules.
  • −Rows are time-ordered and the future must not inform the past: a random permutation is the wrong order (use has_time=True).

Where it's used

Price and demand forecasting

Retail and travel models with store, product, region and calendar categories; CatBoost was developed at Yandex for this kind of heterogeneous tabular data.

Real estate valuation

Rent or sale price from district, building type and numeric size — the worked example at production scale.

Insurance and risk amounts

Claim severity with Tweedie or RMSE loss on policy categories (vehicle make, region, occupation).

Delivery and arrival times

ETA regression with courier, restaurant and zone ids, where high-cardinality ids carry most of the signal.

Check yourself

In permutation order, a row from East is preceded by two East rows with rents 10 and 13. With prior p = 14 and a = 1, its ordered target statistic is…

Check yourself

A symmetric (oblivious) tree of depth 6 has how many leaves, and how many distinct split conditions?

Check yourself

A leaf holds 5 rows whose residuals sum to 21. With l2_leaf_reg = 3 and learning rate 0.3, how much does this tree add to those rows' predictions?

Check yourself

Why does leave-one-out target encoding still leak the target?

Remember

  • ✓Target encoding that includes a row's own target leaks it (prediction shift); leave-one-out leaks it in reverse.
  • ✓Ordered TS: encode row ii from the rows before it in a random permutation, smoothed by ( ∑y+ap)/(k+a)(\,\sum y+ap)/(k+a); new rows use all training rows.
  • ✓Ordered boosting applies the same idea to residuals; Plain mode keeps ordered TS but ordinary residuals.
  • ✓Symmetric trees share one (feature, border) per level: dd tests, 2d2^d leaves, a dd-bit index, chosen by ∑jGj2/(nj+λ)\sum_j G_j^2/(n_j+\lambda).
  • ✓For squared error, leaf value w=G/(n+λ)w=G/(n+\lambda) and update F←F+ηwF\leftarrow F+\eta w; λ shrinks small leaves and changes which splits win.
  • ✓Pass raw categories in cat_features, use early stopping, and tune depth and l2_leaf_reg first.
← back to ML Algorithms roadmap