Definition
CatBoost regression predicts a number as a constant plus a sum of small trees, , like any gradient-boosted model. What is special is what the trees see and how they are shaped: every categorical feature is replaced by an ordered target statistic — the average target of the earlier rows with the same category in a random permutation — and every tree is symmetric (oblivious): one question per level, shared by all nodes on that level.
At a glance
- Task
- Regression: RMSE (squared error) by default; also MAE, Quantile, Huber, MAPE, Poisson, Tweedie, RMSEWithUncertainty
- Input → output
- numeric and raw categorical features → real number
- Learns
- oblivious trees: (feature, border) pairs and leaf values each, plus per-category target statistics stored with the model
- Training
- features quantized into borders; per tree about for features — slower per tree than LightGBM on numeric data, competitive once categories are involved
- Prediction
- : each tree is comparisons that form a -bit leaf index — branch-free and very fast
- Key hyperparameters
iterations,learning_rate(η),depth(default 6),l2_leaf_reg(λ, default 3),random_strength,bagging_temperature/subsample,one_hot_max_size,boosting_type(Plain / Ordered)- Assumes
- a differentiable loss; no scaling; categories need no manual encoding; the fit is piecewise constant and axis-aligned, so no extrapolation beyond the training targets
The problem
Tabular regression data is full of categories: district, product id, store, device, postcode. Trees need numbers. One-hot encoding works for a handful of levels but explodes for thousands, and each indicator column carries almost no signal on its own. The popular alternative is target encoding: replace each category by the mean target of its rows. For a rent model, "district = West" becomes "average West rent".
Target encoding computed on the training rows leaks the target. If row 's own rent is part of the average that replaces its district, the encoded feature contains a piece of . For a category with a single row the encoding is a rescaled . The trees happily split on it, the training error drops, and the model fails on new data, where the encoding has no such piece. The CatBoost paper (Prokhorenkova et al., 2018) calls this prediction shift: the distribution of a feature at training time differs from its distribution at test time.
Boosting itself has the same flaw in a quieter form. The residuals that tree is fitted to are computed by a model that was trained on those same rows, so they are systematically smaller than the residuals the model will make on fresh rows. CatBoost's answer to both problems is the same trick: order the data randomly and let each row use only the rows before it.
| Model | Train RMSE | Test RMSE | corr(encoding, y) on train |
|---|---|---|---|
| GBM on only | 0.801 | 1.041 | — |
| GBM + greedy target encoding | 0.656 | 1.192 | 0.621 |
| GBM + ordered target statistic | 0.774 | 1.053 | 0.052 |
| CatBoost, Plain (raw category) | 0.919 | 1.012 | — |
| CatBoost, Ordered (raw category) | 0.927 | 1.008 | — |
The greedy encoding looks like the best model on the training set and is the worst on the test set: a pure-noise category correlates 0.621 with the target once each row's own target is inside its encoding. The ordered statistic is honest (0.052), and CatBoost's own pipeline does best of all.
The idea
Imagine the training rows arriving one at a time, in a random order, like a stream. When row arrives, you may summarise only what you have already seen. Its district encoding is the average rent of the earlier rows from the same district, smoothed toward a prior. Row 's own rent was never used, so the feature is exactly as informative as it will be for a brand-new flat. The first rows of each district get mostly the prior (noisy but unbiased); later rows get sharper estimates.
The trees are deliberately simple: a symmetric tree of depth asks the same question of every node on a level, so it is really a list of yes/no tests and a table of leaf values. That is a strong regularizer (one split must be good for every branch at once), and it turns prediction into computing a -bit binary number.
Vocabulary
- Target statistic (TS / CTR)
- a number that replaces a category value, computed from the targets of rows with that value — e.g. a smoothed mean rent. CatBoost's docs call them CTRs.
- Prior p, weight a
- the value a TS falls back to when a category has few rows, and how many "virtual rows" it is worth. Here is the mean target and .
- Greedy TS
- a TS computed from all training rows of the category, including the row being encoded — leaks the target
- Leave-one-out TS
- all rows of the category except the row itself — still leaks, in reverse
- Permutation σ
- a random ordering of the training rows; is row 's position. CatBoost draws several.
- Ordered TS
- the TS of row computed only from rows that come before it in
- Prediction shift
- the gap between a feature's (or residual's) distribution on training rows and on new rows, caused by reusing a row's own target
- Ordered boosting
- computing row 's residual with a model that never saw row (only rows before it in ) —
boosting_type="Ordered" - Oblivious (symmetric) tree
- a tree in which every node at depth uses the same feature and border; tests, leaves
- Border
- a candidate threshold. Features are quantized into at most 254 borders by default (
border_count). - Feature combination
- a new categorical feature formed by joining categories (e.g. district × building type), encoded with its own TS
The math
Target statistics. For a categorical feature with value in row , the three candidate encodings differ only in which rows they average:
- the number that replaces row 's category
- 1 if row has the same category as row , else 0
- target (here: rent) of row j
- prior: the value an unseen category gets (here the mean target)
- prior weight: how many pseudo-rows the prior is worth
- row comes before row in the random permutation
Why greedy leaks. If category has rows, the greedy TS contains with weight . For a category seen once, — a straight line in , so a tree can recover the target exactly. On a new row of that category the encoding is a fixed number with no link to the new target. Why leave-one-out leaks. Removing the row gives with the category's target sum. Within one category this is a decreasing linear function of : the row with the highest target always gets the lowest encoding, and one split on separates them perfectly. Why ordered does not leak. is a function of for rows before only, so does not depend on — the same property a test row has.
The boosting step for squared error. Start from . At round the loss and its derivatives for row are
- current prediction for row i
- residual: what the next tree should add
- gradient and hessian; for squared error the hessian is 1, so Newton and gradient leaf steps coincide
A tree assigns every row in leaf the same value . Adding it to the predictions and penalising large values with , the objective for leaf is
- number of training rows in leaf j
- sum of residuals in leaf j (= minus the gradient sum)
l2_leaf_reg(default 3): shrinks small leaves toward 0
With the leaf value is the mean residual; with a leaf of 1 row is shrunk to a quarter of its residual (at ), a leaf of 100 rows hardly at all. Plugging back in, the loss saved by leaf is , so a tree structure is scored by
- feature and border of level — one pair per level, shared by all nodes on it
- bit of the leaf index: 1 = "no, go right"
- number of leaves of a depth-d symmetric tree
Growing a symmetric tree is greedy over levels: at level , try every (feature, border) pair, apply it to all current nodes at once, and keep the pair with the highest score. A normal tree picks the best split per node; the oblivious tree picks the split that is best summed over every node of the level, even if it is useless for some of them. This is also why changes the structure, not just the leaf values: the denominator makes splits that isolate one or two rows score less. (CatBoost's default CPU score is Cosine, a normalised variant of this L2 score; score_function="L2" gives exactly the formula above.)
Ordered boosting. For the residuals, the same no-peeking rule applies: , where is a model trained on the first rows of only. Keeping models would be absurd, so CatBoost keeps supporting models for prefixes of exponentially growing length (1, 2, 4, 8, …) and shares the tree structure between them; only leaf values differ. The final model's leaf values are computed normally from all rows. In Plain mode — the default in current versions — residuals are ordinary, but categorical features still get ordered target statistics.
How each hyperparameter enters the math
- learning_rate η
- scales every tree: . Smaller η needs more iterations but averages more trees, which generalises better. Chosen automatically from data size if not set.
- depth d
- leaves and interactions of up to features per tree. Default 6 (64 leaves); 4–10 is the useful range.
- l2_leaf_reg λ
- denominator in both the leaf value and the split score; larger → smaller steps, fewer splits that isolate tiny groups.
- random_strength
- adds random noise (scaled by this value and shrinking over iterations) to split scores, so near-tied splits are chosen randomly — a regularizer against greedy over-commitment.
- a, p (CTR priors)
- pull rare categories toward the prior; a larger means a category needs more rows before its TS departs from .
- one_hot_max_size
- categories with at most this many levels are one-hot encoded instead of target-encoded
Worked example: rent by district and size
Twelve flats, one categorical feature (district) and one numeric feature (size in m²). Rent is in hundreds of euros per month. Assume the random permutation drawn is simply the row order shown. Mean rent , and the prior weight is .
| # | District | Size (m²) | Rent (×100 €) |
|---|---|---|---|
| 1 | North | 45 | 9 |
| 2 | South | 60 | 14 |
| 3 | North | 70 | 15 |
| 4 | East | 50 | 10 |
| 5 | South | 40 | 11 |
| 6 | West | 80 | 22 |
| 7 | North | 55 | 12 |
| 8 | East | 65 | 13 |
| 9 | South | 75 | 18 |
| 10 | West | 60 | 19 |
| 11 | East | 35 | 7 |
| 12 | North | 85 | 18 |
Step 1 — ordered target statistics
- 1Row 1 (North) is the first North row: no history, so . The same holds for the first South (row 2), East (row 4) and West (row 6) rows: all get the prior, 14.
- 2Row 3 (North) has one earlier North row, rent 9: .
- 3Row 7 (North) sees rows 1 and 3: . Row 12 sees rows 1, 3 and 7: .
- 4East: row 8 sees row 4 → ; row 11 sees rows 4 and 8 → . South: row 5 → ; row 9 → . West: row 10 → .
- 5For comparison, the greedy TS gives every North row and both West rows — each including its own rent.
| # | District | Earlier same-district rents | Ordered TS | Greedy TS | LOO TS | Residual r |
|---|---|---|---|---|---|---|
| 1 | North | — | 14.000 | 13.600 | 14.750 | −5 |
| 2 | South | — | 14.000 | 14.250 | 14.333 | 0 |
| 3 | North | 9 | 11.500 | 13.600 | 13.250 | +1 |
| 4 | East | — | 14.000 | 11.000 | 11.333 | −4 |
| 5 | South | 14 | 14.000 | 14.250 | 15.333 | −3 |
| 6 | West | — | 14.000 | 18.333 | 16.500 | +8 |
| 7 | North | 9, 15 | 12.667 | 13.600 | 14.000 | −2 |
| 8 | East | 10 | 12.000 | 11.000 | 10.333 | −1 |
| 9 | South | 14, 11 | 13.000 | 14.250 | 13.000 | +4 |
| 10 | West | 22 | 18.000 | 18.333 | 18.000 | +5 |
| 11 | East | 10, 13 | 12.333 | 11.000 | 12.333 | −7 |
| 12 | North | 9, 15, 12 | 12.500 | 13.600 | 12.500 | +4 |
Step 2 — one symmetric tree of depth 2 on (size, ordered TS), λ = 3, η = 0.3
- 1Start. ; the residuals sum to 0, so the root's score is . RMSE of : 4.340.
- 2Level 1. Try every border on both features. Best: size ≤ 57.5 — left rows {1, 4, 5, 7, 11} with , ; right rows {2, 3, 6, 8, 9, 10, 12} with , . Score (runner-up: size ≤ 52.5, 84.39).
- 3Level 2. Now each candidate is applied to both halves. Best: TS ≤ 12.17. On the left half it separates nothing (all five left rows have TS above 12.17, leaf 00 is empty); on the right it isolates rows 3 and 8 () from rows {2, 6, 9, 10, 12} (). Score . Next best: size ≤ 72.5 at 101.36.
- 4λ changed the choice. TS ≤ 12.42 would also move row 11 (TS 12.333, ) into its own leaf, which is what plain least squares wants (). With that one-row leaf is worth only , and the total falls to 95.375 — below 110.25.
- 5Leaf values : leaf 00: empty → 0; leaf 01: ; leaf 10: ; leaf 11: .
- 6Update : rows {1, 4, 5, 7, 11} → 13.2125, rows {3, 8} → 14, rows {2, 6, 9, 10, 12} → 14.7875. Training RMSE 4.340 → 3.720.
- 7New flat (South, 70 m²). At prediction time the TS uses all training rows of South: . Size 70 > 57.5 → bit 1; TS 14.25 > 12.17 → bit 1; leaf . Prediction 14.7875, i.e. about 1,479 € per month after one tree.
Running 100 such rounds (same TS, depth 2) drives the training RMSE to 0.337 and prices the new flat at 17.87 — that is badly overfitted with only 12 rows, which is why real CatBoost redraws permutations, adds score noise and is normally stopped early on a validation set.
The algorithm, step by step
CatBoost regression (Plain mode, with ordered TS)
- 1Quantize every numeric feature into at most
border_count(254) borders. Categories with at mostone_hot_max_sizelevels are one-hot encoded; the rest are left for target statistics. - 2Draw permutations of the training rows. Set (
boost_from_average). - 3For each round : pick a permutation; compute the ordered TS of every categorical feature (and of feature combinations built from splits already used) under it; compute residuals — or, in
Orderedmode, with the supporting model that has not seen row . - 4Grow a symmetric tree level by level: for each (feature, border) compute over all nodes, perturb the scores with
random_strengthnoise, keep the best pair. - 5Leaf values (for losses other than RMSE: a Newton step with the hessians). Update .
- 6Stop after
iterations, or earlier when the validation metric (RMSE, MAE, R²) has not improved forearly_stopping_rounds. - 7Predict: encode each category with statistics over all training rows, compute each tree's -bit leaf index, sum the leaf values.
Complexity
- Per level
- one histogram pass over rows for each of features, then to score borders
- Per tree
- about ; ordered TS add per categorical feature per permutation
- Ordered boosting
- a constant-factor slowdown (supporting models on growing prefixes), worth it mainly for small and medium datasets
- Prediction
- comparisons with no branching; vectorizes well on CPU and GPU
Variants and tuning
| loss_function | Leaf value aims at | Use when |
|---|---|---|
RMSE (default) | mean of the residuals, shrunk by λ | errors are roughly symmetric; you report RMSE or R² |
MAE, Quantile:alpha=q | median / q-quantile of the residuals | outliers or prediction intervals (fit q = 0.1 and 0.9) |
Huber:delta=δ | mean near zero, median for large residuals | mostly Gaussian noise with a few gross errors |
Poisson, Tweedie:variance_power=ρ | log-mean of counts / non-negative amounts | counts, insurance claims, sales with many zeros |
MAPE | relative error | targets of very different scales where % error matters |
RMSEWithUncertainty | a mean and a variance per row | you need a predictive standard deviation |
Practical tuning order
- 1. Pass categories raw
- list them in
cat_features; do not one-hot or target-encode yourself — that throws away the ordered statistics - 2. Early stopping
- set a large
iterations, aneval_set, andearly_stopping_rounds(e.g. 100); the auto learning rate is usually a decent start - 3. depth
- try 4, 6, 8. Each extra level doubles the leaves
- 4. l2_leaf_reg
- try 1, 3, 10, 30 — more for small or noisy data
- 5. Randomness
random_strength,bagging_temperature(Bayesian bootstrap) orsubsample(MVS / Bernoulli)- 6. boosting_type
Orderedfor small data (thousands of rows),Plainfor large; compare on validation
In code
1import numpy as np23def ordered_ts(cat, y, perm, a=1.0, prior=None):4 """Ordered target statistic: each row sees only the rows before it in perm."""5 prior = y.mean() if prior is None else prior6 s, k, ts = {}, {}, np.empty(len(y))7 for i in perm:8 c = cat[i]9 ts[i] = (s.get(c, 0.0) + a * prior) / (k.get(c, 0) + a)10 s[c] = s.get(c, 0.0) + y[i]11 k[c] = k.get(c, 0) + 112 return ts, s, k, prior1314def full_ts(cat, s, k, prior, a=1.0):15 """For new rows: statistics over the whole training set."""16 return np.array([(s.get(c, 0.0) + a * prior) / (k.get(c, 0) + a) for c in cat])1718def leaf_index(X, splits):19 """Oblivious tree: the leaf is a binary number, one bit per level."""20 idx = np.zeros(len(X), dtype=int)21 for j, t in splits:22 idx = 2 * idx + (X[:, j] > t)23 return idx2425def fit_oblivious(X, r, depth, lam):26 splits = []27 for _ in range(depth):28 best = (-np.inf, None)29 for j in range(X.shape[1]):30 v = np.unique(X[:, j])31 for t in (v[:-1] + v[1:]) / 2:32 idx = leaf_index(X, splits + [(j, t)])33 G = np.bincount(idx, r, minlength=2 ** (len(splits) + 1))34 n = np.bincount(idx, minlength=2 ** (len(splits) + 1))35 score = (G ** 2 / np.maximum(n + lam, 1e-9)).sum() # same split for every node36 if score > best[0]:37 best = (score, (j, t))38 splits.append(best[1])39 idx = leaf_index(X, splits)40 G = np.bincount(idx, r, minlength=2 ** depth)41 n = np.bincount(idx, minlength=2 ** depth)42 return splits, G / np.maximum(n + lam, 1e-9) # leaf values (0 if empty)4344def fit(X, y, rounds=100, depth=2, eta=0.3, lam=3.0):45 F0 = y.mean()46 F = np.full(len(y), F0)47 trees = []48 for _ in range(rounds):49 splits, w = fit_oblivious(X, y - F, depth, lam) # residual = -gradient of squared error50 F += eta * w[leaf_index(X, splits)]51 trees.append((splits, w))52 return F0, trees5354def predict(model, X, eta=0.3):55 F0, trees = model56 return F0 + eta * sum(w[leaf_index(X, s)] for s, w in trees)5758district = np.array("North South North East South West North East South West East North".split())59size = np.array([45, 60, 70, 50, 40, 80, 55, 65, 75, 60, 35, 85], float)60rent = np.array([9, 14, 15, 10, 11, 22, 12, 13, 18, 19, 7, 18], float)6162ts, s, k, prior = ordered_ts(district, rent, perm=np.arange(12))63print("ordered TS:", np.round(ts, 3))64X = np.c_[size, ts]65Xq = np.c_[[70.0], full_ts(np.array(["South"]), s, k, prior)]6667one = fit(X, rent, rounds=1)68print("tree 1 splits:", [(int(j), round(float(t), 3)) for j, t in one[1][0][0]])69print("tree 1 leaves:", np.round(one[1][0][1], 4))70print("after 1 tree, South 70 m2 ->", round(float(predict(one, Xq)[0]), 4))7172model = fit(X, rent, rounds=100)73rmse = np.sqrt(np.mean((predict(model, X) - rent) ** 2))74print("100 trees: train RMSE", round(float(rmse), 3), "| South 70 m2 ->", round(float(predict(model, Xq)[0]), 3))1ordered TS: [14. 14. 11.5 14. 14. 14. 12.667 12. 13. 18.2 12.333 12.5 ]3tree 1 splits: [(0, 57.5), (1, 12.167)]4tree 1 leaves: [ 0. -2.625 0. 2.625]5after 1 tree, South 70 m2 -> 14.78756100 trees: train RMSE 0.337 | South 70 m2 -> 17.871import numpy as np, pandas as pd2from catboost import CatBoostRegressor34df = pd.DataFrame({5 "district": "North South North East South West North East South West East North".split(),6 "size": [45, 60, 70, 50, 40, 80, 55, 65, 75, 60, 35, 85],7 "rent": [9, 14, 15, 10, 11, 22, 12, 13, 18, 19, 7, 18],8})9new = pd.DataFrame({"district": ["South"], "size": [70]})1011# 1) Reproduce the hand-computed tree: feed the ordered TS as a number, one tree, L2 score.12ts = [14, 14, 11.5, 14, 14, 14, 38 / 3, 12, 13, 18, 37 / 3, 12.5]13X = pd.DataFrame({"size": df["size"], "ts": ts})14one = CatBoostRegressor(iterations=1, depth=2, learning_rate=0.3, l2_leaf_reg=3,15 score_function="L2", random_strength=0, bootstrap_type="No", verbose=0)16one.fit(X, df.rent)17print("1 tree, train predictions:", np.round(one.predict(X), 4))18print("1 tree, South 70 m2:", round(float(one.predict(pd.DataFrame({"size": [70], "ts": [14.25]}))[0]), 4))1920# 2) The normal way: hand CatBoost the raw category and let it build its own statistics.21for mode in ["Plain", "Ordered"]:22 m = CatBoostRegressor(iterations=300, depth=2, learning_rate=0.1, loss_function="RMSE",23 boosting_type=mode, random_seed=0, verbose=0)24 m.fit(df[["district", "size"]], df.rent, cat_features=["district"])25 rmse = np.sqrt(np.mean((m.predict(df[["district", "size"]]) - df.rent) ** 2))26 print(mode, "| train RMSE", round(float(rmse), 3), "| South 70 m2 ->", round(float(m.predict(new)[0]), 2),27 "| importances", {k: round(float(v), 1) for k, v in zip(["district", "size"], m.get_feature_importance())})28print("default regression CTR:", m.get_all_params()["simple_ctr"][0].split(":")[0])11 tree, train predictions: [13.2125 14.7875 14. 13.2125 13.2125 14.7875 13.2125 14. 14.78752 14.7875 13.2125 14.7875]31 tree, South 70 m2: 14.78754Plain | train RMSE 0.411 | South 70 m2 -> 15.38 | importances {'district': 27.5, 'size': 72.5}5Ordered | train RMSE 0.641 | South 70 m2 -> 15.65 | importances {'district': 23.8, 'size': 76.2}6default regression CTR: BordersUse when
- +The data has categorical columns, especially high-cardinality ones (ids, postcodes, SKUs), and you want them used without leakage.
- +You want strong results with default settings and little tuning — symmetric trees and ordered TS are robust regularizers.
- +Prediction latency matters: oblivious trees evaluate as branch-free bit operations.
- +The dataset is small or medium, where prediction shift hurts most and
Orderedboosting helps.
Avoid when
- −All features are numeric and the data is huge: LightGBM is usually faster to train at similar accuracy.
- −The target depends smoothly or linearly on the inputs, or you must extrapolate beyond the training range — tree sums are flat outside it.
- −You need a single readable model: hundreds of trees plus stored category statistics are not interpretable rules.
- −Rows are time-ordered and the future must not inform the past: a random permutation is the wrong order (use
has_time=True).
Where it's used
Price and demand forecasting
Retail and travel models with store, product, region and calendar categories; CatBoost was developed at Yandex for this kind of heterogeneous tabular data.
Real estate valuation
Rent or sale price from district, building type and numeric size — the worked example at production scale.
Insurance and risk amounts
Claim severity with Tweedie or RMSE loss on policy categories (vehicle make, region, occupation).
Delivery and arrival times
ETA regression with courier, restaurant and zone ids, where high-cardinality ids carry most of the signal.
Check yourself
In permutation order, a row from East is preceded by two East rows with rents 10 and 13. With prior p = 14 and a = 1, its ordered target statistic is…
Check yourself
A symmetric (oblivious) tree of depth 6 has how many leaves, and how many distinct split conditions?
Check yourself
A leaf holds 5 rows whose residuals sum to 21. With l2_leaf_reg = 3 and learning rate 0.3, how much does this tree add to those rows' predictions?
Check yourself
Why does leave-one-out target encoding still leak the target?
Remember
- ✓Target encoding that includes a row's own target leaks it (prediction shift); leave-one-out leaks it in reverse.
- ✓Ordered TS: encode row from the rows before it in a random permutation, smoothed by ; new rows use all training rows.
- ✓Ordered boosting applies the same idea to residuals; Plain mode keeps ordered TS but ordinary residuals.
- ✓Symmetric trees share one (feature, border) per level: tests, leaves, a -bit index, chosen by .
- ✓For squared error, leaf value and update ; λ shrinks small leaves and changes which splits win.
- ✓Pass raw categories in
cat_features, use early stopping, and tune depth and l2_leaf_reg first.