# Decision tree regression

> Split to maximize variance reduction; each leaf predicts the mean of its training targets, giving a step function.

Source: https://www.mlfoundry.in/learn/decision-tree-regression  
Section: ML Algorithms › Supervised · Regression  
Level: core · 17 min read · updated 2026-10-07

**A regression tree:** predicts a **number** by asking yes/no questions of the form *is $x_j \le t$?* until it reaches a leaf, and then outputs the **mean target** of the training points that ended up in that leaf. Splits are chosen greedily to make the squared error inside the children as small as possible, so the fitted function is a piecewise-constant **step function**.

- **Task:** Regression (numeric target)
- **Input → output:** feature vector $x$ → the mean $\bar y$ stored in the leaf $x$ lands in
- **Learns:** a feature + threshold per internal node, one constant per leaf
- **Split criterion:** squared error: minimize $\text{SSE}_L+\text{SSE}_R$ (equivalently, maximize variance reduction)
- **Training cost:** $O(d\,n\log n)$ to sort each feature, then $O(d\,n)$ per tree level with running sums
- **Prediction cost:** $O(\text{depth})$ comparisons — independent of $n$
- **Key hyperparameters:** `max_depth`, `min_samples_leaf`, `min_samples_split`, `max_leaf_nodes`, `ccp_alpha`, `criterion`
- **Assumptions:** none about distributions or linearity; implicitly that the target is roughly constant inside axis-aligned boxes

## The problem

Linear regression assumes the target moves at a constant rate with each feature. Many real targets don't. Bike rentals rise as it warms up, peak on pleasant days and **fall again** when it gets hot. A straight line through such data is wrong almost everywhere: on the 12-day dataset used below, the least-squares line $\hat y = 13.56 + 0.50\,T$ explains only $R^2 = 0.32$ of the variance (MSE 56.20).

You *can* repair a linear model with polynomial terms, splines or hand-made interactions, but you have to guess the right shape in advance. A regression tree takes the opposite approach: it makes no assumption about shape at all and simply **chops the input space into regions where the target is roughly constant**. With just four regions it reaches $R^2 = 0.89$ (MSE 9.00) on the same data.

## The idea

If you had to summarize a group of numbers by a single value and were scored on squared error, you would pick their **mean**. A regression tree applies that idea locally. Start with one group (all the data, predict the overall mean). Then look for the one question *is $x_j\le t$?* that splits the group into two halves whose own means fit them best — the halves should be internally similar and different from each other. Repeat inside each half. Stop when the groups are small or the tree is deep enough.

Each leaf is a box in feature space with a flat prediction, so in one dimension the model is a staircase and in two dimensions a set of flat rectangular plateaus. More depth means more, narrower steps — and eventually one step per training point.

**Vocabulary**

- **Node / region:** a set of training points defined by the answers on the path from the root; geometrically an axis-aligned box $R_m$
- **Split:** a pair (feature $j$, threshold $t$) sending points with $x_j\le t$ left and the rest right
- **Candidate threshold:** a midpoint between two consecutive distinct sorted values of a feature — the only places a split can change
- **Leaf value:** the constant $c_m$ a leaf predicts; under squared error, the mean of the leaf's training targets
- **SSE:** sum of squared errors $\sum (y_i-\bar y)^2$ of a node around its own mean — the regression tree's impurity
- **Variance reduction:** parent SSE minus the children's SSE (often divided by n); the score a split is ranked by
- **Step function:** the piecewise-constant function the tree computes: constant inside each leaf, jumping at thresholds

## The math

A tree with leaves $R_1,\dots,R_M$ computes

$$
\hat f(x)=\sum_{m=1}^{M} c_m\,\mathbf 1[x\in R_m]
$$

- $R_m$: the m-th leaf region (an axis-aligned box)
- $c_m$: the constant predicted in that leaf
- $\mathbf 1[\cdot]$: indicator: 1 if x falls in the region, else 0
- $M$: number of leaves

Training minimizes the squared error over both the regions and the constants:

$$
\min_{\{R_m,c_m\}}\ \sum_{m=1}^{M}\ \sum_{i:\,x_i\in R_m}\big(y_i-c_m\big)^2
$$

- $y_i$: target of training point i
- $x_i$: its feature vector

**Leaf values are means.** Fix the regions. Each leaf's term depends only on its own $c_m$, so set its derivative to zero: $\frac{\partial}{\partial c_m}\sum_{i\in R_m}(y_i-c_m)^2=-2\sum_{i\in R_m}(y_i-c_m)=0$, which gives $c_m=\frac{1}{n_m}\sum_{i\in R_m}y_i=\bar y_m$. What is left in each leaf is its sum of squared errors around its mean:

$$
\text{SSE}(S)=\sum_{i\in S}(y_i-\bar y_S)^2=\sum_{i\in S}y_i^2-\frac{\big(\sum_{i\in S}y_i\big)^2}{n_S}=n_S\operatorname{Var}(S)
$$

- $S$: the set of training points in a node
- $n_S,\ \bar y_S$: its size and mean target
- $\operatorname{Var}(S)$: variance of the targets in S (dividing by n)

**Choosing a split.** Searching over all partitions is intractable, so CART grows the tree greedily: at a node $S$ it tries every feature $j$ and threshold $t$, splits $S$ into $L=\{x_j\le t\}$ and $R=\{x_j>t\}$, and keeps the pair with the smallest $\text{SSE}(L)+\text{SSE}(R)$. Equivalently it maximizes the drop

$$
\Delta(j,t)=\text{SSE}(S)-\text{SSE}(L)-\text{SSE}(R)=\frac{n_L\,n_R}{n_S}\,\big(\bar y_L-\bar y_R\big)^2
$$

- $L,\ R$: points with $x_j\le t$ and $x_j>t$
- $n_L,\ n_R$: how many points go each way
- $\bar y_L,\ \bar y_R$: the two children's means (their future leaf values)

The right-hand form follows from expanding the SSE identity above (it is the between-group sum of squares from ANOVA) and says exactly what the tree looks for: **two big groups with very different means**. Dividing by $n_S$ gives the **variance reduction** $\operatorname{Var}(S)-\tfrac{n_L}{n_S}\operatorname{Var}(L)-\tfrac{n_R}{n_S}\operatorname{Var}(R)$, the regression analogue of information gain. Because the parent term is fixed, minimizing children's SSE is also the same as maximizing $\tfrac{(\sum_L y)^2}{n_L}+\tfrac{(\sum_R y)^2}{n_R}$, which needs only running sums — this is what makes an $O(n)$ scan per sorted feature possible.

> **It splits on mean differences, not on where the curve bends:** On the bike data the curve peaks around 23–26 °C, but a split *at* the peak ($T\le 27.5$) is nearly useless: both sides have almost the same mean (22.89 vs 22.67), so $\Delta=\tfrac{9\cdot3}{12}(0.22)^2=0.11$. The best root split is instead at the cold end ($T\le 12.5$), where 4 low days separate from 8 much higher ones: $\Delta=\tfrac{4\cdot 8}{12}(12.5-28)^2=640.67$.

**How each hyperparameter acts on this objective.** Every split can only lower training SSE, so without limits the greedy search continues until each leaf has one distinct target and training error is 0. The hyperparameters cap that:

**Hyperparameters, mathematically**

- **max_depth = D:** at most $2^D$ leaves; the step function has at most $2^D-1$ jumps per path. Lower $D$ → higher bias, lower variance.
- **min_samples_leaf = k:** only thresholds with $n_L\ge k$ and $n_R\ge k$ are candidates. Each leaf mean then averages at least $k$ targets, so its noise variance is at most $\sigma^2/k$.
- **min_samples_split:** a node with fewer points is not split at all (a weaker, parent-side version of the leaf limit)
- **min_impurity_decrease = δ:** split only if $\Delta/N\ge\delta$ ($N$ = training-set size): a minimum variance reduction per split
- **max_leaf_nodes:** grow best-first (largest $\Delta$ anywhere in the tree) and stop at that many leaves
- **ccp_alpha = α:** after growing, prune to minimize $\text{MSE}(T)+\alpha\,|T|$ — every leaf must buy at least $\alpha$ of training MSE

$$
R_\alpha(T)=\frac{1}{N}\sum_{m=1}^{|T|}\text{SSE}(R_m)+\alpha\,|T|,\qquad \alpha_{\text{eff}}(t)=\frac{R(t)-R(T_t)}{|T_t|-1}
$$

- $|T|$: number of leaves of tree T
- $R(t)$: training MSE contribution of node t if it were collapsed into a leaf
- $R(T_t)$: contribution of the subtree rooted at t, as grown
- $\alpha_{\text{eff}}(t)$: error added per leaf removed when t is collapsed; the smallest one is pruned first (weakest link)

## Worked example: bike rentals vs temperature

Twelve days, one feature (noon temperature $T$ in °C) and the target (bike rentals, in hundreds). Rentals climb until the mid-20s and drop on hot days. Overall: $\sum y=274$, $\sum y^2=7254$, so the root mean is $274/12=22.83$ and the root SSE is $7254-274^2/12=997.67$ (variance 83.14).

Training data. Highlighted: the 4 cold days that the first split will separate.

| Day | T (°C) | Rentals y |
| --- | --- | --- |
| 1 | 2 | 8 |
| 2 | 5 | 10 |
| 3 | 8 | 13 |
| 4 | 11 | 19 |
| 5 | 14 | 24 |
| 6 | 17 | 30 |
| 7 | 20 | 33 |
| 8 | 23 | 35 |
| 9 | 26 | 34 |
| 10 | 29 | 29 |
| 11 | 32 | 22 |
| 12 | 35 | 17 |

**Growing a depth-2 tree by hand**

1. **Candidates.** The 12 sorted temperatures give 11 midpoints: 3.5, 6.5, …, 33.5.
2. **Score one.** For $T\le 12.5$: left $\{8,10,13,19\}$ has mean $12.5$ and SSE $4.5^2+2.5^2+0.5^2+6.5^2=69$; right $\{24,30,33,35,34,29,22,17\}$ has mean $28$ and SSE $288$. Children SSE $=357$, so $\Delta=997.67-357=640.67$. Check with the shortcut: $\tfrac{4\cdot 8}{12}(12.5-28)^2=640.67$.
3. **Score all 11** (table below). $T\le 12.5$ has the smallest children SSE, so it becomes the root. Training MSE drops from $83.14$ to $357/12=29.75$.
4. **Recurse left** (4 cold days, SSE 69). Candidates 3.5, 6.5, 9.5 leave SSE 42.00, 20.00, 12.67. Best: $T\le 9.5$ → leaves $\{8,10,13\}$ (mean 10.33) and $\{19\}$ (mean 19).
5. **Recurse right** (8 warm days, SSE 288). The 7 candidates leave SSE 269.71, 285.33, 283.20, 238.00, 151.47, **95.33**, 149.71. Best: $T\le 30.5$ → leaves $\{24,30,33,35,34,29\}$ (mean $185/6=30.83$, SSE 82.83) and $\{22,17\}$ (mean 19.5, SSE 12.5).
6. **Stop** at max_depth 2. Total leaf SSE $=12.67+0+82.83+12.5=108$, so training MSE $=108/12=9.00$ and $R^2=1-9/83.14=0.89$.
7. **Predict** a 27 °C day: $27>12.5$ → right; $27\le 30.5$ → left. Prediction **30.83** hundred rentals (the straight line would say 27.09).

Every root candidate. Children SSE = SSE(left) + SSE(right); the drop is 997.67 minus it. The peak split at 27.5 is almost worthless.

| Split | n left | Left mean (SSE) | Right mean (SSE) | Children SSE | Drop Δ |
| --- | --- | --- | --- | --- | --- |
| T ≤ 3.5 | 1 | 8.00 (0.00) | 24.18 (757.64) | 757.64 | 240.03 |
| T ≤ 6.5 | 2 | 9.00 (2.00) | 25.60 (536.40) | 538.40 | 459.27 |
| T ≤ 9.5 | 3 | 10.33 (12.67) | 27.00 (360.00) | 372.67 | 625.00 |
| **T ≤ 12.5** | **4** | **12.50 (69.00)** | **28.00 (288.00)** | **357.00** | **640.67** |
| T ≤ 15.5 | 5 | 14.80 (174.80) | 28.57 (269.71) | 444.51 | 553.15 |
| T ≤ 18.5 | 6 | 17.33 (367.33) | 28.33 (267.33) | 634.67 | 363.00 |
| T ≤ 21.5 | 7 | 19.57 (577.71) | 27.40 (241.20) | 818.91 | 178.75 |
| T ≤ 24.5 | 8 | 21.50 (786.00) | 25.50 (169.00) | 955.00 | 42.67 |
| T ≤ 27.5 | 9 | 22.89 (924.89) | 22.67 (72.67) | 997.56 | 0.11 |
| T ≤ 30.5 | 10 | 23.50 (958.50) | 19.50 (12.50) | 971.00 | 26.67 |
| T ≤ 33.5 | 11 | 23.36 (960.55) | 17.00 (0.00) | 960.55 | 37.12 |

The fitted depth-2 tree as rules. Each row is one leaf, one step of the step function.

| Leaf (rule) | Days | Prediction (mean) | Leaf SSE |
| --- | --- | --- | --- |
| T ≤ 9.5 | 2, 5, 8 °C | 10.33 | 12.67 |
| 9.5 < T ≤ 12.5 | 11 °C | 19.00 | 0.00 |
| **12.5 < T ≤ 30.5** | 14 … 29 °C | **30.83** | 82.83 |
| T > 30.5 | 32, 35 °C | 19.50 | 12.50 |

> Figure: The tree growing on the 12 days. Top: the tree, each node drawn above the region of temperature it owns. Middle: data, the current step function (green) and residuals (red). Bottom: the SSE drop of each candidate threshold of the node being split. The root scan visits all 11 midpoints and keeps T ≤ 12.5 (drop 640.67); the cold node then splits at 9.5 (56.33) and the warm node at 30.5 (192.67). At depth 2 the train MSE is 9.00 and a 27 °C day gets 30.83; depth 3 gives 7 leaves (MSE 2.40); no limit gives 12 one-day leaves, depth 7, MSE 0.

> **Why the 11 °C day gets its own leaf:** At depth 2 one leaf contains a single day. Nothing in squared error discourages that: isolating 19 from $\{8,10,13\}$ removes 56.33 of SSE. A single-point leaf is exactly the kind of decision that fits noise — `min_samples_leaf=3` would forbid it.

## The algorithm, step by step

**Grow(node)**

1. **Leaf value.** Store the mean of the node's targets (it is the prediction if the node stays a leaf).
2. **Stop?** Make it a leaf if depth = `max_depth`, the node has fewer than `min_samples_split` points, or all its targets are equal (SSE = 0).
3. **Sort and scan.** For each feature, sort the node's points by that feature and sweep left to right, keeping running $n_L,\ \sum_L y,\ \sum_L y^2$ (the right side is total minus left). At each midpoint between distinct values with at least `min_samples_leaf` points on each side, compute $\text{SSE}_L+\text{SSE}_R$ in $O(1)$.
4. **Choose** the (feature, threshold) with the smallest children SSE. If its drop is below `min_impurity_decrease`, make a leaf instead.
5. **Recurse** on the left and right subsets.
6. **(Optional) prune** the grown tree with cost-complexity pruning, choosing `ccp_alpha` by cross-validation.
7. **Predict** by walking from the root, comparing one feature per node, and returning the leaf mean.

**Complexity**

- **Training:** each level touches every point once per feature: $O(d\,n)$ per level after an $O(d\,n\log n)$ sort (or $O(d\,n\log n)$ per level if re-sorting). A balanced tree has $O(\log n)$ levels, a degenerate one up to $O(n)$.
- **Prediction:** $O(\text{depth})$ per query, at most $O(\log_2 M)$ for a balanced tree with $M$ leaves
- **Memory:** one threshold, feature index and value per node — at most 2n − 1 nodes
- **Histogram variant:** bin each feature into $B\approx256$ buckets once; split search becomes $O(d\,B)$ per node — the trick LightGBM and XGBoost's hist mode use

## Variants and tuning

Split criteria available in scikit-learn's DecisionTreeRegressor.

| criterion | Impurity of a node | Leaf value | When to use |
| --- | --- | --- | --- |
| `squared_error` (default) | SSE / variance | mean | the standard choice; fast O(n) scan |
| `friedman_mse` | $\tfrac{n_Ln_R}{n_L+n_R}(\bar y_L-\bar y_R)^2$ improvement score | mean | the default split score inside scikit-learn's gradient boosting; rarely worth choosing for a single tree |
| `absolute_error` | $\sum\lvert y_i-\operatorname{median}\rvert$ | median | heavy-tailed targets or outliers; much slower to fit |
| `poisson` | Poisson deviance | mean (must be > 0) | counts and rates, e.g. claims, visits |

On the bike data, `criterion="absolute_error"` with depth 2 picks the same four regions but predicts medians: 10.00, 19.00, 31.50 and 19.50 instead of 10.33, 19.00, 30.83 and 19.50.

Tuning on a larger, noisy problem: 300 points, $y=\sin x+\varepsilon$, $\varepsilon\sim\mathcal N(0,0.3^2)$, so the best achievable MSE is 0.09. 5-fold cross-validated MSE; best rows highlighted.

| Setting | Leaves | Train MSE | CV MSE |
| --- | --- | --- | --- |
| max_depth = 1 | 2 | 0.397 | 0.402 |
| max_depth = 3 | 8 | 0.138 | 0.158 |
| **max_depth = 5** | 29 | 0.067 | **0.111** |
| max_depth = 8 | 138 | 0.027 | 0.134 |
| no limit | 300 | 0.000 | 0.147 |
| **min_samples_leaf = 10** | 23 | 0.072 | **0.109** |
| min_samples_leaf = 50 | 5 | 0.201 | 0.234 |

**Practical tuning**

- **Start with:** `min_samples_leaf` (try 5–50 or 1–5% of n) and `max_depth` (3–10); they matter far more than the criterion
- **Pruning:** compute `cost_complexity_pruning_path` on the training set and cross-validate over its alphas
- **On the bike data:** the pruning path has $\alpha$ = 4.69 (collapsing the cold node: $(69-12.67)/12$), 16.06 (the warm node: $(288-95.33)/12$) and 53.39 (the root). So `ccp_alpha=5` keeps 3 leaves (train MSE 13.69) and `ccp_alpha=20` keeps only the root split (29.75)
- **Watch:** the gap between train and validation MSE; a deep tree's training MSE is near 0 and tells you nothing
- **Need more accuracy?:** average many trees (see *Random forest regression*) or fit trees to residuals (see *Gradient boosting regression* and *AdaBoost regression*)

## In code

```python
import numpy as np

def best_split(x, y, min_leaf=1):
  """Best threshold on one feature, scanning sorted x with prefix sums: O(n)."""
  order = np.argsort(x)
  xs, ys = x[order], y[order]
  n = len(ys)
  csum, csq = np.cumsum(ys), np.cumsum(ys ** 2)
  best = (np.inf, None)                          # (SSE_L + SSE_R, threshold)
  for i in range(min_leaf - 1, n - min_leaf):    # left = first i+1 points
      if xs[i] == xs[i + 1]:
          continue                               # cannot split between equal x
      nL, nR = i + 1, n - i - 1
      sL, sR = csum[i], csum[-1] - csum[i]
      qL, qR = csq[i], csq[-1] - csq[i]
      sse = (qL - sL ** 2 / nL) + (qR - sR ** 2 / nR)   # SSE = sum y^2 - (sum y)^2 / n
      if sse < best[0]:
          best = (sse, (xs[i] + xs[i + 1]) / 2)
  return best

def grow(X, y, depth=0, max_depth=None, min_leaf=1):
  node = {"value": y.mean(), "n": len(y)}
  if (max_depth is not None and depth >= max_depth) or len(y) < 2 * min_leaf or np.ptp(y) == 0:
      return node
  parent_sse = ((y - y.mean()) ** 2).sum()
  best = (parent_sse, None, None)                 # (SSE, feature, threshold)
  for j in range(X.shape[1]):
      sse, t = best_split(X[:, j], y, min_leaf)
      if t is not None and sse < best[0]:
          best = (sse, j, t)
  sse, j, t = best
  if j is None:
      return node
  L = X[:, j] <= t
  node.update(feature=j, threshold=t, gain=parent_sse - sse,
              left=grow(X[L], y[L], depth + 1, max_depth, min_leaf),
              right=grow(X[~L], y[~L], depth + 1, max_depth, min_leaf))
  return node

def predict(node, x):
  while "threshold" in node:
      node = node["left"] if x[node["feature"]] <= node["threshold"] else node["right"]
  return node["value"]

def show(node, pad=""):
  if "threshold" in node:
      print(f"{pad}x{node['feature']} <= {node['threshold']}  (SSE drop {node['gain']:.2f})")
      show(node["left"], pad + "  "); show(node["right"], pad + "  ")
  else:
      print(f"{pad}leaf: {node['value']:.2f}  (n={node['n']})")

T = np.array([2, 5, 8, 11, 14, 17, 20, 23, 26, 29, 32, 35.])
y = np.array([8, 10, 13, 19, 24, 30, 33, 35, 34, 29, 22, 17.])
tree = grow(T[:, None], y, max_depth=2)
show(tree)
pred = np.array([predict(tree, [t]) for t in T])
print("train MSE:", ((pred - y) ** 2).mean())
print("T = 27 ->", round(predict(tree, [27]), 2))

# x0 <= 12.5  (SSE drop 640.67)
#   x0 <= 9.5  (SSE drop 56.33)
#     leaf: 10.33  (n=3)
#     leaf: 19.00  (n=1)
#   x0 <= 30.5  (SSE drop 192.67)
#     leaf: 30.83  (n=6)
#     leaf: 19.50  (n=2)
# train MSE: 9.0
# T = 27 -> 30.83
```

From scratch in NumPy: prefix-sum split search, recursive growth, prediction. Prints the depth-2 tree from the worked example.

```python
import numpy as np
from sklearn.tree import DecisionTreeRegressor, export_text
from sklearn.metrics import mean_squared_error, r2_score

T = np.array([2, 5, 8, 11, 14, 17, 20, 23, 26, 29, 32, 35.]).reshape(-1, 1)
y = np.array([8, 10, 13, 19, 24, 30, 33, 35, 34, 29, 22, 17.])

tree = DecisionTreeRegressor(max_depth=2, random_state=0).fit(T, y)
print(export_text(tree, feature_names=["temp"]))
pred = tree.predict(T)
print("train MSE:", mean_squared_error(y, pred), " R2:", round(r2_score(y, pred), 4))
print("T = 27 ->", tree.predict([[27]]).round(2), "  T = 45 ->", tree.predict([[45]]).round(2))

# How depth changes the fit
for d in [1, 2, 3, None]:
  m = DecisionTreeRegressor(max_depth=d, random_state=0).fit(T, y)
  print(f"max_depth={d}: {m.get_n_leaves()} leaves, train MSE {mean_squared_error(y, m.predict(T)):.2f}")

# Cost-complexity pruning path (alphas are in MSE units)
path = DecisionTreeRegressor(random_state=0).cost_complexity_pruning_path(T, y)
print("ccp alphas:", [round(float(a), 2) for a in path.ccp_alphas])
for a in [5, 20]:
  m = DecisionTreeRegressor(ccp_alpha=a, random_state=0).fit(T, y)
  print(f"ccp_alpha={a}: {m.get_n_leaves()} leaves, train MSE {mean_squared_error(y, m.predict(T)):.2f}")

# |--- temp <= 12.50
# |   |--- temp <= 9.50
# |   |   |--- value: [10.33]
# |   |--- temp >  9.50
# |   |   |--- value: [19.00]
# |--- temp >  12.50
# |   |--- temp <= 30.50
# |   |   |--- value: [30.83]
# |   |--- temp >  30.50
# |   |   |--- value: [19.50]
# train MSE: 9.0  R2: 0.8917
# T = 27 -> [30.83]   T = 45 -> [19.5]
# max_depth=1: 2 leaves, train MSE 29.75
# max_depth=2: 4 leaves, train MSE 9.00
# max_depth=3: 7 leaves, train MSE 2.40
# max_depth=None: 12 leaves, train MSE 0.00
# ccp alphas: [0.0, 0.04, 0.13, 0.17, 0.89, 1.0, 1.04, 1.07, 4.67, 4.69, 16.06, 53.39]
# ccp_alpha=5: 3 leaves, train MSE 13.69
# ccp_alpha=20: 2 leaves, train MSE 29.75
```

scikit-learn on the same data: the same tree, the same prediction, the effect of depth and the pruning path.

**Strengths**

- The relationship is non-linear, non-monotonic or full of thresholds and interactions you can't specify in advance.
- You need a model a person can read: each prediction is a short rule ending in an average of real training cases.
- Features are on wildly different scales or mixed with ordinal codes — splits only use the order of values, so no scaling is needed.
- You want a fast baseline, or the base learner for a forest or a boosted ensemble.

**Limitations**

- The target is smooth in the features: a staircase needs many leaves (and much data) to approximate even a straight line.
- You must extrapolate: predictions are always averages of training targets, so they never leave the observed range.
- Stability matters: a small change in the data can move the root split and rewrite the whole tree.
- You need the best accuracy from one model: a single deep tree has high variance — use an ensemble.

- **Demand forecasting baselines:** Bike, taxi and energy demand vs weather, hour and weekday — strongly non-monotonic, threshold-heavy relationships.
- **Pricing and valuation rules:** Explainable price bands for houses, insurance premiums or used cars, where each band is an auditable rule.
- **Exploratory analysis:** A shallow tree on a KPI shows which variables and cut-points explain most of its variance.
- **Base learner of ensembles:** Every random forest, gradient boosting, XGBoost, LightGBM and CatBoost regressor is a sum or average of regression trees.

> **Trees cannot extrapolate:** Outside the training range the tree just repeats its edge leaf. The depth-2 bike tree predicts 19.5 at 45 °C and 10.33 at −10 °C, whatever the trend. For time series with a trend (sales growing every year) a tree will predict a flat line at last year's level — detrend first, or use a linear model or linear-leaf trees.

> **Default settings memorize:** With `max_depth=None` and `min_samples_leaf=1`, scikit-learn grows until every leaf is pure: on the bike data, 12 leaves for 12 days and a training MSE of exactly 0. Always set `min_samples_leaf` / `max_depth` or prune with `ccp_alpha`, and judge the tree on validation MSE.

> **Squared error lets outliers grab leaves:** Because the criterion squares residuals, one extreme target can produce the largest $\Delta$ simply by being isolated in its own leaf, and every leaf mean is pulled by outliers. Use `min_samples_leaf`, a log-transformed target, or `criterion="absolute_error"` (median leaves) for heavy-tailed targets.

**Question:** A leaf contains training targets 4, 6 and 11. Under squared error, what does it predict, and what is its SSE?

- A. 6 and 26
- B. 7 and 26
- C. 7 and 8.67
- D. 6 and 7

**Answer:** B. 7 and 26 — The squared-error-optimal constant is the mean, $(4+6+11)/3=7$. SSE $=(-3)^2+(-1)^2+4^2=26$. (6 is the median — the leaf value under absolute error, with absolute deviation 7.)

**Question:** A node has 12 points. Split A sends 4 points with mean 12.5 left and 8 with mean 28 right. Split B sends 9 points with mean 22.89 left and 3 with mean 22.67 right. Which reduces SSE more, and by how much?

- A. B, because it splits at the peak of the curve
- B. A, by about 640.7
- C. A, by about 15.5
- D. They are equal because both use all 12 points

**Answer:** B. A, by about 640.7 — $\Delta=\tfrac{n_Ln_R}{n}(\bar y_L-\bar y_R)^2$. A: $\tfrac{32}{12}(15.5)^2=640.67$. B: $\tfrac{27}{12}(0.22)^2\approx0.11$. Trees reward large groups with very different means.

**Question:** The depth-2 bike tree was trained on temperatures 2–35 °C. What does it predict for a 45 °C day?

- A. Something below 17, continuing the downward trend
- B. 19.5, the mean of the hottest leaf
- C. 22.83, the overall mean
- D. It cannot predict; 45 is out of range

**Answer:** B. 19.5, the mean of the hottest leaf — 45 > 12.5 and 45 > 30.5, so it lands in the T > 30.5 leaf, whose mean of {22, 17} is 19.5. A regression tree's output is always one of its leaf means, so it never extrapolates.

**Question:** With `max_depth = 3`, what is the maximum number of distinct values a regression tree can predict?

- A. 3
- B. 6
- C. 8
- D. Unlimited — it depends on n

**Answer:** C. 8 — A binary tree of depth 3 has at most $2^3=8$ leaves, and each leaf predicts one constant. (On the bike data depth 3 gives only 7, because the 11 °C leaf holds a single day and cannot split.)

**Key points**

- A regression tree is a step function: $\hat f(x)=\sum_m c_m\mathbf 1[x\in R_m]$, with each leaf predicting the **mean** of its training targets (the minimizer of squared error).
- Splits are chosen greedily to minimize $\text{SSE}_L+\text{SSE}_R$; the drop equals $\tfrac{n_Ln_R}{n}(\bar y_L-\bar y_R)^2$, i.e. variance reduction.
- Running sums of y and y² make the threshold scan O(n) per sorted feature.
- Without limits the tree reaches zero training error; control it with `max_depth`, `min_samples_leaf` or cost-complexity pruning (`ccp_alpha`), chosen by cross-validation.
- Trees need no scaling and capture thresholds and interactions, but they cannot extrapolate and have high variance — the reason forests and boosting exist.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
