Random forest regression

Average many decorrelated regression trees grown on bootstrap samples; averaging cuts variance without adding bias.

17 min read2222 wordscore6 viewsupdated 2026-10-07

Definition

Random forest regression predicts a number by averaging the outputs of TT regression trees. Each tree is grown deep on its own bootstrap sample (nn rows drawn with replacement), optionally considering only a random subset of features at each split, and each leaf predicts the mean target of the training rows that reach it. The forest's prediction is the plain average fˉ(x)=1T∑tht(x)\bar f(x)=\tfrac1T\sum_t h_t(x).

At a glance

Task
Regression (numeric target); squared-error loss
Input → output
x∈Rdx\in\mathbb R^d → fˉ(x)\bar f(x), the mean of TT trees' leaf means
Learns
TT deep regression trees: thresholds per internal node, a mean yˉ\bar y per leaf
Training cost
about O(T⋅m nlog⁡n)O(T\cdot m\,n\log n) with mm features tried per split; trees are independent, so it parallelizes perfectly
Prediction cost
O(T⋅depth)O(T\cdot\text{depth}) comparisons, then one average
Key hyperparameters
number of trees n_estimators; features per split max_features; leaf size min_samples_leaf; bootstrap size max_samples
Assumes
nothing distributional; the target can be reached as an average of training targets (no extrapolation)

The problem

A single regression tree (see Decision tree regression) is an excellent low-bias model: grown deep, it can follow any nonlinear shape and any interaction between features. But it pays for that with variance. Its thresholds are chosen greedily from the exact training rows, so moving or removing a handful of points changes the top split, which changes everything underneath. Two trees trained on two samples from the same population can give very different step functions — and at a given xx, very different numbers.

The usual cure for variance — prune harder, make leaves bigger — trades it straight back for bias: big leaves average over regions where the target is not constant. Linear models have the opposite problem: stable, but blind to thresholds and interactions. We want to keep the deep tree's flexibility and remove its instability.

The idea

If you had TT independent training sets, you could grow a tree on each and average their predictions: the average has the same expected value as one tree but TT times less variance. We only have one training set, so we fake new ones with the bootstrap — resample nn rows with replacement — and grow one deep tree per resample. This is bagging (bootstrap aggregating, Breiman 1996).

Bootstrap trees are not independent, though: they share most rows, so they tend to pick the same strong feature at the root and make similar errors. Averaging correlated errors cancels much less. A random forest (Breiman 2001) adds a second source of randomness — at every split, only a random subset of mm of the dd features is searched — so the trees differ more and their errors cancel more. For regression the averaging is literal: no votes, no probabilities, just the arithmetic mean of numbers.

Vocabulary

Bootstrap sample
nn row indices drawn uniformly with replacement from {1,…,n}\{1,\dots,n\}; some rows appear 2–3 times, others not at all
Bagging
fitting the same learner on many bootstrap samples and averaging the fits
In-bag / out-of-bag (OOB)
for a tree tt: the rows in its bootstrap sample / the rows it never saw (about 36.8% of them)
OOB prediction
for row ii: the average over only those trees for which ii was out-of-bag — an honest, held-out prediction
max_features (mtry)
mm, the number of features randomly offered to each split; m=dm=d turns the forest back into plain bagging
min_samples_leaf (nodesize)
smallest number of training rows a leaf may hold; the forest's main smoothing knob in regression
Tree correlation ρ
the correlation, at a fixed xx, between the predictions of two trees of the same forest
Ambiguity
the spread of the trees around their own average, 1T∑t(ht(x)−fˉ(x))2\tfrac1T\sum_t (h_t(x)-\bar f(x))^2

The math

Leaves predict means. Under squared error, a leaf ℓ\ell holding training targets yiy_i (i∈ℓi\in\ell) should predict the constant cc that minimizes ∑i∈ℓ(yi−c)2\sum_{i\in\ell}(y_i-c)^2. Setting the derivative −2∑i∈ℓ(yi−c)-2\sum_{i\in\ell}(y_i-c) to zero gives c=yˉℓc=\bar y_\ell, the leaf mean, and the leftover loss is the leaf's sum of squared errors (SSE). Each tree splits greedily on the threshold that most reduces total SSE — the same variance-reduction rule as a single regression tree. In a bootstrap sample a row that appears twice simply counts twice in both the mean and the SSE.

fˉ(x)=1T∑t=1Tht(x),ht(x)=1∣ℓt(x)∣∑i∈ℓt(x)yi\bar f(x)=\frac1T\sum_{t=1}^{T}h_t(x),\qquad h_t(x)=\frac{1}{|\ell_t(x)|}\sum_{i\in \ell_t(x)} y_i
TT
number of trees
ht(x)h_t(x)
prediction of tree tt: the mean target in the leaf xx falls into
ℓt(x)\ell_t(x)
the multiset of tree tt's in-bag training rows sharing that leaf (duplicates counted)
fˉ(x)\bar f(x)
the forest prediction

Averaging does not change bias. Every tree is grown by the same random procedure, so they all have the same expected prediction E[h(x)]\mathbb E[h(x)]. By linearity of expectation E[fˉ(x)]=1T∑tE[ht(x)]=E[h(x)]\mathbb E[\bar f(x)]=\tfrac1T\sum_t\mathbb E[h_t(x)]=\mathbb E[h(x)]: the forest's bias is exactly the bias of one of its trees. That is why forests use deep, low-bias trees — averaging can only remove variance, so the trees should bring as little bias as possible.

Averaging divides variance — partly. Let each tree's prediction at xx have variance σ2\sigma^2, and any two trees correlation ρ\rho, so Cov⁡(hs,ht)=ρσ2\operatorname{Cov}(h_s,h_t)=\rho\sigma^2 for s≠ts\ne t. The variance of a mean is the sum of all T2T^2 covariances divided by T2T^2: TT diagonal terms equal to σ2\sigma^2 and T(T−1)T(T-1) off-diagonal terms equal to ρσ2\rho\sigma^2.

Var⁡(fˉ(x))=1T2(Tσ2+T(T−1)ρσ2)=ρ σ2+1−ρT σ2\operatorname{Var}\big(\bar f(x)\big)=\frac{1}{T^2}\Big(T\sigma^2+T(T-1)\rho\sigma^2\Big)=\rho\,\sigma^2+\frac{1-\rho}{T}\,\sigma^2
σ2\sigma^2
variance of one tree's prediction at xx (over training sets and bootstrap draws)
ρ\rho
correlation between two trees of the same forest
1−ρTσ2\tfrac{1-\rho}{T}\sigma^2
the part more trees remove; it vanishes as T→∞T\to\infty
ρσ2\rho\sigma^2
the floor no number of trees can remove

Putting the pieces into the bias–variance decomposition of the expected squared error at xx gives the whole story of the method in one line:

E[(y−fˉ(x))2]=σε2⏟noise+(E[h(x)]−f∗(x))2⏟bias2 of one tree+ρσ2+1−ρTσ2⏟forest variance\mathbb E\big[(y-\bar f(x))^2\big]=\underbrace{\sigma_\varepsilon^2}_{\text{noise}}+\underbrace{\big(\mathbb E[h(x)]-f^*(x)\big)^2}_{\text{bias}^2\text{ of one tree}}+\underbrace{\rho\sigma^2+\tfrac{1-\rho}{T}\sigma^2}_{\text{forest variance}}
f∗(x)f^*(x)
the true regression function E[y∣x]\mathbb E[y\mid x]
σε2\sigma_\varepsilon^2
irreducible noise in y

The ambiguity decomposition. For squared error there is also an exact identity on any single point, with no expectations at all (Krogh & Vedelsby, 1995). Expand ht−y=(ht−fˉ)+(fˉ−y)h_t-y=(h_t-\bar f)+(\bar f-y), square, and average over tt; the cross term vanishes because ∑t(ht−fˉ)=0\sum_t (h_t-\bar f)=0:

(fˉ(x)−y)2=1T∑t(ht(x)−y)2⏟average tree error−1T∑t(ht(x)−fˉ(x))2⏟ambiguity  ≥0\big(\bar f(x)-y\big)^2=\underbrace{\frac1T\sum_t\big(h_t(x)-y\big)^2}_{\text{average tree error}}-\underbrace{\frac1T\sum_t\big(h_t(x)-\bar f(x)\big)^2}_{\text{ambiguity}\;\ge 0}
yy
the true target at xx
ambiguityambiguity
how much the trees disagree with their own mean

So the forest is never worse than its average tree, and it beats it by exactly how much the trees disagree. Diversity is not a side effect; under squared loss it is literally the profit. The catch is that decorrelating trees (smaller mm, smaller bootstrap samples) also makes each tree worse, so the first term grows too — the tuning question is which effect wins.

A forest is a weighted nearest-neighbour average. Substituting the leaf means into fˉ\bar f and swapping the sums gives

fˉ(x)=∑i=1nwi(x) yi,wi(x)=1T∑t=1Tcti [ i∈ℓt(x) ]∣ℓt(x)∣\bar f(x)=\sum_{i=1}^{n} w_i(x)\,y_i,\qquad w_i(x)=\frac1T\sum_{t=1}^{T}\frac{c_{ti}\,[\,i\in\ell_t(x)\,]}{|\ell_t(x)|}
wi(x)w_i(x)
weight of training row ii in the prediction at xx; wi≥0w_i\ge0 and ∑iwi=1\sum_i w_i=1
ctic_{ti}
how many times row ii appears in tree tt's bootstrap sample

The weights are non-negative and sum to one, so every forest prediction is a convex combination of training targets: it always lies between min⁡iyi\min_i y_i and max⁡iyi\max_i y_i. Rows that keep landing in the same leaf as xx get large weight — the forest learns its own adaptive neighbourhood, wide along irrelevant features and narrow along important ones. The same weights are what quantile regression forests reuse to estimate the whole conditional distribution instead of only its mean.

Out-of-bag rows. One bootstrap draw misses row ii with probability 1−1n1-\tfrac1n, so all nn draws miss it with probability (1−1n)n→e−1≈0.368(1-\tfrac1n)^n\to e^{-1}\approx0.368 (already 0.349 at n=10n=10). Each row is therefore unseen by about a third of the trees, and averaging just those trees gives an honest held-out prediction. The OOB MSE 1n∑i(y^iOOB−yi)2\tfrac1n\sum_i(\hat y_i^{\text{OOB}}-y_i)^2 estimates test error without a separate validation split.

How the hyperparameters act on the formula

At a glance

n_estimators T
shrinks only 1−ρTσ2\tfrac{1-\rho}{T}\sigma^2. Error falls fast, then plateaus; bias, ρ\rho and σ2\sigma^2 are untouched.
max_features m
smaller mm → trees share fewer splits → ρ\rho falls. But each tree is forced to use weaker features at some splits, so its bias and σ2\sigma^2 rise. Best mm depends on how many features carry signal.
min_samples_leaf
bigger leaves average more targets → σ2\sigma^2 falls, but the step function gets coarser → bias rises. In regression this matters much more than in classification, because leaf means of 1–2 noisy targets are very noisy.
max_samples
smaller bootstrap samples → trees see fewer shared rows → ρ\rho falls, each tree is weaker; also faster.
max_depth
a hard cap with the same effect as bigger leaves; usually left unlimited and controlled through leaf size instead.

To see these numbers for real, we simulated the standard Friedman #1 benchmark (d=10d=10 features, only 5 of which matter, noise sd 1): 25 independent training sets of 500 rows, a 200-tree forest on each, scored against the noise-free truth on 1000 test points. Across the 25 forests we measured each tree's variance σ2\sigma^2, the tree correlation ρ\rho, the bias² and the forest's variance.

SettingOne tree MSETree varianceTree correlationBias²Forest varianceForest MSE
m = 126.4618.770.0167.700.398.09
m = 3 (≈ d/3)16.3812.450.0323.930.454.39
m = 513.129.830.0533.290.563.86
m = 10 (bagging)11.388.290.0913.090.813.90
m = 10, leaf ≥ 59.545.900.1203.640.744.39
m = 10, leaf ≥ 209.823.920.1585.900.636.52
Friedman #1, 200 trees, averaged over 25 training sets. Forest MSE = bias² + forest variance (noise excluded). Lower mm decorrelates the trees (ρ\rho 0.091 → 0.016) but raises bias, because half the features are noise; here m=5m=5 is best and m=1m=1 is clearly worst. Bigger leaves cut σ2\sigma^2 and raise bias.

Read the bagging row: one tree has MSE 11.38 = bias² 3.09 + variance 8.29. The forest keeps the same bias² 3.09 and cuts variance to 0.81 — very close to the floor ρσ2=0.091×8.29≈0.75\rho\sigma^2=0.091\times8.29\approx0.75. Ninety percent of a single tree's variance is gone. With one training set the number of trees behaves as the formula promises: test MSE 10.71 with 1 tree, 4.87 with 5, 3.80 with 25, 3.62 with 100, 3.56 with 500.

Worked example: bike rentals by temperature

Ten days of rentals (in tens of bikes) against the day's temperature. Demand rises with warmth, peaks in the low twenties and falls again on hot days — a hump no straight line can fit. We grow a forest of 3 trees with min_samples_leaf = 3 (a leaf must hold at least 3 bootstrap rows, duplicates included) and predict a 21 °C day.

RowTemp (°C)Rentals (×10)
0412
1818
21125
31431
41738
52044
62347
72645
82937
93329
Training data. Highlighted: the three days that every one of the three bootstrap samples happens to contain, so they will have no out-of-bag prediction.

Tree 1, by hand

  1. 1Bootstrap. Draw 10 row numbers with replacement (NumPy "default_rng(10)"): rows {1,2,2,5,5,7,7,8,8,9}\{1, 2, 2, 5, 5, 7, 7, 8, 8, 9\}. Rows 2, 5, 7, 8 appear twice; rows 0, 3, 4, 6 are out-of-bag.
  2. 2Root node. The 10 in-bag targets have mean 34.90 and SSE 894.90. With at least 3 rows per side, only three thresholds are legal (midpoints between distinct temperatures): 15.515.5, 2323 and 27.527.5.
  3. 3Score them (table below). x≤15.5x\le15.5 leaves SSE 32.67+220.86=253.5232.67+220.86=253.52, by far the lowest, so the root asks temp ≤ 15.5? — removing 641.38 of the 894.90.
  4. 4Left child holds 3 rows (8, 11, 11 °C → 18, 25, 25). Splitting it would leave a side with fewer than 3 rows, so it becomes a leaf predicting 18+25+253=22.67\tfrac{18+25+25}{3}=22.67.
  5. 5Right child holds 7 rows: 20, 20, 26, 26, 29, 29, 33 °C (mean 40.14, SSE 220.86). Only 27.527.5 keeps 3+ rows on each side: leaves {44,44,45,45}\{44,44,45,45\} (mean 44.50, SSE 1.00) and {37,37,29}\{37,37,29\} (mean 34.33, SSE 42.67). SSE drops by 177.19, so split. Neither leaf can split again.
  6. 6Tree 1 is the step function 22.67 for x≤15.5x\le15.5, 44.50 for 15.5<x≤27.515.5<x\le27.5, 34.33 above. At 21 °C it predicts 44.50.
ThresholdLeft rows · mean · SSERight rows · mean · SSETotal SSE
x ≤ 15.53 · 22.67 · 32.677 · 40.14 · 220.86253.52
x ≤ 235 · 31.20 · 578.805 · 38.60 · 179.20758.00
x ≤ 27.57 · 35.14 · 850.863 · 34.33 · 42.67893.52
Tree 1, root node: every legal threshold scored by total SSE of the two children (in-bag rows, duplicates counted).

Trees 2 and 3 are grown the same way on their own bootstrap samples:

TreeBootstrap rowsOut-of-bagLeaves (range → mean, rows)ŷ(21 °C)
11, 2, 2, 5, 5, 7, 7, 8, 8, 90, 3, 4, 6≤15.5 → 22.67 (3) · ≤27.5 → 44.50 (4) · >27.5 → 34.33 (3)44.50
20, 1, 1, 4, 4, 4, 5, 6, 8, 92, 3, 7≤12.5 → 16.00 (3) · ≤21.5 → 39.50 (4) · >21.5 → 37.67 (3)39.50
30, 2, 3, 5, 7, 7, 8, 8, 9, 91, 4, 6≤17 → 22.67 (3) · ≤27.5 → 44.67 (3) · >27.5 → 33.00 (4)44.67
The three trees. Each sees a different sample, so each puts its thresholds in a different place.

The forest

  1. 1Predict 21 °C. fˉ(21)=13(44.50+39.50+44.67)=\bar f(21)=\tfrac13(44.50+39.50+44.67)= 42.89. The trees disagree with sd 2.40 — that disagreement is the variance averaging removes. (One tree with the same leaf rule grown on all 10 rows predicts 40.00.)
  2. 2As neighbour weights. Tree 1's leaf at 21 °C holds 20 °C ×2 and 26 °C ×2; tree 2's holds 17 °C ×3 and 20 °C ×1; tree 3's holds 20 °C and 26 °C ×2. So w17°=13⋅34=0.25w_{17°}=\tfrac13\cdot\tfrac34=0.25, w20°=13(24+14+13)=0.361w_{20°}=\tfrac13(\tfrac24+\tfrac14+\tfrac13)=0.361, w26°=13(24+23)=0.389w_{26°}=\tfrac13(\tfrac24+\tfrac23)=0.389, and 0.25⋅38+0.361⋅44+0.389⋅45=42.890.25\cdot38+0.361\cdot44+0.389\cdot45=42.89 — the same number.
  3. 3OOB predictions. Row 4 (17 °C, 38) was out-of-bag for trees 1 and 3, which predict 44.50 and 22.67 → OOB prediction 33.58, squared error 19.51. Doing this for every row that has at least one OOB tree (rows 5, 8, 9 have none):
  4. 4OOB MSE =17(113.78+21.78+81.00+0.01+19.51+5.84+53.78)==\tfrac17(113.78+21.78+81.00+0.01+19.51+5.84+53.78)= 42.24 — an honest error estimate from just 3 trees.
  5. 5Ambiguity check on the training rows. The three trees' training MSEs are 29.50, 41.22 and 48.36 (average 39.69). Their average spread around the forest is 21.45, so the forest's training MSE is 39.69−21.45=18.2439.69-21.45=18.24 — lower than every one of its trees.
RowTempyOOB treesOOB predictionSquared error
0412122.67113.78
1818322.6721.78
21125216.0081.00
314311, 231.080.01
417381, 333.5819.51
623471, 344.585.84
72645237.6753.78
Out-of-bag predictions from the 3-tree forest. Row 0 is the worst: the only tree that never saw it puts it in a leaf with 8–11 °C days.

See it work

— each tree — forest mean of 3 — tree 10°10°20°30°01020304050temperature → rentals (×10)×2×2×2×2x = 21.0°trees at xŷ 42.89sd 2.40error vs number of trees · OOB MSE · - - one tree (avg)1255075100051
databootstrap
min_samples_leafinspect tree#1
trees T = 3
forest ŷ(21.0°) = 42.89spread of trees (sd) = 2.40tree 1: ŷ = 44.50, 4 rows out-of-bagtrain MSE = 18.24OOB MSE = 42.24
fig.The worked example, live. Thin blue lines are the trees, the bold mint line is their average, and the strip on the right shows every tree's prediction at the dashed query line (drag it). At the start — 3 trees, leaf ≥ 3, query 21 °C — the trees say 44.50, 39.50 and 44.67, and the forest says 42.89 (sd 2.40). Step through trees with ◀ ▶ to see each one's bootstrap counts (×2, ×3) and its out-of-bag days (hollow). Drag T up to 100 and watch the step function smooth out while the OOB curve drops below the dashed one-tree line; switch bootstrap off and every tree becomes identical, so averaging does nothing. On 60 noisy days, test MSE against the true curve falls from 13.33 with one tree to 7.47 with 100 (leaf ≥ 1), and to 4.42 with leaf ≥ 5.

The algorithm, step by step

fit(X, y)

  1. 1For t=1,…,Tt=1,\dots,T (in parallel):
  2. 2Bootstrap. Draw nn (or "max_samples") row indices with replacement; remember which rows are out-of-bag.
  3. 3Grow a regression tree on the sample. At each node: draw max_features candidate features at random; for each, sort the node's values and scan all midpoints, computing left/right SSE from running sums; keep the split with the lowest total SSE.
  4. 4Stop a node when it cannot be split into two children with at least min_samples_leaf rows each (or hits max_depth, or no split reduces SSE). The leaf stores the mean target of its rows.
  5. 5OOB. For each out-of-bag row of tree tt, add ht(xi)h_t(x_i) to that row's running sum and count.
  6. 6Predict xx: drop it down all TT trees and average the TT leaf means. OOB MSE =meani(OOB sumi/counti−yi)2=\text{mean}_i(\text{OOB sum}_i/\text{count}_i-y_i)^2.

Complexity. Sorting dominates: a balanced tree costs about O(m nlog⁡2n)O(m\,n\log^2 n) with per-node sorting, or O(m nlog⁡n)O(m\,n\log n) with pre-sorting or histogram binning; the forest is TT times that, divided by the number of cores. Deep regression trees have roughly n/min_samples_leafn/\text{min\_samples\_leaf} leaves, so memory is O(T n/leaf)O(T\,n/\text{leaf}) nodes — a 500-tree forest on a million rows with leaf size 1 can take gigabytes. Prediction is O(Tlog⁡n)O(T\log n) per row.

Variants and practical tuning

Hyperparameterscikit-learn defaultBreiman / R randomForest (regression)Tune how
n_estimators100500Increase until OOB error flattens; more is never worse, only slower
max_features1.0 (all d)d/3Try {d/3, d/2, 1.0} and √d; fewer when many features are correlated copies of each other
min_samples_leaf15 (nodesize)Try 1, 3, 5, 10, 20; larger for noisy targets
max_samplesn (full bootstrap)n0.5–0.8 to decorrelate and speed up large data
bootstrap / oob_scoreTrue / False—Set oob_score=True for a free validation estimate
Where to start. Library defaults differ, so check them: in scikit-learn the regressor's max_features defaults to 1.0 — all features, i.e. plain bagging.

Relatives

Bagged trees
m=dm=d. Only bootstrap randomness; best when few features matter or features are all informative.
Extra Trees
split thresholds drawn at random too, no bootstrap by default — even lower correlation, a bit more bias, faster to train (see Extra Trees).
Quantile regression forest
keep all leaf targets, not just the mean; use the weights wi(x)w_i(x) to estimate conditional quantiles and give prediction intervals.
Honest / causal forests
use one half of each sample to choose splits and the other to fill leaf means; this gives valid confidence intervals and heterogeneous treatment effects.
Gradient boosting
the other way to combine trees: grow shallow trees sequentially to fix the current residuals. Boosting attacks bias, forests attack variance (see Gradient boosting regression and AdaBoost regression).

In code

python
1import numpy as np
2
3def grow(X, y, min_leaf=3, max_features=None, rng=None):
4 """Regression tree: split on the threshold with the lowest total SSE; leaves store the mean."""
5 n, d = X.shape
6 best = None # (sse, feature, threshold)
7 feats = range(d) if max_features is None else rng.choice(d, max_features, replace=False)
8 for j in feats:
9 order = np.argsort(X[:, j], kind="stable")
10 xs, ys = X[order, j], y[order]
11 cs, cq = np.cumsum(ys), np.cumsum(ys ** 2) # prefix sums -> SSE of every split in O(n)
12 for k in range(min_leaf, n - min_leaf + 1): # k rows go left
13 if xs[k - 1] == xs[k]:
14 continue
15 sl, ql, sr, qr = cs[k - 1], cq[k - 1], cs[-1] - cs[k - 1], cq[-1] - cq[k - 1]
16 sse = ql - sl ** 2 / k + qr - sr ** 2 / (n - k)
17 if best is None or sse < best[0] - 1e-9:
18 best = (sse, j, (xs[k - 1] + xs[k]) / 2)
19 if best is None:
20 return {"value": y.mean(), "n": n}
21 _, j, t = best
22 left = X[:, j] <= t
23 return {"feature": j, "threshold": t,
24 "left": grow(X[left], y[left], min_leaf, max_features, rng),
25 "right": grow(X[~left], y[~left], min_leaf, max_features, rng)}
26
27def predict_tree(node, x):
28 while "value" not in node:
29 node = node["left"] if x[node["feature"]] <= node["threshold"] else node["right"]
30 return node["value"]
31
32class RandomForestRegressor:
33 def __init__(self, n_trees=100, min_leaf=5, max_features=None, seed=0):
34 self.T, self.min_leaf, self.mf, self.seed = n_trees, min_leaf, max_features, seed
35
36 def fit(self, X, y):
37 n = len(y)
38 rng = np.random.default_rng(self.seed)
39 self.trees, self.samples = [], []
40 for _ in range(self.T):
41 idx = rng.integers(0, n, n) # bootstrap: n draws with replacement
42 self.samples.append(idx)
43 self.trees.append(grow(X[idx], y[idx], self.min_leaf, self.mf, rng))
44 # out-of-bag: average only the trees that never saw row i
45 s, c = np.zeros(n), np.zeros(n)
46 for tree, idx in zip(self.trees, self.samples):
47 oob = np.setdiff1d(np.arange(n), idx)
48 s[oob] += [predict_tree(tree, X[i]) for i in oob]
49 c[oob] += 1
50 ok = c > 0
51 self.oob_pred_ = np.where(ok, s / np.maximum(c, 1), np.nan)
52 self.oob_mse_ = np.mean((self.oob_pred_[ok] - y[ok]) ** 2)
53 return self
54
55 def tree_predictions(self, X):
56 return np.array([[predict_tree(t, x) for x in X] for t in self.trees])
57
58 def predict(self, X):
59 return self.tree_predictions(X).mean(axis=0) # the forest = plain average
60
61# Worked example: temperature (°C) -> bike rentals (x10)
62X = np.array([[4], [8], [11], [14], [17], [20], [23], [26], [29], [33]], dtype=float)
63y = np.array([12, 18, 25, 31, 38, 44, 47, 45, 37, 29], dtype=float)
64
65rf = RandomForestRegressor(n_trees=3, min_leaf=3, seed=10).fit(X, y)
66for k, idx in enumerate(rf.samples, 1):
67 print(f"tree {k} bootstrap rows:", sorted(idx.tolist()))
68P = rf.tree_predictions(np.array([[21.0]]))[:, 0]
69print("tree predictions at 21°C:", P.round(2)) # [44.5 39.5 44.67]
70print("forest prediction:", round(P.mean(), 2), " spread (sd):", round(P.std(), 2)) # 42.89, 2.4
71print("OOB predictions:", np.round(rf.oob_pred_, 2))
72print("OOB MSE:", round(rf.oob_mse_, 2)) # 42.24
From scratch with NumPy: SSE splits with prefix sums, bootstrap, feature subsampling, OOB. Running it reproduces the worked example exactly.
python
1import numpy as np
2from sklearn.tree import DecisionTreeRegressor
3from sklearn.ensemble import RandomForestRegressor
4from sklearn.metrics import mean_squared_error
5
6X = np.array([[4], [8], [11], [14], [17], [20], [23], [26], [29], [33]], dtype=float)
7y = np.array([12, 18, 25, 31, 38, 44, 47, 45, 37, 29], dtype=float)
8
9# 1) The three hand-built trees: same bootstrap rows, same leaf rule
10rng = np.random.default_rng(10)
11preds = []
12for k in range(3):
13 idx = rng.integers(0, 10, 10)
14 tree = DecisionTreeRegressor(min_samples_leaf=3).fit(X[idx], y[idx])
15 preds.append(tree.predict([[21.0]])[0])
16print("hand forest at 21°C:", np.round(preds, 2), "->", round(np.mean(preds), 2))
17# hand forest at 21°C: [44.5 39.5 44.67] -> 42.89
18
19# 2) The library forest (its own random bootstraps)
20rf = RandomForestRegressor(n_estimators=500, min_samples_leaf=1, oob_score=True,
21 random_state=0).fit(X, y)
22print("sklearn forest at 21°C:", rf.predict([[21.0]]).round(2)) # [44.2]
23print("OOB R^2:", round(rf.oob_score_, 3), # 0.769
24 " OOB MSE:", round(mean_squared_error(y, rf.oob_prediction_), 2)) # 28.92
25spread = np.array([t.predict([[21.0]])[0] for t in rf.estimators_])
26print("tree spread at 21°C: sd =", round(spread.std(), 2), # 2.26
27 " 10th-90th pct:", np.percentile(spread, [10, 90]).round(1)) # [44. 47.]
scikit-learn: first the three hand-built trees (same bootstrap rows), then a real 500-tree forest on the same 10 days.

Use when

  • +You want a strong nonlinear regressor with almost no tuning: defaults are rarely bad.
  • +Features are mixed in scale and type, with interactions and thresholds; no scaling or transformation needed.
  • +You want a free validation estimate (OOB MSE) and robust permutation importances.
  • +Training must parallelize, or you need a reliable baseline before trying boosting.

Avoid when

  • −Targets will go beyond the training range (trends, growth, time series) — the forest flattens out.
  • −The signal is smooth and close to linear in a few features: a linear or spline model is smaller and more accurate.
  • −You need the last few percent of accuracy on tabular data: tuned gradient boosting usually wins.
  • −Memory or latency is tight: hundreds of deep trees are large and slow to evaluate.

Where it's used

Real-estate valuation

Predicting sale price from size, location, age and amenities, where effects interact and thresholds (e.g. a school-district boundary) matter.

Demand forecasting

Store- or station-level demand from weather, calendar and price features — as long as the future stays inside the range of the past.

Remote sensing & ecology

Biomass, canopy height or species abundance from satellite bands; OOB error and importances are standard in this literature.

Chemistry & materials (QSAR)

Predicting a molecule's solubility or binding affinity from hundreds of descriptors, a classic random-forest benchmark.

Check yourself

Each tree's prediction at x has variance 10 and pairwise correlation 0.2. What is the variance of a 10-tree forest's prediction there?

Check yourself

Three trees predict 30, 36 and 42 for a point whose true value is 40. What is the forest's squared error?

Check yourself

A forest is trained on house prices between 80kand80k and 900k. What can it predict for a mansion far larger than anything in the data?

Check yourself

You switch scikit-learn's RandomForestRegressor from max_features=1.0 to max_features=1/3. What happens to the trees?

Remember

  • ✓Random forest regression averages deep regression trees grown on bootstrap samples, with a random feature subset tried at each split; each leaf predicts a mean.
  • ✓Averaging keeps the bias of one tree and shrinks variance to ρσ2+1−ρTσ2\rho\sigma^2+\tfrac{1-\rho}{T}\sigma^2: more trees remove the second term, decorrelation lowers the first.
  • ✓Under squared error the forest's error equals the average tree error minus the trees' disagreement (ambiguity), so it is never worse than its average tree.
  • ✓A forest is a weighted nearest-neighbour average of training targets: smooth-ish, robust, but unable to extrapolate beyond the training range.
  • ✓Each row is out-of-bag for about 36.8% of trees; OOB MSE is a free test-error estimate. Tune max_features and min_samples_leaf; T only needs to be large enough.
← back to ML Algorithms roadmap