Definition
Random forest regression predicts a number by averaging the outputs of regression trees. Each tree is grown deep on its own bootstrap sample ( rows drawn with replacement), optionally considering only a random subset of features at each split, and each leaf predicts the mean target of the training rows that reach it. The forest's prediction is the plain average .
At a glance
- Task
- Regression (numeric target); squared-error loss
- Input → output
- → , the mean of trees' leaf means
- Learns
- deep regression trees: thresholds per internal node, a mean per leaf
- Training cost
- about with features tried per split; trees are independent, so it parallelizes perfectly
- Prediction cost
- comparisons, then one average
- Key hyperparameters
- number of trees
n_estimators; features per splitmax_features; leaf sizemin_samples_leaf; bootstrap sizemax_samples - Assumes
- nothing distributional; the target can be reached as an average of training targets (no extrapolation)
The problem
A single regression tree (see Decision tree regression) is an excellent low-bias model: grown deep, it can follow any nonlinear shape and any interaction between features. But it pays for that with variance. Its thresholds are chosen greedily from the exact training rows, so moving or removing a handful of points changes the top split, which changes everything underneath. Two trees trained on two samples from the same population can give very different step functions — and at a given , very different numbers.
The usual cure for variance — prune harder, make leaves bigger — trades it straight back for bias: big leaves average over regions where the target is not constant. Linear models have the opposite problem: stable, but blind to thresholds and interactions. We want to keep the deep tree's flexibility and remove its instability.
The idea
If you had independent training sets, you could grow a tree on each and average their predictions: the average has the same expected value as one tree but times less variance. We only have one training set, so we fake new ones with the bootstrap — resample rows with replacement — and grow one deep tree per resample. This is bagging (bootstrap aggregating, Breiman 1996).
Bootstrap trees are not independent, though: they share most rows, so they tend to pick the same strong feature at the root and make similar errors. Averaging correlated errors cancels much less. A random forest (Breiman 2001) adds a second source of randomness — at every split, only a random subset of of the features is searched — so the trees differ more and their errors cancel more. For regression the averaging is literal: no votes, no probabilities, just the arithmetic mean of numbers.
Vocabulary
- Bootstrap sample
- row indices drawn uniformly with replacement from ; some rows appear 2–3 times, others not at all
- Bagging
- fitting the same learner on many bootstrap samples and averaging the fits
- In-bag / out-of-bag (OOB)
- for a tree : the rows in its bootstrap sample / the rows it never saw (about 36.8% of them)
- OOB prediction
- for row : the average over only those trees for which was out-of-bag — an honest, held-out prediction
- max_features (mtry)
- , the number of features randomly offered to each split; turns the forest back into plain bagging
- min_samples_leaf (nodesize)
- smallest number of training rows a leaf may hold; the forest's main smoothing knob in regression
- Tree correlation ρ
- the correlation, at a fixed , between the predictions of two trees of the same forest
- Ambiguity
- the spread of the trees around their own average,
The math
Leaves predict means. Under squared error, a leaf holding training targets () should predict the constant that minimizes . Setting the derivative to zero gives , the leaf mean, and the leftover loss is the leaf's sum of squared errors (SSE). Each tree splits greedily on the threshold that most reduces total SSE — the same variance-reduction rule as a single regression tree. In a bootstrap sample a row that appears twice simply counts twice in both the mean and the SSE.
- number of trees
- prediction of tree : the mean target in the leaf falls into
- the multiset of tree 's in-bag training rows sharing that leaf (duplicates counted)
- the forest prediction
Averaging does not change bias. Every tree is grown by the same random procedure, so they all have the same expected prediction . By linearity of expectation : the forest's bias is exactly the bias of one of its trees. That is why forests use deep, low-bias trees — averaging can only remove variance, so the trees should bring as little bias as possible.
Averaging divides variance — partly. Let each tree's prediction at have variance , and any two trees correlation , so for . The variance of a mean is the sum of all covariances divided by : diagonal terms equal to and off-diagonal terms equal to .
- variance of one tree's prediction at (over training sets and bootstrap draws)
- correlation between two trees of the same forest
- the part more trees remove; it vanishes as
- the floor no number of trees can remove
Putting the pieces into the bias–variance decomposition of the expected squared error at gives the whole story of the method in one line:
- the true regression function
- irreducible noise in y
The ambiguity decomposition. For squared error there is also an exact identity on any single point, with no expectations at all (Krogh & Vedelsby, 1995). Expand , square, and average over ; the cross term vanishes because :
- the true target at
- how much the trees disagree with their own mean
So the forest is never worse than its average tree, and it beats it by exactly how much the trees disagree. Diversity is not a side effect; under squared loss it is literally the profit. The catch is that decorrelating trees (smaller , smaller bootstrap samples) also makes each tree worse, so the first term grows too — the tuning question is which effect wins.
A forest is a weighted nearest-neighbour average. Substituting the leaf means into and swapping the sums gives
- weight of training row in the prediction at ; and
- how many times row appears in tree 's bootstrap sample
The weights are non-negative and sum to one, so every forest prediction is a convex combination of training targets: it always lies between and . Rows that keep landing in the same leaf as get large weight — the forest learns its own adaptive neighbourhood, wide along irrelevant features and narrow along important ones. The same weights are what quantile regression forests reuse to estimate the whole conditional distribution instead of only its mean.
Out-of-bag rows. One bootstrap draw misses row with probability , so all draws miss it with probability (already 0.349 at ). Each row is therefore unseen by about a third of the trees, and averaging just those trees gives an honest held-out prediction. The OOB MSE estimates test error without a separate validation split.
How the hyperparameters act on the formula
At a glance
- n_estimators T
- shrinks only . Error falls fast, then plateaus; bias, and are untouched.
- max_features m
- smaller → trees share fewer splits → falls. But each tree is forced to use weaker features at some splits, so its bias and rise. Best depends on how many features carry signal.
- min_samples_leaf
- bigger leaves average more targets → falls, but the step function gets coarser → bias rises. In regression this matters much more than in classification, because leaf means of 1–2 noisy targets are very noisy.
- max_samples
- smaller bootstrap samples → trees see fewer shared rows → falls, each tree is weaker; also faster.
- max_depth
- a hard cap with the same effect as bigger leaves; usually left unlimited and controlled through leaf size instead.
To see these numbers for real, we simulated the standard Friedman #1 benchmark ( features, only 5 of which matter, noise sd 1): 25 independent training sets of 500 rows, a 200-tree forest on each, scored against the noise-free truth on 1000 test points. Across the 25 forests we measured each tree's variance , the tree correlation , the bias² and the forest's variance.
| Setting | One tree MSE | Tree variance | Tree correlation | Bias² | Forest variance | Forest MSE |
|---|---|---|---|---|---|---|
| m = 1 | 26.46 | 18.77 | 0.016 | 7.70 | 0.39 | 8.09 |
| m = 3 (≈ d/3) | 16.38 | 12.45 | 0.032 | 3.93 | 0.45 | 4.39 |
| m = 5 | 13.12 | 9.83 | 0.053 | 3.29 | 0.56 | 3.86 |
| m = 10 (bagging) | 11.38 | 8.29 | 0.091 | 3.09 | 0.81 | 3.90 |
| m = 10, leaf ≥ 5 | 9.54 | 5.90 | 0.120 | 3.64 | 0.74 | 4.39 |
| m = 10, leaf ≥ 20 | 9.82 | 3.92 | 0.158 | 5.90 | 0.63 | 6.52 |
Read the bagging row: one tree has MSE 11.38 = bias² 3.09 + variance 8.29. The forest keeps the same bias² 3.09 and cuts variance to 0.81 — very close to the floor . Ninety percent of a single tree's variance is gone. With one training set the number of trees behaves as the formula promises: test MSE 10.71 with 1 tree, 4.87 with 5, 3.80 with 25, 3.62 with 100, 3.56 with 500.
Worked example: bike rentals by temperature
Ten days of rentals (in tens of bikes) against the day's temperature. Demand rises with warmth, peaks in the low twenties and falls again on hot days — a hump no straight line can fit. We grow a forest of 3 trees with min_samples_leaf = 3 (a leaf must hold at least 3 bootstrap rows, duplicates included) and predict a 21 °C day.
| Row | Temp (°C) | Rentals (×10) |
|---|---|---|
| 0 | 4 | 12 |
| 1 | 8 | 18 |
| 2 | 11 | 25 |
| 3 | 14 | 31 |
| 4 | 17 | 38 |
| 5 | 20 | 44 |
| 6 | 23 | 47 |
| 7 | 26 | 45 |
| 8 | 29 | 37 |
| 9 | 33 | 29 |
Tree 1, by hand
- 1Bootstrap. Draw 10 row numbers with replacement (NumPy "default_rng(10)"): rows . Rows 2, 5, 7, 8 appear twice; rows 0, 3, 4, 6 are out-of-bag.
- 2Root node. The 10 in-bag targets have mean 34.90 and SSE 894.90. With at least 3 rows per side, only three thresholds are legal (midpoints between distinct temperatures): , and .
- 3Score them (table below). leaves SSE , by far the lowest, so the root asks temp ≤ 15.5? — removing 641.38 of the 894.90.
- 4Left child holds 3 rows (8, 11, 11 °C → 18, 25, 25). Splitting it would leave a side with fewer than 3 rows, so it becomes a leaf predicting .
- 5Right child holds 7 rows: 20, 20, 26, 26, 29, 29, 33 °C (mean 40.14, SSE 220.86). Only keeps 3+ rows on each side: leaves (mean 44.50, SSE 1.00) and (mean 34.33, SSE 42.67). SSE drops by 177.19, so split. Neither leaf can split again.
- 6Tree 1 is the step function 22.67 for , 44.50 for , 34.33 above. At 21 °C it predicts 44.50.
| Threshold | Left rows · mean · SSE | Right rows · mean · SSE | Total SSE |
|---|---|---|---|
| x ≤ 15.5 | 3 · 22.67 · 32.67 | 7 · 40.14 · 220.86 | 253.52 |
| x ≤ 23 | 5 · 31.20 · 578.80 | 5 · 38.60 · 179.20 | 758.00 |
| x ≤ 27.5 | 7 · 35.14 · 850.86 | 3 · 34.33 · 42.67 | 893.52 |
Trees 2 and 3 are grown the same way on their own bootstrap samples:
| Tree | Bootstrap rows | Out-of-bag | Leaves (range → mean, rows) | ŷ(21 °C) |
|---|---|---|---|---|
| 1 | 1, 2, 2, 5, 5, 7, 7, 8, 8, 9 | 0, 3, 4, 6 | ≤15.5 → 22.67 (3) · ≤27.5 → 44.50 (4) · >27.5 → 34.33 (3) | 44.50 |
| 2 | 0, 1, 1, 4, 4, 4, 5, 6, 8, 9 | 2, 3, 7 | ≤12.5 → 16.00 (3) · ≤21.5 → 39.50 (4) · >21.5 → 37.67 (3) | 39.50 |
| 3 | 0, 2, 3, 5, 7, 7, 8, 8, 9, 9 | 1, 4, 6 | ≤17 → 22.67 (3) · ≤27.5 → 44.67 (3) · >27.5 → 33.00 (4) | 44.67 |
The forest
- 1Predict 21 °C. 42.89. The trees disagree with sd 2.40 — that disagreement is the variance averaging removes. (One tree with the same leaf rule grown on all 10 rows predicts 40.00.)
- 2As neighbour weights. Tree 1's leaf at 21 °C holds 20 °C ×2 and 26 °C ×2; tree 2's holds 17 °C ×3 and 20 °C ×1; tree 3's holds 20 °C and 26 °C ×2. So , , , and — the same number.
- 3OOB predictions. Row 4 (17 °C, 38) was out-of-bag for trees 1 and 3, which predict 44.50 and 22.67 → OOB prediction 33.58, squared error 19.51. Doing this for every row that has at least one OOB tree (rows 5, 8, 9 have none):
- 4OOB MSE 42.24 — an honest error estimate from just 3 trees.
- 5Ambiguity check on the training rows. The three trees' training MSEs are 29.50, 41.22 and 48.36 (average 39.69). Their average spread around the forest is 21.45, so the forest's training MSE is — lower than every one of its trees.
| Row | Temp | y | OOB trees | OOB prediction | Squared error |
|---|---|---|---|---|---|
| 0 | 4 | 12 | 1 | 22.67 | 113.78 |
| 1 | 8 | 18 | 3 | 22.67 | 21.78 |
| 2 | 11 | 25 | 2 | 16.00 | 81.00 |
| 3 | 14 | 31 | 1, 2 | 31.08 | 0.01 |
| 4 | 17 | 38 | 1, 3 | 33.58 | 19.51 |
| 6 | 23 | 47 | 1, 3 | 44.58 | 5.84 |
| 7 | 26 | 45 | 2 | 37.67 | 53.78 |
See it work
The algorithm, step by step
fit(X, y)
- 1For (in parallel):
- 2Bootstrap. Draw (or "max_samples") row indices with replacement; remember which rows are out-of-bag.
- 3Grow a regression tree on the sample. At each node: draw
max_featurescandidate features at random; for each, sort the node's values and scan all midpoints, computing left/right SSE from running sums; keep the split with the lowest total SSE. - 4Stop a node when it cannot be split into two children with at least
min_samples_leafrows each (or hitsmax_depth, or no split reduces SSE). The leaf stores the mean target of its rows. - 5OOB. For each out-of-bag row of tree , add to that row's running sum and count.
- 6Predict : drop it down all trees and average the leaf means. OOB MSE .
Complexity. Sorting dominates: a balanced tree costs about with per-node sorting, or with pre-sorting or histogram binning; the forest is times that, divided by the number of cores. Deep regression trees have roughly leaves, so memory is nodes — a 500-tree forest on a million rows with leaf size 1 can take gigabytes. Prediction is per row.
Variants and practical tuning
| Hyperparameter | scikit-learn default | Breiman / R randomForest (regression) | Tune how |
|---|---|---|---|
| n_estimators | 100 | 500 | Increase until OOB error flattens; more is never worse, only slower |
| max_features | 1.0 (all d) | d/3 | Try {d/3, d/2, 1.0} and √d; fewer when many features are correlated copies of each other |
| min_samples_leaf | 1 | 5 (nodesize) | Try 1, 3, 5, 10, 20; larger for noisy targets |
| max_samples | n (full bootstrap) | n | 0.5–0.8 to decorrelate and speed up large data |
| bootstrap / oob_score | True / False | — | Set oob_score=True for a free validation estimate |
Relatives
- Bagged trees
- . Only bootstrap randomness; best when few features matter or features are all informative.
- Extra Trees
- split thresholds drawn at random too, no bootstrap by default — even lower correlation, a bit more bias, faster to train (see Extra Trees).
- Quantile regression forest
- keep all leaf targets, not just the mean; use the weights to estimate conditional quantiles and give prediction intervals.
- Honest / causal forests
- use one half of each sample to choose splits and the other to fill leaf means; this gives valid confidence intervals and heterogeneous treatment effects.
- Gradient boosting
- the other way to combine trees: grow shallow trees sequentially to fix the current residuals. Boosting attacks bias, forests attack variance (see Gradient boosting regression and AdaBoost regression).
In code
1import numpy as np23def grow(X, y, min_leaf=3, max_features=None, rng=None):4 """Regression tree: split on the threshold with the lowest total SSE; leaves store the mean."""5 n, d = X.shape6 best = None # (sse, feature, threshold)7 feats = range(d) if max_features is None else rng.choice(d, max_features, replace=False)8 for j in feats:9 order = np.argsort(X[:, j], kind="stable")10 xs, ys = X[order, j], y[order]11 cs, cq = np.cumsum(ys), np.cumsum(ys ** 2) # prefix sums -> SSE of every split in O(n)12 for k in range(min_leaf, n - min_leaf + 1): # k rows go left13 if xs[k - 1] == xs[k]:14 continue15 sl, ql, sr, qr = cs[k - 1], cq[k - 1], cs[-1] - cs[k - 1], cq[-1] - cq[k - 1]16 sse = ql - sl ** 2 / k + qr - sr ** 2 / (n - k)17 if best is None or sse < best[0] - 1e-9:18 best = (sse, j, (xs[k - 1] + xs[k]) / 2)19 if best is None:20 return {"value": y.mean(), "n": n}21 _, j, t = best22 left = X[:, j] <= t23 return {"feature": j, "threshold": t,24 "left": grow(X[left], y[left], min_leaf, max_features, rng),25 "right": grow(X[~left], y[~left], min_leaf, max_features, rng)}2627def predict_tree(node, x):28 while "value" not in node:29 node = node["left"] if x[node["feature"]] <= node["threshold"] else node["right"]30 return node["value"]3132class RandomForestRegressor:33 def __init__(self, n_trees=100, min_leaf=5, max_features=None, seed=0):34 self.T, self.min_leaf, self.mf, self.seed = n_trees, min_leaf, max_features, seed3536 def fit(self, X, y):37 n = len(y)38 rng = np.random.default_rng(self.seed)39 self.trees, self.samples = [], []40 for _ in range(self.T):41 idx = rng.integers(0, n, n) # bootstrap: n draws with replacement42 self.samples.append(idx)43 self.trees.append(grow(X[idx], y[idx], self.min_leaf, self.mf, rng))44 # out-of-bag: average only the trees that never saw row i45 s, c = np.zeros(n), np.zeros(n)46 for tree, idx in zip(self.trees, self.samples):47 oob = np.setdiff1d(np.arange(n), idx)48 s[oob] += [predict_tree(tree, X[i]) for i in oob]49 c[oob] += 150 ok = c > 051 self.oob_pred_ = np.where(ok, s / np.maximum(c, 1), np.nan)52 self.oob_mse_ = np.mean((self.oob_pred_[ok] - y[ok]) ** 2)53 return self5455 def tree_predictions(self, X):56 return np.array([[predict_tree(t, x) for x in X] for t in self.trees])5758 def predict(self, X):59 return self.tree_predictions(X).mean(axis=0) # the forest = plain average6061# Worked example: temperature (°C) -> bike rentals (x10)62X = np.array([[4], [8], [11], [14], [17], [20], [23], [26], [29], [33]], dtype=float)63y = np.array([12, 18, 25, 31, 38, 44, 47, 45, 37, 29], dtype=float)6465rf = RandomForestRegressor(n_trees=3, min_leaf=3, seed=10).fit(X, y)66for k, idx in enumerate(rf.samples, 1):67 print(f"tree {k} bootstrap rows:", sorted(idx.tolist()))68P = rf.tree_predictions(np.array([[21.0]]))[:, 0]69print("tree predictions at 21°C:", P.round(2)) # [44.5 39.5 44.67]70print("forest prediction:", round(P.mean(), 2), " spread (sd):", round(P.std(), 2)) # 42.89, 2.471print("OOB predictions:", np.round(rf.oob_pred_, 2))72print("OOB MSE:", round(rf.oob_mse_, 2)) # 42.241import numpy as np2from sklearn.tree import DecisionTreeRegressor3from sklearn.ensemble import RandomForestRegressor4from sklearn.metrics import mean_squared_error56X = np.array([[4], [8], [11], [14], [17], [20], [23], [26], [29], [33]], dtype=float)7y = np.array([12, 18, 25, 31, 38, 44, 47, 45, 37, 29], dtype=float)89# 1) The three hand-built trees: same bootstrap rows, same leaf rule10rng = np.random.default_rng(10)11preds = []12for k in range(3):13 idx = rng.integers(0, 10, 10)14 tree = DecisionTreeRegressor(min_samples_leaf=3).fit(X[idx], y[idx])15 preds.append(tree.predict([[21.0]])[0])16print("hand forest at 21°C:", np.round(preds, 2), "->", round(np.mean(preds), 2))17# hand forest at 21°C: [44.5 39.5 44.67] -> 42.891819# 2) The library forest (its own random bootstraps)20rf = RandomForestRegressor(n_estimators=500, min_samples_leaf=1, oob_score=True,21 random_state=0).fit(X, y)22print("sklearn forest at 21°C:", rf.predict([[21.0]]).round(2)) # [44.2]23print("OOB R^2:", round(rf.oob_score_, 3), # 0.76924 " OOB MSE:", round(mean_squared_error(y, rf.oob_prediction_), 2)) # 28.9225spread = np.array([t.predict([[21.0]])[0] for t in rf.estimators_])26print("tree spread at 21°C: sd =", round(spread.std(), 2), # 2.2627 " 10th-90th pct:", np.percentile(spread, [10, 90]).round(1)) # [44. 47.]Use when
- +You want a strong nonlinear regressor with almost no tuning: defaults are rarely bad.
- +Features are mixed in scale and type, with interactions and thresholds; no scaling or transformation needed.
- +You want a free validation estimate (OOB MSE) and robust permutation importances.
- +Training must parallelize, or you need a reliable baseline before trying boosting.
Avoid when
- −Targets will go beyond the training range (trends, growth, time series) — the forest flattens out.
- −The signal is smooth and close to linear in a few features: a linear or spline model is smaller and more accurate.
- −You need the last few percent of accuracy on tabular data: tuned gradient boosting usually wins.
- −Memory or latency is tight: hundreds of deep trees are large and slow to evaluate.
Where it's used
Real-estate valuation
Predicting sale price from size, location, age and amenities, where effects interact and thresholds (e.g. a school-district boundary) matter.
Demand forecasting
Store- or station-level demand from weather, calendar and price features — as long as the future stays inside the range of the past.
Remote sensing & ecology
Biomass, canopy height or species abundance from satellite bands; OOB error and importances are standard in this literature.
Chemistry & materials (QSAR)
Predicting a molecule's solubility or binding affinity from hundreds of descriptors, a classic random-forest benchmark.
Check yourself
Each tree's prediction at x has variance 10 and pairwise correlation 0.2. What is the variance of a 10-tree forest's prediction there?
Check yourself
Three trees predict 30, 36 and 42 for a point whose true value is 40. What is the forest's squared error?
Check yourself
A forest is trained on house prices between 900k. What can it predict for a mansion far larger than anything in the data?
Check yourself
You switch scikit-learn's RandomForestRegressor from max_features=1.0 to max_features=1/3. What happens to the trees?
Remember
- ✓Random forest regression averages deep regression trees grown on bootstrap samples, with a random feature subset tried at each split; each leaf predicts a mean.
- ✓Averaging keeps the bias of one tree and shrinks variance to : more trees remove the second term, decorrelation lowers the first.
- ✓Under squared error the forest's error equals the average tree error minus the trees' disagreement (ambiguity), so it is never worse than its average tree.
- ✓A forest is a weighted nearest-neighbour average of training targets: smooth-ish, robust, but unable to extrapolate beyond the training range.
- ✓Each row is out-of-bag for about 36.8% of trees; OOB MSE is a free test-error estimate. Tune max_features and min_samples_leaf; T only needs to be large enough.