Definition
AdaBoost.R2 (Drucker, 1997) is the regression version of AdaBoost. It trains a sequence of regression trees, each one on the original targets but with the training points reweighted so that points the previous trees predicted badly count more. Each tree gets a confidence , and the ensemble predicts the weighted median of the trees' outputs — not their sum.
At a glance
- Task
- Regression (a numeric target)
- Input → output
- → the -weighted median of
- Learns
- regression trees and one confidence per tree
- Training
- sequential tree fits: about ; rounds cannot run in parallel
- Prediction
- tree walks plus a sort of numbers:
- Hyperparameters
n_estimators, base-treemax_depth,loss(linear / square / exponential),learning_rate- Assumes
- each tree alone is better than "chance": its weighted average loss ; targets are not dominated by gross outliers
The problem
A single regression tree (see decision-tree-regression) is a step function fitted by least squares. Grow it shallow and it misses whole regions; grow it deep and it chases noise. Bagging (random-forest-regression) averages many deep trees to cut variance, but every tree sees the data the same way, so a region that all trees find hard stays hard.
Classification AdaBoost solved this for labels: after each round, up-weight the points you got wrong so the next learner is forced to look at them. But "wrong" is binary for a label and continuous for a number — an error of 0.1 and an error of 3 are both "wrong". AdaBoost.R2 needs (1) a way to turn a real-valued error into a loss on a fixed scale, so the AdaBoost weight formula still applies, and (2) a way to combine real-valued predictions that does not let one confidently wrong tree drag the answer away. Its answers are the relative loss and the weighted median.
The idea
Picture ten people each guessing a number, where every guesser was trained on a slightly different emphasis of the same data. You do not add their guesses; you sort them, give each a vote proportional to how reliable they were on training data, and take the guess where the votes reach the halfway point. That is the weighted median: a vote, robust to a few wild guesses.
The emphasis comes from the weights. After a tree is fitted, every training point gets a loss between 0 (predicted perfectly) and 1 (the worst point of this round). The weights of well-predicted points are multiplied by a factor — the better the tree was overall, the smaller — while the worst point keeps its weight. After renormalising, the hard points hold a larger share, and the next tree, which minimises weighted squared error, bends towards them.
Vocabulary
- Sample weight wᵢ
- importance of training point in round ; non-negative and summing to 1. Starts at .
- Base learner hₜ
- the regression tree fitted in round by weighted least squares (or, in scikit-learn, on a weighted bootstrap resample)
- Absolute error eᵢ
- on every training point
- Max error Eₜ
- — the scale that makes losses relative
- Relative loss Lᵢ
- passed through the chosen loss (linear, square or exponential); always in
- Average loss L̄ₜ
- — the weighted badness of tree . Must be .
- Confidence βₜ
- , in for a useful tree: small = good tree
- Tree weight αₜ
- — the tree's vote in the weighted median
- Weighted median
- sort the predictions; output the first one at which the cumulative reaches half of
The math
Step 1 — relative loss. After fitting , compute every training error and scale it by the largest one, so the worst point has loss exactly 1 whatever the units of :
- target and input of training point
- prediction of the round- tree
- largest absolute training error of that tree
- loss shape: (linear), (square) or (exponential)
The shape decides how sharply the algorithm separates "slightly off" from "badly off". With and : linear gives , square , exponential . Square pushes all but the worst points towards 0, so their weights shrink more and attention concentrates on the few largest errors. Exponential caps the worst loss at , so even the worst point's weight is shrunk a little.
Step 2 — how good was this tree? Average the losses under the current weights and turn the average into a confidence:
- weight of point in round ()
- weighted average loss, in
- confidence ratio;
- the tree's vote in the final median
- learning rate in (scikit-learn's
learning_rate)
is the odds of the average loss. A tree with has and ; a tree with has and — it gets no vote. If then , so the algorithm stops and discards that tree (keeping it only if it is the very first). This is the regression analogue of AdaBoost's "weak learner must beat 50% error" condition.
Step 3 — reweight. Every point's weight is multiplied by raised to how well it was predicted, then renormalised:
- how well point was predicted: 1 for a perfect fit, 0 for the worst point
- normaliser that makes the new weights sum to 1
Taking logs shows what really happens: . Each point's log-weight drops by the tree's confidence times how well that point was fitted. The worst point () loses nothing; a perfectly fitted point loses . A confident tree (large ) moves weights a lot; a barely useful tree hardly moves them. The learning rate scales both the vote and this log-weight change, so smaller means gentler reweighting and more rounds needed.
Step 4 — combine with a weighted median. Why a median? Ask which single number is closest to the trees' predictions in weighted absolute distance:
- ensemble prediction at input x
- prediction of tree at
- tree weight from step 2
The derivative is (weight of trees below ) minus (weight of trees above ). It changes sign at the point where the sorted cumulative weight first reaches half the total — the weighted median. Concretely: sort ascending, accumulate , and return the first prediction whose cumulative weight is . A weighted mean would minimise squared distance instead, and one tree with a wild prediction could pull it arbitrarily far; the median ignores how far away the outvoted trees are.
Worked example: ad spend and sales
Ten weeks of advertising spend (thousand dollars) and sales (thousand units). The relationship is S-shaped: little effect at first, a jump around , then steady growth. We run 3 rounds of AdaBoost.R2 with depth-2 trees, linear loss and . To keep every number reproducible by hand, each tree is fitted by weighted least squares (the leaf value is the weighted mean of its points) instead of on a random resample.
| x (spend, thousand dollars) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
|---|---|---|---|---|---|---|---|---|---|---|
| y (sales, k units) | 0.8 | 1.5 | 2.5 | 2.2 | 3.0 | 5.3 | 7.4 | 7.9 | 8.2 | 10.5 |
Round 1 by hand
- 1Weights. All , so tree 1 is an ordinary depth-2 least-squares tree: : ? ( ? 1.15 : 2.567) : ( ? 7.2 : 10.5).
- 2Errors. . The worst point is (true 5.3, predicted 7.2), so .
- 3Relative losses : .
- 4Average loss. , so the tree is kept.
- 5Confidence. , .
- 6Reweight. Multiply each weight by : keeps its (exponent 0); , fitted perfectly, becomes ; () becomes . The ten products sum to ; dividing gives — nearly double its fair share.
Rounds 2 and 3
- 1Tree 2 is fitted with . The heavy point now pays for its own leaf: : ? ( ? 1.15 : 2.575) : ( ? 5.3 : 8.393). Leaves are weighted means, e.g. is the -weighted mean of at .
- 2Its worst point is now (10.5 vs 8.393): , , , . The weight of jumps from 0.0739 to 0.1919.
- 3Tree 3 with : : ? ( ? 1.992 : 5.3) : ( ? 7.819 : 10.5). Worst point (), , , .
| x | y | w⁽¹⁾ | w⁽²⁾ | w⁽³⁾ | w⁽⁴⁾ | h₁(x) | h₂(x) | h₃(x) |
|---|---|---|---|---|---|---|---|---|
| 1 | 0.8 | 0.1 | 0.0878 | 0.0770 | 0.1378 | 1.150 | 1.150 | 1.992 |
| 2 | 1.5 | 0.1 | 0.0878 | 0.0770 | 0.0820 | 1.150 | 1.150 | 1.992 |
| 3 | 2.5 | 0.1 | 0.0764 | 0.0565 | 0.0609 | 2.567 | 2.575 | 1.992 |
| 4 | 2.2 | 0.1 | 0.0885 | 0.0788 | 0.0680 | 2.567 | 2.575 | 1.992 |
| 5 | 3.0 | 0.1 | 0.0914 | 0.0840 | 0.1311 | 2.567 | 2.575 | 1.992 |
| 6 | 5.3 | 0.1 | 0.1878 | 0.1326 | 0.0981 | 7.200 | 5.300 | 5.300 |
| 7 | 7.4 | 0.1 | 0.0815 | 0.1064 | 0.1074 | 7.200 | 8.393 | 7.819 |
| 8 | 7.9 | 0.1 | 0.1042 | 0.0998 | 0.0784 | 7.200 | 8.393 | 7.819 |
| 9 | 8.2 | 0.1 | 0.1207 | 0.0961 | 0.0943 | 7.200 | 8.393 | 7.819 |
| 10 | 10.5 | 0.1 | 0.0739 | 0.1919 | 0.1419 | 10.500 | 8.393 | 10.500 |
| Round | Worst point | E_t | L̄_t | β_t | α_t | Tree's own train MSE |
|---|---|---|---|---|---|---|
| 1 | x = 6 | 1.900 | 0.2825 | 0.3936 | 0.9323 | 0.571 |
| 2 | x = 10 | 2.107 | 0.2139 | 0.2721 | 1.3016 | 0.628 |
| 3 | x = 1 | 1.192 | 0.2923 | 0.4130 | 0.8842 | 0.331 |
| Ensemble | Σα = 3.1181 | 0.090 |
Predicting sales for a new week with x = 7.5
- 1Tree outputs: , , .
- 2Sort with their votes: , , . Half of the total vote is .
- 3Cumulative vote: after the first (not yet half), after the second.
- 4Prediction: thousand units. (The weighted mean would be 7.874; here the two agree closely, but on the training set the median is much better: MSE 0.090 vs 0.219.)
Notice what the median did on the training set: at two trees say 5.3 and outvote tree 1's 7.2; at two trees say 10.5 and outvote tree 2's 8.393. Each tree has one region it gets badly wrong, the reweighting made sure those regions differ, and the vote picks the majority view point by point. The ensemble's train is 0.991.
β = L̄/(1−L̄) = 0.4130 · α = ln(1/β) = 0.8842 · train MSE: this tree 0.331, ensemble 0.090
all 10 rounds ran with L̄ < 0.5.
The algorithm, step by step
AdaBoost.R2(X, y, T, loss, ν)
- 1Initialise for every training point.
- 2Fit tree to using weights — either directly via
sample_weightor, as in Drucker's paper and scikit-learn, on a bootstrap resample of size drawn with probabilities . - 3Score: , (if the tree is perfect: keep it and stop), , .
- 4Stop if : discard (unless it is the first tree) and end training.
- 5Vote: , .
- 6Reweight: , then divide by the sum. Repeat from step 2 until trees.
- 7Predict at : sort , return the first one whose cumulative reaches .
Complexity
- Training
- tree fits, each for a depth-limited CART tree, plus per round for errors and weights. Strictly sequential: round needs round 's weights.
- Prediction
- to evaluate the trees plus to sort for the median, per input.
- Memory
- small trees and floats ; the weights are only needed during training.
Hyperparameters and variants
To see the knobs at a realistic size, take (), 200 training and 1000 test points. A single depth-4 tree has test MSE 0.1608. AdaBoost.R2 with 300 depth-4 trees (scikit-learn defaults otherwise) reaches 0.0378; the curve over rounds is 0.2364 after 1 tree, 0.0779 after 10, 0.0382 after 100, 0.0378 after 300.
| Setting | Trees actually kept | Test MSE |
|---|---|---|
| Single tree, depth 4 | 1 | 0.1608 |
| R2, depth 1 (stumps) | 9 (stopped) | 0.5187 |
| R2, depth 2 | 7 (stopped) | 0.3685 |
| R2, depth 3 | 81 (stopped) | 0.1315 |
| R2, depth 4, linear loss | 300 | 0.0378 |
| R2, depth 4, square loss | 300 | 0.0322 |
| R2, depth 4, exponential loss | 300 | 0.0402 |
| R2, depth 4, linear, ν = 0.1 | 300 | 0.0451 |
| R2, depth 6 | 300 | 0.0266 |
| Gradient boosting, depth 4, ν = 0.1, 300 trees | 300 | 0.0278 |
Tuning guide
- Base-tree depth
- The most important knob. Each tree models itself, so it must reach alone. Stumps stop after a few rounds (above: 9 trees, MSE 0.52). Start at depth 3–4 (sklearn's default base is depth 3).
- n_estimators (T)
- More trees rarely hurt much — the median is robust — but returns flatten (0.0382 at 100, 0.0378 at 300). Use
staged_predicton a validation set to pick . - loss
- linear (default) is a safe start. square focuses harder on the largest errors (best here, 0.0322). exponential is gentlest on the worst point. Differences are usually small; tune depth first.
- learning_rate (ν)
- Scales and the reweighting exponent. In the worked example turns into . Smaller needs more trees; unlike gradient boosting it rarely improves accuracy (here 0.0451 at vs 0.0378 at 1).
- Resampling vs reweighting
- Drucker's R2 and scikit-learn fit each tree on a weighted bootstrap resample, which adds randomness and diversity; fitting with
sample_weightdirectly is deterministic (our worked example). Setrandom_statefor reproducibility. - Relatives
- AdaBoost.RT (Solomatine & Shrestha) thresholds the relative error into right/wrong and reuses AdaBoost.M1; gradient boosting replaces reweighting with residual fitting and is usually the stronger default today.
In code
1import numpy as np23def fit_tree(x, y, w, depth):4 """Weighted least-squares regression tree on one feature (x sorted)."""5 mean = (w * y).sum() / w.sum()6 sse = lambda m: (w[m] * (y[m] - (w[m] * y[m]).sum() / w[m].sum()) ** 2).sum()7 if depth == 0 or len(x) < 2:8 return mean9 best, t_best = sse(np.ones(len(x), bool)), None10 for t in (x[:-1] + x[1:]) / 2:11 L = x <= t12 s = sse(L) + sse(~L)13 if s < best - 1e-12:14 best, t_best = s, t15 if t_best is None:16 return mean17 L = x <= t_best18 return (t_best, fit_tree(x[L], y[L], w[L], depth - 1), fit_tree(x[~L], y[~L], w[~L], depth - 1))1920def predict_tree(node, x):21 while isinstance(node, tuple):22 t, left, right = node23 node = left if x <= t else right24 return node2526def adaboost_r2(x, y, M=3, depth=2, loss="linear", lr=1.0):27 n = len(x)28 w = np.full(n, 1 / n)29 trees, alphas = [], []30 for m in range(M):31 tree = fit_tree(x, y, w, depth)32 pred = np.array([predict_tree(tree, xi) for xi in x])33 err = np.abs(y - pred)34 if err.max() == 0: # perfect fit: keep it and stop35 trees.append(tree); alphas.append(1.0); break36 L = err / err.max() # relative loss in [0, 1]37 if loss == "square": L = L ** 238 if loss == "exponential": L = 1 - np.exp(-L)39 Lbar = (w * L).sum() # weighted average loss40 if Lbar >= 0.5: # no better than chance: stop41 if not trees: trees.append(tree); alphas.append(1.0)42 break43 beta = Lbar / (1 - Lbar) # confidence: small beta = good learner44 trees.append(tree); alphas.append(lr * np.log(1 / beta))45 print(f"round {m+1}: Lbar={Lbar:.4f} beta={beta:.4f} alpha={alphas[-1]:.4f}")46 w = w * beta ** ((1 - L) * lr) # shrink weights of well-fit points47 w = w / w.sum()48 print(" new weights", np.round(w, 4))49 return trees, np.array(alphas)5051def predict(trees, alphas, x_new):52 preds = np.array([predict_tree(t, x_new) for t in trees])53 order = np.argsort(preds, kind="stable")54 cum = np.cumsum(alphas[order])55 k = np.searchsorted(cum, 0.5 * cum[-1]) # first cumulative weight >= half56 return preds[order[k]], preds5758x = np.arange(1, 11, dtype=float)59y = np.array([0.8, 1.5, 2.5, 2.2, 3.0, 5.3, 7.4, 7.9, 8.2, 10.5])60trees, alphas = adaboost_r2(x, y)61fit = np.array([predict(trees, alphas, xi)[0] for xi in x])62print("train MSE", round(((fit - y) ** 2).mean(), 4)) # 0.089963yhat, members = predict(trees, alphas, 7.5)64print("x=7.5 members", np.round(members, 3), "-> weighted median", round(yhat, 3))65# x=7.5 members [7.2 8.393 7.819] -> weighted median 7.8191import numpy as np2from sklearn.ensemble import AdaBoostRegressor3from sklearn.tree import DecisionTreeRegressor4from sklearn.metrics import mean_squared_error56# 1) The worked example. sklearn resamples by weight, so its trees differ from the hand run.7X = np.arange(1, 11, dtype=float).reshape(-1, 1)8y = np.array([0.8, 1.5, 2.5, 2.2, 3.0, 5.3, 7.4, 7.9, 8.2, 10.5])9ada = AdaBoostRegressor(DecisionTreeRegressor(max_depth=2), n_estimators=3,10 loss="linear", learning_rate=1.0, random_state=0).fit(X, y)11print("alpha ", ada.estimator_weights_.round(4)) # [1.3988 1.3998 0.7881]12print("weighted loss", ada.estimator_errors_.round(4)) # [0.198 0.1978 0.3126]13print("train MSE ", round(mean_squared_error(y, ada.predict(X)), 4)) # 0.620114print("x = 7.5 -> ", ada.predict([[7.5]]).round(3)) # [8.]1516# 2) A realistic size: y = sin x + sin 6x + noise, 200 train / 1000 test points.17rng = np.random.RandomState(1)18f = lambda X: np.sin(X).ravel() + np.sin(6 * X).ravel()19Xtr = np.sort(rng.uniform(0, 6, 200))[:, None]; ytr = f(Xtr) + rng.normal(0, 0.1, 200)20Xte = np.sort(rng.uniform(0, 6, 1000))[:, None]; yte = f(Xte) + rng.normal(0, 0.1, 1000)2122tree = DecisionTreeRegressor(max_depth=4).fit(Xtr, ytr)23print("one depth-4 tree test MSE", round(mean_squared_error(yte, tree.predict(Xte)), 4)) # 0.160824for loss in ["linear", "square", "exponential"]:25 ada = AdaBoostRegressor(DecisionTreeRegressor(max_depth=4), n_estimators=300,26 loss=loss, random_state=0).fit(Xtr, ytr)27 print(f"AdaBoost.R2 {loss:<11} test MSE", round(mean_squared_error(yte, ada.predict(Xte)), 4))28# linear 0.0378 · square 0.0322 · exponential 0.04022930# staged_predict gives the ensemble after 1, 2, ... trees: choose n_estimators on validation data31ada = AdaBoostRegressor(DecisionTreeRegressor(max_depth=4), n_estimators=300, random_state=0).fit(Xtr, ytr)32curve = [mean_squared_error(yte, p) for p in ada.staged_predict(Xte)]33print("after 1/10/100/300 trees", [round(curve[i], 4) for i in (0, 9, 99, 299)])Use when
- +You want a quick, strong upgrade over a single regression tree with almost no tuning (depth 3–4, a few hundred trees).
- +Targets are clean and you want an ensemble whose output is always one of its members' predictions (the median never averages into implausible values).
- +You want a combination rule that is robust to a few badly wrong trees.
- +You are teaching or comparing boosting schemes: R2 shows reweighting cleanly, separate from residual fitting.
Avoid when
- −Targets contain gross outliers or label noise: the reweighting concentrates on them (test MSE 1.39 vs 0.64 for a single tree above).
- −You need the best accuracy on tabular data: gradient boosting, XGBoost, LightGBM and CatBoost usually win and have robust losses.
- −Training time matters: rounds are sequential and cannot be parallelised across trees.
- −You need smooth predictions or calibrated uncertainty: the output is a step function picked by a vote.
Where it's used
Electricity load forecasting
Boosted trees over weather and calendar features; the weighted median resists the occasional tree that overreacts to a heat wave.
Hydrology
River-flow and rainfall–runoff prediction, where AdaBoost.R2 and its variant AdaBoost.RT were studied as upgrades to single regression trees and neural nets.
Materials and chemistry
Predicting a property (strength, band gap, solubility) from descriptors with small datasets, as a strong baseline alongside random forests.
Image-based regression
Boosted regressors for age estimation or head-pose angles from face features, a classic use of boosting in vision.
Check yourself
A tree's weighted average loss is . What are and its vote (with )?
Check yourself
Three trees predict 4, 6 and 9 at some , with votes . What does AdaBoost.R2 output?
Check yourself
A point's absolute error is 2 and the round's maximum error is 4. What is its loss under the square loss?
Check yourself
Training stops early with far fewer trees than n_estimators. What is the most likely cause?
Remember
- ✓Each round fits a tree to the original targets under sample weights; nothing is fitted to residuals and nothing is added up.
- ✓Errors become relative losses by dividing by the round's maximum error (then linear, square or exponential shaping).
- ✓; well-fitted points are multiplied by , so hard points gain weight. Training stops when .
- ✓Prediction is the -weighted median (), the minimiser of weighted absolute deviation — a vote that ignores far-off trees.
- ✓Use base trees of depth 3–4; stumps stall. Tune depth, then the number of trees with staged_predict.
- ✓Outliers attract weight round after round: clean targets first, or prefer gradient boosting with a robust loss.