AdaBoost regression

AdaBoost.R2 reweights training points by their relative error and combines learners with a weighted median.

16 min read1826 wordsadvanced10 viewsupdated 2026-10-07

Definition

AdaBoost.R2 (Drucker, 1997) is the regression version of AdaBoost. It trains a sequence of regression trees, each one on the original targets but with the training points reweighted so that points the previous trees predicted badly count more. Each tree gets a confidence αt\alpha_t, and the ensemble predicts the weighted median of the trees' outputs — not their sum.

At a glance

Task
Regression (a numeric target)
Input → output
xx → the α\alpha-weighted median of h1(x),…,hT(x)h_1(x),\dots,h_T(x)
Learns
TT regression trees hth_t and one confidence αt=ln⁡(1/βt)\alpha_t=\ln(1/\beta_t) per tree
Training
TT sequential tree fits: about O(T d nlog⁡n)O(T\,d\,n\log n); rounds cannot run in parallel
Prediction
TT tree walks plus a sort of TT numbers: O(T depth+Tlog⁡T)O(T\,\text{depth}+T\log T)
Hyperparameters
n_estimators TT, base-tree max_depth, loss (linear / square / exponential), learning_rate ν\nu
Assumes
each tree alone is better than "chance": its weighted average loss Lˉt<0.5\bar L_t<0.5; targets are not dominated by gross outliers

The problem

A single regression tree (see decision-tree-regression) is a step function fitted by least squares. Grow it shallow and it misses whole regions; grow it deep and it chases noise. Bagging (random-forest-regression) averages many deep trees to cut variance, but every tree sees the data the same way, so a region that all trees find hard stays hard.

Classification AdaBoost solved this for labels: after each round, up-weight the points you got wrong so the next learner is forced to look at them. But "wrong" is binary for a label and continuous for a number — an error of 0.1 and an error of 3 are both "wrong". AdaBoost.R2 needs (1) a way to turn a real-valued error into a loss on a fixed [0,1][0,1] scale, so the AdaBoost weight formula still applies, and (2) a way to combine real-valued predictions that does not let one confidently wrong tree drag the answer away. Its answers are the relative loss Li=∣ei∣/max⁡j∣ej∣L_i=|e_i|/\max_j|e_j| and the weighted median.

The idea

Picture ten people each guessing a number, where every guesser was trained on a slightly different emphasis of the same data. You do not add their guesses; you sort them, give each a vote proportional to how reliable they were on training data, and take the guess where the votes reach the halfway point. That is the weighted median: a vote, robust to a few wild guesses.

The emphasis comes from the weights. After a tree is fitted, every training point gets a loss between 0 (predicted perfectly) and 1 (the worst point of this round). The weights of well-predicted points are multiplied by a factor β<1\beta<1 — the better the tree was overall, the smaller β\beta — while the worst point keeps its weight. After renormalising, the hard points hold a larger share, and the next tree, which minimises weighted squared error, bends towards them.

Vocabulary

Sample weight wᵢ
importance of training point ii in round tt; non-negative and summing to 1. Starts at 1/n1/n.
Base learner hₜ
the regression tree fitted in round tt by weighted least squares (or, in scikit-learn, on a weighted bootstrap resample)
Absolute error eᵢ
∣yi−ht(xi)∣|y_i-h_t(x_i)| on every training point
Max error Eₜ
max⁡iei\max_i e_i — the scale that makes losses relative
Relative loss Lᵢ
ei/Ete_i/E_t passed through the chosen loss (linear, square or exponential); always in [0,1][0,1]
Average loss L̄ₜ
∑iwiLi\sum_i w_i L_i — the weighted badness of tree tt. Must be <0.5<0.5.
Confidence βₜ
Lˉt/(1−Lˉt)\bar L_t/(1-\bar L_t), in (0,1)(0,1) for a useful tree: small = good tree
Tree weight αₜ
ln⁡(1/βt)\ln(1/\beta_t) — the tree's vote in the weighted median
Weighted median
sort the TT predictions; output the first one at which the cumulative α\alpha reaches half of ∑tαt\sum_t\alpha_t

The math

Step 1 — relative loss. After fitting hth_t, compute every training error and scale it by the largest one, so the worst point has loss exactly 1 whatever the units of yy:

Li=g ⁣(∣yi−ht(xi)∣Et),Et=max⁡j∣yj−ht(xj)∣L_i=g\!\left(\frac{|y_i-h_t(x_i)|}{E_t}\right),\qquad E_t=\max_j |y_j-h_t(x_j)|
yi,  xiy_i,\;x_i
target and input of training point ii
ht(xi)h_t(x_i)
prediction of the round-tt tree
EtE_t
largest absolute training error of that tree
g(u)g(u)
loss shape: uu (linear), u2u^2 (square) or 1−e−u1-e^{-u} (exponential)

The shape gg decides how sharply the algorithm separates "slightly off" from "badly off". With ∣e∣=1|e|=1 and E=4E=4: linear gives L=0.25L=0.25, square 0.06250.0625, exponential 0.22120.2212. Square pushes all but the worst points towards 0, so their weights shrink more and attention concentrates on the few largest errors. Exponential caps the worst loss at 1−e−1=0.63211-e^{-1}=0.6321, so even the worst point's weight is shrunk a little.

Step 2 — how good was this tree? Average the losses under the current weights and turn the average into a confidence:

Lˉt=∑i=1nwi(t)Li,βt=Lˉt1−Lˉt,αt=ν ln⁡1βt\bar L_t=\sum_{i=1}^{n} w_i^{(t)}L_i,\qquad \beta_t=\frac{\bar L_t}{1-\bar L_t},\qquad \alpha_t=\nu\,\ln\frac{1}{\beta_t}
wi(t)w_i^{(t)}
weight of point ii in round tt (∑iwi(t)=1\sum_i w_i^{(t)}=1)
Lˉt\bar L_t
weighted average loss, in [0,1][0,1]
βt\beta_t
confidence ratio; βt<1  ⟺  Lˉt<0.5\beta_t<1 \iff \bar L_t<0.5
αt\alpha_t
the tree's vote in the final median
ν\nu
learning rate in (0,1](0,1] (scikit-learn's learning_rate)

βt\beta_t is the odds of the average loss. A tree with Lˉt=0.25\bar L_t=0.25 has βt=1/3\beta_t=1/3 and αt=ln⁡3=1.0986\alpha_t=\ln 3=1.0986; a tree with Lˉt→0.5\bar L_t\to0.5 has βt→1\beta_t\to1 and αt→0\alpha_t\to0 — it gets no vote. If Lˉt≥0.5\bar L_t\ge0.5 then αt≤0\alpha_t\le0, so the algorithm stops and discards that tree (keeping it only if it is the very first). This is the regression analogue of AdaBoost's "weak learner must beat 50% error" condition.

Step 3 — reweight. Every point's weight is multiplied by βt\beta_t raised to how well it was predicted, then renormalised:

wi(t+1)=wi(t) βt ν(1−Li)Zt,Zt=∑jwj(t) βt ν(1−Lj)w_i^{(t+1)}=\frac{w_i^{(t)}\,\beta_t^{\,\nu(1-L_i)}}{Z_t},\qquad Z_t=\sum_j w_j^{(t)}\,\beta_t^{\,\nu(1-L_j)}
1−Li1-L_i
how well point ii was predicted: 1 for a perfect fit, 0 for the worst point
ZtZ_t
normaliser that makes the new weights sum to 1

Taking logs shows what really happens: ln⁡wi←ln⁡wi−αt(1−Li)−ln⁡Zt\ln w_i \leftarrow \ln w_i-\alpha_t(1-L_i)-\ln Z_t. Each point's log-weight drops by the tree's confidence times how well that point was fitted. The worst point (Li=1L_i=1) loses nothing; a perfectly fitted point loses αt\alpha_t. A confident tree (large αt\alpha_t) moves weights a lot; a barely useful tree hardly moves them. The learning rate ν\nu scales both the vote and this log-weight change, so smaller ν\nu means gentler reweighting and more rounds needed.

Step 4 — combine with a weighted median. Why a median? Ask which single number y^\hat y is closest to the trees' predictions in weighted absolute distance:

y^(x)=arg⁡min⁡c∑t=1Tαt ∣c−ht(x)∣    ⇒    ∂∂c=∑t: ht(x)<cαt−∑t: ht(x)>cαt=0\hat y(x)=\arg\min_{c}\sum_{t=1}^{T}\alpha_t\,\bigl|c-h_t(x)\bigr|\;\;\Rightarrow\;\;\frac{\partial}{\partial c}=\sum_{t:\,h_t(x)<c}\alpha_t-\sum_{t:\,h_t(x)>c}\alpha_t=0
y^(x)\hat y(x)
ensemble prediction at input x
ht(x)h_t(x)
prediction of tree tt at xx
αt\alpha_t
tree weight from step 2

The derivative is (weight of trees below cc) minus (weight of trees above cc). It changes sign at the point where the sorted cumulative weight first reaches half the total — the weighted median. Concretely: sort ht(x)h_t(x) ascending, accumulate α\alpha, and return the first prediction whose cumulative weight is ≥12∑tαt\ge \tfrac12\sum_t\alpha_t. A weighted mean would minimise squared distance instead, and one tree with a wild prediction could pull it arbitrarily far; the median ignores how far away the outvoted trees are.

Worked example: ad spend and sales

Ten weeks of advertising spend xx (thousand dollars) and sales yy (thousand units). The relationship is S-shaped: little effect at first, a jump around x=6x=6, then steady growth. We run 3 rounds of AdaBoost.R2 with depth-2 trees, linear loss and ν=1\nu=1. To keep every number reproducible by hand, each tree is fitted by weighted least squares (the leaf value is the weighted mean of its points) instead of on a random resample.

x (spend, thousand dollars)12345678910
y (sales, k units)0.81.52.52.23.05.37.47.98.210.5
Training data: ten weeks of spend and sales. Watch x = 6 (the jump) and x = 10 (the last, highest point): they end up driving the reweighting.

Round 1 by hand

  1. 1Weights. All wi(1)=1/10=0.1w_i^{(1)}=1/10=0.1, so tree 1 is an ordinary depth-2 least-squares tree: h1h_1: x≤5.5x\le5.5 ? (x≤2.5x\le2.5 ? 1.15 : 2.567) : (x≤9.5x\le9.5 ? 7.2 : 10.5).
  2. 2Errors. ∣ei∣=0.35, 0.35, 0.067, 0.367, 0.433, 1.9, 0.2, 0.7, 1.0, 0|e_i| = 0.35,\,0.35,\,0.067,\,0.367,\,0.433,\,\mathbf{1.9},\,0.2,\,0.7,\,1.0,\,0. The worst point is x=6x=6 (true 5.3, predicted 7.2), so E1=1.9E_1=1.9.
  3. 3Relative losses Li=∣ei∣/1.9L_i=|e_i|/1.9: 0.184, 0.184, 0.035, 0.193, 0.228, 1, 0.105, 0.368, 0.526, 00.184,\,0.184,\,0.035,\,0.193,\,0.228,\,\mathbf{1},\,0.105,\,0.368,\,0.526,\,0.
  4. 4Average loss. Lˉ1=0.1×(0.184+0.184+⋯+0)=0.1×2.825=0.2825<0.5\bar L_1=0.1\times(0.184+0.184+\dots+0)=0.1\times2.825=0.2825<0.5, so the tree is kept.
  5. 5Confidence. β1=0.2825/0.7175=0.3936\beta_1=0.2825/0.7175=0.3936, α1=ln⁡(1/0.3936)=0.9323\alpha_1=\ln(1/0.3936)=0.9323.
  6. 6Reweight. Multiply each weight by β11−Li\beta_1^{1-L_i}: x=6x=6 keeps its 0.10.1 (exponent 0); x=10x=10, fitted perfectly, becomes 0.1×0.3936=0.03940.1\times0.3936=0.0394; x=9x=9 (L=0.526L=0.526) becomes 0.1×0.39360.474=0.06430.1\times0.3936^{0.474}=0.0643. The ten products sum to Z1=0.5326Z_1=0.5326; dividing gives w6(2)=0.1/0.5326=0.1878w^{(2)}_6=0.1/0.5326=0.1878 — nearly double its fair share.

Rounds 2 and 3

  1. 1Tree 2 is fitted with w(2)w^{(2)}. The heavy point x=6x=6 now pays for its own leaf: h2h_2: x≤5.5x\le5.5 ? (x≤2.5x\le2.5 ? 1.15 : 2.575) : (x≤6.5x\le6.5 ? 5.3 : 8.393). Leaves are weighted means, e.g. 8.3938.393 is the w(2)w^{(2)}-weighted mean of yy at x=7…10x=7\ldots10.
  2. 2Its worst point is now x=10x=10 (10.5 vs 8.393): E2=2.107E_2=2.107, Lˉ2=0.2139\bar L_2=0.2139, β2=0.2721\beta_2=0.2721, α2=1.3016\alpha_2=1.3016. The weight of x=10x=10 jumps from 0.0739 to 0.1919.
  3. 3Tree 3 with w(3)w^{(3)}: h3h_3: x≤6.5x\le6.5 ? (x≤5.5x\le5.5 ? 1.992 : 5.3) : (x≤9.5x\le9.5 ? 7.819 : 10.5). Worst point x=1x=1 (E3=1.192E_3=1.192), Lˉ3=0.2923\bar L_3=0.2923, β3=0.4130\beta_3=0.4130, α3=0.8842\alpha_3=0.8842.
xyw⁽¹⁾w⁽²⁾w⁽³⁾w⁽⁴⁾h₁(x)h₂(x)h₃(x)
10.80.10.08780.07700.13781.1501.1501.992
21.50.10.08780.07700.08201.1501.1501.992
32.50.10.07640.05650.06092.5672.5751.992
42.20.10.08850.07880.06802.5672.5751.992
53.00.10.09140.08400.13112.5672.5751.992
65.30.10.18780.13260.09817.2005.3005.300
77.40.10.08150.10640.10747.2008.3937.819
87.90.10.10420.09980.07847.2008.3937.819
98.20.10.12070.09610.09437.2008.3937.819
1010.50.10.07390.19190.141910.5008.39310.500
Weights before each round. Bold: the point each tree fitted worst (Li=1L_i=1), which therefore keeps its raw weight and grows after normalising.
RoundWorst pointE_tL̄_tβ_tα_tTree's own train MSE
1x = 61.9000.28250.39360.93230.571
2x = 102.1070.21390.27211.30160.628
3x = 11.1920.29230.41300.88420.331
EnsembleΣα = 3.11810.090
Round summary. The ensemble (weighted median of the three trees) has train MSE 0.090 — far below any single tree, and below the α-weighted mean of the same trees (0.219).

Predicting sales for a new week with x = 7.5

  1. 1Tree outputs: h1(7.5)=7.2h_1(7.5)=7.2, h2(7.5)=8.393h_2(7.5)=8.393, h3(7.5)=7.819h_3(7.5)=7.819.
  2. 2Sort with their votes: 7.2 (α=0.9323)7.2\,(\alpha=0.9323), 7.819 (0.8842)7.819\,(0.8842), 8.393 (1.3016)8.393\,(1.3016). Half of the total vote is 3.1181/2=1.55903.1181/2=1.5590.
  3. 3Cumulative vote: 0.93230.9323 after the first (not yet half), 0.9323+0.8842=1.8165≥1.55900.9323+0.8842=1.8165\ge1.5590 after the second.
  4. 4Prediction: y^(7.5)=7.819\hat y(7.5)=7.819 thousand units. (The weighted mean would be 7.874; here the two agree closely, but on the training set the median is much better: MSE 0.090 vs 0.219.)

Notice what the median did on the training set: at x=6x=6 two trees say 5.3 and outvote tree 1's 7.2; at x=10x=10 two trees say 10.5 and outvote tree 2's 8.393. Each tree has one region it gets badly wrong, the reweighting made sure those regions differ, and the vote picks the majority view point by point. The ensemble's train R2R^2 is 0.991.

ad spend → sales · dot area = weight w⁽³⁾ · drag dots024681012345678910↔- - tree h₃— weighted median of h₁…h₃loss Lᵢ = |eᵢ|/E · E = max|eᵢ| = 1.192 · L̄ = Σ wᵢLᵢ = 0.29231.000.410.430.170.850.000.350.070.320.00next wᵢ ∝ wᵢ·β^(1−Lᵢ), β = 0.4130 · high-loss points keep their weightat x = 7.5: trees sorted by prediction, width = α · half of Σα = 1.559h₁ 7.20h₃ 7.82h₂ 8.39weighted median = 7.819 · weighted mean = 7.874α: h₁ 0.93 h₂ 1.30 h₃ 0.88
round3/10depthcombine
loss
h₃: (x≤6.5 ? (x≤5.5 ? 1.99 : 5.30) : (x≤9.5 ? 7.82 : 10.50))
β = L̄/(1−L̄) = 0.4130 · α = ln(1/β) = 0.8842 · train MSE: this tree 0.331, ensemble 0.090
all 10 rounds ran with L̄ < 0.5.
fig.The worked example, live. Dot area is each point's weight in the selected round, colour and bars show its relative loss Lᵢ, the dashed orange line is that round's tree and the green line is the weighted median of trees 1…m. At round 3 (default) α = 0.93, 1.30, 0.88, the ensemble's train MSE is 0.090, and at x = 7.5 the median strip picks h₃ = 7.82 (mean 7.87). Step through rounds, drag dots up or down (try turning x = 8 into an outlier and watch its weight take over), drag the ↔ handle to move the query, and switch depth: with stumps (depth 1) the later trees have L̄ near 0.5 and almost no vote.

The algorithm, step by step

AdaBoost.R2(X, y, T, loss, ν)

  1. 1Initialise wi=1/nw_i=1/n for every training point.
  2. 2Fit tree hth_t to (xi,yi)(x_i,y_i) using weights ww — either directly via sample_weight or, as in Drucker's paper and scikit-learn, on a bootstrap resample of size nn drawn with probabilities ww.
  3. 3Score: ei=∣yi−ht(xi)∣e_i=|y_i-h_t(x_i)|, Et=max⁡ieiE_t=\max_i e_i (if Et=0E_t=0 the tree is perfect: keep it and stop), Li=g(ei/Et)L_i=g(e_i/E_t), Lˉt=∑iwiLi\bar L_t=\sum_i w_iL_i.
  4. 4Stop if Lˉt≥0.5\bar L_t\ge0.5: discard hth_t (unless it is the first tree) and end training.
  5. 5Vote: βt=Lˉt/(1−Lˉt)\beta_t=\bar L_t/(1-\bar L_t), αt=νln⁡(1/βt)\alpha_t=\nu\ln(1/\beta_t).
  6. 6Reweight: wi←wi βtν(1−Li)w_i\leftarrow w_i\,\beta_t^{\nu(1-L_i)}, then divide by the sum. Repeat from step 2 until TT trees.
  7. 7Predict at xx: sort h1(x),…,hT(x)h_1(x),\dots,h_T(x), return the first one whose cumulative α\alpha reaches 12∑tαt\tfrac12\sum_t\alpha_t.

Complexity

Training
TT tree fits, each O(d nlog⁡n)O(d\,n\log n) for a depth-limited CART tree, plus O(n)O(n) per round for errors and weights. Strictly sequential: round t+1t+1 needs round tt's weights.
Prediction
O(T⋅depth)O(T\cdot\text{depth}) to evaluate the trees plus O(Tlog⁡T)O(T\log T) to sort for the median, per input.
Memory
TT small trees and TT floats αt\alpha_t; the weights are only needed during training.

Hyperparameters and variants

To see the knobs at a realistic size, take y=sin⁡x+sin⁡6x+εy=\sin x+\sin 6x+\varepsilon (ε∼N(0,0.12)\varepsilon\sim\mathcal N(0,0.1^2)), 200 training and 1000 test points. A single depth-4 tree has test MSE 0.1608. AdaBoost.R2 with 300 depth-4 trees (scikit-learn defaults otherwise) reaches 0.0378; the curve over rounds is 0.2364 after 1 tree, 0.0779 after 10, 0.0382 after 100, 0.0378 after 300.

SettingTrees actually keptTest MSE
Single tree, depth 410.1608
R2, depth 1 (stumps)9 (stopped)0.5187
R2, depth 27 (stopped)0.3685
R2, depth 381 (stopped)0.1315
R2, depth 4, linear loss3000.0378
R2, depth 4, square loss3000.0322
R2, depth 4, exponential loss3000.0402
R2, depth 4, linear, ν = 0.13000.0451
R2, depth 63000.0266
Gradient boosting, depth 4, ν = 0.1, 300 trees3000.0278
Test MSE on the sin x + sin 6x benchmark (noise variance 0.01). Shallow trees stop early because their weighted loss reaches 0.5.

Tuning guide

Base-tree depth
The most important knob. Each tree models yy itself, so it must reach Lˉ<0.5\bar L<0.5 alone. Stumps stop after a few rounds (above: 9 trees, MSE 0.52). Start at depth 3–4 (sklearn's default base is depth 3).
n_estimators (T)
More trees rarely hurt much — the median is robust — but returns flatten (0.0382 at 100, 0.0378 at 300). Use staged_predict on a validation set to pick TT.
loss
linear (default) is a safe start. square focuses harder on the largest errors (best here, 0.0322). exponential is gentlest on the worst point. Differences are usually small; tune depth first.
learning_rate (ν)
Scales αt\alpha_t and the reweighting exponent. In the worked example ν=0.5\nu=0.5 turns α=0.93,1.30,0.88\alpha=0.93,1.30,0.88 into 0.47,0.56,0.360.47,0.56,0.36. Smaller ν\nu needs more trees; unlike gradient boosting it rarely improves accuracy (here 0.0451 at ν=0.1\nu=0.1 vs 0.0378 at 1).
Resampling vs reweighting
Drucker's R2 and scikit-learn fit each tree on a weighted bootstrap resample, which adds randomness and diversity; fitting with sample_weight directly is deterministic (our worked example). Set random_state for reproducibility.
Relatives
AdaBoost.RT (Solomatine & Shrestha) thresholds the relative error into right/wrong and reuses AdaBoost.M1; gradient boosting replaces reweighting with residual fitting and is usually the stronger default today.

In code

python
1import numpy as np
2
3def fit_tree(x, y, w, depth):
4 """Weighted least-squares regression tree on one feature (x sorted)."""
5 mean = (w * y).sum() / w.sum()
6 sse = lambda m: (w[m] * (y[m] - (w[m] * y[m]).sum() / w[m].sum()) ** 2).sum()
7 if depth == 0 or len(x) < 2:
8 return mean
9 best, t_best = sse(np.ones(len(x), bool)), None
10 for t in (x[:-1] + x[1:]) / 2:
11 L = x <= t
12 s = sse(L) + sse(~L)
13 if s < best - 1e-12:
14 best, t_best = s, t
15 if t_best is None:
16 return mean
17 L = x <= t_best
18 return (t_best, fit_tree(x[L], y[L], w[L], depth - 1), fit_tree(x[~L], y[~L], w[~L], depth - 1))
19
20def predict_tree(node, x):
21 while isinstance(node, tuple):
22 t, left, right = node
23 node = left if x <= t else right
24 return node
25
26def adaboost_r2(x, y, M=3, depth=2, loss="linear", lr=1.0):
27 n = len(x)
28 w = np.full(n, 1 / n)
29 trees, alphas = [], []
30 for m in range(M):
31 tree = fit_tree(x, y, w, depth)
32 pred = np.array([predict_tree(tree, xi) for xi in x])
33 err = np.abs(y - pred)
34 if err.max() == 0: # perfect fit: keep it and stop
35 trees.append(tree); alphas.append(1.0); break
36 L = err / err.max() # relative loss in [0, 1]
37 if loss == "square": L = L ** 2
38 if loss == "exponential": L = 1 - np.exp(-L)
39 Lbar = (w * L).sum() # weighted average loss
40 if Lbar >= 0.5: # no better than chance: stop
41 if not trees: trees.append(tree); alphas.append(1.0)
42 break
43 beta = Lbar / (1 - Lbar) # confidence: small beta = good learner
44 trees.append(tree); alphas.append(lr * np.log(1 / beta))
45 print(f"round {m+1}: Lbar={Lbar:.4f} beta={beta:.4f} alpha={alphas[-1]:.4f}")
46 w = w * beta ** ((1 - L) * lr) # shrink weights of well-fit points
47 w = w / w.sum()
48 print(" new weights", np.round(w, 4))
49 return trees, np.array(alphas)
50
51def predict(trees, alphas, x_new):
52 preds = np.array([predict_tree(t, x_new) for t in trees])
53 order = np.argsort(preds, kind="stable")
54 cum = np.cumsum(alphas[order])
55 k = np.searchsorted(cum, 0.5 * cum[-1]) # first cumulative weight >= half
56 return preds[order[k]], preds
57
58x = np.arange(1, 11, dtype=float)
59y = np.array([0.8, 1.5, 2.5, 2.2, 3.0, 5.3, 7.4, 7.9, 8.2, 10.5])
60trees, alphas = adaboost_r2(x, y)
61fit = np.array([predict(trees, alphas, xi)[0] for xi in x])
62print("train MSE", round(((fit - y) ** 2).mean(), 4)) # 0.0899
63yhat, members = predict(trees, alphas, 7.5)
64print("x=7.5 members", np.round(members, 3), "-> weighted median", round(yhat, 3))
65# x=7.5 members [7.2 8.393 7.819] -> weighted median 7.819
From scratch: AdaBoost.R2 with weighted regression trees on one feature. Prints the worked example exactly (α = 0.9323, 1.3016, 0.8842; prediction 7.819 at x = 7.5; train MSE 0.0899).
python
1import numpy as np
2from sklearn.ensemble import AdaBoostRegressor
3from sklearn.tree import DecisionTreeRegressor
4from sklearn.metrics import mean_squared_error
5
6# 1) The worked example. sklearn resamples by weight, so its trees differ from the hand run.
7X = np.arange(1, 11, dtype=float).reshape(-1, 1)
8y = np.array([0.8, 1.5, 2.5, 2.2, 3.0, 5.3, 7.4, 7.9, 8.2, 10.5])
9ada = AdaBoostRegressor(DecisionTreeRegressor(max_depth=2), n_estimators=3,
10 loss="linear", learning_rate=1.0, random_state=0).fit(X, y)
11print("alpha ", ada.estimator_weights_.round(4)) # [1.3988 1.3998 0.7881]
12print("weighted loss", ada.estimator_errors_.round(4)) # [0.198 0.1978 0.3126]
13print("train MSE ", round(mean_squared_error(y, ada.predict(X)), 4)) # 0.6201
14print("x = 7.5 -> ", ada.predict([[7.5]]).round(3)) # [8.]
15
16# 2) A realistic size: y = sin x + sin 6x + noise, 200 train / 1000 test points.
17rng = np.random.RandomState(1)
18f = lambda X: np.sin(X).ravel() + np.sin(6 * X).ravel()
19Xtr = np.sort(rng.uniform(0, 6, 200))[:, None]; ytr = f(Xtr) + rng.normal(0, 0.1, 200)
20Xte = np.sort(rng.uniform(0, 6, 1000))[:, None]; yte = f(Xte) + rng.normal(0, 0.1, 1000)
21
22tree = DecisionTreeRegressor(max_depth=4).fit(Xtr, ytr)
23print("one depth-4 tree test MSE", round(mean_squared_error(yte, tree.predict(Xte)), 4)) # 0.1608
24for loss in ["linear", "square", "exponential"]:
25 ada = AdaBoostRegressor(DecisionTreeRegressor(max_depth=4), n_estimators=300,
26 loss=loss, random_state=0).fit(Xtr, ytr)
27 print(f"AdaBoost.R2 {loss:<11} test MSE", round(mean_squared_error(yte, ada.predict(Xte)), 4))
28# linear 0.0378 · square 0.0322 · exponential 0.0402
29
30# staged_predict gives the ensemble after 1, 2, ... trees: choose n_estimators on validation data
31ada = AdaBoostRegressor(DecisionTreeRegressor(max_depth=4), n_estimators=300, random_state=0).fit(Xtr, ytr)
32curve = [mean_squared_error(yte, p) for p in ada.staged_predict(Xte)]
33print("after 1/10/100/300 trees", [round(curve[i], 4) for i in (0, 9, 99, 299)])
scikit-learn's AdaBoostRegressor on the worked example and on the sin x + sin 6x benchmark. It resamples by weight, so on 10 points its trees (and numbers) differ from the deterministic hand run.

Use when

  • +You want a quick, strong upgrade over a single regression tree with almost no tuning (depth 3–4, a few hundred trees).
  • +Targets are clean and you want an ensemble whose output is always one of its members' predictions (the median never averages into implausible values).
  • +You want a combination rule that is robust to a few badly wrong trees.
  • +You are teaching or comparing boosting schemes: R2 shows reweighting cleanly, separate from residual fitting.

Avoid when

  • −Targets contain gross outliers or label noise: the reweighting concentrates on them (test MSE 1.39 vs 0.64 for a single tree above).
  • −You need the best accuracy on tabular data: gradient boosting, XGBoost, LightGBM and CatBoost usually win and have robust losses.
  • −Training time matters: rounds are sequential and cannot be parallelised across trees.
  • −You need smooth predictions or calibrated uncertainty: the output is a step function picked by a vote.

Where it's used

Electricity load forecasting

Boosted trees over weather and calendar features; the weighted median resists the occasional tree that overreacts to a heat wave.

Hydrology

River-flow and rainfall–runoff prediction, where AdaBoost.R2 and its variant AdaBoost.RT were studied as upgrades to single regression trees and neural nets.

Materials and chemistry

Predicting a property (strength, band gap, solubility) from descriptors with small datasets, as a strong baseline alongside random forests.

Image-based regression

Boosted regressors for age estimation or head-pose angles from face features, a classic use of boosting in vision.

Check yourself

A tree's weighted average loss is Lˉt=0.25\bar L_t=0.25. What are βt\beta_t and its vote αt\alpha_t (with ν=1\nu=1)?

Check yourself

Three trees predict 4, 6 and 9 at some xx, with votes α=0.5, 0.8, 1.0\alpha=0.5,\,0.8,\,1.0. What does AdaBoost.R2 output?

Check yourself

A point's absolute error is 2 and the round's maximum error is 4. What is its loss LiL_i under the square loss?

Check yourself

Training stops early with far fewer trees than n_estimators. What is the most likely cause?

Remember

  • ✓Each round fits a tree to the original targets under sample weights; nothing is fitted to residuals and nothing is added up.
  • ✓Errors become relative losses Li∈[0,1]L_i\in[0,1] by dividing by the round's maximum error (then linear, square or exponential shaping).
  • ✓βt=Lˉt/(1−Lˉt)\beta_t=\bar L_t/(1-\bar L_t); well-fitted points are multiplied by βt1−Li\beta_t^{1-L_i}, so hard points gain weight. Training stops when Lˉt≥0.5\bar L_t\ge0.5.
  • ✓Prediction is the α\alpha-weighted median (αt=ln⁡1/βt\alpha_t=\ln 1/\beta_t), the minimiser of weighted absolute deviation — a vote that ignores far-off trees.
  • ✓Use base trees of depth 3–4; stumps stall. Tune depth, then the number of trees with staged_predict.
  • ✓Outliers attract weight round after round: clean targets first, or prefer gradient boosting with a robust loss.
← back to ML Algorithms roadmap