# AdaBoost regression

> AdaBoost.R2 reweights training points by their relative error and combines learners with a weighted median.

Source: https://www.mlfoundry.in/learn/adaboost-regression  
Section: ML Algorithms › Supervised · Regression  
Level: advanced · 16 min read · updated 2026-10-07

**AdaBoost.R2:** (Drucker, 1997) is the regression version of AdaBoost. It trains a sequence of regression trees, each one on the **original targets** but with the training points **reweighted** so that points the previous trees predicted badly count more. Each tree gets a confidence $\alpha_t$, and the ensemble predicts the **weighted median** of the trees' outputs — not their sum.

- **Task:** Regression (a numeric target)
- **Input → output:** $x$ → the $\alpha$-weighted median of $h_1(x),\dots,h_T(x)$
- **Learns:** $T$ regression trees $h_t$ and one confidence $\alpha_t=\ln(1/\beta_t)$ per tree
- **Training:** $T$ sequential tree fits: about $O(T\,d\,n\log n)$; rounds cannot run in parallel
- **Prediction:** $T$ tree walks plus a sort of $T$ numbers: $O(T\,\text{depth}+T\log T)$
- **Hyperparameters:** `n_estimators` $T$, base-tree `max_depth`, `loss` (linear / square / exponential), `learning_rate` $\nu$
- **Assumes:** each tree alone is better than "chance": its weighted average loss $\bar L_t<0.5$; targets are not dominated by gross outliers

## The problem

A single regression tree (see **decision-tree-regression**) is a step function fitted by least squares. Grow it shallow and it misses whole regions; grow it deep and it chases noise. Bagging (**random-forest-regression**) averages many deep trees to cut variance, but every tree sees the data the same way, so a region that all trees find hard stays hard.

Classification AdaBoost solved this for labels: after each round, *up-weight the points you got wrong* so the next learner is forced to look at them. But "wrong" is binary for a label and continuous for a number — an error of 0.1 and an error of 3 are both "wrong". AdaBoost.R2 needs (1) a way to turn a real-valued error into a loss on a fixed $[0,1]$ scale, so the AdaBoost weight formula still applies, and (2) a way to combine real-valued predictions that does not let one confidently wrong tree drag the answer away. Its answers are the **relative loss** $L_i=|e_i|/\max_j|e_j|$ and the **weighted median**.

## The idea

Picture ten people each guessing a number, where every guesser was trained on a slightly different emphasis of the same data. You do not add their guesses; you sort them, give each a vote proportional to how reliable they were on training data, and take the guess where the votes reach the halfway point. That is the weighted median: a **vote**, robust to a few wild guesses.

The emphasis comes from the weights. After a tree is fitted, every training point gets a loss between 0 (predicted perfectly) and 1 (the worst point of this round). The weights of well-predicted points are multiplied by a factor $\beta<1$ — the better the tree was overall, the smaller $\beta$ — while the worst point keeps its weight. After renormalising, the hard points hold a larger share, and the next tree, which minimises **weighted** squared error, bends towards them.

> **Not residual fitting:** Gradient boosting (**gradient-boosting-regression**) fits each new tree to the *residuals* $y-F(x)$ and **adds** it to the model, so trees are small corrections. AdaBoost.R2 fits every tree to the *original* $y$; only the weights change. Each tree must therefore be a decent model of the whole function on its own — which is why stumps work poorly here and depth-3 or depth-4 trees are typical.

**Vocabulary**

- **Sample weight wᵢ:** importance of training point $i$ in round $t$; non-negative and summing to 1. Starts at $1/n$.
- **Base learner hₜ:** the regression tree fitted in round $t$ by weighted least squares (or, in scikit-learn, on a weighted bootstrap resample)
- **Absolute error eᵢ:** $|y_i-h_t(x_i)|$ on every training point
- **Max error Eₜ:** $\max_i e_i$ — the scale that makes losses relative
- **Relative loss Lᵢ:** $e_i/E_t$ passed through the chosen loss (linear, square or exponential); always in $[0,1]$
- **Average loss L̄ₜ:** $\sum_i w_i L_i$ — the weighted badness of tree $t$. Must be $<0.5$.
- **Confidence βₜ:** $\bar L_t/(1-\bar L_t)$, in $(0,1)$ for a useful tree: small = good tree
- **Tree weight αₜ:** $\ln(1/\beta_t)$ — the tree's vote in the weighted median
- **Weighted median:** sort the $T$ predictions; output the first one at which the cumulative $\alpha$ reaches half of $\sum_t\alpha_t$

## The math

**Step 1 — relative loss.** After fitting $h_t$, compute every training error and scale it by the largest one, so the worst point has loss exactly 1 whatever the units of $y$:

$$
L_i=g\!\left(\frac{|y_i-h_t(x_i)|}{E_t}\right),\qquad E_t=\max_j |y_j-h_t(x_j)|
$$

- $y_i,\;x_i$: target and input of training point $i$
- $h_t(x_i)$: prediction of the round-$t$ tree
- $E_t$: largest absolute training error of that tree
- $g(u)$: loss shape: $u$ (linear), $u^2$ (square) or $1-e^{-u}$ (exponential)

The shape $g$ decides how sharply the algorithm separates "slightly off" from "badly off". With $|e|=1$ and $E=4$: linear gives $L=0.25$, square $0.0625$, exponential $0.2212$. Square pushes all but the worst points towards 0, so their weights shrink more and attention concentrates on the few largest errors. Exponential caps the worst loss at $1-e^{-1}=0.6321$, so even the worst point's weight is shrunk a little.

**Step 2 — how good was this tree?** Average the losses under the current weights and turn the average into a confidence:

$$
\bar L_t=\sum_{i=1}^{n} w_i^{(t)}L_i,\qquad \beta_t=\frac{\bar L_t}{1-\bar L_t},\qquad \alpha_t=\nu\,\ln\frac{1}{\beta_t}
$$

- $w_i^{(t)}$: weight of point $i$ in round $t$ ($\sum_i w_i^{(t)}=1$)
- $\bar L_t$: weighted average loss, in $[0,1]$
- $\beta_t$: confidence ratio; $\beta_t<1 \iff \bar L_t<0.5$
- $\alpha_t$: the tree's vote in the final median
- $\nu$: learning rate in $(0,1]$ (scikit-learn's `learning_rate`)

$\beta_t$ is the odds of the average loss. A tree with $\bar L_t=0.25$ has $\beta_t=1/3$ and $\alpha_t=\ln 3=1.0986$; a tree with $\bar L_t\to0.5$ has $\beta_t\to1$ and $\alpha_t\to0$ — it gets no vote. If $\bar L_t\ge0.5$ then $\alpha_t\le0$, so the algorithm **stops** and discards that tree (keeping it only if it is the very first). This is the regression analogue of AdaBoost's "weak learner must beat 50% error" condition.

**Step 3 — reweight.** Every point's weight is multiplied by $\beta_t$ raised to *how well it was predicted*, then renormalised:

$$
w_i^{(t+1)}=\frac{w_i^{(t)}\,\beta_t^{\,\nu(1-L_i)}}{Z_t},\qquad Z_t=\sum_j w_j^{(t)}\,\beta_t^{\,\nu(1-L_j)}
$$

- $1-L_i$: how *well* point $i$ was predicted: 1 for a perfect fit, 0 for the worst point
- $Z_t$: normaliser that makes the new weights sum to 1

Taking logs shows what really happens: $\ln w_i \leftarrow \ln w_i-\alpha_t(1-L_i)-\ln Z_t$. Each point's log-weight drops by the tree's confidence times how well that point was fitted. The worst point ($L_i=1$) loses nothing; a perfectly fitted point loses $\alpha_t$. A confident tree (large $\alpha_t$) moves weights a lot; a barely useful tree hardly moves them. The learning rate $\nu$ scales both the vote and this log-weight change, so smaller $\nu$ means gentler reweighting and more rounds needed.

> **It contains classification AdaBoost:** If every $L_i$ is 0 or 1 (right or wrong), then $\bar L_t$ is the weighted error rate $\varepsilon_t$, $\beta_t=\varepsilon_t/(1-\varepsilon_t)$, and correctly handled points are multiplied by $\beta_t$ while mistakes keep their weight — exactly AdaBoost.M1 (see **adaboost**). R2 simply lets the loss be any value in between.

**Step 4 — combine with a weighted median.** Why a median? Ask which single number $\hat y$ is closest to the trees' predictions in weighted *absolute* distance:

$$
\hat y(x)=\arg\min_{c}\sum_{t=1}^{T}\alpha_t\,\bigl|c-h_t(x)\bigr|\;\;\Rightarrow\;\;\frac{\partial}{\partial c}=\sum_{t:\,h_t(x)<c}\alpha_t-\sum_{t:\,h_t(x)>c}\alpha_t=0
$$

- $\hat y(x)$: ensemble prediction at input x
- $h_t(x)$: prediction of tree $t$ at $x$
- $\alpha_t$: tree weight from step 2

The derivative is (weight of trees below $c$) minus (weight of trees above $c$). It changes sign at the point where the sorted cumulative weight first reaches half the total — the **weighted median**. Concretely: sort $h_t(x)$ ascending, accumulate $\alpha$, and return the first prediction whose cumulative weight is $\ge \tfrac12\sum_t\alpha_t$. A weighted *mean* would minimise squared distance instead, and one tree with a wild prediction could pull it arbitrarily far; the median ignores how far away the outvoted trees are.

## Worked example: ad spend and sales

Ten weeks of advertising spend $x$ (thousand dollars) and sales $y$ (thousand units). The relationship is S-shaped: little effect at first, a jump around $x=6$, then steady growth. We run **3 rounds** of AdaBoost.R2 with depth-2 trees, linear loss and $\nu=1$. To keep every number reproducible by hand, each tree is fitted by **weighted** least squares (the leaf value is the weighted mean of its points) instead of on a random resample.

Training data: ten weeks of spend and sales. Watch x = 6 (the jump) and x = 10 (the last, highest point): they end up driving the reweighting.

| x (spend, thousand dollars) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| y (sales, k units) | 0.8 | 1.5 | 2.5 | 2.2 | 3.0 | 5.3 | 7.4 | 7.9 | 8.2 | 10.5 |

**Round 1 by hand**

1. **Weights.** All $w_i^{(1)}=1/10=0.1$, so tree 1 is an ordinary depth-2 least-squares tree: $h_1$: $x\le5.5$ ? ($x\le2.5$ ? **1.15** : **2.567**) : ($x\le9.5$ ? **7.2** : **10.5**).
2. **Errors.** $|e_i| = 0.35,\,0.35,\,0.067,\,0.367,\,0.433,\,\mathbf{1.9},\,0.2,\,0.7,\,1.0,\,0$. The worst point is $x=6$ (true 5.3, predicted 7.2), so $E_1=1.9$.
3. **Relative losses** $L_i=|e_i|/1.9$: $0.184,\,0.184,\,0.035,\,0.193,\,0.228,\,\mathbf{1},\,0.105,\,0.368,\,0.526,\,0$.
4. **Average loss.** $\bar L_1=0.1\times(0.184+0.184+\dots+0)=0.1\times2.825=0.2825<0.5$, so the tree is kept.
5. **Confidence.** $\beta_1=0.2825/0.7175=0.3936$, $\alpha_1=\ln(1/0.3936)=0.9323$.
6. **Reweight.** Multiply each weight by $\beta_1^{1-L_i}$: $x=6$ keeps its $0.1$ (exponent 0); $x=10$, fitted perfectly, becomes $0.1\times0.3936=0.0394$; $x=9$ ($L=0.526$) becomes $0.1\times0.3936^{0.474}=0.0643$. The ten products sum to $Z_1=0.5326$; dividing gives $w^{(2)}_6=0.1/0.5326=0.1878$ — nearly double its fair share.

**Rounds 2 and 3**

1. **Tree 2** is fitted with $w^{(2)}$. The heavy point $x=6$ now pays for its own leaf: $h_2$: $x\le5.5$ ? ($x\le2.5$ ? 1.15 : 2.575) : ($x\le6.5$ ? **5.3** : **8.393**). Leaves are weighted means, e.g. $8.393$ is the $w^{(2)}$-weighted mean of $y$ at $x=7\ldots10$.
2. Its worst point is now $x=10$ (10.5 vs 8.393): $E_2=2.107$, $\bar L_2=0.2139$, $\beta_2=0.2721$, $\alpha_2=1.3016$. The weight of $x=10$ jumps from 0.0739 to **0.1919**.
3. **Tree 3** with $w^{(3)}$: $h_3$: $x\le6.5$ ? ($x\le5.5$ ? 1.992 : 5.3) : ($x\le9.5$ ? **7.819** : **10.5**). Worst point $x=1$ ($E_3=1.192$), $\bar L_3=0.2923$, $\beta_3=0.4130$, $\alpha_3=0.8842$.

Weights before each round. Bold: the point each tree fitted worst ($L_i=1$), which therefore keeps its raw weight and grows after normalising.

| x | y | w⁽¹⁾ | w⁽²⁾ | w⁽³⁾ | w⁽⁴⁾ | h₁(x) | h₂(x) | h₃(x) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 1 | 0.8 | 0.1 | 0.0878 | 0.0770 | **0.1378** | 1.150 | 1.150 | 1.992 |
| 2 | 1.5 | 0.1 | 0.0878 | 0.0770 | 0.0820 | 1.150 | 1.150 | 1.992 |
| 3 | 2.5 | 0.1 | 0.0764 | 0.0565 | 0.0609 | 2.567 | 2.575 | 1.992 |
| 4 | 2.2 | 0.1 | 0.0885 | 0.0788 | 0.0680 | 2.567 | 2.575 | 1.992 |
| 5 | 3.0 | 0.1 | 0.0914 | 0.0840 | 0.1311 | 2.567 | 2.575 | 1.992 |
| 6 | 5.3 | 0.1 | **0.1878** | 0.1326 | 0.0981 | 7.200 | 5.300 | 5.300 |
| 7 | 7.4 | 0.1 | 0.0815 | 0.1064 | 0.1074 | 7.200 | 8.393 | 7.819 |
| 8 | 7.9 | 0.1 | 0.1042 | 0.0998 | 0.0784 | 7.200 | 8.393 | 7.819 |
| 9 | 8.2 | 0.1 | 0.1207 | 0.0961 | 0.0943 | 7.200 | 8.393 | 7.819 |
| 10 | 10.5 | 0.1 | 0.0739 | **0.1919** | 0.1419 | 10.500 | 8.393 | 10.500 |

Round summary. The ensemble (weighted median of the three trees) has train MSE 0.090 — far below any single tree, and below the α-weighted mean of the same trees (0.219).

| Round | Worst point | E_t | L̄_t | β_t | α_t | Tree's own train MSE |
| --- | --- | --- | --- | --- | --- | --- |
| 1 | x = 6 | 1.900 | 0.2825 | 0.3936 | 0.9323 | 0.571 |
| 2 | x = 10 | 2.107 | 0.2139 | 0.2721 | 1.3016 | 0.628 |
| 3 | x = 1 | 1.192 | 0.2923 | 0.4130 | 0.8842 | 0.331 |
| **Ensemble** |  |  |  |  | Σα = 3.1181 | **0.090** |

**Predicting sales for a new week with x = 7.5**

1. Tree outputs: $h_1(7.5)=7.2$, $h_2(7.5)=8.393$, $h_3(7.5)=7.819$.
2. Sort with their votes: $7.2\,(\alpha=0.9323)$, $7.819\,(0.8842)$, $8.393\,(1.3016)$. Half of the total vote is $3.1181/2=1.5590$.
3. Cumulative vote: $0.9323$ after the first (not yet half), $0.9323+0.8842=1.8165\ge1.5590$ after the second.
4. **Prediction: $\hat y(7.5)=7.819$** thousand units. (The weighted mean would be 7.874; here the two agree closely, but on the training set the median is much better: MSE 0.090 vs 0.219.)

Notice what the median did on the training set: at $x=6$ two trees say 5.3 and outvote tree 1's 7.2; at $x=10$ two trees say 10.5 and outvote tree 2's 8.393. Each tree has one region it gets badly wrong, the reweighting made sure those regions differ, and the vote picks the majority view point by point. The ensemble's train $R^2$ is 0.991.

> Figure: The worked example, live. Dot area is each point's weight in the selected round, colour and bars show its relative loss Lᵢ, the dashed orange line is that round's tree and the green line is the weighted median of trees 1…m. At round 3 (default) α = 0.93, 1.30, 0.88, the ensemble's train MSE is 0.090, and at x = 7.5 the median strip picks h₃ = 7.82 (mean 7.87). Step through rounds, drag dots up or down (try turning x = 8 into an outlier and watch its weight take over), drag the ↔ handle to move the query, and switch depth: with stumps (depth 1) the later trees have L̄ near 0.5 and almost no vote.

## The algorithm, step by step

**AdaBoost.R2(X, y, T, loss, ν)**

1. **Initialise** $w_i=1/n$ for every training point.
2. **Fit** tree $h_t$ to $(x_i,y_i)$ using weights $w$ — either directly via `sample_weight` or, as in Drucker's paper and scikit-learn, on a bootstrap resample of size $n$ drawn with probabilities $w$.
3. **Score**: $e_i=|y_i-h_t(x_i)|$, $E_t=\max_i e_i$ (if $E_t=0$ the tree is perfect: keep it and stop), $L_i=g(e_i/E_t)$, $\bar L_t=\sum_i w_iL_i$.
4. **Stop** if $\bar L_t\ge0.5$: discard $h_t$ (unless it is the first tree) and end training.
5. **Vote**: $\beta_t=\bar L_t/(1-\bar L_t)$, $\alpha_t=\nu\ln(1/\beta_t)$.
6. **Reweight**: $w_i\leftarrow w_i\,\beta_t^{\nu(1-L_i)}$, then divide by the sum. Repeat from step 2 until $T$ trees.
7. **Predict** at $x$: sort $h_1(x),\dots,h_T(x)$, return the first one whose cumulative $\alpha$ reaches $\tfrac12\sum_t\alpha_t$.

**Complexity**

- **Training:** $T$ tree fits, each $O(d\,n\log n)$ for a depth-limited CART tree, plus $O(n)$ per round for errors and weights. Strictly sequential: round $t+1$ needs round $t$'s weights.
- **Prediction:** $O(T\cdot\text{depth})$ to evaluate the trees plus $O(T\log T)$ to sort for the median, per input.
- **Memory:** $T$ small trees and $T$ floats $\alpha_t$; the weights are only needed during training.

## Hyperparameters and variants

To see the knobs at a realistic size, take $y=\sin x+\sin 6x+\varepsilon$ ($\varepsilon\sim\mathcal N(0,0.1^2)$), 200 training and 1000 test points. A single depth-4 tree has test MSE **0.1608**. AdaBoost.R2 with 300 depth-4 trees (scikit-learn defaults otherwise) reaches **0.0378**; the curve over rounds is 0.2364 after 1 tree, 0.0779 after 10, 0.0382 after 100, 0.0378 after 300.

Test MSE on the sin x + sin 6x benchmark (noise variance 0.01). Shallow trees stop early because their weighted loss reaches 0.5.

| Setting | Trees actually kept | Test MSE |
| --- | --- | --- |
| Single tree, depth 4 | 1 | 0.1608 |
| R2, depth 1 (stumps) | 9 (stopped) | 0.5187 |
| R2, depth 2 | 7 (stopped) | 0.3685 |
| R2, depth 3 | 81 (stopped) | 0.1315 |
| **R2, depth 4, linear loss** | 300 | **0.0378** |
| R2, depth 4, square loss | 300 | 0.0322 |
| R2, depth 4, exponential loss | 300 | 0.0402 |
| R2, depth 4, linear, ν = 0.1 | 300 | 0.0451 |
| R2, depth 6 | 300 | 0.0266 |
| Gradient boosting, depth 4, ν = 0.1, 300 trees | 300 | 0.0278 |

**Tuning guide**

- **Base-tree depth:** The most important knob. Each tree models $y$ itself, so it must reach $\bar L<0.5$ alone. Stumps stop after a few rounds (above: 9 trees, MSE 0.52). Start at depth 3–4 (sklearn's default base is depth 3).
- **n_estimators (T):** More trees rarely hurt much — the median is robust — but returns flatten (0.0382 at 100, 0.0378 at 300). Use `staged_predict` on a validation set to pick $T$.
- **loss:** *linear* (default) is a safe start. *square* focuses harder on the largest errors (best here, 0.0322). *exponential* is gentlest on the worst point. Differences are usually small; tune depth first.
- **learning_rate (ν):** Scales $\alpha_t$ and the reweighting exponent. In the worked example $\nu=0.5$ turns $\alpha=0.93,1.30,0.88$ into $0.47,0.56,0.36$. Smaller $\nu$ needs more trees; unlike gradient boosting it rarely improves accuracy (here 0.0451 at $\nu=0.1$ vs 0.0378 at 1).
- **Resampling vs reweighting:** Drucker's R2 and scikit-learn fit each tree on a weighted bootstrap resample, which adds randomness and diversity; fitting with `sample_weight` directly is deterministic (our worked example). Set `random_state` for reproducibility.
- **Relatives:** **AdaBoost.RT** (Solomatine & Shrestha) thresholds the relative error into right/wrong and reuses AdaBoost.M1; **gradient boosting** replaces reweighting with residual fitting and is usually the stronger default today.

> **Outliers take over the weights:** A point with a huge error is the worst point ($L_i=1$) round after round, so it is never down-weighted while everything else is. Its weight snowballs, later trees spend their splits on it, and because $E_t$ is that one error, every *other* loss looks tiny. On the benchmark above, shifting 10 of 200 training targets by ±5 raised AdaBoost.R2's test MSE from 0.0378 to **1.39** (linear loss) — worse than a single depth-4 tree on the same corrupted data (0.64). Clean or clip targets first, or use a robust loss in gradient boosting.

> **Weak learners that are too weak:** The stopping rule $\bar L_t<0.5$ is not a formality. In the worked example, stumps reach $\bar L=0.358$, then 0.491, 0.497, 0.499 — after round 1 the votes $\alpha_t$ collapse towards 0 and the ensemble is essentially one stump (train MSE 1.68 vs 0.090 with depth 2). If scikit-learn returns far fewer estimators than `n_estimators`, the trees are too shallow.

> **Why the previous tree looks bad after reweighting:** In classification AdaBoost the update makes the last learner's weighted error exactly 0.5 under the new weights, so the next learner cannot simply copy it. R2 does this only approximately: in the worked example tree 1's weighted loss rises from 0.2825 to 0.3712 under $w^{(2)}$, tree 2's from 0.2139 to 0.3328. The new weights push the next tree away from its predecessor's mistakes — that is where the ensemble's diversity comes from.

## In code

```python
import numpy as np

def fit_tree(x, y, w, depth):
  """Weighted least-squares regression tree on one feature (x sorted)."""
  mean = (w * y).sum() / w.sum()
  sse = lambda m: (w[m] * (y[m] - (w[m] * y[m]).sum() / w[m].sum()) ** 2).sum()
  if depth == 0 or len(x) < 2:
      return mean
  best, t_best = sse(np.ones(len(x), bool)), None
  for t in (x[:-1] + x[1:]) / 2:
      L = x <= t
      s = sse(L) + sse(~L)
      if s < best - 1e-12:
          best, t_best = s, t
  if t_best is None:
      return mean
  L = x <= t_best
  return (t_best, fit_tree(x[L], y[L], w[L], depth - 1), fit_tree(x[~L], y[~L], w[~L], depth - 1))

def predict_tree(node, x):
  while isinstance(node, tuple):
      t, left, right = node
      node = left if x <= t else right
  return node

def adaboost_r2(x, y, M=3, depth=2, loss="linear", lr=1.0):
  n = len(x)
  w = np.full(n, 1 / n)
  trees, alphas = [], []
  for m in range(M):
      tree = fit_tree(x, y, w, depth)
      pred = np.array([predict_tree(tree, xi) for xi in x])
      err = np.abs(y - pred)
      if err.max() == 0:                      # perfect fit: keep it and stop
          trees.append(tree); alphas.append(1.0); break
      L = err / err.max()                     # relative loss in [0, 1]
      if loss == "square":      L = L ** 2
      if loss == "exponential": L = 1 - np.exp(-L)
      Lbar = (w * L).sum()                    # weighted average loss
      if Lbar >= 0.5:                         # no better than chance: stop
          if not trees: trees.append(tree); alphas.append(1.0)
          break
      beta = Lbar / (1 - Lbar)                # confidence: small beta = good learner
      trees.append(tree); alphas.append(lr * np.log(1 / beta))
      print(f"round {m+1}: Lbar={Lbar:.4f} beta={beta:.4f} alpha={alphas[-1]:.4f}")
      w = w * beta ** ((1 - L) * lr)          # shrink weights of well-fit points
      w = w / w.sum()
      print("   new weights", np.round(w, 4))
  return trees, np.array(alphas)

def predict(trees, alphas, x_new):
  preds = np.array([predict_tree(t, x_new) for t in trees])
  order = np.argsort(preds, kind="stable")
  cum = np.cumsum(alphas[order])
  k = np.searchsorted(cum, 0.5 * cum[-1])     # first cumulative weight >= half
  return preds[order[k]], preds

x = np.arange(1, 11, dtype=float)
y = np.array([0.8, 1.5, 2.5, 2.2, 3.0, 5.3, 7.4, 7.9, 8.2, 10.5])
trees, alphas = adaboost_r2(x, y)
fit = np.array([predict(trees, alphas, xi)[0] for xi in x])
print("train MSE", round(((fit - y) ** 2).mean(), 4))          # 0.0899
yhat, members = predict(trees, alphas, 7.5)
print("x=7.5 members", np.round(members, 3), "-> weighted median", round(yhat, 3))
# x=7.5 members [7.2   8.393 7.819] -> weighted median 7.819
```

From scratch: AdaBoost.R2 with weighted regression trees on one feature. Prints the worked example exactly (α = 0.9323, 1.3016, 0.8842; prediction 7.819 at x = 7.5; train MSE 0.0899).

```python
import numpy as np
from sklearn.ensemble import AdaBoostRegressor
from sklearn.tree import DecisionTreeRegressor
from sklearn.metrics import mean_squared_error

# 1) The worked example. sklearn resamples by weight, so its trees differ from the hand run.
X = np.arange(1, 11, dtype=float).reshape(-1, 1)
y = np.array([0.8, 1.5, 2.5, 2.2, 3.0, 5.3, 7.4, 7.9, 8.2, 10.5])
ada = AdaBoostRegressor(DecisionTreeRegressor(max_depth=2), n_estimators=3,
                        loss="linear", learning_rate=1.0, random_state=0).fit(X, y)
print("alpha        ", ada.estimator_weights_.round(4))   # [1.3988 1.3998 0.7881]
print("weighted loss", ada.estimator_errors_.round(4))    # [0.198  0.1978 0.3126]
print("train MSE    ", round(mean_squared_error(y, ada.predict(X)), 4))  # 0.6201
print("x = 7.5 ->   ", ada.predict([[7.5]]).round(3))     # [8.]

# 2) A realistic size: y = sin x + sin 6x + noise, 200 train / 1000 test points.
rng = np.random.RandomState(1)
f = lambda X: np.sin(X).ravel() + np.sin(6 * X).ravel()
Xtr = np.sort(rng.uniform(0, 6, 200))[:, None]; ytr = f(Xtr) + rng.normal(0, 0.1, 200)
Xte = np.sort(rng.uniform(0, 6, 1000))[:, None]; yte = f(Xte) + rng.normal(0, 0.1, 1000)

tree = DecisionTreeRegressor(max_depth=4).fit(Xtr, ytr)
print("one depth-4 tree   test MSE", round(mean_squared_error(yte, tree.predict(Xte)), 4))  # 0.1608
for loss in ["linear", "square", "exponential"]:
  ada = AdaBoostRegressor(DecisionTreeRegressor(max_depth=4), n_estimators=300,
                          loss=loss, random_state=0).fit(Xtr, ytr)
  print(f"AdaBoost.R2 {loss:<11} test MSE", round(mean_squared_error(yte, ada.predict(Xte)), 4))
# linear 0.0378 · square 0.0322 · exponential 0.0402

# staged_predict gives the ensemble after 1, 2, ... trees: choose n_estimators on validation data
ada = AdaBoostRegressor(DecisionTreeRegressor(max_depth=4), n_estimators=300, random_state=0).fit(Xtr, ytr)
curve = [mean_squared_error(yte, p) for p in ada.staged_predict(Xte)]
print("after 1/10/100/300 trees", [round(curve[i], 4) for i in (0, 9, 99, 299)])
```

scikit-learn's AdaBoostRegressor on the worked example and on the sin x + sin 6x benchmark. It resamples by weight, so on 10 points its trees (and numbers) differ from the deterministic hand run.

> **Why scikit-learn's 10-point result is worse:** Each sklearn tree is trained on a bootstrap resample of the 10 points drawn with probabilities $w$, so a tree may never see some points (train MSE 0.6201 with 3 trees). The randomness pays off on real dataset sizes, where it adds diversity between rounds; on toy data it mostly adds noise. Its `estimator_weights_` are the $\alpha_t$ and its `estimator_errors_` are the $\bar L_t$ from the math above.

**Strengths**

- You want a quick, strong upgrade over a single regression tree with almost no tuning (depth 3–4, a few hundred trees).
- Targets are clean and you want an ensemble whose output is always one of its members' predictions (the median never averages into implausible values).
- You want a combination rule that is robust to a few badly wrong trees.
- You are teaching or comparing boosting schemes: R2 shows reweighting cleanly, separate from residual fitting.

**Limitations**

- Targets contain gross outliers or label noise: the reweighting concentrates on them (test MSE 1.39 vs 0.64 for a single tree above).
- You need the best accuracy on tabular data: gradient boosting, XGBoost, LightGBM and CatBoost usually win and have robust losses.
- Training time matters: rounds are sequential and cannot be parallelised across trees.
- You need smooth predictions or calibrated uncertainty: the output is a step function picked by a vote.

- **Electricity load forecasting:** Boosted trees over weather and calendar features; the weighted median resists the occasional tree that overreacts to a heat wave.
- **Hydrology:** River-flow and rainfall–runoff prediction, where AdaBoost.R2 and its variant AdaBoost.RT were studied as upgrades to single regression trees and neural nets.
- **Materials and chemistry:** Predicting a property (strength, band gap, solubility) from descriptors with small datasets, as a strong baseline alongside random forests.
- **Image-based regression:** Boosted regressors for age estimation or head-pose angles from face features, a classic use of boosting in vision.

**Question:** A tree's weighted average loss is $\bar L_t=0.25$. What are $\beta_t$ and its vote $\alpha_t$ (with $\nu=1$)?

- A. $\beta=0.25,\ \alpha=1.386$
- B. $\beta=1/3,\ \alpha=1.099$
- C. $\beta=3,\ \alpha=-1.099$
- D. $\beta=0.75,\ \alpha=0.288$

**Answer:** B. $\beta=1/3,\ \alpha=1.099$ — $\beta=\bar L/(1-\bar L)=0.25/0.75=1/3$ and $\alpha=\ln(1/\beta)=\ln 3=1.0986$. A perfectly predicted point's weight would be multiplied by $1/3$; the worst point's weight is left unchanged.

**Question:** Three trees predict 4, 6 and 9 at some $x$, with votes $\alpha=0.5,\,0.8,\,1.0$. What does AdaBoost.R2 output?

- A. 4
- B. 6
- C. 6.87 (the weighted mean)
- D. 9

**Answer:** B. 6 — Half of the total vote is $2.3/2=1.15$. Sorted cumulative votes: 0.5 after 4 (not enough), 1.3 after 6 — the first to reach 1.15. So the weighted median is 6. (The weighted mean would be $(2+4.8+9)/2.3=6.87$, which is not what R2 returns.)

**Question:** A point's absolute error is 2 and the round's maximum error is 4. What is its loss $L_i$ under the square loss?

- A. 0.5
- B. 0.25
- C. 0.3935
- D. 4

**Answer:** B. 0.25 — $L_i=(2/4)^2=0.25$. Linear would give 0.5 and exponential $1-e^{-0.5}=0.3935$. Square loss makes moderate errors look small, so attention concentrates on the largest ones.

**Question:** Training stops early with far fewer trees than n_estimators. What is the most likely cause?

- A. The learning rate is too large
- B. The base trees are too shallow, so a tree's weighted average loss reached 0.5
- C. The weighted median had a tie
- D. The training targets were standardised

**Answer:** B. The base trees are too shallow, so a tree's weighted average loss reached 0.5 — R2 stops as soon as $\bar L_t\ge0.5$ ($\beta_t\ge1$, so $\alpha_t\le0$). Each tree must model $y$ on its own; stumps quickly hit this limit (9 trees on the benchmark; depth 4 used all 300).

**Key points**

- Each round fits a tree to the **original** targets under sample weights; nothing is fitted to residuals and nothing is added up.
- Errors become relative losses $L_i\in[0,1]$ by dividing by the round's maximum error (then linear, square or exponential shaping).
- $\beta_t=\bar L_t/(1-\bar L_t)$; well-fitted points are multiplied by $\beta_t^{1-L_i}$, so hard points gain weight. Training stops when $\bar L_t\ge0.5$.
- Prediction is the $\alpha$-weighted **median** ($\alpha_t=\ln 1/\beta_t$), the minimiser of weighted absolute deviation — a vote that ignores far-off trees.
- Use base trees of depth 3–4; stumps stall. Tune depth, then the number of trees with staged_predict.
- Outliers attract weight round after round: clean targets first, or prefer gradient boosting with a robust loss.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
