Definition
LightGBM regression predicts a number as a constant plus a sum of shrunken regression trees, — the same model as every gradient-boosted tree ensemble. What makes it LightGBM is how each tree is grown: every feature is bucketed once into at most a few hundred bins, splits are searched over bin edges using per-bin gradient histograms, and the tree grows leaf-wise — always splitting the single leaf, anywhere in the tree, whose split lowers the loss most.
At a glance
- Task
- Regression (squared error by default; also L1, Huber, quantile, Poisson, Gamma, Tweedie, MAPE objectives)
- Input → output
- feature vector → real number
- Learns
- bin edges per feature (once, before training); then trees, each with a (feature, bin edge) per internal node and a weight per leaf
- Training
- binning once: ; per tree: building histograms (less with subtraction and GOSS), scanning them — independent of
- Prediction
- comparisons; depth is not bounded by default, so leaf-wise trees can be deep and lopsided
- Key hyperparameters
num_leaves,learning_rate,n_estimators,min_data_in_leaf,max_bin,lambda_l2/lambda_l1,max_depth,feature_fraction,bagging_fraction- Assumes
- a differentiable loss (absolute / quantile losses get a leaf re-fit); nothing about feature distributions; no scaling needed, only the order of values (and their bin) matters; the fit is axis-aligned and piecewise constant
The problem
A gradient-boosted tree ensemble (see Gradient boosting regression for the residual-fitting idea and XGBoost regression for the second-order leaf weights) spends almost all of its training time in one place: finding the best split. The exact greedy method sorts every feature and tries every midpoint between consecutive values — candidates per feature per node. With rows and features, every node of every tree walks a billion (row, feature) pairs, and the pre-sorted index arrays alone are several times the size of the data.
A second, quieter waste is how trees are grown. Classic implementations grow level by level: every leaf at depth 2 is split before any leaf at depth 3, even leaves whose best split barely helps. With a fixed budget of leaves, a level-wise tree spends many of them on near-useless splits while a high-error region of the data waits for the next level.
LightGBM (Ke et al., Microsoft, 2017) attacks both. It gives up on testing every threshold — it tests only bin edges, and finds that this almost never hurts accuracy — and it gives up on balanced trees, spending each leaf where the loss drops most. On the 50,000-row benchmark later in this article, it matches an exact-split learner's test error while training about 80× faster.
The idea
Think of a split search as asking, for each candidate threshold, "how much would the squared error drop if I cut here?". To answer that you only need two running totals on each side: the sum of gradients and the sum of hessians (for squared error, is just the row count). Those totals do not care about the exact values, only which side of the cut each row falls. So LightGBM pre-sorts nothing; it drops every row into a bucket (bin) once, and for each tree node it adds up and per bucket. A split is then a cut between buckets, and its score comes from prefix sums over at most a few hundred numbers — no matter whether the node holds 50 rows or 50 million.
Three more tricks then stack on top. The histogram of one child can be subtracted from the parent's to get the other child for free. Growth is leaf-wise: compare the best split of every current leaf and make only the best one. And two sampling tricks shrink the histogram-building work itself: GOSS drops many rows whose gradient is already small, and EFB merges sparse features that are never non-zero together into one.
Vocabulary
- Bin
- a contiguous range of one feature's values; each row stores only its bin number (a small integer) per feature
- Bin edge (upper bound)
- the threshold between two neighbouring bins, placed halfway between data values. The only thresholds LightGBM ever tests
- max_bin
- maximum number of bins per feature (default 255). Bins are roughly equal-count, so dense regions get narrow bins
- Gradient histogram
- for one leaf and one feature: per bin, the sums and (and the row count) of the rows in that leaf that fall in bin
- Histogram subtraction
- : after a split, build only the smaller child's histogram
- Leaf-wise (best-first) growth
- repeatedly split the leaf with the largest gain anywhere in the tree, until the tree has
num_leavesleaves or no split gains anything - Level-wise (depth-wise) growth
- split all leaves at one depth before going deeper (XGBoost's classic default)
- num_leaves
- the leaf budget per tree (default 31); the main complexity knob, instead of depth
- min_data_in_leaf
- minimum rows a leaf may hold (default 20);
min_sum_hessian_in_leafis the same on H - GOSS
- Gradient-based One-Side Sampling: keep the rows with the largest |g|, subsample the rest and up-weight them
- EFB
- Exclusive Feature Bundling: pack sparse features that are (almost) never non-zero on the same row into one feature with offset bins
The math
What the trees minimize. LightGBM uses the same regularized second-order objective as XGBoost; the derivation is spelled out in XGBoost regression, so here is the short version for squared error. The model after trees is ; the next tree adds .
- target and feature vector of training row i
- the current ensemble's prediction for row i
- the new tree; it outputs weight for every row in its leaf
- first and second derivative of with respect to : minus the residual, and 1
- L2 penalty on leaf weights (
lambda_l2, default 0)
Leaf weight. All rows in leaf get the same , so that leaf's part of is (up to a constant) with , . For squared error this is exact, not an approximation. Setting the derivative to zero:
- sum of gradients and of hessians (= row count) in leaf
- best leaf output. With it is the leaf's mean residual ; λ shrinks it toward 0 as if λ extra rows with residual 0 were in the leaf
- the lowest loss that leaf can reach (relative to w = 0)
Split gain. Splitting a leaf into and replaces one by two. LightGBM reports the reduction without the :
- gradient and hessian sums of the rows the split sends left
- the same for the right side
With and , for a node with mean residual , and the gain is exactly the drop in the sum of squared errors that the split would buy if each side predicted its own mean residual. So a LightGBM regression tree is a variance-reduction tree on the residuals, like a CART regression tree — what changes is which thresholds are scored and in what order leaves are split.
Histograms make the gain cheap. Map each value to its bin once. For a leaf and feature , accumulate
- bin index of row on feature
- histogram entries: gradient and hessian sums of the leaf's rows in bin
- prefix sum — the left side of the split "bins go left";
Building the histogram is one pass over the leaf's rows, per feature, with cheap integer indexing instead of sorted-order access. Scanning all candidate splits is per feature. Compare the exact method's scan plus keeping sorted orders for every node. Because histograms are sums, they are additive across disjoint row sets:
- parent leaf and its two children; L is chosen as the child with fewer rows
so after each split only the smaller child is scanned — at most half the parent's rows — and the larger child costs . Each row is therefore touched in at most histogram builds per tree, instead of once per level of a deep tree.
Leaf-wise growth is a greedy rule on the same gain. Let be the current set of leaves; the next split is
- a current leaf, a feature, and a bin edge of that feature
min_gain_to_split(default 0)
Every split lowers the training loss by its gain, and the gains of different leaves add, so for a fixed number of leaves the leaf-wise order greedily removes the most squared error per leaf. A level-wise tree with the same leaf count has to split some low-gain leaves first. The price: leaf-wise trees can grow long, thin branches into small groups of rows, which is exactly how they overfit.
GOSS. Rows with small are already fitted well and contribute little to the gain. GOSS keeps the top rows by ("top_rate", default 0.2), samples of the rest uniformly ("other_rate", default 0.1), and multiplies the sampled small-gradient rows' and by so the sums stay unbiased:
- large-gradient rows that go left (always kept)
- sampled small-gradient rows that go left
- up-weighting factor: with a = 0.2, b = 0.1 each sampled row stands for 8 rows
Histograms are then built from rows — 30% with the defaults. For squared error, large means large residuals, so GOSS focuses each tree on the houses the model currently prices worst.
How each hyperparameter enters the math. adds to every denominator: it shrinks and lowers the gain of splits on small leaves the most (a leaf with and keeps only a third of its weight). (lambda_l1) soft-thresholds the gradient sum, , so leaves whose residuals nearly cancel output exactly 0. min_data_in_leaf and min_sum_hessian_in_leaf remove candidate edges whose or is too small. max_bin sets : fewer bins mean fewer candidate thresholds, a faster scan, and a coarser function — with one feature, the model can take at most different values, however many trees you add. num_leaves caps ; learning_rate multiplies every before it is added.
Worked example: pricing flats by floor area
Twelve flats, one feature (floor area in m²) and the price in thousands of dollars. The mean price is exactly 260, so . We grow the first tree by hand with 4 bins, num_leaves = 3, , min_data_in_leaf = 1 and learning rate .
| Flat | Size (m²) | Price (k$) | g = 260 − y | Bin |
|---|---|---|---|---|
| 1 | 42 | 150 | +110 | 0 |
| 2 | 48 | 162 | +98 | 0 |
| 3 | 55 | 158 | +102 | 0 |
| 4 | 61 | 175 | +85 | 1 |
| 5 | 66 | 180 | +80 | 1 |
| 6 | 72 | 186 | +74 | 1 |
| 7 | 79 | 300 | −40 | 2 |
| 8 | 85 | 312 | −52 | 2 |
| 9 | 93 | 318 | −58 | 2 |
| 10 | 104 | 340 | −80 | 3 |
| 11 | 118 | 425 | −165 | 3 |
| 12 | 130 | 414 | −154 | 3 |
Growing tree 1
- 1Bin once. 12 sorted sizes into 4 equal-count bins; the edges sit halfway between the 3rd/4th, 6th/7th and 9th/10th values: , , .
- 2Gradients. , . They sum to 0 because is the mean.
- 3Root histogram (one pass over 12 rows): for bins 0–3, . Total , , so the parent term .
- 4Scan 3 edges instead of 11 midpoints. Edge 58: . Edge 75.5: , 100,467.0. Edge 98.5: . Best: size ≤ 75.5. (An exact learner tries all 11 midpoints; 75.5 is also its best root split, so binning cost nothing here.)
- 5Children. Left (flats 1–6) gets its histogram from a pass over its 6 rows: . Right = parent − left — no pass over flats 7–12.
- 6Score every leaf. Left leaf, only edge 58: . Right leaf, only edge 98.5: 10,333.5. An exact learner would cut the right leaf at 111 m² (gain 13,872.0), but 111 is inside bin 3, so LightGBM cannot see it — this is the "little exactness" being traded.
- 7Leaf-wise choice. , so split the right leaf at 98.5. The tree now has 3 leaves = num_leaves, so it stops. A level-wise tree with the same 3 leaves would have split the left leaf (840.2) instead.
- 8Leaf weights : size ≤ 75.5: ; 75.5–98.5: ; > 98.5: .
- 9Update. : 250.85, 265.00, 273.30. Train RMSE drops from 98.41 to 89.05 (the level-wise tree would reach 89.89).
- 10Predict a new 100 m² flat: → bin 3 → leaf → 273.3 k$.
| Histogram | bin 0 | bin 1 | bin 2 | bin 3 | Best edge (gain) |
|---|---|---|---|---|---|
| Root (12 rows) | +310 / 3 | +239 / 3 | −150 / 3 | −399 / 3 | 75.5 (100,467.0) |
| Left child: built from 6 rows | +310 / 3 | +239 / 3 | 0 / 0 | 0 / 0 | 58 (840.2) |
| Right child = root − left | 0 / 0 | 0 / 0 | −150 / 3 | −399 / 3 | 98.5 (10,333.5) |
Keep boosting and every new tree repeats the same procedure on the new residuals. After 10 trees the 100 m² flat is priced at 346.63, after 100 trees at 392.99 — almost exactly 393, the mean price of bin 3. With 4 bins and one feature the model can only ever output 4 values, so after 100 rounds the training RMSE stalls at 19.53, the spread of prices within the bins. Give it 255 bins and the same 100 trees fit the 12 flats to an RMSE of 0.37 — memorization. max_bin is a speed knob and a regularizer.
- 1bin x once into max_bin bins
- 2F ← mean(y)
- 3for t = 1 … T:
- 4 gᵢ ← F(xᵢ) − yᵢ , hᵢ ← 1
- 5 hist[bin] ← (ΣG, ΣH)
- 6 gain ← G²_L/H_L + G²_R/H_R − G²/H
- 7 split best (leaf, bin) anywhere
- 8 sibling hist ← parent − child
- 9 at num_leaves: w ← −G/(H+λ)
- 10 F ← F + η·w
- 11predict: x → bin → leaf → F
Before any tree is grown, the 12 sizes are bucketed once into 4 equal-count bins with edges at 58, 75.5, 98.5 m². From now on only bin numbers 0–3 are used.
The algorithm, step by step
LightGBM regression (squared error)
- 1Bin. For each feature, find at most
max_binequal-count bin edges from (a sample of) the training values —bin_construct_sample_cnt= 200,000 rows by default — and replace every value with its bin index. Bundle mutually exclusive sparse features (EFB). Done once. - 2Initialize (
boost_from_average). - 3For each round : compute , ; optionally keep only a GOSS or bagging subset of rows.
- 4Root. Build the root's histogram for every feature (one pass over the rows) and find its best (feature, edge) by prefix-sum scans.
- 5Grow leaf-wise. Repeat until the tree has
num_leavesleaves or the best gain is ≤min_gain_to_split: split the leaf whose best split has the highest gain (respectingmin_data_in_leaf,max_depth); build the smaller child's histogram, subtract it from the parent's for the larger child; find the best split of both children. - 6Leaf weights (with soft-thresholding if set); add to every training row's .
- 7Stop after
n_estimatorsrounds or when the validation metric (L2, RMSE, L1…) has not improved forearly_stopping_rounds. - 8Predict: bin the new row with the stored edges, walk each tree, sum .
Cost per tree (n rows, d features, B bins, L leaves)
- Exact greedy (pre-sorted)
- scan per tree level plus sorted index maintenance; memory about the data for sorted indices
- Histogram build
- for the root; with subtraction each row is re-scanned only when it lands in the smaller child, at most times
- Split search
- — independent of
- With GOSS / EFB
- shrinks to about rows; shrinks to the number of bundles
- Memory
- one byte per value when (vs 4–8 bytes for floats)
Leaf-wise vs depth-wise
On the worked example the two growth orders share the root split, then disagree: leaf-wise spends the third leaf on the expensive right half (gain 10,333.5), level-wise on the cheap left half (840.2). One tree later the train RMSE is 89.05 vs 89.89. On large, rich data that difference compounds — leaf-wise reaches a given loss with far fewer leaves. On small or noisy data the same greed builds deep chains that isolate a handful of rows, so LightGBM pairs num_leaves with min_data_in_leaf and an optional max_depth.
Regression objectives
Only and change between objectives; binning, histograms and leaf-wise growth are untouched. For losses whose second derivative is zero or undefined (L1, quantile), LightGBM uses and afterwards re-fits each leaf to the right statistic of its residuals (the median, or the α-quantile).
| objective | Loss | gᵢ | hᵢ | Use for |
|---|---|---|---|---|
regression (L2) | ½ r² | −r | 1 | default; predicts the conditional mean |
regression_l1 | |r| | −sign(r) | 1, leaves re-fit to median | heavy-tailed noise; predicts the median |
huber | ½ r² if |r| ≤ α, else α|r| − ½α² | −r, clipped to ±α | 1 | mostly Gaussian noise with some outliers |
quantile | α·r if r ≥ 0, else (α − 1)·r | −α or 1 − α | 1, leaves re-fit to α-quantile | prediction intervals (e.g. α = 0.1 and 0.9) |
poisson | e^F − yF | e^F − y | e^F (scaled) | counts ≥ 0 |
gamma / tweedie | log-link deviance | from deviance | from deviance | positive skewed amounts, insurance claims |
| objective | ŷ(50 m²) | ŷ(70 m²) | ŷ(100 m²) |
|---|---|---|---|
| regression (L2) | 156.67 | 418.33 | 393.00 |
| regression_l1 | 158.00 | 180.00 | 414.00 |
| huber, α = 30 | 156.67 | 192.50 | 404.50 |
Hyperparameters and tuning
What to tune, in order
- learning_rate + n_estimators
- Fix η around 0.05–0.1 and let early stopping on a validation set pick the number of trees. Halving η roughly doubles the trees needed.
- num_leaves
- Main capacity knob (default 31). Try 15–255; larger needs more data and a larger
min_data_in_leaf. - min_data_in_leaf
- Default 20. The main brake on leaf-wise overfitting; hundreds or thousands for large noisy data.
- max_bin
- Default 255. Lower (15–63) is faster and regularizes; higher only helps when fine thresholds matter on very large data.
- feature_fraction, bagging_fraction + bagging_freq
- Column and row subsampling per tree (e.g. 0.8): decorrelates trees and speeds up training.
- lambda_l2, lambda_l1, min_gain_to_split
- Shrink leaf weights and veto weak splits; useful once the above are set.
- max_depth
- Off (−1) by default; set it (e.g. 6–10) as a safety net against very deep leaf-wise chains.
- data_sample_strategy = goss
- Pays off on millions of rows; on small data it can be slower than plain histograms.
| Model | num_leaves | max_bin | Test RMSE | Fit time |
|---|---|---|---|---|
| LightGBM | 3 | 255 | 1.128 | 0.40 s |
| LightGBM | 7 | 255 | 1.055 | 0.54 s |
| LightGBM | 31 | 15 | 1.153 | 0.70 s |
| LightGBM | 31 | 63 | 1.057 | 0.74 s |
| LightGBM (defaults) | 31 | 255 | 1.058 | 0.92 s |
| LightGBM | 127 | 255 | 1.065 | 1.83 s |
| LightGBM | 511 | 255 | 1.108 | 4.56 s |
| LightGBM, GOSS | 31 | 255 | 1.064 | 1.36 s |
| scikit-learn GradientBoosting (exact) | 31 | — | 1.059 | 73.53 s |
Two lessons from the table. Bins: 15 bins cost accuracy (1.153), 63 are already as good as 255. Leaves: past the true complexity of the function, more leaves only overfit (511 leaves: 1.108) and cost time. GOSS was slower here — with only 50,000 rows the sampling overhead outweighs the saved histogram work.
In code
1import numpy as np23def make_bins(x, max_bins):4 """Equal-frequency bin edges, placed midway between neighbouring sorted values."""5 v = np.unique(x)6 if len(v) <= max_bins:7 return (v[:-1] + v[1:]) / 28 xs = np.sort(x)9 cuts = [round(k * len(xs) / max_bins) for k in range(1, max_bins)]10 return np.unique([(xs[c - 1] + xs[c]) / 2 for c in cuts])1112def histogram(B, g, rows, n_bins):13 """Per feature and bin: sum of gradients, sum of hessians (=count for L2), count."""14 d = B.shape[1]15 Hist = np.zeros((3, d, n_bins))16 for j in range(d):17 np.add.at(Hist[0, j], B[rows, j], g[rows])18 np.add.at(Hist[1, j], B[rows, j], 1.0) # h_i = 1 for squared error19 np.add.at(Hist[2, j], B[rows, j], 1.0)20 return Hist2122def best_split(Hist, lam, min_leaf):23 best = (0.0, None, None) # (gain, feature, last bin on the left)24 G, H, N = Hist25 for j in range(G.shape[0]):26 GL, HL, NL = (np.cumsum(a[j])[:-1] for a in (G, H, N))27 Gt, Ht, Nt = G[j].sum(), H[j].sum(), N[j].sum()28 with np.errstate(divide="ignore", invalid="ignore"):29 gain = GL**2 / (HL + lam) + (Gt - GL)**2 / (Ht - HL + lam) - Gt**2 / (Ht + lam)30 gain[(NL < min_leaf) | (Nt - NL < min_leaf) | np.isnan(gain)] = -np.inf31 if len(gain) and gain.max() > best[0]:32 best = (gain.max(), j, int(gain.argmax()))33 return best3435def grow_tree(B, g, n_bins, num_leaves, lam, min_leaf):36 """Leaf-wise growth with histogram subtraction."""37 root = np.arange(len(g))38 leaves = {0: (root, histogram(B, g, root, n_bins))}39 nodes = {} # node id -> (feature, bin, left id, right id)40 while len(leaves) < num_leaves:41 cand = {i: best_split(h, lam, min_leaf) for i, (_, h) in leaves.items()}42 i = max(cand, key=lambda k: cand[k][0]) # the best leaf ANYWHERE in the tree43 gain, j, k = cand[i]44 if j is None:45 break # no split with positive gain left46 rows, hist = leaves.pop(i)47 L, R = rows[B[rows, j] <= k], rows[B[rows, j] > k]48 if len(L) <= len(R): # build the smaller child's histogram,49 hL = histogram(B, g, L, n_bins); hR = hist - hL # get the larger one by subtraction50 else:51 hR = histogram(B, g, R, n_bins); hL = hist - hR52 a, b = 2 * len(nodes) + 1, 2 * len(nodes) + 253 nodes[i] = (j, k, a, b)54 leaves[a], leaves[b] = (L, hL), (R, hR)55 value = {i: -h[0, 0].sum() / (h[1, 0].sum() + lam) for i, (_, h) in leaves.items()}56 return nodes, value # leaf output w* = -G / (H + lambda)5758def route(nodes, value, B):59 node = np.zeros(len(B), int)60 for i, (j, k, a, b) in nodes.items(): # parents were stored before children61 at = node == i62 node[at] = np.where(B[at, j] <= k, a, b)63 return np.array([value[i] for i in node])6465class TinyLightGBMRegressor:66 def __init__(self, n_estimators=100, learning_rate=0.1, num_leaves=31,67 max_bin=255, lambda_l2=0.0, min_data_in_leaf=20):68 self.T, self.eta, self.leaves = n_estimators, learning_rate, num_leaves69 self.max_bin, self.lam, self.min_leaf = max_bin, lambda_l2, min_data_in_leaf7071 def _bin(self, X):72 return np.stack([np.searchsorted(e, X[:, j]) for j, e in enumerate(self.edges)], 1)7374 def fit(self, X, y):75 self.edges = [make_bins(X[:, j], self.max_bin) for j in range(X.shape[1])]76 B, n_bins = self._bin(X), max(len(e) for e in self.edges) + 1 # bin ONCE, up front77 self.f0 = y.mean()78 F = np.full(len(y), self.f0)79 self.trees = []80 for _ in range(self.T):81 g = F - y # gradient of 1/2 (F - y)^282 nodes, value = grow_tree(B, g, n_bins, self.leaves, self.lam, self.min_leaf)83 F += self.eta * route(nodes, value, B)84 self.trees.append((nodes, value))85 return self8687 def predict(self, X):88 B = self._bin(np.asarray(X, float))89 return self.f0 + self.eta * sum(route(n, v, B) for n, v in self.trees)9091size = np.array([42, 48, 55, 61, 66, 72, 79, 85, 93, 104, 118, 130], float)92price = np.array([150, 162, 158, 175, 180, 186, 300, 312, 318, 340, 425, 414], float)93X = size.reshape(-1, 1)94for T in [1, 100]:95 m = TinyLightGBMRegressor(T, 0.1, num_leaves=3, max_bin=4, min_data_in_leaf=1).fit(X, price)96 rmse = np.sqrt(np.mean((m.predict(X) - price) ** 2))97 print(f"T={T:3d} bin edges={m.edges[0]} train RMSE={rmse:.3f} f(100 m2)={m.predict([[100.]])[0]:.3f}")98# T= 1 bin edges=[58. 75.5 98.5] train RMSE=89.053 f(100 m2)=273.30099# T=100 bin edges=[58. 75.5 98.5] train RMSE=19.531 f(100 m2)=392.9921import numpy as np2import lightgbm as lgb34size = np.array([42, 48, 55, 61, 66, 72, 79, 85, 93, 104, 118, 130], float)5price = np.array([150, 162, 158, 175, 180, 186, 300, 312, 318, 340, 425, 414], float)6X = size.reshape(-1, 1)78params = dict(objective="regression", num_leaves=3, learning_rate=0.1,9 max_bin=5, min_data_in_bin=1, # 4 bins: 58 | 75.5 | 98.510 min_data_in_leaf=1, min_sum_hessian_in_leaf=0,11 lambda_l2=0.0, verbose=-1)1213booster = lgb.train(params, lgb.Dataset(X, price, params=params), num_boost_round=1)14print(booster.trees_to_dataframe()[["node_depth", "threshold", "split_gain", "value", "count"]])15# node_depth threshold split_gain value count16# 0 1 75.5 100467.0 260.00 1217# 1 2 NaN NaN 250.85 618# 2 2 98.5 10333.5 269.15 619# 3 3 NaN NaN 265.00 320# 4 3 NaN NaN 273.30 321print(booster.predict([[100.0]])) # [273.3]2223for rounds in [10, 100]:24 b = lgb.train(params, lgb.Dataset(X, price, params=params), num_boost_round=rounds)25 rmse = np.sqrt(np.mean((b.predict(X) - price) ** 2))26 print(rounds, round(rmse, 3), round(b.predict([[100.0]])[0], 3))27# 10 39.674 346.62628# 100 19.531 392.9922930# On real data: scikit-learn API + early stopping on a validation set31# model = lgb.LGBMRegressor(n_estimators=5000, learning_rate=0.05, num_leaves=31,32# min_child_samples=20, subsample=0.8, subsample_freq=1,33# colsample_bytree=0.8)34# model.fit(X_tr, y_tr, eval_set=[(X_va, y_va)], eval_metric="l2",35# callbacks=[lgb.early_stopping(100)])Use when
- +Large tabular data (10⁵ rows and up): histogram splits and leaf-wise growth train an order of magnitude or more faster than exact boosting at the same accuracy.
- +Many sparse or one-hot features: EFB bundles them, and bins are 1 byte each.
- +You need robust or distributional losses (L1, Huber, quantile, Poisson, Tweedie) with the same fast trainer.
- +Native categorical features (
categorical_feature) and missing values are needed without preprocessing.
Avoid when
- −The dataset is small (hundreds of rows): leaf-wise trees overfit easily and the defaults may not split at all.
- −You need smooth predictions or extrapolation: the fit is piecewise constant and flat beyond the training range.
- −The relationship is mostly linear in a few features: a regularized linear model is simpler and extrapolates.
- −You need a single interpretable model: hundreds of deep, lopsided trees are not readable.
Where it's used
Demand and sales forecasting
Retail and supply-chain teams regress daily units sold on calendar, price and lag features over millions of rows; LightGBM was the workhorse of most top solutions in the M5 forecasting competition.
Ad and ranking signals
Predicting click-through or dwell time on huge, sparse feature sets — the workload LightGBM was built for at Microsoft.
Insurance claim severity
Gamma and Tweedie objectives model positive, right-skewed claim amounts with many zeros.
Real-estate and pricing models
Automated valuation from size, location and amenities; quantile objectives give price intervals instead of a single number.
Check yourself
A leaf's histogram scan reaches an edge with G_L = −120, H_L = 4 on the left and G_R = +120, H_R = 6 on the right (λ = 0). What gain does LightGBM report?
Check yourself
A model has one feature binned into 8 bins. After 10,000 boosting rounds, how many different predictions can it output at most?
Check yourself
A tree has 3 leaves. Their best splits gain 300 (leaf A), 1,200 (leaf B, at depth 4) and nothing (leaf C, at depth 1). With num_leaves = 4, what does LightGBM do next?
Check yourself
GOSS runs with top_rate a = 0.2 and other_rate b = 0.1. By what factor are the sampled small-gradient rows' gradients multiplied?
Remember
- ✓For squared error , : leaf weight is a shrunken mean residual, and the split gain (λ = 0) is exactly the drop in squared error.
- ✓Features are binned once; splits are scored only at bin edges from per-bin gradient histograms, so split search costs O(bins), not O(rows).
- ✓Histogram subtraction builds only the smaller child; leaf-wise growth spends each leaf on the largest gain anywhere in the tree.
- ✓num_leaves and min_data_in_leaf are the main capacity controls; max_bin trades speed and smoothness (one feature with B bins gives at most B output values).
- ✓GOSS and EFB cut rows and features for very large or sparse data; on small data, lower the defaults or use a simpler model.