Definition
Lasso regression (least absolute shrinkage and selection operator, Tibshirani 1996) is linear regression whose squared-error loss carries an extra penalty on the absolute size of the weights. The penalty shrinks every weight toward zero and sets the weakest ones exactly to zero, so fitting the model also picks which features to keep.
At a glance
- Task
- Regression (numeric target); the same penalty is used for L1-regularized logistic regression
- Input → output
- feature vector →
- Learns
- a sparse weight vector (many entries exactly 0) and an unpenalized intercept
- Loss
- — convex, but not differentiable where any
- Training
- no closed form; coordinate descent costs per sweep, and a whole path of values costs little more than one fit thanks to warm starts
- Prediction
- for non-zero weights — dropped features need not even be measured
- Hyperparameters
- (scikit-learn: "alpha"), chosen by cross-validation; tolerance and max iterations of the solver
- Assumes
- a roughly linear relationship; features on a common scale (standardize first); the true signal uses a few of the features (sparsity)
The problem
Ordinary least squares uses every feature you give it, and on small or noisy data it happily assigns weight to columns that mean nothing. In the worked example below, 10 house prices are regressed on size, bedrooms, age and the house's door number. OLS says each extra standard deviation of door number adds 3.77 k. Both are noise. Its training error is low (MSE 258.9) but its leave-one-out error is 1276.8 — five times worse.
The classical fix is best-subset selection: try every subset of features and keep the one with the best validation score. With features there are subsets — about a million at — and the problem is NP-hard in general. Greedy forward/backward stepwise selection is cheaper but unstable. Ridge regression (penalty , see ridge-regression) is cheap and stable but never removes anything: on the same data with scikit-learn's ridge at alpha = 10 the door weight is −0.008, small but not zero. Lasso is the convex middle ground: one smooth optimisation problem that both shrinks and selects.
The idea
Charge every weight a fixed price per unit of size, , regardless of how big it already is. A feature is worth including only if moving its weight away from 0 lowers the squared error by more than per unit. Features whose correlation with what is left of the target is below that price are not worth a single unit, so their weight stays at exactly 0. That is the difference from ridge, whose price per unit vanishes as , so every feature can always afford a little weight.
Geometrically (the full picture is in l1-l2-regularization): the penalized problem is the same as minimizing squared error inside the diamond . The elliptical error contours usually first touch the diamond at a corner, and the corners lie on the axes, where some coordinates are 0. Ridge's ball is round and has no corners, so the touching point is almost never on an axis.
Vocabulary
- L1 norm
- , the sum of absolute values. Its unit ball is a diamond (a cross-polytope in dimensions)
- λ (alpha)
- the penalty strength. is OLS; large gives all-zero weights
- Sparse
- most weights exactly 0. The model is then a regression on the few features that survive
- Active set
- the features with at a given
- Partial residual
- : what is left of the target after every feature except has made its prediction
- ρⱼ
- , how strongly feature correlates with its partial residual; the quantity lasso compares with
- Soft-thresholding
- : move toward 0 by , and stop at 0
- Regularization path
- the weights as functions of . For lasso they are piecewise linear in
- λ_max
- the smallest at which every weight is 0
- Subgradient
- the set of slopes of at a kink: if , the whole interval at
The math
- number of training rows and of features
- feature of row , standardized so that and
- target of row i
- intercept — not penalized; with centred features its optimum is
- weight of feature j
- penalty strength (scikit-learn and glmnet call it alpha / lambda with exactly this 1/(2n) scaling)
Because has a kink at 0 there is no gradient to set to zero for the whole vector, and so no closed form like OLS's . But the problem is easy one coordinate at a time. Fix every weight except . Using , the objective in alone is
- partial residual of row :
- the least-squares weight feature j would get if it had to explain the partial residual on its own
This is a parabola plus a V. At the minimum, 0 must lie in the subgradient: . Three cases:
Solving the one-dimensional problem
- 1: , so . This is consistent only if .
- 2: , so , consistent only if .
- 3: can be anything in , so we need , i.e. . This case covers a whole interval of values, not a single point. That interval is where the exact zeros come from.
- 4Together: , the soft-thresholding operator below.
- soft-threshold: shrink toward 0 by ; anything inside becomes exactly 0
Cycling this update over until nothing changes is coordinate descent. Each update decreases (it minimizes exactly along one axis), is convex, and its non-smooth part is a sum of one-coordinate terms, so the cycle converges to the global minimum. At that minimum the update changes nothing, which gives the optimality (KKT) conditions with the full residual :
- correlation of feature j with the final residual
Read it as: every active feature has exactly the same correlation with the residual (up to sign), and every dropped feature has less. Some consequences:
What λ does
- λ = 0
- OLS (when it is unique).
- λ ≥ λ_max
- all weights 0. At the residual is , so the condition is — the largest marginal correlation.
- Between
- as falls from , features enter the active set one at a time and the path is piecewise linear between those knots (the LARS algorithm walks exactly these segments). A feature can also leave again.
- Orthonormal features
- if , then is just the OLS weight, so while : lasso subtracts, ridge divides.
- Bias
- surviving weights are pulled toward 0 by , even large, important ones. That is why practitioners often refit OLS on the selected features (the relaxed lasso).
- At most n
- when , a lasso solution has at most non-zero weights.
Worked example: which features matter for house prices?
Ten houses with four features. Price was generated from size and age only, plus noise; bedrooms is partly correlated with size (0.61), and door number is pure noise. We fit lasso with and check whether it finds that.
| House | Size (m²) | Beds | Age (yr) | Door no. | Price (k$) |
|---|---|---|---|---|---|
| 1 | 50 | 2 | 30 | 7 | 185 |
| 2 | 60 | 1 | 5 | 3 | 235 |
| 3 | 65 | 3 | 20 | 12 | 185 |
| 4 | 70 | 2 | 10 | 5 | 210 |
| 5 | 80 | 2 | 25 | 9 | 220 |
| 6 | 85 | 3 | 8 | 14 | 285 |
| 7 | 90 | 2 | 15 | 2 | 250 |
| 8 | 100 | 4 | 40 | 8 | 265 |
| 9 | 110 | 3 | 12 | 11 | 305 |
| 10 | 120 | 3 | 22 | 4 | 300 |
| House | z size | z beds | z age | z door | y − 244 |
|---|---|---|---|---|---|
| 1 | −1.545 | −0.620 | 1.092 | −0.131 | −59 |
| 2 | −1.077 | −1.861 | −1.324 | −1.176 | −9 |
| 3 | −0.843 | 0.620 | 0.126 | 1.176 | −59 |
| 4 | −0.609 | −0.620 | −0.841 | −0.653 | −34 |
| 5 | −0.140 | −0.620 | 0.609 | 0.392 | −24 |
| 6 | 0.094 | 0.620 | −1.034 | 1.698 | 41 |
| 7 | 0.328 | −0.620 | −0.358 | −1.437 | 6 |
| 8 | 0.796 | 1.861 | 2.059 | 0.131 | 21 |
| 9 | 1.264 | 0.620 | −0.648 | 0.914 | 61 |
| 10 | 1.733 | 0.620 | 0.319 | −0.914 | 56 |
Coordinate descent by hand, λ = 6, starting from w = 0
- 1Where the path starts. At the correlations are size 37.136, beds 18.605, age −6.893, door 3.005. The largest magnitude gives : for any above it the answer is all zeros. With , three of the four exceed the threshold marginally, but marginal correlations are not what lasso finally compares.
- 2Sweep 1, size. Nothing else is in the model, so the partial residual is and . Then .
- 3Sweep 1, beds. Now subtract size's prediction: . Beds' correlation with this residual is . Its marginal correlation 18.605 came almost entirely from being correlated with size. , so .
- 4Sweep 1, age. (larger than its marginal −6.893, because the size term has removed variation that was masking it). .
- 5Sweep 1, door. , inside the dead zone, so . After one sweep , objective .
- 6Sweep 2 recomputes each with the updated weights: size 37.706 → 31.706; beds 1.711 → 0; age −10.754 → −4.754; door 2.657 → 0. .
- 7Sweep 3 moves the weights by less than 0.01 (size 37.714 → 31.714, age −10.755 → −4.755). Converged: .
- 8Check the KKT conditions. With the final residual, . The two active features sit exactly at ; the two dropped ones are strictly inside . This is the optimum.
- 9Predict a new house: 95 m², 3 beds, 10 years, door 6. Standardized: , (beds and door do not matter). k\hat y=129.33+1.4852\cdot\text{size}-0.4596\cdot\text{age}$.
| Sweep | ρ size → w | ρ beds → w | ρ age → w | ρ door → w | Objective J | Train MSE |
|---|---|---|---|---|---|---|
| 1 | 37.136 → 31.136 | −0.384 → 0 | −10.684 → −4.684 | 2.664 → 0 | 393.816 | 357.8 |
| 2 | 37.706 → 31.706 | 1.711 → 0 | −10.754 → −4.754 | 2.657 → 0 | 393.651 | 349.8 |
| 3 | 37.714 → 31.714 | 1.742 → 0 | −10.755 → −4.755 | 2.657 → 0 | 393.651 | 349.7 |
| Model | w size | w beds | w age | w door | Train MSE | LOOCV MSE | ŷ new house |
|---|---|---|---|---|---|---|---|
| OLS | 39.480 | −1.768 | −10.963 | 3.766 | 258.9 | 1276.8 | 272.83 |
| Lasso λ = 6 | 31.715 | 0 | −4.755 | 0 | 349.7 | 625.0 | 265.82 |
Lasso pays about 91 points of training MSE and gets less than half the out-of-sample error in return. It dropped exactly the two features that had no effect on the price. Scanning in steps of 0.25, the LOOCV error is lowest at (620.7), so is nearly optimal.
The algorithm, step by step
LassoCD(X, y, λ)
- 1Standardize each column to mean 0 and ; set and ; start at (or a warm start from a nearby ).
- 2For each feature in turn: (adding back turns the full residual into the partial residual).
- 3Soft-threshold: .
- 4Update the residual in : ; set .
- 5Repeat sweeps until the largest change in any weight (or the duality gap) is below the tolerance.
- 6Un-standardize: , .
Cost
- One sweep
- , and only for sparse . No matrix is ever formed or inverted.
- Sweeps needed
- a handful when features are nearly uncorrelated (in the example, 3 sweeps reach 2-decimal accuracy and 7 reach a 10⁻⁸ tolerance); many more when features are strongly correlated, because each coordinate step can only move along one axis of a narrow valley.
- Whole path
- fit (log-spaced, typically to ) using each solution as the next warm start. With active-set tricks (only sweep the non-zero features until they converge, then check the rest) this is barely slower than a single fit.
- LARS-lasso
- computes the exact piecewise-linear path knot by knot, about the cost of one OLS fit, for . Good when is moderate and you want the whole path.
- Prediction
- per row.
Variants and tuning
| Setting | Meaning | Practical advice |
|---|---|---|
| alpha | λ in the objective above | pick by CV on a log grid from λ_max down to 10⁻³ λ_max (LassoCV does this with n_alphas = 100) |
| Scaling (StandardScaler) | not part of Lasso itself | always standardize in a pipeline: the penalty treats a weight in metres and a weight in millimetres the same |
| fit_intercept | leave the intercept unpenalized | keep True unless the data are already centred |
| max_iter, tol | coordinate descent stopping rule (duality gap) | raise max_iter if you see a ConvergenceWarning, which is common with correlated features and tiny alpha |
| selection | cyclic or random coordinate order | random can converge faster when d is large |
| positive | constrain w ≥ 0 | for quantities that cannot be negative (mixture proportions, prices of components) |
| warm_start | start from the previous solution | speeds up manual λ sweeps |
Variants
- LassoCV / 1-SE rule
- choose with the lowest CV error, or the largest within one standard error of it. The 1-SE choice gives a sparser, more stable model at almost the same accuracy.
- LassoLarsIC
- choose λ by AIC or BIC instead of CV. Very fast, but relies on a noise-variance estimate and needs n > d.
- Relaxed lasso
- use lasso only to choose the active set, then refit OLS (or a lightly penalized fit) on those features to remove the shrinkage bias.
- Adaptive lasso
- weight the penalty per feature, : features that look strong at first are penalized less. Under some conditions this recovers the correct features where plain lasso does not.
- Elastic net
- add a ridge term, . It keeps groups of correlated features together; see elastic-net-regression.
- Group / fused lasso
- penalize groups of coefficients together (e.g. all dummies of one categorical variable in or out), or differences between neighbouring coefficients (piecewise-constant signals).
In code
1import numpy as np23def soft_threshold(rho, lam):4 return np.sign(rho) * np.maximum(np.abs(rho) - lam, 0.0)56def lasso_cd(X, y, lam, tol=1e-8, max_sweeps=10_000):7 """Minimize (1/2n)||y - b - Xw||^2 + lam*||w||_1 by cyclic coordinate descent."""8 n, d = X.shape9 mu, sd = X.mean(0), X.std(0) # standardize: every column has (1/n)sum z^2 = 110 Z, b = (X - mu) / sd, y.mean() # the intercept is never penalized11 w, r = np.zeros(d), y - b # r = current residual y - b - Zw12 for sweep in range(max_sweeps):13 biggest = 0.014 for j in range(d):15 rho = Z[:, j] @ r / n + w[j] # = (1/n) z_j . (partial residual without feature j)16 new = soft_threshold(rho, lam)17 r -= Z[:, j] * (new - w[j]) # O(n) residual update18 biggest = max(biggest, abs(new - w[j]))19 w[j] = new20 if biggest < tol:21 break22 coef = w / sd # back to original units23 return coef, b - mu @ coef, w, sweep + 12425X = np.array([[50, 2, 30, 7], [60, 1, 5, 3], [65, 3, 20, 12], [70, 2, 10, 5], [80, 2, 25, 9],26 [85, 3, 8, 14], [90, 2, 15, 2], [100, 4, 40, 8], [110, 3, 12, 11], [120, 3, 22, 4]], float)27y = np.array([185, 235, 185, 210, 220, 285, 250, 265, 305, 300], float)2829coef, intercept, w_std, sweeps = lasso_cd(X, y, lam=6.0)30print("standardized w:", w_std.round(3), "sweeps:", sweeps) # [31.715 0. -4.755 0.] 731print("raw coef:", coef.round(4), "intercept:", round(intercept, 3)) # [1.4852 0. -0.4596 0.] 129.32632print("predict [95, 3, 10, 6]:", round(intercept + np.array([95, 3, 10, 6]) @ coef, 2)) # 265.8233lam_max = np.abs(((X - X.mean(0)) / X.std(0)).T @ (y - y.mean()) / len(y)).max()34print("lambda_max:", round(lam_max, 3)) # 37.1361import numpy as np2from sklearn.linear_model import Lasso, LinearRegression, lasso_path3from sklearn.model_selection import LeaveOneOut, cross_val_score4from sklearn.pipeline import make_pipeline5from sklearn.preprocessing import StandardScaler67X = np.array([[50, 2, 30, 7], [60, 1, 5, 3], [65, 3, 20, 12], [70, 2, 10, 5], [80, 2, 25, 9],8 [85, 3, 8, 14], [90, 2, 15, 2], [100, 4, 40, 8], [110, 3, 12, 11], [120, 3, 22, 4]], float)9y = np.array([185, 235, 185, 210, 220, 285, 250, 265, 305, 300], float)10names = ["size", "beds", "age", "door"]11new = np.array([[95, 3, 10, 6]])1213lasso = make_pipeline(StandardScaler(), Lasso(alpha=6.0)).fit(X, y)14ols = make_pipeline(StandardScaler(), LinearRegression()).fit(X, y)15print("lasso w:", dict(zip(names, lasso[-1].coef_.round(3)))) # size 31.715, beds 0, age -4.755, door 016print("ols w:", dict(zip(names, ols[-1].coef_.round(3)))) # 39.48, -1.768, -10.963, 3.76617print("predict new house: lasso %.2f ols %.2f" % (lasso.predict(new)[0], ols.predict(new)[0])) # 265.82 272.831819loo = LeaveOneOut()20for name, model in [("ols", ols), ("lasso a=6", lasso), ("lasso a=5.5", make_pipeline(StandardScaler(), Lasso(alpha=5.5)))]:21 cv = -cross_val_score(model, X, y, cv=loo, scoring="neg_mean_squared_error").mean()22 train = np.mean((model.fit(X, y).predict(X) - y) ** 2)23 print(f"{name:12s} train MSE {train:7.1f} LOOCV MSE {cv:7.1f}")24# ols train MSE 258.9 LOOCV MSE 1276.825# lasso a=6 train MSE 349.7 LOOCV MSE 625.026# lasso a=5.5 train MSE 336.6 LOOCV MSE 620.72728alphas, coefs, _ = lasso_path(StandardScaler().fit_transform(X), y - y.mean(), alphas=np.linspace(40, 0.01, 4000))29for j, n in enumerate(names):30 print(n, "enters at alpha ~", round(alphas[np.argmax(coefs[j] != 0)], 2))31# size 37.13, beds 0.18, age 10.17, door 2.77 (grid approximations of the exact knots)Use when
- +You suspect only a few of many features matter and want a model that names them.
- +Prediction or measurement is expensive per feature: zero weights mean those inputs never need collecting.
- +d is large, even larger than n (genomics, text, sensor banks). Coordinate descent scales and the solution has at most n non-zeros.
- +You need a fast, convex, reproducible alternative to stepwise selection.
Avoid when
- −Many features carry small real effects (dense signal). Ridge usually predicts better.
- −Features come in highly correlated groups and you want the whole group. Lasso picks one somewhat arbitrarily; use elastic net.
- −You need unbiased effect sizes. Surviving weights are shrunk by λ; refit, or use the relaxed lasso.
- −The relationship is strongly non-linear or interaction-heavy. Lasso only selects among the features you give it.
Where it's used
Genomics
Predict a trait or drug response from tens of thousands of gene-expression levels measured on a few hundred patients, and report the short list of genes with non-zero weight.
Compressed sensing & MRI
Reconstruct a signal that is sparse in some basis from far fewer measurements than unknowns. L1 minimization is the core of the recovery.
Econometrics & forecasting
Pick a handful of predictors from hundreds of candidate macro indicators or lags (as in double-selection lasso for causal effect estimation).
Lean production models
Credit scoring, demand forecasting and ad click-through models with thousands of sparse one-hot features, where the L1 penalty keeps the deployed model small and auditable.
Check yourself
During coordinate descent with λ = 6, a feature's partial-residual correlation is ρ = −10.684. Its new weight is…
Check yourself
On the standardized house data the marginal correlations (1/n)zⱼᵀ(y − ȳ) are 37.136, 18.605, −6.893 and 3.005. What is the smallest λ for which lasso sets all four weights to zero?
Check yourself
Bedrooms starts with ρ = 18.6 (three times λ = 6), yet its final lasso weight is exactly 0. Why?
Check yourself
You append an exact duplicate of the size column and refit lasso (λ = 6). What happens?
Remember
- ✓Lasso minimizes . It is convex, has no closed form, and gives sparse solutions.
- ✓One coordinate at a time the solution is soft-thresholding, : any feature whose partial-residual correlation satisfies gets exactly 0.
- ✓At the optimum every active feature has correlation exactly with the residual. All weights are zero for .
- ✓Standardize features first, choose λ by cross-validation along a warm-started path, and remember that the surviving weights are biased toward 0.
- ✓Selection depends on partial correlation, and among correlated features the choice is arbitrary. Use elastic net when groups matter and ridge when the signal is dense.