Lasso regression

Add λ‖w‖₁ to the squared error; the corners of the L1 ball push some weights to exactly zero, selecting features.

17 min read1458 wordscore10 viewsupdated 2026-10-07

Definition

Lasso regression (least absolute shrinkage and selection operator, Tibshirani 1996) is linear regression whose squared-error loss carries an extra penalty λ∑j∣wj∣\lambda\sum_j|w_j| on the absolute size of the weights. The penalty shrinks every weight toward zero and sets the weakest ones exactly to zero, so fitting the model also picks which features to keep.

At a glance

Task
Regression (numeric target); the same penalty is used for L1-regularized logistic regression
Input → output
feature vector x∈Rdx\in\mathbb{R}^d → y^=b+w⊤x\hat y = b + w^\top x
Learns
a sparse weight vector ww (many entries exactly 0) and an unpenalized intercept bb
Loss
12n∥y−b−Xw∥22+λ∥w∥1\tfrac{1}{2n}\lVert y-b-Xw\rVert_2^2+\lambda\lVert w\rVert_1 — convex, but not differentiable where any wj=0w_j=0
Training
no closed form; coordinate descent costs O(nd)O(nd) per sweep, and a whole path of λ\lambda values costs little more than one fit thanks to warm starts
Prediction
O(k)O(k) for kk non-zero weights — dropped features need not even be measured
Hyperparameters
λ\lambda (scikit-learn: "alpha"), chosen by cross-validation; tolerance and max iterations of the solver
Assumes
a roughly linear relationship; features on a common scale (standardize first); the true signal uses a few of the features (sparsity)

The problem

Ordinary least squares uses every feature you give it, and on small or noisy data it happily assigns weight to columns that mean nothing. In the worked example below, 10 house prices are regressed on size, bedrooms, age and the house's door number. OLS says each extra standard deviation of door number adds 3.77 ktotheprice,andthatmorebedrooms∗lower∗itby1.77k to the price, and that more bedrooms *lower* it by 1.77 k. Both are noise. Its training error is low (MSE 258.9) but its leave-one-out error is 1276.8 — five times worse.

The classical fix is best-subset selection: try every subset of features and keep the one with the best validation score. With dd features there are 2d2^d subsets — about a million at d=20d=20 — and the problem is NP-hard in general. Greedy forward/backward stepwise selection is cheaper but unstable. Ridge regression (penalty λ∥w∥22\lambda\lVert w\rVert_2^2, see ridge-regression) is cheap and stable but never removes anything: on the same data with scikit-learn's ridge at alpha = 10 the door weight is −0.008, small but not zero. Lasso is the convex middle ground: one smooth optimisation problem that both shrinks and selects.

The idea

Charge every weight a fixed price per unit of size, λ\lambda, regardless of how big it already is. A feature is worth including only if moving its weight away from 0 lowers the squared error by more than λ\lambda per unit. Features whose correlation with what is left of the target is below that price are not worth a single unit, so their weight stays at exactly 0. That is the difference from ridge, whose price 2λwj2\lambda w_j per unit vanishes as wj→0w_j\to0, so every feature can always afford a little weight.

Geometrically (the full picture is in l1-l2-regularization): the penalized problem is the same as minimizing squared error inside the diamond ∥w∥1≤t\lVert w\rVert_1\le t. The elliptical error contours usually first touch the diamond at a corner, and the corners lie on the axes, where some coordinates are 0. Ridge's ball is round and has no corners, so the touching point is almost never on an axis.

Vocabulary

L1 norm
∥w∥1=∑j∣wj∣\lVert w\rVert_1=\sum_j|w_j|, the sum of absolute values. Its unit ball is a diamond (a cross-polytope in dd dimensions)
λ (alpha)
the penalty strength. λ=0\lambda=0 is OLS; large λ\lambda gives all-zero weights
Sparse
most weights exactly 0. The model is then a regression on the few features that survive
Active set
the features with wj≠0w_j\neq 0 at a given λ\lambda
Partial residual
r(j)=y−b−∑k≠jxkwkr^{(j)}=y-b-\sum_{k\ne j}x_k w_k: what is left of the target after every feature except jj has made its prediction
ρⱼ
1nxj⊤r(j)\tfrac1n x_j^\top r^{(j)}, how strongly feature jj correlates with its partial residual; the quantity lasso compares with λ\lambda
Soft-thresholding
S(ρ,λ)=sign⁡(ρ)max⁡(∣ρ∣−λ,0)S(\rho,\lambda)=\operatorname{sign}(\rho)\max(|\rho|-\lambda,0): move ρ\rho toward 0 by λ\lambda, and stop at 0
Regularization path
the weights w(λ)w(\lambda) as functions of λ\lambda. For lasso they are piecewise linear in λ\lambda
λ_max
the smallest λ\lambda at which every weight is 0
Subgradient
the set of slopes of ∣w∣|w| at a kink: {sign⁡(w)}\{\operatorname{sign}(w)\} if w≠0w\ne0, the whole interval [−1,1][-1,1] at w=0w=0

The math

min⁡b, w    J(w)=12n∑i=1n(yi−b−∑j=1dxijwj)2+λ∑j=1d∣wj∣\min_{b,\,w}\;\; J(w)=\frac{1}{2n}\sum_{i=1}^{n}\Big(y_i-b-\sum_{j=1}^{d}x_{ij}w_j\Big)^2+\lambda\sum_{j=1}^{d}|w_j|
n, dn,\ d
number of training rows and of features
xijx_{ij}
feature jj of row ii, standardized so that 1n∑ixij=0\tfrac1n\sum_i x_{ij}=0 and 1n∑ixij2=1\tfrac1n\sum_i x_{ij}^2=1
yiy_i
target of row i
bb
intercept — not penalized; with centred features its optimum is b=yˉb=\bar y
wjw_j
weight of feature j
λ≥0\lambda\ge 0
penalty strength (scikit-learn and glmnet call it alpha / lambda with exactly this 1/(2n) scaling)

Because ∣wj∣|w_j| has a kink at 0 there is no gradient to set to zero for the whole vector, and so no closed form like OLS's (X⊤X)−1X⊤y(X^\top X)^{-1}X^\top y. But the problem is easy one coordinate at a time. Fix every weight except wjw_j. Using 1n∑ixij2=1\tfrac1n\sum_i x_{ij}^2=1, the objective in wjw_j alone is

J(wj)=12wj2−ρj wj+λ∣wj∣+const,ρj=1n∑i=1nxij ri(j)J(w_j)=\frac{1}{2}w_j^2-\rho_j\,w_j+\lambda|w_j|+\text{const},\qquad \rho_j=\frac1n\sum_{i=1}^n x_{ij}\,r^{(j)}_i
ri(j)r^{(j)}_i
partial residual of row ii: yi−b−∑k≠jxikwky_i-b-\sum_{k\neq j}x_{ik}w_k
ρj\rho_j
the least-squares weight feature j would get if it had to explain the partial residual on its own

This is a parabola plus a V. At the minimum, 0 must lie in the subgradient: 0∈wj−ρj+λ ∂∣wj∣0\in w_j-\rho_j+\lambda\,\partial|w_j|. Three cases:

Solving the one-dimensional problem

  1. 1wj>0w_j>0: ∂∣wj∣=1\partial|w_j|=1, so wj=ρj−λw_j=\rho_j-\lambda. This is consistent only if ρj>λ\rho_j>\lambda.
  2. 2wj<0w_j<0: ∂∣wj∣=−1\partial|w_j|=-1, so wj=ρj+λw_j=\rho_j+\lambda, consistent only if ρj<−λ\rho_j<-\lambda.
  3. 3wj=0w_j=0: ∂∣wj∣\partial|w_j| can be anything in [−1,1][-1,1], so we need ρj∈[−λ,λ]\rho_j\in[-\lambda,\lambda], i.e. ∣ρj∣≤λ|\rho_j|\le\lambda. This case covers a whole interval of ρj\rho_j values, not a single point. That interval is where the exact zeros come from.
  4. 4Together: wj=S(ρj,λ)w_j=S(\rho_j,\lambda), the soft-thresholding operator below.
wj←S(ρj,λ)=sign⁡(ρj) max⁡(∣ρj∣−λ,  0)w_j\leftarrow S(\rho_j,\lambda)=\operatorname{sign}(\rho_j)\,\max\big(|\rho_j|-\lambda,\;0\big)
S(⋅,λ)S(\cdot,\lambda)
soft-threshold: shrink toward 0 by λ\lambda; anything inside [−λ,λ][-\lambda,\lambda] becomes exactly 0

Cycling this update over j=1,…,dj=1,\dots,d until nothing changes is coordinate descent. Each update decreases JJ (it minimizes JJ exactly along one axis), JJ is convex, and its non-smooth part is a sum of one-coordinate terms, so the cycle converges to the global minimum. At that minimum the update changes nothing, which gives the optimality (KKT) conditions with the full residual r=y−b−Xwr=y-b-Xw:

1nxj⊤r=λ sign⁡(wj)  if wj≠0,∣1nxj⊤r∣≤λ  if wj=0\tfrac1n x_j^\top r=\lambda\,\operatorname{sign}(w_j)\ \text{ if } w_j\neq0,\qquad \big|\tfrac1n x_j^\top r\big|\le\lambda\ \text{ if } w_j=0
1nxj⊤r\tfrac1n x_j^\top r
correlation of feature j with the final residual

Read it as: every active feature has exactly the same correlation λ\lambda with the residual (up to sign), and every dropped feature has less. Some consequences:

What λ does

λ = 0
OLS (when it is unique).
λ ≥ λ_max
all weights 0. At w=0w=0 the residual is y−yˉy-\bar y, so the condition is λ≥λmax⁡=max⁡j1n∣xj⊤(y−yˉ)∣\lambda\ge\lambda_{\max}=\max_j\tfrac1n|x_j^\top(y-\bar y)| — the largest marginal correlation.
Between
as λ\lambda falls from λmax⁡\lambda_{\max}, features enter the active set one at a time and the path w(λ)w(\lambda) is piecewise linear between those knots (the LARS algorithm walks exactly these segments). A feature can also leave again.
Orthonormal features
if 1nX⊤X=I\tfrac1n X^\top X=I, then ρj\rho_j is just the OLS weight, so wjlasso=S(wjOLS,λ)w_j^{\text{lasso}}=S(w_j^{\text{OLS}},\lambda) while wjridge=wjOLS/(1+λ′)w_j^{\text{ridge}}=w_j^{\text{OLS}}/(1+\lambda'): lasso subtracts, ridge divides.
Bias
surviving weights are pulled toward 0 by λ\lambda, even large, important ones. That is why practitioners often refit OLS on the selected features (the relaxed lasso).
At most n
when d>nd>n, a lasso solution has at most nn non-zero weights.

Worked example: which features matter for house prices?

Ten houses with four features. Price was generated from size and age only, plus noise; bedrooms is partly correlated with size (0.61), and door number is pure noise. We fit lasso with λ=6\lambda=6 and check whether it finds that.

HouseSize (m²)BedsAge (yr)Door no.Price (k$)
1502307185
260153235
36532012185
4702105210
5802259220
6853814285
7902152250
81004408265
911031211305
101203224300
Training data. Mean price yˉ=244\bar y=244. Feature means: size 83, beds 2.5, age 18.7, door 7.5; population standard deviations 21.354, 0.806, 10.345, 3.828.
Housez sizez bedsz agez doory − 244
1−1.545−0.6201.092−0.131−59
2−1.077−1.861−1.324−1.176−9
3−0.8430.6200.1261.176−59
4−0.609−0.620−0.841−0.653−34
5−0.140−0.6200.6090.392−24
60.0940.620−1.0341.69841
70.328−0.620−0.358−1.4376
80.7961.8612.0590.13121
91.2640.620−0.6480.91461
101.7330.6200.319−0.91456
Standardized features z=(x−mean)/sdz=(x-\text{mean})/\text{sd} and centred target y−yˉy-\bar y. Every z column has mean 0 and mean square 1, which is what makes the update simply wj=S(ρj,λ)w_j=S(\rho_j,\lambda).

Coordinate descent by hand, λ = 6, starting from w = 0

  1. 1Where the path starts. At w=0w=0 the correlations 1nzj⊤(y−yˉ)\tfrac1n z_j^\top(y-\bar y) are size 37.136, beds 18.605, age −6.893, door 3.005. The largest magnitude gives λmax⁡=37.136\lambda_{\max}=37.136: for any λ\lambda above it the answer is all zeros. With λ=6\lambda=6, three of the four exceed the threshold marginally, but marginal correlations are not what lasso finally compares.
  2. 2Sweep 1, size. Nothing else is in the model, so the partial residual is y−yˉy-\bar y and ρsize=110((−1.545)(−59)+(−1.077)(−9)+⋯+(1.733)(56))=37.136\rho_{\text{size}}=\tfrac1{10}\big((-1.545)(-59)+(-1.077)(-9)+\dots+(1.733)(56)\big)=37.136. Then wsize=S(37.136,6)=37.136−6=31.136w_{\text{size}}=S(37.136,6)=37.136-6=31.136.
  3. 3Sweep 1, beds. Now subtract size's prediction: r(beds)=y−yˉ−31.136 zsizer^{(\text{beds})}=y-\bar y-31.136\,z_{\text{size}}. Beds' correlation with this residual is ρbeds=−0.384\rho_{\text{beds}}=-0.384. Its marginal correlation 18.605 came almost entirely from being correlated with size. ∣−0.384∣≤6|{-0.384}|\le6, so wbeds=0w_{\text{beds}}=0.
  4. 4Sweep 1, age. ρage=−10.684\rho_{\text{age}}=-10.684 (larger than its marginal −6.893, because the size term has removed variation that was masking it). wage=−10.684+6=−4.684w_{\text{age}}=-10.684+6=-4.684.
  5. 5Sweep 1, door. ρdoor=2.664\rho_{\text{door}}=2.664, inside the dead zone, so wdoor=0w_{\text{door}}=0. After one sweep w=(31.136, 0, −4.684, 0)w=(31.136,\,0,\,-4.684,\,0), objective J=393.816J=393.816.
  6. 6Sweep 2 recomputes each ρ\rho with the updated weights: size 37.706 → 31.706; beds 1.711 → 0; age −10.754 → −4.754; door 2.657 → 0. J=393.651J=393.651.
  7. 7Sweep 3 moves the weights by less than 0.01 (size 37.714 → 31.714, age −10.755 → −4.755). Converged: w=(31.715, 0, −4.755, 0)w=(31.715,\,0,\,-4.755,\,0).
  8. 8Check the KKT conditions. With the final residual, 1nzj⊤r=(6.000, 1.743, −6.000, 2.657)\tfrac1n z_j^\top r=(6.000,\ 1.743,\ -6.000,\ 2.657). The two active features sit exactly at ±λ\pm\lambda; the two dropped ones are strictly inside (−6,6)(-6,6). This is the optimum.
  9. 9Predict a new house: 95 m², 3 beds, 10 years, door 6. Standardized: zsize=(95−83)/21.354=0.562z_{\text{size}}=(95-83)/21.354=0.562, zage=(10−18.7)/10.345=−0.841z_{\text{age}}=(10-18.7)/10.345=-0.841 (beds and door do not matter). y^=244+31.715(0.562)−4.755(−0.841)=244+17.82+4.00=265.82\hat y=244+31.715(0.562)-4.755(-0.841)=244+17.82+4.00=265.82 k.Inoriginalunitsthemodelis. In original units the model is \hat y=129.33+1.4852\cdot\text{size}-0.4596\cdot\text{age}$.
Sweepρ size → wρ beds → wρ age → wρ door → wObjective JTrain MSE
137.136 → 31.136−0.384 → 0−10.684 → −4.6842.664 → 0393.816357.8
237.706 → 31.7061.711 → 0−10.754 → −4.7542.657 → 0393.651349.8
337.714 → 31.7141.742 → 0−10.755 → −4.7552.657 → 0393.651349.7
Every coordinate update at λ = 6. A weight is exactly 0 whenever |ρ| ≤ 6. Sweep 3 differs from sweep 2 only in the third decimal.
Modelw sizew bedsw agew doorTrain MSELOOCV MSEŷ new house
OLS39.480−1.768−10.9633.766258.91276.8272.83
Lasso λ = 631.7150−4.7550349.7625.0265.82
OLS against lasso (λ = 6) on the same 10 houses, standardized weights. Leave-one-out MSE fits on 9 houses and predicts the 10th, 10 times.

Lasso pays about 91 points of training MSE and gets less than half the out-of-sample error in return. It dropped exactly the two features that had no effect on the price. Scanning λ\lambda in steps of 0.25, the LOOCV error is lowest at λ=5.5\lambda=5.5 (620.7), so λ=6\lambda=6 is nearly optimal.

coefficient path w(λ) · size beds age door -10010203040λmax 37.1410.18leave-one-out CV error (MSE) · min 621 at λ = 5.50 · OLS (λ=0) 12770510152025303540λ →λ = 6.00w = S(ρ, λ): |ρ| ≤ λ ⇒ w = 0ρwdead zone ±6.00sizebedsagedoorw now (filled) vs OLS (outline)31.7size0beds-4.8age0door
λ = 6.00
non-zero 2/4‖w‖₁ 36.47train MSE 349.7objective 393.65LOOCV 625.0
coordinate descent from w = 0:
converged at λ = 6.00: w = [31.71, 0, -4.75, 0] · each dot is a feature at (ρⱼ, wⱼ)
fig.The 10-house example, live. Drag λ: top, the four standardized weights along the whole path (dots on the zero line mark where each feature enters: size at λmax = 37.14, age 10.18, door 2.77, beds 0.19); middle, leave-one-out error, lowest (620.7) at λ = 5.5 against 1276.8 for OLS at λ = 0; bottom left, each feature plotted at (ρⱼ, wⱼ) on the soft-threshold curve, with the shaded dead zone |ρ| ≤ λ where weights are exactly 0; bottom right, the weights against the OLS outlines. Press play to run coordinate descent from w = 0 one update at a time. At λ = 6, beds and door land in the dead zone in every sweep.

The algorithm, step by step

LassoCD(X, y, λ)

  1. 1Standardize each column to mean 0 and 1n∑xij2=1\tfrac1n\sum x_{ij}^2=1; set b=yˉb=\bar y and r=y−br=y-b; start at w=0w=0 (or a warm start from a nearby λ\lambda).
  2. 2For each feature jj in turn: ρj=1nxj⊤r+wj\rho_j=\tfrac1n x_j^\top r+w_j (adding wjw_j back turns the full residual into the partial residual).
  3. 3Soft-threshold: wjnew=S(ρj,λ)w_j^{\text{new}}=S(\rho_j,\lambda).
  4. 4Update the residual in O(n)O(n): r←r−xj (wjnew−wj)r\leftarrow r-x_j\,(w_j^{\text{new}}-w_j); set wj←wjneww_j\leftarrow w_j^{\text{new}}.
  5. 5Repeat sweeps until the largest change in any weight (or the duality gap) is below the tolerance.
  6. 6Un-standardize: wjraw=wj/sdjw_j^{\text{raw}}=w_j/\text{sd}_j, braw=yˉ−∑jwjraw meanjb^{\text{raw}}=\bar y-\sum_j w_j^{\text{raw}}\,\text{mean}_j.

Cost

One sweep
O(nd)O(nd), and only O(n⋅nnz)O(n\cdot\text{nnz}) for sparse XX. No d×dd\times d matrix is ever formed or inverted.
Sweeps needed
a handful when features are nearly uncorrelated (in the example, 3 sweeps reach 2-decimal accuracy and 7 reach a 10⁻⁸ tolerance); many more when features are strongly correlated, because each coordinate step can only move along one axis of a narrow valley.
Whole path
fit λmax⁡>λ1>⋯>λK\lambda_{\max}>\lambda_1>\dots>\lambda_K (log-spaced, typically to 10−3λmax⁡10^{-3}\lambda_{\max}) using each solution as the next warm start. With active-set tricks (only sweep the non-zero features until they converge, then check the rest) this is barely slower than a single fit.
LARS-lasso
computes the exact piecewise-linear path knot by knot, about the cost of one OLS fit, O(nd2)O(nd^2) for n>dn>d. Good when dd is moderate and you want the whole path.
Prediction
O(nnz(w))O(\text{nnz}(w)) per row.

Variants and tuning

SettingMeaningPractical advice
alphaλ in the objective abovepick by CV on a log grid from λ_max down to 10⁻³ λ_max (LassoCV does this with n_alphas = 100)
Scaling (StandardScaler)not part of Lasso itselfalways standardize in a pipeline: the penalty treats a weight in metres and a weight in millimetres the same
fit_interceptleave the intercept unpenalizedkeep True unless the data are already centred
max_iter, tolcoordinate descent stopping rule (duality gap)raise max_iter if you see a ConvergenceWarning, which is common with correlated features and tiny alpha
selectioncyclic or random coordinate orderrandom can converge faster when d is large
positiveconstrain w ≥ 0for quantities that cannot be negative (mixture proportions, prices of components)
warm_startstart from the previous solutionspeeds up manual λ sweeps
Hyperparameters of scikit-learn's Lasso / LassoCV and what they do.

Variants

LassoCV / 1-SE rule
choose λ\lambda with the lowest CV error, or the largest λ\lambda within one standard error of it. The 1-SE choice gives a sparser, more stable model at almost the same accuracy.
LassoLarsIC
choose λ by AIC or BIC instead of CV. Very fast, but relies on a noise-variance estimate and needs n > d.
Relaxed lasso
use lasso only to choose the active set, then refit OLS (or a lightly penalized fit) on those features to remove the shrinkage bias.
Adaptive lasso
weight the penalty per feature, λ∑j∣wj∣/∣w^jinit∣\lambda\sum_j|w_j|/|\hat w_j^{\text{init}}|: features that look strong at first are penalized less. Under some conditions this recovers the correct features where plain lasso does not.
Elastic net
add a ridge term, λ(α∥w∥1+1−α2∥w∥22)\lambda\big(\alpha\lVert w\rVert_1+\tfrac{1-\alpha}{2}\lVert w\rVert_2^2\big). It keeps groups of correlated features together; see elastic-net-regression.
Group / fused lasso
penalize groups of coefficients together (e.g. all dummies of one categorical variable in or out), or differences between neighbouring coefficients (piecewise-constant signals).

In code

python
1import numpy as np
2
3def soft_threshold(rho, lam):
4 return np.sign(rho) * np.maximum(np.abs(rho) - lam, 0.0)
5
6def lasso_cd(X, y, lam, tol=1e-8, max_sweeps=10_000):
7 """Minimize (1/2n)||y - b - Xw||^2 + lam*||w||_1 by cyclic coordinate descent."""
8 n, d = X.shape
9 mu, sd = X.mean(0), X.std(0) # standardize: every column has (1/n)sum z^2 = 1
10 Z, b = (X - mu) / sd, y.mean() # the intercept is never penalized
11 w, r = np.zeros(d), y - b # r = current residual y - b - Zw
12 for sweep in range(max_sweeps):
13 biggest = 0.0
14 for j in range(d):
15 rho = Z[:, j] @ r / n + w[j] # = (1/n) z_j . (partial residual without feature j)
16 new = soft_threshold(rho, lam)
17 r -= Z[:, j] * (new - w[j]) # O(n) residual update
18 biggest = max(biggest, abs(new - w[j]))
19 w[j] = new
20 if biggest < tol:
21 break
22 coef = w / sd # back to original units
23 return coef, b - mu @ coef, w, sweep + 1
24
25X = np.array([[50, 2, 30, 7], [60, 1, 5, 3], [65, 3, 20, 12], [70, 2, 10, 5], [80, 2, 25, 9],
26 [85, 3, 8, 14], [90, 2, 15, 2], [100, 4, 40, 8], [110, 3, 12, 11], [120, 3, 22, 4]], float)
27y = np.array([185, 235, 185, 210, 220, 285, 250, 265, 305, 300], float)
28
29coef, intercept, w_std, sweeps = lasso_cd(X, y, lam=6.0)
30print("standardized w:", w_std.round(3), "sweeps:", sweeps) # [31.715 0. -4.755 0.] 7
31print("raw coef:", coef.round(4), "intercept:", round(intercept, 3)) # [1.4852 0. -0.4596 0.] 129.326
32print("predict [95, 3, 10, 6]:", round(intercept + np.array([95, 3, 10, 6]) @ coef, 2)) # 265.82
33lam_max = np.abs(((X - X.mean(0)) / X.std(0)).T @ (y - y.mean()) / len(y)).max()
34print("lambda_max:", round(lam_max, 3)) # 37.136
From scratch: coordinate descent with soft-thresholding and an O(n) residual update. Prints w = [31.715 0 −4.755 0], intercept 129.326, prediction 265.82, λ_max 37.136.
python
1import numpy as np
2from sklearn.linear_model import Lasso, LinearRegression, lasso_path
3from sklearn.model_selection import LeaveOneOut, cross_val_score
4from sklearn.pipeline import make_pipeline
5from sklearn.preprocessing import StandardScaler
6
7X = np.array([[50, 2, 30, 7], [60, 1, 5, 3], [65, 3, 20, 12], [70, 2, 10, 5], [80, 2, 25, 9],
8 [85, 3, 8, 14], [90, 2, 15, 2], [100, 4, 40, 8], [110, 3, 12, 11], [120, 3, 22, 4]], float)
9y = np.array([185, 235, 185, 210, 220, 285, 250, 265, 305, 300], float)
10names = ["size", "beds", "age", "door"]
11new = np.array([[95, 3, 10, 6]])
12
13lasso = make_pipeline(StandardScaler(), Lasso(alpha=6.0)).fit(X, y)
14ols = make_pipeline(StandardScaler(), LinearRegression()).fit(X, y)
15print("lasso w:", dict(zip(names, lasso[-1].coef_.round(3)))) # size 31.715, beds 0, age -4.755, door 0
16print("ols w:", dict(zip(names, ols[-1].coef_.round(3)))) # 39.48, -1.768, -10.963, 3.766
17print("predict new house: lasso %.2f ols %.2f" % (lasso.predict(new)[0], ols.predict(new)[0])) # 265.82 272.83
18
19loo = LeaveOneOut()
20for name, model in [("ols", ols), ("lasso a=6", lasso), ("lasso a=5.5", make_pipeline(StandardScaler(), Lasso(alpha=5.5)))]:
21 cv = -cross_val_score(model, X, y, cv=loo, scoring="neg_mean_squared_error").mean()
22 train = np.mean((model.fit(X, y).predict(X) - y) ** 2)
23 print(f"{name:12s} train MSE {train:7.1f} LOOCV MSE {cv:7.1f}")
24# ols train MSE 258.9 LOOCV MSE 1276.8
25# lasso a=6 train MSE 349.7 LOOCV MSE 625.0
26# lasso a=5.5 train MSE 336.6 LOOCV MSE 620.7
27
28alphas, coefs, _ = lasso_path(StandardScaler().fit_transform(X), y - y.mean(), alphas=np.linspace(40, 0.01, 4000))
29for j, n in enumerate(names):
30 print(n, "enters at alpha ~", round(alphas[np.argmax(coefs[j] != 0)], 2))
31# size 37.13, beds 0.18, age 10.17, door 2.77 (grid approximations of the exact knots)
scikit-learn: the same fit inside a scaling pipeline, compared with OLS by leave-one-out CV, and the λ at which each feature enters the path.

Use when

  • +You suspect only a few of many features matter and want a model that names them.
  • +Prediction or measurement is expensive per feature: zero weights mean those inputs never need collecting.
  • +d is large, even larger than n (genomics, text, sensor banks). Coordinate descent scales and the solution has at most n non-zeros.
  • +You need a fast, convex, reproducible alternative to stepwise selection.

Avoid when

  • −Many features carry small real effects (dense signal). Ridge usually predicts better.
  • −Features come in highly correlated groups and you want the whole group. Lasso picks one somewhat arbitrarily; use elastic net.
  • −You need unbiased effect sizes. Surviving weights are shrunk by λ; refit, or use the relaxed lasso.
  • −The relationship is strongly non-linear or interaction-heavy. Lasso only selects among the features you give it.

Where it's used

Genomics

Predict a trait or drug response from tens of thousands of gene-expression levels measured on a few hundred patients, and report the short list of genes with non-zero weight.

Compressed sensing & MRI

Reconstruct a signal that is sparse in some basis from far fewer measurements than unknowns. L1 minimization is the core of the recovery.

Econometrics & forecasting

Pick a handful of predictors from hundreds of candidate macro indicators or lags (as in double-selection lasso for causal effect estimation).

Lean production models

Credit scoring, demand forecasting and ad click-through models with thousands of sparse one-hot features, where the L1 penalty keeps the deployed model small and auditable.

Check yourself

During coordinate descent with λ = 6, a feature's partial-residual correlation is ρ = −10.684. Its new weight is…

Check yourself

On the standardized house data the marginal correlations (1/n)zⱼᵀ(y − ȳ) are 37.136, 18.605, −6.893 and 3.005. What is the smallest λ for which lasso sets all four weights to zero?

Check yourself

Bedrooms starts with ρ = 18.6 (three times λ = 6), yet its final lasso weight is exactly 0. Why?

Check yourself

You append an exact duplicate of the size column and refit lasso (λ = 6). What happens?

Remember

  • ✓Lasso minimizes 12n∥y−b−Xw∥2+λ∥w∥1\tfrac{1}{2n}\lVert y-b-Xw\rVert^2+\lambda\lVert w\rVert_1. It is convex, has no closed form, and gives sparse solutions.
  • ✓One coordinate at a time the solution is soft-thresholding, wj=S(ρj,λ)w_j=S(\rho_j,\lambda): any feature whose partial-residual correlation satisfies ∣ρj∣≤λ|\rho_j|\le\lambda gets exactly 0.
  • ✓At the optimum every active feature has correlation exactly ±λ\pm\lambda with the residual. All weights are zero for λ≥λmax⁡=max⁡j1n∣xj⊤(y−yˉ)∣\lambda\ge\lambda_{\max}=\max_j\tfrac1n|x_j^\top(y-\bar y)|.
  • ✓Standardize features first, choose λ by cross-validation along a warm-started path, and remember that the surviving weights are biased toward 0.
  • ✓Selection depends on partial correlation, and among correlated features the choice is arbitrary. Use elastic net when groups matter and ridge when the signal is dense.
← back to ML Algorithms roadmap