# Lasso regression

> Add λ‖w‖₁ to the squared error; the corners of the L1 ball push some weights to exactly zero, selecting features.

Source: https://www.mlfoundry.in/learn/lasso-regression  
Section: ML Algorithms › Supervised · Regression  
Level: core · 17 min read · updated 2026-10-07

**Lasso regression:** (*least absolute shrinkage and selection operator*, Tibshirani 1996) is linear regression whose squared-error loss carries an extra penalty $\lambda\sum_j|w_j|$ on the absolute size of the weights. The penalty shrinks every weight toward zero and sets the weakest ones **exactly** to zero, so fitting the model also picks which features to keep.

- **Task:** Regression (numeric target); the same penalty is used for L1-regularized logistic regression
- **Input → output:** feature vector $x\in\mathbb{R}^d$ → $\hat y = b + w^\top x$
- **Learns:** a **sparse** weight vector $w$ (many entries exactly 0) and an unpenalized intercept $b$
- **Loss:** $\tfrac{1}{2n}\lVert y-b-Xw\rVert_2^2+\lambda\lVert w\rVert_1$ — convex, but not differentiable where any $w_j=0$
- **Training:** no closed form; coordinate descent costs $O(nd)$ per sweep, and a whole path of $\lambda$ values costs little more than one fit thanks to warm starts
- **Prediction:** $O(k)$ for $k$ non-zero weights — dropped features need not even be measured
- **Hyperparameters:** $\lambda$ (scikit-learn: "alpha"), chosen by cross-validation; tolerance and max iterations of the solver
- **Assumes:** a roughly linear relationship; features on a common scale (standardize first); the true signal uses a few of the features (sparsity)

## The problem

Ordinary least squares uses **every** feature you give it, and on small or noisy data it happily assigns weight to columns that mean nothing. In the worked example below, 10 house prices are regressed on size, bedrooms, age and the house's *door number*. OLS says each extra standard deviation of door number adds 3.77 k$ to the price, and that more bedrooms *lower* it by 1.77 k$. Both are noise. Its training error is low (MSE 258.9) but its leave-one-out error is 1276.8 — five times worse.

The classical fix is **best-subset selection**: try every subset of features and keep the one with the best validation score. With $d$ features there are $2^d$ subsets — about a million at $d=20$ — and the problem is NP-hard in general. Greedy forward/backward stepwise selection is cheaper but unstable. **Ridge regression** (penalty $\lambda\lVert w\rVert_2^2$, see *ridge-regression*) is cheap and stable but never removes anything: on the same data with scikit-learn's ridge at alpha = 10 the door weight is −0.008, small but not zero. Lasso is the convex middle ground: one smooth optimisation problem that both shrinks and selects.

## The idea

Charge every weight a **fixed price per unit of size**, $\lambda$, regardless of how big it already is. A feature is worth including only if moving its weight away from 0 lowers the squared error by more than $\lambda$ per unit. Features whose correlation with what is left of the target is below that price are not worth a single unit, so their weight stays at exactly 0. That is the difference from ridge, whose price $2\lambda w_j$ per unit vanishes as $w_j\to0$, so every feature can always afford a little weight.

Geometrically (the full picture is in *l1-l2-regularization*): the penalized problem is the same as minimizing squared error inside the diamond $\lVert w\rVert_1\le t$. The elliptical error contours usually first touch the diamond at a **corner**, and the corners lie on the axes, where some coordinates are 0. Ridge's ball is round and has no corners, so the touching point is almost never on an axis.

**Vocabulary**

- **L1 norm:** $\lVert w\rVert_1=\sum_j|w_j|$, the sum of absolute values. Its unit ball is a diamond (a cross-polytope in $d$ dimensions)
- **λ (alpha):** the penalty strength. $\lambda=0$ is OLS; large $\lambda$ gives all-zero weights
- **Sparse:** most weights exactly 0. The model is then a regression on the few features that survive
- **Active set:** the features with $w_j\neq 0$ at a given $\lambda$
- **Partial residual:** $r^{(j)}=y-b-\sum_{k\ne j}x_k w_k$: what is left of the target after every feature *except* $j$ has made its prediction
- **ρⱼ:** $\tfrac1n x_j^\top r^{(j)}$, how strongly feature $j$ correlates with its partial residual; the quantity lasso compares with $\lambda$
- **Soft-thresholding:** $S(\rho,\lambda)=\operatorname{sign}(\rho)\max(|\rho|-\lambda,0)$: move $\rho$ toward 0 by $\lambda$, and stop at 0
- **Regularization path:** the weights $w(\lambda)$ as functions of $\lambda$. For lasso they are piecewise linear in $\lambda$
- **λ_max:** the smallest $\lambda$ at which every weight is 0
- **Subgradient:** the set of slopes of $|w|$ at a kink: $\{\operatorname{sign}(w)\}$ if $w\ne0$, the whole interval $[-1,1]$ at $w=0$

## The math

$$
\min_{b,\,w}\;\; J(w)=\frac{1}{2n}\sum_{i=1}^{n}\Big(y_i-b-\sum_{j=1}^{d}x_{ij}w_j\Big)^2+\lambda\sum_{j=1}^{d}|w_j|
$$

- $n,\ d$: number of training rows and of features
- $x_{ij}$: feature $j$ of row $i$, standardized so that $\tfrac1n\sum_i x_{ij}=0$ and $\tfrac1n\sum_i x_{ij}^2=1$
- $y_i$: target of row i
- $b$: intercept — not penalized; with centred features its optimum is $b=\bar y$
- $w_j$: weight of feature j
- $\lambda\ge 0$: penalty strength (scikit-learn and glmnet call it alpha / lambda with exactly this 1/(2n) scaling)

Because $|w_j|$ has a kink at 0 there is no gradient to set to zero for the whole vector, and so no closed form like OLS's $(X^\top X)^{-1}X^\top y$. But the problem *is* easy one coordinate at a time. Fix every weight except $w_j$. Using $\tfrac1n\sum_i x_{ij}^2=1$, the objective in $w_j$ alone is

$$
J(w_j)=\frac{1}{2}w_j^2-\rho_j\,w_j+\lambda|w_j|+\text{const},\qquad \rho_j=\frac1n\sum_{i=1}^n x_{ij}\,r^{(j)}_i
$$

- $r^{(j)}_i$: partial residual of row $i$: $y_i-b-\sum_{k\neq j}x_{ik}w_k$
- $\rho_j$: the least-squares weight feature j would get if it had to explain the partial residual on its own

This is a parabola plus a V. At the minimum, 0 must lie in the subgradient: $0\in w_j-\rho_j+\lambda\,\partial|w_j|$. Three cases:

**Solving the one-dimensional problem**

1. **$w_j>0$**: $\partial|w_j|=1$, so $w_j=\rho_j-\lambda$. This is consistent only if $\rho_j>\lambda$.
2. **$w_j<0$**: $\partial|w_j|=-1$, so $w_j=\rho_j+\lambda$, consistent only if $\rho_j<-\lambda$.
3. **$w_j=0$**: $\partial|w_j|$ can be anything in $[-1,1]$, so we need $\rho_j\in[-\lambda,\lambda]$, i.e. $|\rho_j|\le\lambda$. This case covers a whole **interval** of $\rho_j$ values, not a single point. That interval is where the exact zeros come from.
4. Together: $w_j=S(\rho_j,\lambda)$, the soft-thresholding operator below.

$$
w_j\leftarrow S(\rho_j,\lambda)=\operatorname{sign}(\rho_j)\,\max\big(|\rho_j|-\lambda,\;0\big)
$$

- $S(\cdot,\lambda)$: soft-threshold: shrink toward 0 by $\lambda$; anything inside $[-\lambda,\lambda]$ becomes exactly 0

Cycling this update over $j=1,\dots,d$ until nothing changes is **coordinate descent**. Each update decreases $J$ (it minimizes $J$ exactly along one axis), $J$ is convex, and its non-smooth part is a sum of one-coordinate terms, so the cycle converges to the global minimum. At that minimum the update changes nothing, which gives the optimality (KKT) conditions with the full residual $r=y-b-Xw$:

$$
\tfrac1n x_j^\top r=\lambda\,\operatorname{sign}(w_j)\ \text{ if } w_j\neq0,\qquad \big|\tfrac1n x_j^\top r\big|\le\lambda\ \text{ if } w_j=0
$$

- $\tfrac1n x_j^\top r$: correlation of feature j with the final residual

Read it as: **every active feature has exactly the same correlation $\lambda$ with the residual** (up to sign), and every dropped feature has less. Some consequences:

**What λ does**

- **λ = 0:** OLS (when it is unique).
- **λ ≥ λ_max:** all weights 0. At $w=0$ the residual is $y-\bar y$, so the condition is $\lambda\ge\lambda_{\max}=\max_j\tfrac1n|x_j^\top(y-\bar y)|$ — the largest marginal correlation.
- **Between:** as $\lambda$ falls from $\lambda_{\max}$, features enter the active set one at a time and the path $w(\lambda)$ is piecewise linear between those knots (the LARS algorithm walks exactly these segments). A feature can also leave again.
- **Orthonormal features:** if $\tfrac1n X^\top X=I$, then $\rho_j$ is just the OLS weight, so $w_j^{\text{lasso}}=S(w_j^{\text{OLS}},\lambda)$ while $w_j^{\text{ridge}}=w_j^{\text{OLS}}/(1+\lambda')$: lasso subtracts, ridge divides.
- **Bias:** surviving weights are pulled toward 0 by $\lambda$, even large, important ones. That is why practitioners often refit OLS on the selected features (the *relaxed lasso*).
- **At most n:** when $d>n$, a lasso solution has at most $n$ non-zero weights.

> **Two scalings of λ:** Textbooks often write $\lVert y-Xw\rVert^2+\lambda\lVert w\rVert_1$ without the $\tfrac{1}{2n}$. That $\lambda$ equals $2n$ times the one used here and in scikit-learn. Always check which form a paper or library uses before copying a value.

## Worked example: which features matter for house prices?

Ten houses with four features. Price was generated from size and age only, plus noise; bedrooms is partly correlated with size (0.61), and door number is pure noise. We fit lasso with $\lambda=6$ and check whether it finds that.

Training data. Mean price $\bar y=244$. Feature means: size 83, beds 2.5, age 18.7, door 7.5; population standard deviations 21.354, 0.806, 10.345, 3.828.

| House | Size (m²) | Beds | Age (yr) | Door no. | Price (k$) |
| --- | --- | --- | --- | --- | --- |
| 1 | 50 | 2 | 30 | 7 | 185 |
| 2 | 60 | 1 | 5 | 3 | 235 |
| 3 | 65 | 3 | 20 | 12 | 185 |
| 4 | 70 | 2 | 10 | 5 | 210 |
| 5 | 80 | 2 | 25 | 9 | 220 |
| 6 | 85 | 3 | 8 | 14 | 285 |
| 7 | 90 | 2 | 15 | 2 | 250 |
| 8 | 100 | 4 | 40 | 8 | 265 |
| 9 | 110 | 3 | 12 | 11 | 305 |
| 10 | 120 | 3 | 22 | 4 | 300 |

Standardized features $z=(x-\text{mean})/\text{sd}$ and centred target $y-\bar y$. Every z column has mean 0 and mean square 1, which is what makes the update simply $w_j=S(\rho_j,\lambda)$.

| House | z size | z beds | z age | z door | y − 244 |
| --- | --- | --- | --- | --- | --- |
| 1 | −1.545 | −0.620 | 1.092 | −0.131 | −59 |
| 2 | −1.077 | −1.861 | −1.324 | −1.176 | −9 |
| 3 | −0.843 | 0.620 | 0.126 | 1.176 | −59 |
| 4 | −0.609 | −0.620 | −0.841 | −0.653 | −34 |
| 5 | −0.140 | −0.620 | 0.609 | 0.392 | −24 |
| 6 | 0.094 | 0.620 | −1.034 | 1.698 | 41 |
| 7 | 0.328 | −0.620 | −0.358 | −1.437 | 6 |
| 8 | 0.796 | 1.861 | 2.059 | 0.131 | 21 |
| 9 | 1.264 | 0.620 | −0.648 | 0.914 | 61 |
| 10 | 1.733 | 0.620 | 0.319 | −0.914 | 56 |

**Coordinate descent by hand, λ = 6, starting from w = 0**

1. **Where the path starts.** At $w=0$ the correlations $\tfrac1n z_j^\top(y-\bar y)$ are size 37.136, beds 18.605, age −6.893, door 3.005. The largest magnitude gives $\lambda_{\max}=37.136$: for any $\lambda$ above it the answer is all zeros. With $\lambda=6$, three of the four exceed the threshold *marginally*, but marginal correlations are not what lasso finally compares.
2. **Sweep 1, size.** Nothing else is in the model, so the partial residual is $y-\bar y$ and $\rho_{\text{size}}=\tfrac1{10}\big((-1.545)(-59)+(-1.077)(-9)+\dots+(1.733)(56)\big)=37.136$. Then $w_{\text{size}}=S(37.136,6)=37.136-6=31.136$.
3. **Sweep 1, beds.** Now subtract size's prediction: $r^{(\text{beds})}=y-\bar y-31.136\,z_{\text{size}}$. Beds' correlation with *this* residual is $\rho_{\text{beds}}=-0.384$. Its marginal correlation 18.605 came almost entirely from being correlated with size. $|{-0.384}|\le6$, so $w_{\text{beds}}=0$.
4. **Sweep 1, age.** $\rho_{\text{age}}=-10.684$ (larger than its marginal −6.893, because the size term has removed variation that was masking it). $w_{\text{age}}=-10.684+6=-4.684$.
5. **Sweep 1, door.** $\rho_{\text{door}}=2.664$, inside the dead zone, so $w_{\text{door}}=0$. After one sweep $w=(31.136,\,0,\,-4.684,\,0)$, objective $J=393.816$.
6. **Sweep 2** recomputes each $\rho$ with the updated weights: size 37.706 → 31.706; beds 1.711 → 0; age −10.754 → −4.754; door 2.657 → 0. $J=393.651$.
7. **Sweep 3** moves the weights by less than 0.01 (size 37.714 → 31.714, age −10.755 → −4.755). Converged: $w=(31.715,\,0,\,-4.755,\,0)$.
8. **Check the KKT conditions.** With the final residual, $\tfrac1n z_j^\top r=(6.000,\ 1.743,\ -6.000,\ 2.657)$. The two active features sit exactly at $\pm\lambda$; the two dropped ones are strictly inside $(-6,6)$. This is the optimum.
9. **Predict** a new house: 95 m², 3 beds, 10 years, door 6. Standardized: $z_{\text{size}}=(95-83)/21.354=0.562$, $z_{\text{age}}=(10-18.7)/10.345=-0.841$ (beds and door do not matter). $\hat y=244+31.715(0.562)-4.755(-0.841)=244+17.82+4.00=265.82$ k$. In original units the model is $\hat y=129.33+1.4852\cdot\text{size}-0.4596\cdot\text{age}$.

Every coordinate update at λ = 6. A weight is exactly 0 whenever |ρ| ≤ 6. Sweep 3 differs from sweep 2 only in the third decimal.

| Sweep | ρ size → w | ρ beds → w | ρ age → w | ρ door → w | Objective J | Train MSE |
| --- | --- | --- | --- | --- | --- | --- |
| 1 | 37.136 → 31.136 | −0.384 → 0 | −10.684 → −4.684 | 2.664 → 0 | 393.816 | 357.8 |
| 2 | 37.706 → 31.706 | 1.711 → 0 | −10.754 → −4.754 | 2.657 → 0 | 393.651 | 349.8 |
| 3 | 37.714 → 31.714 | 1.742 → 0 | −10.755 → −4.755 | 2.657 → 0 | 393.651 | 349.7 |

OLS against lasso (λ = 6) on the same 10 houses, standardized weights. Leave-one-out MSE fits on 9 houses and predicts the 10th, 10 times.

| Model | w size | w beds | w age | w door | Train MSE | LOOCV MSE | ŷ new house |
| --- | --- | --- | --- | --- | --- | --- | --- |
| OLS | 39.480 | −1.768 | −10.963 | 3.766 | 258.9 | 1276.8 | 272.83 |
| **Lasso λ = 6** | **31.715** | **0** | **−4.755** | **0** | 349.7 | **625.0** | 265.82 |

Lasso pays about 91 points of training MSE and gets less than half the out-of-sample error in return. It dropped exactly the two features that had no effect on the price. Scanning $\lambda$ in steps of 0.25, the LOOCV error is lowest at $\lambda=5.5$ (620.7), so $\lambda=6$ is nearly optimal.

> Figure: The 10-house example, live. Drag λ: top, the four standardized weights along the whole path (dots on the zero line mark where each feature enters: size at λmax = 37.14, age 10.18, door 2.77, beds 0.19); middle, leave-one-out error, lowest (620.7) at λ = 5.5 against 1276.8 for OLS at λ = 0; bottom left, each feature plotted at (ρⱼ, wⱼ) on the soft-threshold curve, with the shaded dead zone |ρ| ≤ λ where weights are exactly 0; bottom right, the weights against the OLS outlines. Press play to run coordinate descent from w = 0 one update at a time. At λ = 6, beds and door land in the dead zone in every sweep.

> **Lasso selects on partial, not marginal, correlation:** Bedrooms has a marginal correlation of 0.44 with price and a starting $\rho$ of 18.6, three times $\lambda$, yet lasso drops it. Once size is in the model, bedrooms explains almost nothing extra ($\rho=-0.38$). Lasso asks *does this feature help given the others?* A univariate filter (keep features correlated with $y$) would have kept bedrooms.

## The algorithm, step by step

**LassoCD(X, y, λ)**

1. **Standardize** each column to mean 0 and $\tfrac1n\sum x_{ij}^2=1$; set $b=\bar y$ and $r=y-b$; start at $w=0$ (or a warm start from a nearby $\lambda$).
2. **For each feature $j$** in turn: $\rho_j=\tfrac1n x_j^\top r+w_j$ (adding $w_j$ back turns the full residual into the partial residual).
3. **Soft-threshold:** $w_j^{\text{new}}=S(\rho_j,\lambda)$.
4. **Update the residual** in $O(n)$: $r\leftarrow r-x_j\,(w_j^{\text{new}}-w_j)$; set $w_j\leftarrow w_j^{\text{new}}$.
5. **Repeat sweeps** until the largest change in any weight (or the duality gap) is below the tolerance.
6. **Un-standardize:** $w_j^{\text{raw}}=w_j/\text{sd}_j$, $b^{\text{raw}}=\bar y-\sum_j w_j^{\text{raw}}\,\text{mean}_j$.

**Cost**

- **One sweep:** $O(nd)$, and only $O(n\cdot\text{nnz})$ for sparse $X$. No $d\times d$ matrix is ever formed or inverted.
- **Sweeps needed:** a handful when features are nearly uncorrelated (in the example, 3 sweeps reach 2-decimal accuracy and 7 reach a 10⁻⁸ tolerance); many more when features are strongly correlated, because each coordinate step can only move along one axis of a narrow valley.
- **Whole path:** fit $\lambda_{\max}>\lambda_1>\dots>\lambda_K$ (log-spaced, typically to $10^{-3}\lambda_{\max}$) using each solution as the next warm start. With active-set tricks (only sweep the non-zero features until they converge, then check the rest) this is barely slower than a single fit.
- **LARS-lasso:** computes the exact piecewise-linear path knot by knot, about the cost of one OLS fit, $O(nd^2)$ for $n>d$. Good when $d$ is moderate and you want the whole path.
- **Prediction:** $O(\text{nnz}(w))$ per row.

## Variants and tuning

Hyperparameters of scikit-learn's Lasso / LassoCV and what they do.

| Setting | Meaning | Practical advice |
| --- | --- | --- |
| alpha | λ in the objective above | pick by CV on a log grid from λ_max down to 10⁻³ λ_max (LassoCV does this with n_alphas = 100) |
| Scaling (StandardScaler) | not part of Lasso itself | always standardize in a pipeline: the penalty treats a weight in metres and a weight in millimetres the same |
| fit_intercept | leave the intercept unpenalized | keep True unless the data are already centred |
| max_iter, tol | coordinate descent stopping rule (duality gap) | raise max_iter if you see a ConvergenceWarning, which is common with correlated features and tiny alpha |
| selection | cyclic or random coordinate order | random can converge faster when d is large |
| positive | constrain w ≥ 0 | for quantities that cannot be negative (mixture proportions, prices of components) |
| warm_start | start from the previous solution | speeds up manual λ sweeps |

**Variants**

- **LassoCV / 1-SE rule:** choose $\lambda$ with the lowest CV error, or the *largest* $\lambda$ within one standard error of it. The 1-SE choice gives a sparser, more stable model at almost the same accuracy.
- **LassoLarsIC:** choose λ by AIC or BIC instead of CV. Very fast, but relies on a noise-variance estimate and needs n > d.
- **Relaxed lasso:** use lasso only to choose the active set, then refit OLS (or a lightly penalized fit) on those features to remove the shrinkage bias.
- **Adaptive lasso:** weight the penalty per feature, $\lambda\sum_j|w_j|/|\hat w_j^{\text{init}}|$: features that look strong at first are penalized less. Under some conditions this recovers the correct features where plain lasso does not.
- **Elastic net:** add a ridge term, $\lambda\big(\alpha\lVert w\rVert_1+\tfrac{1-\alpha}{2}\lVert w\rVert_2^2\big)$. It keeps groups of correlated features together; see *elastic-net-regression*.
- **Group / fused lasso:** penalize groups of coefficients together (e.g. all dummies of one categorical variable in or out), or differences between neighbouring coefficients (piecewise-constant signals).

> **Forgetting to standardize changes which features survive:** The penalty $\lambda|w_j|$ is the same price for every feature, but the size of $w_j$ depends on the units of $x_j$. Measure size in cm² instead of m² and its weight shrinks 10 000-fold, so lasso hardly penalizes it at all. Measure it in km² and it gets dropped. Put a StandardScaler (fit on the training fold only) in front of Lasso.

> **With correlated features, the chosen one is arbitrary:** Add an exact copy of the size column to the example and lasso at $\lambda=6$ gives the two copies 31.136 and 0.579. Any split with the same total and sign has the same objective, so the solution is not unique, and which copy gets the weight depends on solver order. With nearly duplicated features a different bootstrap sample can flip the choice. Do not read lasso's selection as *the* causal features; check stability across resamples, or use elastic net to keep the group.

> **Inference after selection:** p-values and confidence intervals from refitting OLS on lasso-selected features are too optimistic, because the same data chose the features. Use a held-out split for inference, or selective-inference methods.

## In code

```python
import numpy as np

def soft_threshold(rho, lam):
    return np.sign(rho) * np.maximum(np.abs(rho) - lam, 0.0)

def lasso_cd(X, y, lam, tol=1e-8, max_sweeps=10_000):
    """Minimize (1/2n)||y - b - Xw||^2 + lam*||w||_1 by cyclic coordinate descent."""
    n, d = X.shape
    mu, sd = X.mean(0), X.std(0)            # standardize: every column has (1/n)sum z^2 = 1
    Z, b = (X - mu) / sd, y.mean()          # the intercept is never penalized
    w, r = np.zeros(d), y - b               # r = current residual y - b - Zw
    for sweep in range(max_sweeps):
        biggest = 0.0
        for j in range(d):
            rho = Z[:, j] @ r / n + w[j]    # = (1/n) z_j . (partial residual without feature j)
            new = soft_threshold(rho, lam)
            r -= Z[:, j] * (new - w[j])     # O(n) residual update
            biggest = max(biggest, abs(new - w[j]))
            w[j] = new
        if biggest < tol:
            break
    coef = w / sd                           # back to original units
    return coef, b - mu @ coef, w, sweep + 1

X = np.array([[50, 2, 30, 7], [60, 1, 5, 3], [65, 3, 20, 12], [70, 2, 10, 5], [80, 2, 25, 9],
              [85, 3, 8, 14], [90, 2, 15, 2], [100, 4, 40, 8], [110, 3, 12, 11], [120, 3, 22, 4]], float)
y = np.array([185, 235, 185, 210, 220, 285, 250, 265, 305, 300], float)

coef, intercept, w_std, sweeps = lasso_cd(X, y, lam=6.0)
print("standardized w:", w_std.round(3), "sweeps:", sweeps)   # [31.715  0. -4.755  0.]  7
print("raw coef:", coef.round(4), "intercept:", round(intercept, 3))  # [1.4852 0. -0.4596 0.]  129.326
print("predict [95, 3, 10, 6]:", round(intercept + np.array([95, 3, 10, 6]) @ coef, 2))  # 265.82
lam_max = np.abs(((X - X.mean(0)) / X.std(0)).T @ (y - y.mean()) / len(y)).max()
print("lambda_max:", round(lam_max, 3))                       # 37.136
```

From scratch: coordinate descent with soft-thresholding and an O(n) residual update. Prints w = [31.715 0 −4.755 0], intercept 129.326, prediction 265.82, λ_max 37.136.

```python
import numpy as np
from sklearn.linear_model import Lasso, LinearRegression, lasso_path
from sklearn.model_selection import LeaveOneOut, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X = np.array([[50, 2, 30, 7], [60, 1, 5, 3], [65, 3, 20, 12], [70, 2, 10, 5], [80, 2, 25, 9],
              [85, 3, 8, 14], [90, 2, 15, 2], [100, 4, 40, 8], [110, 3, 12, 11], [120, 3, 22, 4]], float)
y = np.array([185, 235, 185, 210, 220, 285, 250, 265, 305, 300], float)
names = ["size", "beds", "age", "door"]
new = np.array([[95, 3, 10, 6]])

lasso = make_pipeline(StandardScaler(), Lasso(alpha=6.0)).fit(X, y)
ols = make_pipeline(StandardScaler(), LinearRegression()).fit(X, y)
print("lasso w:", dict(zip(names, lasso[-1].coef_.round(3))))  # size 31.715, beds 0, age -4.755, door 0
print("ols   w:", dict(zip(names, ols[-1].coef_.round(3))))    # 39.48, -1.768, -10.963, 3.766
print("predict new house: lasso %.2f  ols %.2f" % (lasso.predict(new)[0], ols.predict(new)[0]))  # 265.82 272.83

loo = LeaveOneOut()
for name, model in [("ols", ols), ("lasso a=6", lasso), ("lasso a=5.5", make_pipeline(StandardScaler(), Lasso(alpha=5.5)))]:
    cv = -cross_val_score(model, X, y, cv=loo, scoring="neg_mean_squared_error").mean()
    train = np.mean((model.fit(X, y).predict(X) - y) ** 2)
    print(f"{name:12s} train MSE {train:7.1f}   LOOCV MSE {cv:7.1f}")
# ols          train MSE   258.9   LOOCV MSE  1276.8
# lasso a=6    train MSE   349.7   LOOCV MSE   625.0
# lasso a=5.5  train MSE   336.6   LOOCV MSE   620.7

alphas, coefs, _ = lasso_path(StandardScaler().fit_transform(X), y - y.mean(), alphas=np.linspace(40, 0.01, 4000))
for j, n in enumerate(names):
    print(n, "enters at alpha ~", round(alphas[np.argmax(coefs[j] != 0)], 2))
# size 37.13, beds 0.18, age 10.17, door 2.77   (grid approximations of the exact knots)
```

scikit-learn: the same fit inside a scaling pipeline, compared with OLS by leave-one-out CV, and the λ at which each feature enters the path.

> **In practice: LassoCV:** On real data replace the fixed alpha with make_pipeline(StandardScaler(), LassoCV(cv=5)). It builds the λ grid from λ_max, fits the path with warm starts in each fold, and stores the chosen value in alpha_. Use an outer CV loop if you also want an honest estimate of the tuned model's error.

**Strengths**

- You suspect only a few of many features matter and want a model that names them.
- Prediction or measurement is expensive per feature: zero weights mean those inputs never need collecting.
- d is large, even larger than n (genomics, text, sensor banks). Coordinate descent scales and the solution has at most n non-zeros.
- You need a fast, convex, reproducible alternative to stepwise selection.

**Limitations**

- Many features carry small real effects (dense signal). Ridge usually predicts better.
- Features come in highly correlated groups and you want the whole group. Lasso picks one somewhat arbitrarily; use elastic net.
- You need unbiased effect sizes. Surviving weights are shrunk by λ; refit, or use the relaxed lasso.
- The relationship is strongly non-linear or interaction-heavy. Lasso only selects among the features you give it.

- **Genomics:** Predict a trait or drug response from tens of thousands of gene-expression levels measured on a few hundred patients, and report the short list of genes with non-zero weight.
- **Compressed sensing & MRI:** Reconstruct a signal that is sparse in some basis from far fewer measurements than unknowns. L1 minimization is the core of the recovery.
- **Econometrics & forecasting:** Pick a handful of predictors from hundreds of candidate macro indicators or lags (as in double-selection lasso for causal effect estimation).
- **Lean production models:** Credit scoring, demand forecasting and ad click-through models with thousands of sparse one-hot features, where the L1 penalty keeps the deployed model small and auditable.

**Question:** During coordinate descent with λ = 6, a feature's partial-residual correlation is ρ = −10.684. Its new weight is…

- A. −10.684
- B. −4.684
- C. 0
- D. −16.684

**Answer:** B. −4.684 — $S(-10.684,6)=\operatorname{sign}(-10.684)\max(10.684-6,0)=-4.684$. Soft-thresholding moves $\rho$ toward zero by $\lambda$. This is age's update in sweep 1 of the worked example.

**Question:** On the standardized house data the marginal correlations (1/n)zⱼᵀ(y − ȳ) are 37.136, 18.605, −6.893 and 3.005. What is the smallest λ for which lasso sets all four weights to zero?

- A. 6.893
- B. 16.41 (their mean absolute value)
- C. 37.136
- D. 65.639 (their absolute sum)

**Answer:** C. 37.136 — At $w=0$ the KKT condition is $|\tfrac1n z_j^\top(y-\bar y)|\le\lambda$ for every $j$, so $\lambda_{\max}=\max_j|\cdot|=37.136$. Just below it, size, the feature with the largest correlation, enters first.

**Question:** Bedrooms starts with ρ = 18.6 (three times λ = 6), yet its final lasso weight is exactly 0. Why?

- A. Its standard deviation is the smallest of the four features
- B. After size is in the model, its correlation with the partial residual is only about ±1.7, inside the dead zone
- C. Lasso always drops the second feature in the cyclic order
- D. Its OLS coefficient is negative

**Answer:** B. After size is in the model, its correlation with the partial residual is only about ±1.7, inside the dead zone — Lasso compares $\lambda$ with the correlation to the **partial** residual. Bedrooms is correlated with size (0.61), so once size explains its share of the price, bedrooms' $\rho$ drops to −0.384 in sweep 1 and about 1.74 at convergence, both with $|\rho|\le 6$.

**Question:** You append an exact duplicate of the size column and refit lasso (λ = 6). What happens?

- A. Each copy gets 31.715, doubling size's effect
- B. Both copies are dropped because they are collinear
- C. The two weights sum to 31.715, but how they split is arbitrary (here 31.136 and 0.579)
- D. The solver fails because XᵀX is singular

**Answer:** C. The two weights sum to 31.715, but how they split is arbitrary (here 31.136 and 0.579) — Only the total $w_a+w_b$ affects predictions, and $|w_a|+|w_b|$ is the same for every same-sign split, so the objective has a whole segment of minimizers. Coordinate descent needs no matrix inverse, so nothing fails; it just returns one point on that segment. Elastic net's ridge term would split the weight evenly.

**Key points**

- Lasso minimizes $\tfrac{1}{2n}\lVert y-b-Xw\rVert^2+\lambda\lVert w\rVert_1$. It is convex, has no closed form, and gives sparse solutions.
- One coordinate at a time the solution is soft-thresholding, $w_j=S(\rho_j,\lambda)$: any feature whose partial-residual correlation satisfies $|\rho_j|\le\lambda$ gets exactly 0.
- At the optimum every active feature has correlation exactly $\pm\lambda$ with the residual. All weights are zero for $\lambda\ge\lambda_{\max}=\max_j\tfrac1n|x_j^\top(y-\bar y)|$.
- Standardize features first, choose λ by cross-validation along a warm-started path, and remember that the surviving weights are biased toward 0.
- Selection depends on partial correlation, and among correlated features the choice is arbitrary. Use elastic net when groups matter and ridge when the signal is dense.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
