# Elastic Net regression

> Mix the L1 and L2 penalties to get Lasso's sparsity while keeping groups of correlated features together.

Source: https://www.mlfoundry.in/learn/elastic-net-regression  
Section: ML Algorithms › Supervised · Regression  
Level: core · 17 min read · updated 2026-10-07

**Elastic Net regression:** is linear least squares with **two** penalties added at once: an L1 term $\lambda_1\|w\|_1$, which can set weights to exactly zero, and an L2 term $\tfrac{\lambda_2}{2}\|w\|_2^2$, which shrinks weights smoothly and makes strongly correlated features share their weight. Set $\lambda_2=0$ and you get Lasso. Set $\lambda_1=0$ and you get Ridge.

- **Task:** Regression (the same penalty also works with logistic loss for classification)
- **Input → output:** feature vector $x\in\mathbb R^p$ → a real number $\hat y=b+w^\top x$
- **Learns:** a weight vector $w$, usually **sparse** with correlated features **grouped**, plus an unpenalized intercept $b$
- **Training:** coordinate descent: $O(np)$ per sweep over all features; a whole path of 100 $\alpha$ values with warm starts costs only a few times one fit
- **Prediction:** $O(p)$, or $O(\#\text{nonzero})$ for a sparse $w$
- **Hyperparameters:** `alpha` (overall penalty strength) and `l1_ratio` ρ (the L1 share of the penalty)
- **Assumes:** a roughly linear relationship, and **standardized features**, since the penalty compares the sizes of the weights directly

## The problem

**Lasso** (see the *Lasso regression* article) does feature selection: its L1 penalty drives unhelpful weights to exactly zero. But it handles **correlated features** badly. When two features carry nearly the same information, such as a house's floor area and its number of rooms, Lasso usually keeps one and zeroes the other. *Which one* it keeps depends on tiny details of the sample, so refitting on slightly different data can swap the choice. Lasso has two more limits: when $p>n$ it can select at most $n$ features, and when predictors are highly correlated its accuracy is often worse than Ridge's.

**Ridge** (see *Ridge regression*) has the opposite profile. It is stable under correlation and spreads weight evenly over correlated features, but it never sets a weight to zero, so it cannot select features. In genomics, text, and sensor data you often want both: a short list of features, where a whole correlated *group* gets in or stays out together. Elastic Net (Zou & Hastie, 2005) provides this.

## The idea

Use both penalties at once. The L1 part keeps Lasso's **corners**, so weights can land exactly on zero. The L2 part makes the penalty **strictly convex**: among many nearly equivalent ways to split weight between two near-duplicate features, it strictly prefers the even split. The resulting penalty ball is a diamond with rounded edges. Its corners still sit on the axes, which gives sparsity. Its edges bulge outward, so the loss contours can touch it at a point where both correlated weights are nonzero.

**Vocabulary**

- **α (alpha):** overall regularization strength. $\alpha=0$ is ordinary least squares; large $\alpha$ drives every weight to 0
- **ρ (l1_ratio):** how the penalty is split. $\rho=1$ is pure Lasso, $\rho=0$ is pure Ridge, and values in between are Elastic Net
- **λ₁, λ₂:** the actual L1 and L2 strengths: $\lambda_1=\alpha\rho$, $\lambda_2=\alpha(1-\rho)$
- **Grouping effect:** highly correlated features receive nearly equal weights and enter or leave the model together
- **Soft-thresholding:** $S(v,t)=\operatorname{sign}(v)\max(|v|-t,0)$: move $v$ toward 0 by $t$, and clip it to 0 if it would cross. This is what produces exact zeros
- **Active set:** the features whose weight is currently nonzero
- **Regularization path:** the weights as a function of $\alpha$, from all-zero (huge $\alpha$) to the least-squares fit ($\alpha\to0$)
- **Naive vs rescaled:** the plain solution below is the *naive* Elastic Net. Zou & Hastie multiply it by $(1+\lambda_2)$ to undo the extra shrinkage from the L2 term

## The math

We use scikit-learn's parameterization, so the numbers here match `ElasticNet(alpha, l1_ratio)` exactly. The columns of $X$ are standardized, $y$ is centred, and the intercept is fitted separately and never penalized.

$$
\min_{w}\;\; \frac{1}{2n}\|y-Xw\|_2^2\;+\;\alpha\rho\,\|w\|_1\;+\;\frac{\alpha(1-\rho)}{2}\,\|w\|_2^2
$$

- $X$: $n\times p$ matrix of standardized features (each column has mean 0 and variance 1)
- $y$: centred targets; the intercept is $b=\bar y$
- $w$: weights, one per feature
- $\|w\|_1$: $\sum_j|w_j|$, the L1 norm, which produces zeros
- $\|w\|_2^2$: $\sum_j w_j^2$, the squared L2 norm, which shrinks weights and groups correlated features
- $\alpha,\ \rho$: overall strength and L1 share; $\lambda_1=\alpha\rho$, $\lambda_2=\alpha(1-\rho)$

**Deriving the coordinate update.** Fix every weight except $w_j$. Write $r_{jk}=\tfrac1n x_j^\top x_k$ for the correlation between features $j$ and $k$; standardization makes $r_{jj}=1$. Write $c_j=\tfrac1n x_j^\top y$ for each feature's correlation with the target. As a function of $w_j$ alone, the objective is $\tfrac12 w_j^2 - z_j w_j + \lambda_1|w_j| + \tfrac{\lambda_2}{2}w_j^2 + \text{const}$, where $z_j$ is the *partial-residual correlation*: how strongly feature $j$ correlates with what the other features have not yet explained.

$$
z_j \;=\; c_j-\sum_{k\neq j} r_{jk}\,w_k \;=\; \tfrac1n\,x_j^\top\Big(y-\sum_{k\ne j}x_k w_k\Big)
$$

- $c_j$: $\tfrac1n x_j^\top y$, the correlation of feature $j$ with the target
- $r_{jk}$: $\tfrac1n x_j^\top x_k$, the correlation between features $j$ and $k$

For $w_j\neq0$, setting the derivative to zero gives $(1+\lambda_2)w_j = z_j-\lambda_1\operatorname{sign}(w_j)$. At $w_j=0$ the subgradient of $|w_j|$ can be anything in $[-1,1]$, so zero is optimal whenever $|z_j|\le\lambda_1$. Putting both cases together gives the update:

$$
w_j \;\leftarrow\; \frac{S(z_j,\ \lambda_1)}{1+\lambda_2},\qquad S(v,t)=\operatorname{sign}(v)\max(|v|-t,\,0)
$$

- $S(\cdot,\lambda_1)$: soft-thresholding, the L1 part: subtract $\lambda_1$, and return exactly 0 if $|z_j|\le\lambda_1$
- $1/(1+\lambda_2)$: proportional shrinkage, the L2 part: divide every surviving weight by the same factor

This one line shows each hyperparameter's effect. **λ₁ sets a threshold**: a feature whose partial correlation is at most $\lambda_1$ is switched off. **λ₂ sets a divisor**: every active weight is divided by $1+\lambda_2$. Ridge's update is $z_j/(1+\lambda_2)$, which has a divisor but no threshold. Lasso's is $S(z_j,\lambda_1)$, which has a threshold but no divisor. Raising α at a fixed ρ raises both. Raising ρ at a fixed α moves strength from the divisor to the threshold, which gives more zeros and less proportional shrinkage.

## Why correlated features get grouped

Suppose features 1 and 2 are both active with positive weights, and their correlation is $r$. Their two optimality conditions are $(1+\lambda_2)w_1 + r\,w_2 = c_1-\lambda_1-\ldots$ and $r\,w_1+(1+\lambda_2)w_2 = c_2-\lambda_1-\ldots$, where "…" is the same small contribution from any other features. Subtract the second from the first:

$$
w_1-w_2 \;=\; \frac{c_1-c_2}{\,1+\lambda_2-r\,}
$$

- $c_1-c_2$: how differently the two features correlate with the target, which is small for near-duplicates
- $1+\lambda_2-r$: for Lasso this is $1-r$, which tends to 0 as $r\to1$; Elastic Net keeps it at least $\lambda_2$

For **Lasso** ($\lambda_2=0$) the denominator $1-r$ goes to zero as the features become duplicates. A tiny difference $c_1-c_2$ then becomes a large weight difference. In practice one weight takes everything and the other hits zero, and at exactly $r=1$ the split is undetermined. For **Elastic Net** the denominator never drops below $\lambda_2$, so the weights stay close together. Zou and Hastie state the general version as a bound: for any two features $i,j$ in the active set, $|w_i-w_j|\le \sqrt{2(1-r_{ij})}\;\text{RMSE}/\lambda_2$, where RMSE is the root-mean-square residual of the fit. Perfectly correlated features ($r=1$) get **identical** weights.

> **Elastic Net is a Lasso on augmented data:** Append $p$ fake rows to $X$, namely $\sqrt{n\lambda_2}\,I$, each with target 0. On those rows the squared error adds exactly $\tfrac{\lambda_2}{2}\|w\|^2$, so a plain Lasso on the augmented data *is* the Elastic Net. Two facts follow. The augmented matrix has rank $p$ whatever $n$ is, so Elastic Net can select **more than $n$ features** when $p\gg n$, which Lasso cannot. And any Lasso solver, including LARS, also solves Elastic Net. We checked this on the worked example below: the augmented Lasso returns (22.04, 19.20, 0), the same as the Elastic Net.

## Worked example: size or rooms?

Eight houses. Floor area and number of rooms are almost the same information (correlation $r=0.976$). Days on market is a feature with nothing to say about price. We fit Elastic Net with $\alpha=4$ and $\rho=0.9$, so $\lambda_1=3.6$ and $\lambda_2=0.4$, and compare it with Lasso at the same $\alpha$.

House prices. Size and rooms are nearly collinear; days on market is uncorrelated with both (r = 0.011 and 0.022 with size and rooms).

| House | Size (m²) | Rooms | Days on market | Price ($k) |
| --- | --- | --- | --- | --- |
| 1 | 50 | 2 | 30 | 215 |
| 2 | 60 | 2 | 40 | 240 |
| 3 | 70 | 3 | 20 | 265 |
| 4 | 80 | 3 | 15 | 285 |
| 5 | 90 | 4 | 45 | 315 |
| 6 | 100 | 4 | 10 | 340 |
| 7 | 110 | 5 | 20 | 355 |
| 8 | 120 | 5 | 45 | 375 |

After standardizing (population standard deviation) and centring the price on its mean of 298.75.

| House | z size | z rooms | z days | y − ȳ |
| --- | --- | --- | --- | --- |
| 1 | −1.53 | −1.34 | 0.14 | −83.75 |
| 2 | −1.09 | −1.34 | 0.92 | −58.75 |
| 3 | −0.65 | −0.45 | −0.63 | −33.75 |
| 4 | −0.22 | −0.45 | −1.01 | −13.75 |
| 5 | 0.22 | 0.45 | 1.30 | 16.25 |
| 6 | 0.65 | 0.45 | −1.40 | 41.25 |
| 7 | 1.09 | 1.34 | −0.63 | 56.25 |
| 8 | 1.53 | 1.34 | 1.30 | 76.25 |

**Elastic Net by hand (α = 4, ρ = 0.9)**

1. **Standardize.** Means $\mu=(85,\ 3.5,\ 28.125)$, standard deviations $\sigma=(22.913,\ 1.118,\ 12.975)$, and $\bar y=298.75$, which becomes the intercept in standardized units.
2. **Correlations.** $r_{\text{size,rooms}}=0.9759$, $r_{\text{size,days}}=0.0105$, $r_{\text{rooms,days}}=0.0215$. Target correlations $c=(53.191,\ 51.989,\ -0.422)$.
3. **Sweep 1, size.** $z_1=c_1=53.191>\lambda_1$, so $w_1=(53.191-3.6)/1.4=49.591/1.4=35.422$.
4. **Sweep 1, rooms.** $z_2=51.989-0.9759(35.422)=17.420$, so $w_2=(17.420-3.6)/1.4=9.872$.
5. **Sweep 1, days.** $z_3=-0.422-0.0105(35.422)-0.0215(9.872)=-1.006$. Since $|z_3|\le3.6$, $w_3=0$.
6. **Keep sweeping.** Size moves back down and rooms moves up: $w=(28.54,14.67,0)$ after sweep 2, then $(25.20,17.00,0)$ after sweep 3. Each correction is about $0.9759/1.4=0.70$ times the previous one, and the iteration converges in 15 sweeps to a tolerance of $10^{-3}$.
7. **Or solve the fixed point directly.** With size and rooms active and positive: $1.4w_1+0.9759w_2=49.591$ and $0.9759w_1+1.4w_2=48.389$. Adding the two equations gives $w_1+w_2=97.979/2.3759=41.239$. Subtracting gives $w_1-w_2=1.202/0.4241=2.834$. So $w_1=22.037$ and $w_2=19.202$.
8. **Check days (KKT).** $|z_3|=|-0.422-0.0105(22.037)-0.0215(19.202)|=1.067\le\lambda_1=3.6$, so $w_3=0$ is optimal. The noise feature is dropped.
9. **Back to original units.** Size: $22.037/22.913=0.962$ per m². Rooms: $19.202/1.118=17.17$ per room. Intercept: $298.75-0.962(85)-17.17(3.5)=156.89$.
10. **Predict** a 100 m² house with 3 rooms that has been on the market 25 days: $156.89+0.962(100)+17.17(3)=156.89+96.18+51.52=\mathbf{304.6}$ thousand dollars.

The same data and α = 4 under three penalties. Lasso puts all the weight on size; Elastic Net splits it about 53/47; Ridge (here ρ = 0) splits it evenly but keeps a small nonzero weight on days.

| Model | w size | w rooms | w days | Nonzero | Train MSE | Price of new house |
| --- | --- | --- | --- | --- | --- | --- |
| OLS (α = 0) | 51.33 | 1.91 | −1.00 | 3 | 11.8 | 331.7 |
| Lasso (ρ = 1) | 49.19 | **0** | 0 | 1 | 28.9 | 331.0 |
| **Elastic Net (ρ = 0.9)** | **22.04** | **19.20** | **0** | 2 | 181.6 | **304.6** |
| Ridge-like (ρ = 0) | 8.95 | 8.65 | −0.14 | 3 | 1296.4 | 300.8 |

Why does Lasso drop rooms? Its fit is $w_1=c_1-\lambda=53.191-4=49.191$, which leaves rooms with $|z_2|=|51.989-0.9759(49.191)|=3.98$. That is just under the threshold of 4, so rooms stays at zero. With $r=0.976$, Lasso drops rooms for every $\alpha\ge3.31$. On the new house, which has a large floor area but few rooms, the two models disagree by 26k: Lasso only sees the size, while Elastic Net weighs both signals. The models are also not equally stable. Refit on each of the 8 leave-one-out subsets: Lasso keeps rooms in only **3 of 8** refits, and its size weight swings from 40.4 to 52.6. Elastic Net keeps rooms in **all 8**.

> Figure: Elastic Net on the eight houses. Left: the coefficient plane (w_size, w_rooms). Grey ellipses are squared-error contours, which are long and diagonal because r = 0.976; violet is the penalty ball through the current solution (a diamond at ρ = 1, a circle at ρ = 0); the dashed curve traces the solution as α varies; ○ marks Lasso at the same α. Right: the regularization path for the current ρ. At the default α = 4, ρ = 0.9 the solution is (22.04, 19.20, 0). Click 'lasso ρ=1' and it jumps to (49.19, 0, 0); turn on coordinate descent to watch the 15-sweep zigzag from the origin.

Things to try. Drag ρ from 1 down to 0 at α = 4: the diamond's corner rounds off, and the solution slides from the size axis toward the diagonal $w_{\text{size}}=w_{\text{rooms}}$. Pick Lasso and sweep α: rooms gets a small weight at small α but leaves the model at α ≈ 3.3. With ρ = 0.9, rooms and size stay together along the whole path. With coordinate descent on, compare ρ = 0.9 with ρ = 0.5 (6 sweeps). A larger λ₂ divides each correction by more, so the zigzag dies out faster.

## The algorithm, step by step

**Pathwise coordinate descent (glmnet / scikit-learn)**

1. **Prepare.** Standardize each column of $X$ and centre $y$. Remember $\mu$, $\sigma$ and $\bar y$ so you can convert the weights back to original units afterwards.
2. **Start at the top of the path.** $\alpha_{\max}=\max_j|c_j|/\rho$ is the smallest $\alpha$ at which every weight is zero (it requires $\rho>0$). Make a grid of about 100 values down to $10^{-3}\alpha_{\max}$, spaced on a log scale.
3. **Warm start.** For each $\alpha$ in decreasing order, start from the previous solution, which is already close.
4. **Sweep.** For $j=1..p$, compute $z_j=\tfrac1n x_j^\top(\text{residual})+w_j$, set $w_j\leftarrow S(z_j,\lambda_1)/(1+\lambda_2)$, and update the residual by $-x_j\,\Delta w_j$. Cost: $O(n)$ per feature, so $O(np)$ per sweep.
5. **Stop** when the largest weight change in a sweep, or the duality gap in scikit-learn, falls below the tolerance. Then move on to the next $\alpha$.
6. **Choose** $(\alpha,\rho)$ by cross-validation over the path, then refit on all the data.

**Complexity**

- **One sweep:** $O(np)$ with residual updates, or $O(p\cdot|\text{active}|)$ with "covariance updates" after an $O(np^2)$ precomputation (useful when $n\gg p$)
- **Whole path:** warm starts plus the strong-rule screening of inactive features make 100 values of $\alpha$ cost only a few times a single fit
- **Convergence:** each correction contracts by about $r/(1+\lambda_2)$ between correlated features, so the L2 part improves conditioning: 15 sweeps at ρ = 0.9 versus 6 at ρ = 0.5 in the example
- **Memory:** $O(np)$ for the data and $O(p)$ for the weights; sparse $X$ is supported

## Variants and tuning

Hyperparameters and closely related models.

| Knob / variant | Effect | Practical advice |
| --- | --- | --- |
| `alpha` | Raises both the threshold λ₁ and the divisor 1+λ₂: fewer features, smaller weights | Search it on a log grid with ElasticNetCV; the path is cheap |
| `l1_ratio` ρ | Near 1: Lasso-like and sparse. Near 0: Ridge-like and dense | Try [.1, .5, .7, .9, .95, .99, 1]; good values are often close to 1 |
| Rescaled Elastic Net | Multiply the naive $w$ by $1+\lambda_2$ to undo the double shrinkage | Example: (22.04, 19.20) → (30.85, 26.88). Equivalently, refit OLS on the selected features (relaxed fit) |
| Adaptive Elastic Net | Per-feature L1 weights $1/\|\hat w_j\|^\gamma$ taken from an initial fit | Better feature-selection consistency; needs a good first-stage estimate |
| Group Lasso / sparse group | Penalizes predefined groups (e.g. one-hot levels) as units | Use it when you already know the groups; Elastic Net finds them from correlation |
| Elastic-net logistic / Cox / multi-task | The same penalty on another loss | `LogisticRegression(penalty="elasticnet", solver="saga")`, glmnet, MultiTaskElasticNet |

> **Two conventions for the same model:** scikit-learn writes the penalty as α·ρ‖w‖₁ + α(1−ρ)/2‖w‖² with a 1/(2n) loss. glmnet uses the same α/λ form but calls ρ `alpha`, which is confusing. Zou & Hastie's paper uses λ₁‖w‖₁ + λ₂‖w‖² with an unscaled loss. `ElasticNet(alpha=4, l1_ratio=0)` matches `Ridge(alpha=32)` here: Ridge's loss is not divided by 2n, so its alpha must be n·α = 8·4 = 32. Check which convention your tool uses before you copy hyperparameters between them.

## In code

```python
import numpy as np

def soft_threshold(v, t):
    return np.sign(v) * np.maximum(np.abs(v) - t, 0.0)

def elastic_net(X, y, alpha=1.0, l1_ratio=0.5, tol=1e-8, max_sweeps=10_000):
    """Minimise (1/2n)||y - Xw - b||^2 + alpha*l1_ratio*||w||_1 + alpha*(1-l1_ratio)/2*||w||^2."""
    n, p = X.shape
    mu, sd = X.mean(0), X.std(0)
    Z = (X - mu) / sd                       # standardise: the penalty must see equal scales
    yc = y - y.mean()                       # the intercept is never penalised
    l1, l2 = alpha * l1_ratio, alpha * (1 - l1_ratio)
    z2 = (Z ** 2).sum(0) / n                # = 1 for standardised columns
    w = np.zeros(p)
    resid = yc.copy()                       # resid = yc - Z @ w, kept up to date
    for sweep in range(1, max_sweeps + 1):
        biggest = 0.0
        for j in range(p):
            rho_j = Z[:, j] @ resid / n + z2[j] * w[j]   # partial-residual correlation
            new = soft_threshold(rho_j, l1) / (z2[j] + l2)
            resid -= Z[:, j] * (new - w[j])
            biggest = max(biggest, abs(new - w[j]))
            w[j] = new
        if biggest < tol:
            break
    coef = w / sd                           # back to original units
    intercept = y.mean() - coef @ mu
    return coef, intercept, w, sweep

X = np.array([[50, 2, 30], [60, 2, 40], [70, 3, 20], [80, 3, 15],
              [90, 4, 45], [100, 4, 10], [110, 5, 20], [120, 5, 45]], float)
y = np.array([215, 240, 265, 285, 315, 340, 355, 375], float)

for name, ratio in [("elastic net", 0.9), ("lasso", 1.0)]:
    coef, b, w_std, sweeps = elastic_net(X, y, alpha=4.0, l1_ratio=ratio)
    print(f"{name:11s} standardised w = {np.round(w_std, 2)}  sweeps = {sweeps}")
    print(f"{'':11s} price = {b:.2f} + {np.round(coef, 4)} . x   ->  new house: {b + coef @ [100, 3, 25]:.1f}")

# elastic net standardised w = [22.04 19.2  -0.  ]  sweeps = 31
#             price = 156.89 + [ 0.9618 17.175  -0.    ] . x   ->  new house: 304.6
# lasso       standardised w = [49.19  0.   -0.  ]  sweeps = 2
#             price = 116.27 + [ 2.1469  0.     -0.    ] . x   ->  new house: 331.0
```

From scratch: coordinate descent with soft-thresholding and residual updates. Prints the worked-example numbers.

```python
import numpy as np
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import ElasticNet, Lasso, ElasticNetCV

X = np.array([[50, 2, 30], [60, 2, 40], [70, 3, 20], [80, 3, 15],
              [90, 4, 45], [100, 4, 10], [110, 5, 20], [120, 5, 45]], float)
y = np.array([215, 240, 265, 285, 315, 340, 355, 375], float)
new = np.array([[100, 3, 25]])

for name, model in [("elastic net", ElasticNet(alpha=4.0, l1_ratio=0.9)),
                    ("lasso", Lasso(alpha=4.0))]:
    pipe = make_pipeline(StandardScaler(), model).fit(X, y)
    m = pipe[-1]
    print(f"{name:11s} w = {np.round(m.coef_, 2)}  b = {m.intercept_:.2f}  "
          f"new house = {pipe.predict(new)[0]:.1f}")
# elastic net w = [22.04 19.2  -0.  ]  b = 298.75  new house = 304.6
# lasso       w = [49.19  0.   -0.  ]  b = 298.75  new house = 331.0

# On real data, choose alpha AND l1_ratio by cross-validation:
# cv = make_pipeline(StandardScaler(),
#                    ElasticNetCV(l1_ratio=[0.1, 0.5, 0.7, 0.9, 0.95, 0.99, 1.0],
#                                 alphas=100, cv=5)).fit(X_train, y_train)
# print("CV picked alpha =", round(cv[-1].alpha_, 3), " l1_ratio =", cv[-1].l1_ratio_)
```

scikit-learn: put StandardScaler and ElasticNet in one pipeline so the scaling is learned inside each CV fold.

**Strengths**

- You want a sparse, interpretable linear model, but the features come in correlated groups (genes, pixels, word n-grams, lagged sensors).
- $p\gg n$: Elastic Net can select more than $n$ features, and it stays stable where Lasso flips between near-duplicates.
- You cannot decide between Lasso and Ridge. Cross-validating ρ covers both as special cases.
- You need a fast, convex model with a single global optimum and a cheap full regularization path.

**Limitations**

- The relationship is strongly non-linear or has interactions. Use trees or boosting (see *Gradient Boosting Regression*), or add basis features first.
- You need the smallest possible feature set and the features are nearly independent. Plain Lasso is simpler and selects fewer features.
- You need unbiased effect sizes or p-values. Both penalties bias weights toward 0, so refit or use inference methods designed for penalized models.
- Two hyperparameters to tune is a real cost on very small datasets, where CV is noisy.

- **Genomics:** Selecting groups of co-expressed genes that predict a phenotype when there are thousands of genes and a few hundred samples. This was the original motivation for the method.
- **Quant finance:** Factor models and index tracking with many correlated returns: the model picks a sparse basket while correlated assets share weight.
- **Text regression:** Predicting ratings or prices from bag-of-words features, where synonyms and overlapping n-grams are highly correlated.
- **Neuroimaging & spectroscopy:** Neighbouring voxels or wavelengths are nearly collinear. Elastic Net selects contiguous regions instead of isolated single pixels.

> **Not standardizing first:** The penalty compares the raw sizes of the weights. Measure size in m² instead of in hundreds of m², and its weight becomes 100× smaller, so it is penalized far less. Always standardize, and do it inside a `Pipeline` so the cross-validation folds don't leak scaling statistics.

> **Double shrinkage:** Both penalties pull weights toward 0. In the example the naive Elastic Net's size and rooms weights sum to 41.2, while the least-squares weights sum to 53.2. Training MSE rises from 28.9 for Lasso to 181.6. If prediction accuracy suffers, either let CV push ρ toward 1, or use the selected features but rescale the weights by $1+\lambda_2$ or refit them without the L2 term.

> **Selected ≠ causal:** Elastic Net keeps or drops a whole correlated group, but the group as a whole is what predicts. Saying "rooms matter, size doesn't", or the reverse, is not supported by either model when r = 0.976.

**Question:** In scikit-learn, ElasticNet(alpha=2, l1_ratio=0.25) uses which L1 and L2 strengths?

- A. λ₁ = 0.25, λ₂ = 0.75
- B. λ₁ = 0.5, λ₂ = 1.5
- C. λ₁ = 1.5, λ₂ = 0.5
- D. λ₁ = 2, λ₂ = 2

**Answer:** B. λ₁ = 0.5, λ₂ = 1.5 — $\lambda_1=\alpha\rho=2\times0.25=0.5$ and $\lambda_2=\alpha(1-\rho)=2\times0.75=1.5$.

**Question:** A standardized feature has partial-residual correlation z = 5. With λ₁ = 2 and λ₂ = 0.5, one coordinate-descent update sets its weight to…

- A. 5
- B. 3
- C. 2
- D. 0

**Answer:** C. 2 — $w=S(5,2)/(1+0.5)=3/1.5=2$. The threshold removes 2, then the divisor scales what is left.

**Question:** You duplicate the size column, so two identical features have r = 1. What do Lasso and Elastic Net (α = 1, ρ = 0.5) do?

- A. Both split the weight equally
- B. Lasso puts all the weight on one copy (52.19, 0); Elastic Net gives both copies the same weight (21.08, 21.08)
- C. Both drop both copies
- D. Elastic Net picks one copy at random

**Answer:** B. Lasso puts all the weight on one copy (52.19, 0); Elastic Net gives both copies the same weight (21.08, 21.08) — With $r=1$, Lasso's objective is flat along $w_1+w_2=\text{const}$ and the solver ends at a corner. Elastic Net's L2 term is strictly convex, so the equal split is the unique minimum, and $w_1-w_2=(c_1-c_2)/(1+\lambda_2-r)=0$. We checked both results with scikit-learn.

**Question:** Why can Elastic Net select more than n features when p > n, while Lasso cannot?

- A. Because it uses a different solver
- B. Because it is the same as a Lasso on data augmented with p extra rows √(nλ₂)·I, which has rank p
- C. Because the L1 penalty is weaker
- D. It cannot; both are limited to n

**Answer:** B. Because it is the same as a Lasso on data augmented with p extra rows √(nλ₂)·I, which has rank p — Lasso's solution has at most n nonzeros because X has rank at most n. The augmented matrix [X; √(nλ₂) I] has n + p rows and full column rank, which removes that limit.

**Key points**

- Objective: $\tfrac{1}{2n}\|y-Xw\|^2+\alpha\rho\|w\|_1+\tfrac{\alpha(1-\rho)}{2}\|w\|^2$. ρ = 1 gives Lasso and ρ = 0 gives Ridge.
- Coordinate update $w_j\leftarrow S(z_j,\lambda_1)/(1+\lambda_2)$: the L1 part sets a threshold that produces zeros, and the L2 part divides every surviving weight by the same factor.
- Grouping effect: $w_1-w_2=(c_1-c_2)/(1+\lambda_2-r)$. The L2 term stops the denominator from collapsing, so correlated features share weight. In the example, Lasso gives (49.19, 0) and Elastic Net gives (22.04, 19.20).
- Elastic Net is a Lasso on augmented data, so it can select more than $n$ features and any Lasso solver can fit it.
- Standardize inside a pipeline, tune α and ρ together with ElasticNetCV, and watch for double shrinkage.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
