Elastic Net regression

Mix the L1 and L2 penalties to get Lasso's sparsity while keeping groups of correlated features together.

17 min read1642 wordscore9 viewsupdated 2026-10-07

Definition

Elastic Net regression is linear least squares with two penalties added at once: an L1 term λ1∥w∥1\lambda_1\|w\|_1, which can set weights to exactly zero, and an L2 term λ22∥w∥22\tfrac{\lambda_2}{2}\|w\|_2^2, which shrinks weights smoothly and makes strongly correlated features share their weight. Set λ2=0\lambda_2=0 and you get Lasso. Set λ1=0\lambda_1=0 and you get Ridge.

At a glance

Task
Regression (the same penalty also works with logistic loss for classification)
Input → output
feature vector x∈Rpx\in\mathbb R^p → a real number y^=b+w⊤x\hat y=b+w^\top x
Learns
a weight vector ww, usually sparse with correlated features grouped, plus an unpenalized intercept bb
Training
coordinate descent: O(np)O(np) per sweep over all features; a whole path of 100 α\alpha values with warm starts costs only a few times one fit
Prediction
O(p)O(p), or O(#nonzero)O(\#\text{nonzero}) for a sparse ww
Hyperparameters
alpha (overall penalty strength) and l1_ratio ρ (the L1 share of the penalty)
Assumes
a roughly linear relationship, and standardized features, since the penalty compares the sizes of the weights directly

The problem

Lasso (see the Lasso regression article) does feature selection: its L1 penalty drives unhelpful weights to exactly zero. But it handles correlated features badly. When two features carry nearly the same information, such as a house's floor area and its number of rooms, Lasso usually keeps one and zeroes the other. Which one it keeps depends on tiny details of the sample, so refitting on slightly different data can swap the choice. Lasso has two more limits: when p>np>n it can select at most nn features, and when predictors are highly correlated its accuracy is often worse than Ridge's.

Ridge (see Ridge regression) has the opposite profile. It is stable under correlation and spreads weight evenly over correlated features, but it never sets a weight to zero, so it cannot select features. In genomics, text, and sensor data you often want both: a short list of features, where a whole correlated group gets in or stays out together. Elastic Net (Zou & Hastie, 2005) provides this.

The idea

Use both penalties at once. The L1 part keeps Lasso's corners, so weights can land exactly on zero. The L2 part makes the penalty strictly convex: among many nearly equivalent ways to split weight between two near-duplicate features, it strictly prefers the even split. The resulting penalty ball is a diamond with rounded edges. Its corners still sit on the axes, which gives sparsity. Its edges bulge outward, so the loss contours can touch it at a point where both correlated weights are nonzero.

Vocabulary

α (alpha)
overall regularization strength. α=0\alpha=0 is ordinary least squares; large α\alpha drives every weight to 0
ρ (l1_ratio)
how the penalty is split. ρ=1\rho=1 is pure Lasso, ρ=0\rho=0 is pure Ridge, and values in between are Elastic Net
λ₁, λ₂
the actual L1 and L2 strengths: λ1=αρ\lambda_1=\alpha\rho, λ2=α(1−ρ)\lambda_2=\alpha(1-\rho)
Grouping effect
highly correlated features receive nearly equal weights and enter or leave the model together
Soft-thresholding
S(v,t)=sign⁡(v)max⁡(∣v∣−t,0)S(v,t)=\operatorname{sign}(v)\max(|v|-t,0): move vv toward 0 by tt, and clip it to 0 if it would cross. This is what produces exact zeros
Active set
the features whose weight is currently nonzero
Regularization path
the weights as a function of α\alpha, from all-zero (huge α\alpha) to the least-squares fit (α→0\alpha\to0)
Naive vs rescaled
the plain solution below is the naive Elastic Net. Zou & Hastie multiply it by (1+λ2)(1+\lambda_2) to undo the extra shrinkage from the L2 term

The math

We use scikit-learn's parameterization, so the numbers here match ElasticNet(alpha, l1_ratio) exactly. The columns of XX are standardized, yy is centred, and the intercept is fitted separately and never penalized.

min⁡w    12n∥y−Xw∥22  +  αρ ∥w∥1  +  α(1−ρ)2 ∥w∥22\min_{w}\;\; \frac{1}{2n}\|y-Xw\|_2^2\;+\;\alpha\rho\,\|w\|_1\;+\;\frac{\alpha(1-\rho)}{2}\,\|w\|_2^2
XX
n×pn\times p matrix of standardized features (each column has mean 0 and variance 1)
yy
centred targets; the intercept is b=yˉb=\bar y
ww
weights, one per feature
∥w∥1\|w\|_1
∑j∣wj∣\sum_j|w_j|, the L1 norm, which produces zeros
∥w∥22\|w\|_2^2
∑jwj2\sum_j w_j^2, the squared L2 norm, which shrinks weights and groups correlated features
α, ρ\alpha,\ \rho
overall strength and L1 share; λ1=αρ\lambda_1=\alpha\rho, λ2=α(1−ρ)\lambda_2=\alpha(1-\rho)

Deriving the coordinate update. Fix every weight except wjw_j. Write rjk=1nxj⊤xkr_{jk}=\tfrac1n x_j^\top x_k for the correlation between features jj and kk; standardization makes rjj=1r_{jj}=1. Write cj=1nxj⊤yc_j=\tfrac1n x_j^\top y for each feature's correlation with the target. As a function of wjw_j alone, the objective is 12wj2−zjwj+λ1∣wj∣+λ22wj2+const\tfrac12 w_j^2 - z_j w_j + \lambda_1|w_j| + \tfrac{\lambda_2}{2}w_j^2 + \text{const}, where zjz_j is the partial-residual correlation: how strongly feature jj correlates with what the other features have not yet explained.

zj  =  cj−∑k≠jrjk wk  =  1n xj⊤(y−∑k≠jxkwk)z_j \;=\; c_j-\sum_{k\neq j} r_{jk}\,w_k \;=\; \tfrac1n\,x_j^\top\Big(y-\sum_{k\ne j}x_k w_k\Big)
cjc_j
1nxj⊤y\tfrac1n x_j^\top y, the correlation of feature jj with the target
rjkr_{jk}
1nxj⊤xk\tfrac1n x_j^\top x_k, the correlation between features jj and kk

For wj≠0w_j\neq0, setting the derivative to zero gives (1+λ2)wj=zj−λ1sign⁡(wj)(1+\lambda_2)w_j = z_j-\lambda_1\operatorname{sign}(w_j). At wj=0w_j=0 the subgradient of ∣wj∣|w_j| can be anything in [−1,1][-1,1], so zero is optimal whenever ∣zj∣≤λ1|z_j|\le\lambda_1. Putting both cases together gives the update:

wj  ←  S(zj, λ1)1+λ2,S(v,t)=sign⁡(v)max⁡(∣v∣−t, 0)w_j \;\leftarrow\; \frac{S(z_j,\ \lambda_1)}{1+\lambda_2},\qquad S(v,t)=\operatorname{sign}(v)\max(|v|-t,\,0)
S(⋅,λ1)S(\cdot,\lambda_1)
soft-thresholding, the L1 part: subtract λ1\lambda_1, and return exactly 0 if ∣zj∣≤λ1|z_j|\le\lambda_1
1/(1+λ2)1/(1+\lambda_2)
proportional shrinkage, the L2 part: divide every surviving weight by the same factor

This one line shows each hyperparameter's effect. λ₁ sets a threshold: a feature whose partial correlation is at most λ1\lambda_1 is switched off. λ₂ sets a divisor: every active weight is divided by 1+λ21+\lambda_2. Ridge's update is zj/(1+λ2)z_j/(1+\lambda_2), which has a divisor but no threshold. Lasso's is S(zj,λ1)S(z_j,\lambda_1), which has a threshold but no divisor. Raising α at a fixed ρ raises both. Raising ρ at a fixed α moves strength from the divisor to the threshold, which gives more zeros and less proportional shrinkage.

Why correlated features get grouped

Suppose features 1 and 2 are both active with positive weights, and their correlation is rr. Their two optimality conditions are (1+λ2)w1+r w2=c1−λ1−…(1+\lambda_2)w_1 + r\,w_2 = c_1-\lambda_1-\ldots and r w1+(1+λ2)w2=c2−λ1−…r\,w_1+(1+\lambda_2)w_2 = c_2-\lambda_1-\ldots, where "…" is the same small contribution from any other features. Subtract the second from the first:

w1−w2  =  c1−c2 1+λ2−r w_1-w_2 \;=\; \frac{c_1-c_2}{\,1+\lambda_2-r\,}
c1−c2c_1-c_2
how differently the two features correlate with the target, which is small for near-duplicates
1+λ2−r1+\lambda_2-r
for Lasso this is 1−r1-r, which tends to 0 as r→1r\to1; Elastic Net keeps it at least λ2\lambda_2

For Lasso (λ2=0\lambda_2=0) the denominator 1−r1-r goes to zero as the features become duplicates. A tiny difference c1−c2c_1-c_2 then becomes a large weight difference. In practice one weight takes everything and the other hits zero, and at exactly r=1r=1 the split is undetermined. For Elastic Net the denominator never drops below λ2\lambda_2, so the weights stay close together. Zou and Hastie state the general version as a bound: for any two features i,ji,j in the active set, ∣wi−wj∣≤2(1−rij)  RMSE/λ2|w_i-w_j|\le \sqrt{2(1-r_{ij})}\;\text{RMSE}/\lambda_2, where RMSE is the root-mean-square residual of the fit. Perfectly correlated features (r=1r=1) get identical weights.

Worked example: size or rooms?

Eight houses. Floor area and number of rooms are almost the same information (correlation r=0.976r=0.976). Days on market is a feature with nothing to say about price. We fit Elastic Net with α=4\alpha=4 and ρ=0.9\rho=0.9, so λ1=3.6\lambda_1=3.6 and λ2=0.4\lambda_2=0.4, and compare it with Lasso at the same α\alpha.

HouseSize (m²)RoomsDays on marketPrice ($k)
150230215
260240240
370320265
480315285
590445315
6100410340
7110520355
8120545375
House prices. Size and rooms are nearly collinear; days on market is uncorrelated with both (r = 0.011 and 0.022 with size and rooms).
Housez sizez roomsz daysy − ȳ
1−1.53−1.340.14−83.75
2−1.09−1.340.92−58.75
3−0.65−0.45−0.63−33.75
4−0.22−0.45−1.01−13.75
50.220.451.3016.25
60.650.45−1.4041.25
71.091.34−0.6356.25
81.531.341.3076.25
After standardizing (population standard deviation) and centring the price on its mean of 298.75.

Elastic Net by hand (α = 4, ρ = 0.9)

  1. 1Standardize. Means μ=(85, 3.5, 28.125)\mu=(85,\ 3.5,\ 28.125), standard deviations σ=(22.913, 1.118, 12.975)\sigma=(22.913,\ 1.118,\ 12.975), and yˉ=298.75\bar y=298.75, which becomes the intercept in standardized units.
  2. 2Correlations. rsize,rooms=0.9759r_{\text{size,rooms}}=0.9759, rsize,days=0.0105r_{\text{size,days}}=0.0105, rrooms,days=0.0215r_{\text{rooms,days}}=0.0215. Target correlations c=(53.191, 51.989, −0.422)c=(53.191,\ 51.989,\ -0.422).
  3. 3Sweep 1, size. z1=c1=53.191>λ1z_1=c_1=53.191>\lambda_1, so w1=(53.191−3.6)/1.4=49.591/1.4=35.422w_1=(53.191-3.6)/1.4=49.591/1.4=35.422.
  4. 4Sweep 1, rooms. z2=51.989−0.9759(35.422)=17.420z_2=51.989-0.9759(35.422)=17.420, so w2=(17.420−3.6)/1.4=9.872w_2=(17.420-3.6)/1.4=9.872.
  5. 5Sweep 1, days. z3=−0.422−0.0105(35.422)−0.0215(9.872)=−1.006z_3=-0.422-0.0105(35.422)-0.0215(9.872)=-1.006. Since ∣z3∣≤3.6|z_3|\le3.6, w3=0w_3=0.
  6. 6Keep sweeping. Size moves back down and rooms moves up: w=(28.54,14.67,0)w=(28.54,14.67,0) after sweep 2, then (25.20,17.00,0)(25.20,17.00,0) after sweep 3. Each correction is about 0.9759/1.4=0.700.9759/1.4=0.70 times the previous one, and the iteration converges in 15 sweeps to a tolerance of 10−310^{-3}.
  7. 7Or solve the fixed point directly. With size and rooms active and positive: 1.4w1+0.9759w2=49.5911.4w_1+0.9759w_2=49.591 and 0.9759w1+1.4w2=48.3890.9759w_1+1.4w_2=48.389. Adding the two equations gives w1+w2=97.979/2.3759=41.239w_1+w_2=97.979/2.3759=41.239. Subtracting gives w1−w2=1.202/0.4241=2.834w_1-w_2=1.202/0.4241=2.834. So w1=22.037w_1=22.037 and w2=19.202w_2=19.202.
  8. 8Check days (KKT). ∣z3∣=∣−0.422−0.0105(22.037)−0.0215(19.202)∣=1.067≤λ1=3.6|z_3|=|-0.422-0.0105(22.037)-0.0215(19.202)|=1.067\le\lambda_1=3.6, so w3=0w_3=0 is optimal. The noise feature is dropped.
  9. 9Back to original units. Size: 22.037/22.913=0.96222.037/22.913=0.962 per m². Rooms: 19.202/1.118=17.1719.202/1.118=17.17 per room. Intercept: 298.75−0.962(85)−17.17(3.5)=156.89298.75-0.962(85)-17.17(3.5)=156.89.
  10. 10Predict a 100 m² house with 3 rooms that has been on the market 25 days: 156.89+0.962(100)+17.17(3)=156.89+96.18+51.52=304.6156.89+0.962(100)+17.17(3)=156.89+96.18+51.52=\mathbf{304.6} thousand dollars.
Modelw sizew roomsw daysNonzeroTrain MSEPrice of new house
OLS (α = 0)51.331.91−1.00311.8331.7
Lasso (ρ = 1)49.1900128.9331.0
Elastic Net (ρ = 0.9)22.0419.2002181.6304.6
Ridge-like (ρ = 0)8.958.65−0.1431296.4300.8
The same data and α = 4 under three penalties. Lasso puts all the weight on size; Elastic Net splits it about 53/47; Ridge (here ρ = 0) splits it evenly but keeps a small nonzero weight on days.

Why does Lasso drop rooms? Its fit is w1=c1−λ=53.191−4=49.191w_1=c_1-\lambda=53.191-4=49.191, which leaves rooms with ∣z2∣=∣51.989−0.9759(49.191)∣=3.98|z_2|=|51.989-0.9759(49.191)|=3.98. That is just under the threshold of 4, so rooms stays at zero. With r=0.976r=0.976, Lasso drops rooms for every α≥3.31\alpha\ge3.31. On the new house, which has a large floor area but few rooms, the two models disagree by 26k: Lasso only sees the size, while Elastic Net weighs both signals. The models are also not equally stable. Refit on each of the 8 leave-one-out subsets: Lasso keeps rooms in only 3 of 8 refits, and its size weight swings from 40.4 to 52.6. Elastic Net keeps rooms in all 8.

coefficients: x = wsize, y = wrooms00202040406060OLSpenalty ball · loss contour · ○ lassopath vs log₁₀ α · ρ = 0.900204060-2-1012● size● rooms● days← weaker penalty · stronger →
λ₁ = αρ = 3.60 · λ₂ = α(1−ρ) = 0.40
w = (22.04, 19.20, 0.00) · nonzero 2/3
lasso at same α: (49.19, 0.00, 0.00)
gap wsize−wrooms = (c₁−c₂)/(1+λ₂−r) = 2.83
train MSE 181.6 · price(100 m², 3 rooms, 25 d) = 304.6k
fig.Elastic Net on the eight houses. Left: the coefficient plane (w_size, w_rooms). Grey ellipses are squared-error contours, which are long and diagonal because r = 0.976; violet is the penalty ball through the current solution (a diamond at ρ = 1, a circle at ρ = 0); the dashed curve traces the solution as α varies; ○ marks Lasso at the same α. Right: the regularization path for the current ρ. At the default α = 4, ρ = 0.9 the solution is (22.04, 19.20, 0). Click 'lasso ρ=1' and it jumps to (49.19, 0, 0); turn on coordinate descent to watch the 15-sweep zigzag from the origin.

Things to try. Drag ρ from 1 down to 0 at α = 4: the diamond's corner rounds off, and the solution slides from the size axis toward the diagonal wsize=wroomsw_{\text{size}}=w_{\text{rooms}}. Pick Lasso and sweep α: rooms gets a small weight at small α but leaves the model at α ≈ 3.3. With ρ = 0.9, rooms and size stay together along the whole path. With coordinate descent on, compare ρ = 0.9 with ρ = 0.5 (6 sweeps). A larger λ₂ divides each correction by more, so the zigzag dies out faster.

The algorithm, step by step

Pathwise coordinate descent (glmnet / scikit-learn)

  1. 1Prepare. Standardize each column of XX and centre yy. Remember μ\mu, σ\sigma and yˉ\bar y so you can convert the weights back to original units afterwards.
  2. 2Start at the top of the path. αmax⁡=max⁡j∣cj∣/ρ\alpha_{\max}=\max_j|c_j|/\rho is the smallest α\alpha at which every weight is zero (it requires ρ>0\rho>0). Make a grid of about 100 values down to 10−3αmax⁡10^{-3}\alpha_{\max}, spaced on a log scale.
  3. 3Warm start. For each α\alpha in decreasing order, start from the previous solution, which is already close.
  4. 4Sweep. For j=1..pj=1..p, compute zj=1nxj⊤(residual)+wjz_j=\tfrac1n x_j^\top(\text{residual})+w_j, set wj←S(zj,λ1)/(1+λ2)w_j\leftarrow S(z_j,\lambda_1)/(1+\lambda_2), and update the residual by −xj Δwj-x_j\,\Delta w_j. Cost: O(n)O(n) per feature, so O(np)O(np) per sweep.
  5. 5Stop when the largest weight change in a sweep, or the duality gap in scikit-learn, falls below the tolerance. Then move on to the next α\alpha.
  6. 6Choose (α,ρ)(\alpha,\rho) by cross-validation over the path, then refit on all the data.

Complexity

One sweep
O(np)O(np) with residual updates, or O(p⋅∣active∣)O(p\cdot|\text{active}|) with "covariance updates" after an O(np2)O(np^2) precomputation (useful when n≫pn\gg p)
Whole path
warm starts plus the strong-rule screening of inactive features make 100 values of α\alpha cost only a few times a single fit
Convergence
each correction contracts by about r/(1+λ2)r/(1+\lambda_2) between correlated features, so the L2 part improves conditioning: 15 sweeps at ρ = 0.9 versus 6 at ρ = 0.5 in the example
Memory
O(np)O(np) for the data and O(p)O(p) for the weights; sparse XX is supported

Variants and tuning

Knob / variantEffectPractical advice
alphaRaises both the threshold λ₁ and the divisor 1+λ₂: fewer features, smaller weightsSearch it on a log grid with ElasticNetCV; the path is cheap
l1_ratio ρNear 1: Lasso-like and sparse. Near 0: Ridge-like and denseTry [.1, .5, .7, .9, .95, .99, 1]; good values are often close to 1
Rescaled Elastic NetMultiply the naive ww by 1+λ21+\lambda_2 to undo the double shrinkageExample: (22.04, 19.20) → (30.85, 26.88). Equivalently, refit OLS on the selected features (relaxed fit)
Adaptive Elastic NetPer-feature L1 weights 1/∣w^j∣γ1/|\hat w_j|^\gamma taken from an initial fitBetter feature-selection consistency; needs a good first-stage estimate
Group Lasso / sparse groupPenalizes predefined groups (e.g. one-hot levels) as unitsUse it when you already know the groups; Elastic Net finds them from correlation
Elastic-net logistic / Cox / multi-taskThe same penalty on another lossLogisticRegression(penalty="elasticnet", solver="saga"), glmnet, MultiTaskElasticNet
Hyperparameters and closely related models.

In code

python
1import numpy as np
2
3def soft_threshold(v, t):
4 return np.sign(v) * np.maximum(np.abs(v) - t, 0.0)
5
6def elastic_net(X, y, alpha=1.0, l1_ratio=0.5, tol=1e-8, max_sweeps=10_000):
7 """Minimise (1/2n)||y - Xw - b||^2 + alpha*l1_ratio*||w||_1 + alpha*(1-l1_ratio)/2*||w||^2."""
8 n, p = X.shape
9 mu, sd = X.mean(0), X.std(0)
10 Z = (X - mu) / sd # standardise: the penalty must see equal scales
11 yc = y - y.mean() # the intercept is never penalised
12 l1, l2 = alpha * l1_ratio, alpha * (1 - l1_ratio)
13 z2 = (Z ** 2).sum(0) / n # = 1 for standardised columns
14 w = np.zeros(p)
15 resid = yc.copy() # resid = yc - Z @ w, kept up to date
16 for sweep in range(1, max_sweeps + 1):
17 biggest = 0.0
18 for j in range(p):
19 rho_j = Z[:, j] @ resid / n + z2[j] * w[j] # partial-residual correlation
20 new = soft_threshold(rho_j, l1) / (z2[j] + l2)
21 resid -= Z[:, j] * (new - w[j])
22 biggest = max(biggest, abs(new - w[j]))
23 w[j] = new
24 if biggest < tol:
25 break
26 coef = w / sd # back to original units
27 intercept = y.mean() - coef @ mu
28 return coef, intercept, w, sweep
29
30X = np.array([[50, 2, 30], [60, 2, 40], [70, 3, 20], [80, 3, 15],
31 [90, 4, 45], [100, 4, 10], [110, 5, 20], [120, 5, 45]], float)
32y = np.array([215, 240, 265, 285, 315, 340, 355, 375], float)
33
34for name, ratio in [("elastic net", 0.9), ("lasso", 1.0)]:
35 coef, b, w_std, sweeps = elastic_net(X, y, alpha=4.0, l1_ratio=ratio)
36 print(f"{name:11s} standardised w = {np.round(w_std, 2)} sweeps = {sweeps}")
37 print(f"{'':11s} price = {b:.2f} + {np.round(coef, 4)} . x -> new house: {b + coef @ [100, 3, 25]:.1f}")
38
39# elastic net standardised w = [22.04 19.2 -0. ] sweeps = 31
40# price = 156.89 + [ 0.9618 17.175 -0. ] . x -> new house: 304.6
41# lasso standardised w = [49.19 0. -0. ] sweeps = 2
42# price = 116.27 + [ 2.1469 0. -0. ] . x -> new house: 331.0
From scratch: coordinate descent with soft-thresholding and residual updates. Prints the worked-example numbers.
python
1import numpy as np
2from sklearn.pipeline import make_pipeline
3from sklearn.preprocessing import StandardScaler
4from sklearn.linear_model import ElasticNet, Lasso, ElasticNetCV
5
6X = np.array([[50, 2, 30], [60, 2, 40], [70, 3, 20], [80, 3, 15],
7 [90, 4, 45], [100, 4, 10], [110, 5, 20], [120, 5, 45]], float)
8y = np.array([215, 240, 265, 285, 315, 340, 355, 375], float)
9new = np.array([[100, 3, 25]])
10
11for name, model in [("elastic net", ElasticNet(alpha=4.0, l1_ratio=0.9)),
12 ("lasso", Lasso(alpha=4.0))]:
13 pipe = make_pipeline(StandardScaler(), model).fit(X, y)
14 m = pipe[-1]
15 print(f"{name:11s} w = {np.round(m.coef_, 2)} b = {m.intercept_:.2f} "
16 f"new house = {pipe.predict(new)[0]:.1f}")
17# elastic net w = [22.04 19.2 -0. ] b = 298.75 new house = 304.6
18# lasso w = [49.19 0. -0. ] b = 298.75 new house = 331.0
19
20# On real data, choose alpha AND l1_ratio by cross-validation:
21# cv = make_pipeline(StandardScaler(),
22# ElasticNetCV(l1_ratio=[0.1, 0.5, 0.7, 0.9, 0.95, 0.99, 1.0],
23# alphas=100, cv=5)).fit(X_train, y_train)
24# print("CV picked alpha =", round(cv[-1].alpha_, 3), " l1_ratio =", cv[-1].l1_ratio_)
scikit-learn: put StandardScaler and ElasticNet in one pipeline so the scaling is learned inside each CV fold.

Use when

  • +You want a sparse, interpretable linear model, but the features come in correlated groups (genes, pixels, word n-grams, lagged sensors).
  • +p≫np\gg n: Elastic Net can select more than nn features, and it stays stable where Lasso flips between near-duplicates.
  • +You cannot decide between Lasso and Ridge. Cross-validating ρ covers both as special cases.
  • +You need a fast, convex model with a single global optimum and a cheap full regularization path.

Avoid when

  • −The relationship is strongly non-linear or has interactions. Use trees or boosting (see Gradient Boosting Regression), or add basis features first.
  • −You need the smallest possible feature set and the features are nearly independent. Plain Lasso is simpler and selects fewer features.
  • −You need unbiased effect sizes or p-values. Both penalties bias weights toward 0, so refit or use inference methods designed for penalized models.
  • −Two hyperparameters to tune is a real cost on very small datasets, where CV is noisy.

Where it's used

Genomics

Selecting groups of co-expressed genes that predict a phenotype when there are thousands of genes and a few hundred samples. This was the original motivation for the method.

Quant finance

Factor models and index tracking with many correlated returns: the model picks a sparse basket while correlated assets share weight.

Text regression

Predicting ratings or prices from bag-of-words features, where synonyms and overlapping n-grams are highly correlated.

Neuroimaging & spectroscopy

Neighbouring voxels or wavelengths are nearly collinear. Elastic Net selects contiguous regions instead of isolated single pixels.

Check yourself

In scikit-learn, ElasticNet(alpha=2, l1_ratio=0.25) uses which L1 and L2 strengths?

Check yourself

A standardized feature has partial-residual correlation z = 5. With λ₁ = 2 and λ₂ = 0.5, one coordinate-descent update sets its weight to…

Check yourself

You duplicate the size column, so two identical features have r = 1. What do Lasso and Elastic Net (α = 1, ρ = 0.5) do?

Check yourself

Why can Elastic Net select more than n features when p > n, while Lasso cannot?

Remember

  • ✓Objective: 12n∥y−Xw∥2+αρ∥w∥1+α(1−ρ)2∥w∥2\tfrac{1}{2n}\|y-Xw\|^2+\alpha\rho\|w\|_1+\tfrac{\alpha(1-\rho)}{2}\|w\|^2. ρ = 1 gives Lasso and ρ = 0 gives Ridge.
  • ✓Coordinate update wj←S(zj,λ1)/(1+λ2)w_j\leftarrow S(z_j,\lambda_1)/(1+\lambda_2): the L1 part sets a threshold that produces zeros, and the L2 part divides every surviving weight by the same factor.
  • ✓Grouping effect: w1−w2=(c1−c2)/(1+λ2−r)w_1-w_2=(c_1-c_2)/(1+\lambda_2-r). The L2 term stops the denominator from collapsing, so correlated features share weight. In the example, Lasso gives (49.19, 0) and Elastic Net gives (22.04, 19.20).
  • ✓Elastic Net is a Lasso on augmented data, so it can select more than nn features and any Lasso solver can fit it.
  • ✓Standardize inside a pipeline, tune α and ρ together with ElasticNetCV, and watch for double shrinkage.
← back to ML Algorithms roadmap