Definition
Elastic Net regression is linear least squares with two penalties added at once: an L1 term , which can set weights to exactly zero, and an L2 term , which shrinks weights smoothly and makes strongly correlated features share their weight. Set and you get Lasso. Set and you get Ridge.
At a glance
- Task
- Regression (the same penalty also works with logistic loss for classification)
- Input → output
- feature vector → a real number
- Learns
- a weight vector , usually sparse with correlated features grouped, plus an unpenalized intercept
- Training
- coordinate descent: per sweep over all features; a whole path of 100 values with warm starts costs only a few times one fit
- Prediction
- , or for a sparse
- Hyperparameters
alpha(overall penalty strength) andl1_ratioρ (the L1 share of the penalty)- Assumes
- a roughly linear relationship, and standardized features, since the penalty compares the sizes of the weights directly
The problem
Lasso (see the Lasso regression article) does feature selection: its L1 penalty drives unhelpful weights to exactly zero. But it handles correlated features badly. When two features carry nearly the same information, such as a house's floor area and its number of rooms, Lasso usually keeps one and zeroes the other. Which one it keeps depends on tiny details of the sample, so refitting on slightly different data can swap the choice. Lasso has two more limits: when it can select at most features, and when predictors are highly correlated its accuracy is often worse than Ridge's.
Ridge (see Ridge regression) has the opposite profile. It is stable under correlation and spreads weight evenly over correlated features, but it never sets a weight to zero, so it cannot select features. In genomics, text, and sensor data you often want both: a short list of features, where a whole correlated group gets in or stays out together. Elastic Net (Zou & Hastie, 2005) provides this.
The idea
Use both penalties at once. The L1 part keeps Lasso's corners, so weights can land exactly on zero. The L2 part makes the penalty strictly convex: among many nearly equivalent ways to split weight between two near-duplicate features, it strictly prefers the even split. The resulting penalty ball is a diamond with rounded edges. Its corners still sit on the axes, which gives sparsity. Its edges bulge outward, so the loss contours can touch it at a point where both correlated weights are nonzero.
Vocabulary
- α (alpha)
- overall regularization strength. is ordinary least squares; large drives every weight to 0
- ρ (l1_ratio)
- how the penalty is split. is pure Lasso, is pure Ridge, and values in between are Elastic Net
- λ₁, λ₂
- the actual L1 and L2 strengths: ,
- Grouping effect
- highly correlated features receive nearly equal weights and enter or leave the model together
- Soft-thresholding
- : move toward 0 by , and clip it to 0 if it would cross. This is what produces exact zeros
- Active set
- the features whose weight is currently nonzero
- Regularization path
- the weights as a function of , from all-zero (huge ) to the least-squares fit ()
- Naive vs rescaled
- the plain solution below is the naive Elastic Net. Zou & Hastie multiply it by to undo the extra shrinkage from the L2 term
The math
We use scikit-learn's parameterization, so the numbers here match ElasticNet(alpha, l1_ratio) exactly. The columns of are standardized, is centred, and the intercept is fitted separately and never penalized.
- matrix of standardized features (each column has mean 0 and variance 1)
- centred targets; the intercept is
- weights, one per feature
- , the L1 norm, which produces zeros
- , the squared L2 norm, which shrinks weights and groups correlated features
- overall strength and L1 share; ,
Deriving the coordinate update. Fix every weight except . Write for the correlation between features and ; standardization makes . Write for each feature's correlation with the target. As a function of alone, the objective is , where is the partial-residual correlation: how strongly feature correlates with what the other features have not yet explained.
- , the correlation of feature with the target
- , the correlation between features and
For , setting the derivative to zero gives . At the subgradient of can be anything in , so zero is optimal whenever . Putting both cases together gives the update:
- soft-thresholding, the L1 part: subtract , and return exactly 0 if
- proportional shrinkage, the L2 part: divide every surviving weight by the same factor
This one line shows each hyperparameter's effect. λ₁ sets a threshold: a feature whose partial correlation is at most is switched off. λ₂ sets a divisor: every active weight is divided by . Ridge's update is , which has a divisor but no threshold. Lasso's is , which has a threshold but no divisor. Raising α at a fixed ρ raises both. Raising ρ at a fixed α moves strength from the divisor to the threshold, which gives more zeros and less proportional shrinkage.
Why correlated features get grouped
Suppose features 1 and 2 are both active with positive weights, and their correlation is . Their two optimality conditions are and , where "…" is the same small contribution from any other features. Subtract the second from the first:
- how differently the two features correlate with the target, which is small for near-duplicates
- for Lasso this is , which tends to 0 as ; Elastic Net keeps it at least
For Lasso () the denominator goes to zero as the features become duplicates. A tiny difference then becomes a large weight difference. In practice one weight takes everything and the other hits zero, and at exactly the split is undetermined. For Elastic Net the denominator never drops below , so the weights stay close together. Zou and Hastie state the general version as a bound: for any two features in the active set, , where RMSE is the root-mean-square residual of the fit. Perfectly correlated features () get identical weights.
Worked example: size or rooms?
Eight houses. Floor area and number of rooms are almost the same information (correlation ). Days on market is a feature with nothing to say about price. We fit Elastic Net with and , so and , and compare it with Lasso at the same .
| House | Size (m²) | Rooms | Days on market | Price ($k) |
|---|---|---|---|---|
| 1 | 50 | 2 | 30 | 215 |
| 2 | 60 | 2 | 40 | 240 |
| 3 | 70 | 3 | 20 | 265 |
| 4 | 80 | 3 | 15 | 285 |
| 5 | 90 | 4 | 45 | 315 |
| 6 | 100 | 4 | 10 | 340 |
| 7 | 110 | 5 | 20 | 355 |
| 8 | 120 | 5 | 45 | 375 |
| House | z size | z rooms | z days | y − ȳ |
|---|---|---|---|---|
| 1 | −1.53 | −1.34 | 0.14 | −83.75 |
| 2 | −1.09 | −1.34 | 0.92 | −58.75 |
| 3 | −0.65 | −0.45 | −0.63 | −33.75 |
| 4 | −0.22 | −0.45 | −1.01 | −13.75 |
| 5 | 0.22 | 0.45 | 1.30 | 16.25 |
| 6 | 0.65 | 0.45 | −1.40 | 41.25 |
| 7 | 1.09 | 1.34 | −0.63 | 56.25 |
| 8 | 1.53 | 1.34 | 1.30 | 76.25 |
Elastic Net by hand (α = 4, ρ = 0.9)
- 1Standardize. Means , standard deviations , and , which becomes the intercept in standardized units.
- 2Correlations. , , . Target correlations .
- 3Sweep 1, size. , so .
- 4Sweep 1, rooms. , so .
- 5Sweep 1, days. . Since , .
- 6Keep sweeping. Size moves back down and rooms moves up: after sweep 2, then after sweep 3. Each correction is about times the previous one, and the iteration converges in 15 sweeps to a tolerance of .
- 7Or solve the fixed point directly. With size and rooms active and positive: and . Adding the two equations gives . Subtracting gives . So and .
- 8Check days (KKT). , so is optimal. The noise feature is dropped.
- 9Back to original units. Size: per m². Rooms: per room. Intercept: .
- 10Predict a 100 m² house with 3 rooms that has been on the market 25 days: thousand dollars.
| Model | w size | w rooms | w days | Nonzero | Train MSE | Price of new house |
|---|---|---|---|---|---|---|
| OLS (α = 0) | 51.33 | 1.91 | −1.00 | 3 | 11.8 | 331.7 |
| Lasso (ρ = 1) | 49.19 | 0 | 0 | 1 | 28.9 | 331.0 |
| Elastic Net (ρ = 0.9) | 22.04 | 19.20 | 0 | 2 | 181.6 | 304.6 |
| Ridge-like (ρ = 0) | 8.95 | 8.65 | −0.14 | 3 | 1296.4 | 300.8 |
Why does Lasso drop rooms? Its fit is , which leaves rooms with . That is just under the threshold of 4, so rooms stays at zero. With , Lasso drops rooms for every . On the new house, which has a large floor area but few rooms, the two models disagree by 26k: Lasso only sees the size, while Elastic Net weighs both signals. The models are also not equally stable. Refit on each of the 8 leave-one-out subsets: Lasso keeps rooms in only 3 of 8 refits, and its size weight swings from 40.4 to 52.6. Elastic Net keeps rooms in all 8.
Things to try. Drag ρ from 1 down to 0 at α = 4: the diamond's corner rounds off, and the solution slides from the size axis toward the diagonal . Pick Lasso and sweep α: rooms gets a small weight at small α but leaves the model at α ≈ 3.3. With ρ = 0.9, rooms and size stay together along the whole path. With coordinate descent on, compare ρ = 0.9 with ρ = 0.5 (6 sweeps). A larger λ₂ divides each correction by more, so the zigzag dies out faster.
The algorithm, step by step
Pathwise coordinate descent (glmnet / scikit-learn)
- 1Prepare. Standardize each column of and centre . Remember , and so you can convert the weights back to original units afterwards.
- 2Start at the top of the path. is the smallest at which every weight is zero (it requires ). Make a grid of about 100 values down to , spaced on a log scale.
- 3Warm start. For each in decreasing order, start from the previous solution, which is already close.
- 4Sweep. For , compute , set , and update the residual by . Cost: per feature, so per sweep.
- 5Stop when the largest weight change in a sweep, or the duality gap in scikit-learn, falls below the tolerance. Then move on to the next .
- 6Choose by cross-validation over the path, then refit on all the data.
Complexity
- One sweep
- with residual updates, or with "covariance updates" after an precomputation (useful when )
- Whole path
- warm starts plus the strong-rule screening of inactive features make 100 values of cost only a few times a single fit
- Convergence
- each correction contracts by about between correlated features, so the L2 part improves conditioning: 15 sweeps at ρ = 0.9 versus 6 at ρ = 0.5 in the example
- Memory
- for the data and for the weights; sparse is supported
Variants and tuning
| Knob / variant | Effect | Practical advice |
|---|---|---|
alpha | Raises both the threshold λ₁ and the divisor 1+λ₂: fewer features, smaller weights | Search it on a log grid with ElasticNetCV; the path is cheap |
l1_ratio ρ | Near 1: Lasso-like and sparse. Near 0: Ridge-like and dense | Try [.1, .5, .7, .9, .95, .99, 1]; good values are often close to 1 |
| Rescaled Elastic Net | Multiply the naive by to undo the double shrinkage | Example: (22.04, 19.20) → (30.85, 26.88). Equivalently, refit OLS on the selected features (relaxed fit) |
| Adaptive Elastic Net | Per-feature L1 weights taken from an initial fit | Better feature-selection consistency; needs a good first-stage estimate |
| Group Lasso / sparse group | Penalizes predefined groups (e.g. one-hot levels) as units | Use it when you already know the groups; Elastic Net finds them from correlation |
| Elastic-net logistic / Cox / multi-task | The same penalty on another loss | LogisticRegression(penalty="elasticnet", solver="saga"), glmnet, MultiTaskElasticNet |
In code
1import numpy as np23def soft_threshold(v, t):4 return np.sign(v) * np.maximum(np.abs(v) - t, 0.0)56def elastic_net(X, y, alpha=1.0, l1_ratio=0.5, tol=1e-8, max_sweeps=10_000):7 """Minimise (1/2n)||y - Xw - b||^2 + alpha*l1_ratio*||w||_1 + alpha*(1-l1_ratio)/2*||w||^2."""8 n, p = X.shape9 mu, sd = X.mean(0), X.std(0)10 Z = (X - mu) / sd # standardise: the penalty must see equal scales11 yc = y - y.mean() # the intercept is never penalised12 l1, l2 = alpha * l1_ratio, alpha * (1 - l1_ratio)13 z2 = (Z ** 2).sum(0) / n # = 1 for standardised columns14 w = np.zeros(p)15 resid = yc.copy() # resid = yc - Z @ w, kept up to date16 for sweep in range(1, max_sweeps + 1):17 biggest = 0.018 for j in range(p):19 rho_j = Z[:, j] @ resid / n + z2[j] * w[j] # partial-residual correlation20 new = soft_threshold(rho_j, l1) / (z2[j] + l2)21 resid -= Z[:, j] * (new - w[j])22 biggest = max(biggest, abs(new - w[j]))23 w[j] = new24 if biggest < tol:25 break26 coef = w / sd # back to original units27 intercept = y.mean() - coef @ mu28 return coef, intercept, w, sweep2930X = np.array([[50, 2, 30], [60, 2, 40], [70, 3, 20], [80, 3, 15],31 [90, 4, 45], [100, 4, 10], [110, 5, 20], [120, 5, 45]], float)32y = np.array([215, 240, 265, 285, 315, 340, 355, 375], float)3334for name, ratio in [("elastic net", 0.9), ("lasso", 1.0)]:35 coef, b, w_std, sweeps = elastic_net(X, y, alpha=4.0, l1_ratio=ratio)36 print(f"{name:11s} standardised w = {np.round(w_std, 2)} sweeps = {sweeps}")37 print(f"{'':11s} price = {b:.2f} + {np.round(coef, 4)} . x -> new house: {b + coef @ [100, 3, 25]:.1f}")3839# elastic net standardised w = [22.04 19.2 -0. ] sweeps = 3140# price = 156.89 + [ 0.9618 17.175 -0. ] . x -> new house: 304.641# lasso standardised w = [49.19 0. -0. ] sweeps = 242# price = 116.27 + [ 2.1469 0. -0. ] . x -> new house: 331.01import numpy as np2from sklearn.pipeline import make_pipeline3from sklearn.preprocessing import StandardScaler4from sklearn.linear_model import ElasticNet, Lasso, ElasticNetCV56X = np.array([[50, 2, 30], [60, 2, 40], [70, 3, 20], [80, 3, 15],7 [90, 4, 45], [100, 4, 10], [110, 5, 20], [120, 5, 45]], float)8y = np.array([215, 240, 265, 285, 315, 340, 355, 375], float)9new = np.array([[100, 3, 25]])1011for name, model in [("elastic net", ElasticNet(alpha=4.0, l1_ratio=0.9)),12 ("lasso", Lasso(alpha=4.0))]:13 pipe = make_pipeline(StandardScaler(), model).fit(X, y)14 m = pipe[-1]15 print(f"{name:11s} w = {np.round(m.coef_, 2)} b = {m.intercept_:.2f} "16 f"new house = {pipe.predict(new)[0]:.1f}")17# elastic net w = [22.04 19.2 -0. ] b = 298.75 new house = 304.618# lasso w = [49.19 0. -0. ] b = 298.75 new house = 331.01920# On real data, choose alpha AND l1_ratio by cross-validation:21# cv = make_pipeline(StandardScaler(),22# ElasticNetCV(l1_ratio=[0.1, 0.5, 0.7, 0.9, 0.95, 0.99, 1.0],23# alphas=100, cv=5)).fit(X_train, y_train)24# print("CV picked alpha =", round(cv[-1].alpha_, 3), " l1_ratio =", cv[-1].l1_ratio_)Use when
- +You want a sparse, interpretable linear model, but the features come in correlated groups (genes, pixels, word n-grams, lagged sensors).
- +: Elastic Net can select more than features, and it stays stable where Lasso flips between near-duplicates.
- +You cannot decide between Lasso and Ridge. Cross-validating ρ covers both as special cases.
- +You need a fast, convex model with a single global optimum and a cheap full regularization path.
Avoid when
- −The relationship is strongly non-linear or has interactions. Use trees or boosting (see Gradient Boosting Regression), or add basis features first.
- −You need the smallest possible feature set and the features are nearly independent. Plain Lasso is simpler and selects fewer features.
- −You need unbiased effect sizes or p-values. Both penalties bias weights toward 0, so refit or use inference methods designed for penalized models.
- −Two hyperparameters to tune is a real cost on very small datasets, where CV is noisy.
Where it's used
Genomics
Selecting groups of co-expressed genes that predict a phenotype when there are thousands of genes and a few hundred samples. This was the original motivation for the method.
Quant finance
Factor models and index tracking with many correlated returns: the model picks a sparse basket while correlated assets share weight.
Text regression
Predicting ratings or prices from bag-of-words features, where synonyms and overlapping n-grams are highly correlated.
Neuroimaging & spectroscopy
Neighbouring voxels or wavelengths are nearly collinear. Elastic Net selects contiguous regions instead of isolated single pixels.
Check yourself
In scikit-learn, ElasticNet(alpha=2, l1_ratio=0.25) uses which L1 and L2 strengths?
Check yourself
A standardized feature has partial-residual correlation z = 5. With λ₁ = 2 and λ₂ = 0.5, one coordinate-descent update sets its weight to…
Check yourself
You duplicate the size column, so two identical features have r = 1. What do Lasso and Elastic Net (α = 1, ρ = 0.5) do?
Check yourself
Why can Elastic Net select more than n features when p > n, while Lasso cannot?
Remember
- ✓Objective: . ρ = 1 gives Lasso and ρ = 0 gives Ridge.
- ✓Coordinate update : the L1 part sets a threshold that produces zeros, and the L2 part divides every surviving weight by the same factor.
- ✓Grouping effect: . The L2 term stops the denominator from collapsing, so correlated features share weight. In the example, Lasso gives (49.19, 0) and Elastic Net gives (22.04, 19.20).
- ✓Elastic Net is a Lasso on augmented data, so it can select more than features and any Lasso solver can fit it.
- ✓Standardize inside a pipeline, tune α and ρ together with ElasticNetCV, and watch for double shrinkage.