---
title: Ridge, Lasso, and Elastic Net in Python: The Math of Regularization Explained
source: https://app.sythra.ai/learn/machine-learning/ridge-lasso-elastic-net-regularization-python
topic: Machine Learning
updated: 2026-08-28
publisher: Sythra (https://app.sythra.ai)
---

# Ridge, Lasso, and Elastic Net in Python: The Math of Regularization Explained

Regularization prevents overfitting in linear regression by adding a penalty term to the cost function: Ridge (L2) squares weights to shrink them smoothly, Lasso (L1) uses absolute values to zero out irrelevant features, and Elastic Net blends both.

_Source: [https://app.sythra.ai/learn/machine-learning/ridge-lasso-elastic-net-regularization-python](https://app.sythra.ai/learn/machine-learning/ridge-lasso-elastic-net-regularization-python) — free to read on Sythra._

## Key points

- Explains how regularization controls overfitting and multicollinearity.
- Breaks down the geometric diamond vs. circle constraint intuition.
- Derives mathematical loss functions and subgradients for Ridge, Lasso, and Elastic Net.
- Provides from-scratch NumPy implementations alongside RidgeCV, LassoCV, and ElasticNetCV.

**Regularization** is a technique that penalizes overly large feature weights during model training, preventing linear models from overfitting to noise in the training data. **Ridge (L2)**, **Lasso (L1)**, and **Elastic Net** are the three primary regularization algorithms used in regression modeling.

Imagine grading a student's essay. If you tell them: _"Use as many complex vocabulary words as you want — no penalties,"_ they will stuff every sentence with unnecessary jargon to sound smart, resulting in a convoluted essay that confuses readers (overfitting). But if you say: _"Every complex word costs you a small penalty — use them only if they are genuinely necessary,"_ the student becomes deliberate, keeping only what truly adds meaning. That is exactly how regularization restrains regression weights.

## 1. Why Regularization Is Necessary (Combating Overfitting)

Standard Ordinary Least Squares (OLS) regression minimizes only the training prediction error. When a dataset contains many features, noise, or correlated columns (**multicollinearity**), OLS assigns gigantic positive and negative weights to cancel out noise, causing the model to collapse when making predictions on unseen test data.

Regularization adds a second objective: **keep model weights as small as possible**. A feature can only maintain a large weight if it improves predictions enough to justify its penalty cost.

## 2. Ridge vs. Lasso vs. Elastic Net: The Decision Matrix

| Model | Penalty Type | Weight Behavior | Multicollinearity Handling | Best Used For |
| --- | --- | --- | --- | --- |
| **Ridge Regression** | L2 Norm ($\lambda \sum w_j^2$) | Shrinks weights smoothly toward zero (never exactly 0). | **Stable:** Groups correlated features and shrinks them together. | When many features all contribute small-to-moderate predictive signals. |
| **Lasso Regression** | L1 Norm ($\lambda \sum \|w_j\|$) | Forces irrelevant weights to **exactly zero** (sparse model). | **Unstable:** Arbitrarily selects 1 correlated feature and zeroes the rest. | Automatic feature selection when only a few features truly matter. |
| **Elastic Net** | L1 + L2 Blend ($\lambda [r\sum\|w_j\| + (1-r)\sum w_j^2]$) | Selects groups of correlated features while enforcing sparsity. | **Best of both:** Maintains grouped feature selection without dropping clusters. | High-dimensional data ($p > n$) or datasets with strongly correlated feature clusters. |

## 3. The Geometric Intuition: Why Lasso Zeroes Out Weights

Why does Lasso (L1) force weights to exactly zero while Ridge (L2) only shrinks them?

- **Ridge (L2 Circle Constraint $\sum w_j^2 \le C$):** The constraint region is a smooth circle. The elliptical contours of the MSE loss function touch the circle at smooth points off the axes, shrinking weights evenly without landing on the axes ($w_j \neq 0$).
- **Lasso (L1 Diamond Constraint $\sum |w_j| \le C$):** The constraint region is a diamond with sharp corners located directly on the coordinate axes. The elliptical MSE loss contours almost always intersect the diamond at one of these sharp corners, forcing one or more weights to **exactly zero** ($w_j = 0$).

## 4. The Mathematical Formulations

### 1. Baseline OLS Loss Function

$$J(w) = \frac{1}{2m} \sum_{i=1}^{m} (\hat{y}^{(i)} - y^{(i)})^2$$

### 2. Ridge Regression (L2 Regularization)

$$J_{\text{ridge}}(w) = \frac{1}{2m} \sum_{i=1}^{m} (\hat{y}^{(i)} - y^{(i)})^2 + \lambda \sum_{j=1}^{n} w_j^2$$

**Ridge Gradient Update Rule:**

$$w_j := w_j - \alpha \left[ \frac{1}{m}\sum_{i=1}^m (\hat{y}^{(i)} - y^{(i)})x_j^{(i)} + 2\lambda w_j \right]$$

### 3. Lasso Regression (L1 Regularization)

$$J_{\text{lasso}}(w) = \frac{1}{2m} \sum_{i=1}^{m} (\hat{y}^{(i)} - y^{(i)})^2 + \lambda \sum_{j=1}^{n} |w_j|$$

**Lasso Subgradient:**

$$\frac{\partial J}{\partial w_j} = \frac{1}{m}\sum_{i=1}^m (\hat{y}^{(i)} - y^{(i)})x_j^{(i)} + \lambda \cdot \text{sign}(w_j)$$

### 4. Elastic Net (L1 + L2 Hybrid)

$$J_{\text{elastic}}(w) = \frac{1}{2m} \sum_{i=1}^{m} (\hat{y}^{(i)} - y^{(i)})^2 + \lambda \left[ r \sum_{j=1}^{n} |w_j| + (1-r) \sum_{j=1}^{n} w_j^2 \right]$$

## 5. Worked Example (The Mathematical Pull Toward Zero)

Let's compare the derivative penalty pull for a weight with $\lambda = 0.5$ as $w$ shrinks:

| Method | Derivative Formula | Pull at $w = 4.0$ | Pull at $w = 0.1$ | Result at Small Weights |
| --- | --- | --- | --- | --- |
| **Ridge ($L_2$)** | $2\lambda w$ | $2(0.5)(4) = 4.0$ | $2(0.5)(0.1) = 0.1$ | Pull vanishes near zero — weight idles near zero but never reaches 0. |
| **Lasso ($L_1$)** | $\lambda \cdot \text{sign}(w)$ | $0.5$ | $0.5$ | Constant pull toward zero — drives weight directly into 0. |

## 6. Code: Python Implementation From Scratch & Scikit-Learn

### 1. From-Scratch Regularized Gradient Descent (NumPy)

```python
import numpy as np

np.random.seed(42)
m, n = 100, 3

# True relationship: y = 3*x1 + 0*x2 - 1.5*x3 (feature 2 is useless noise)
X = np.random.randn(m, n)
true_w = np.array([3.0, 0.0, -1.5])
y = X @ true_w + np.random.randn(m) * 0.5

def train_regularized(X, y, penalty="ridge", alpha=0.05, lam=0.1, r=0.5, epochs=2000):
    m, n = X.shape
    w = np.zeros(n)
    b = 0.0
    
    for _ in range(epochs):
        y_pred = X @ w + b
        error = y_pred - y
        
        grad_w = (1 / m) * (X.T @ error)
        grad_b = (1 / m) * np.sum(error)
        
        # Apply specific regularization penalty gradient
        if penalty == "ridge":
            grad_w += 2 * lam * w
        elif penalty == "lasso":
            grad_w += lam * np.sign(w)
        elif penalty == "elastic":
            grad_w += lam * (r * np.sign(w) + (1 - r) * 2 * w)
            
        w -= alpha * grad_w
        b -= alpha * grad_b
        
    return w, b

w_ridge, _ = train_regularized(X, y, penalty="ridge", lam=0.3)
w_lasso, _ = train_regularized(X, y, penalty="lasso", lam=0.1)
w_elastic, _ = train_regularized(X, y, penalty="elastic", lam=0.2, r=0.5)

print("Ridge weights:  ", np.round(w_ridge, 3))
print("Lasso weights:  ", np.round(w_lasso, 3))
print("Elastic weights:", np.round(w_elastic, 3))
```

### 2. Production Scikit-Learn with Automated Cross-Validation

```python
from sklearn.linear_model import RidgeCV, LassoCV, ElasticNetCV
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

# 1. RidgeCV (L2 with automated optimal alpha search)
ridge_pipe = make_pipeline(StandardScaler(), RidgeCV(alphas=[0.01, 0.1, 1.0, 10.0]))
ridge_pipe.fit(X, y)
print("Ridge Optimal Alpha:", ridge_pipe.named_steps["ridgecv"].alpha_)
print("Ridge Learned Weights:", np.round(ridge_pipe.named_steps["ridgecv"].coef_, 3))

# 2. LassoCV (L1 feature selection)
lasso_pipe = make_pipeline(StandardScaler(), LassoCV(cv=5, random_state=42))
lasso_pipe.fit(X, y)
print("Lasso Optimal Alpha:", np.round(lasso_pipe.named_steps["lassocv"].alpha_, 4))
print("Lasso Learned Weights:", np.round(lasso_pipe.named_steps["lassocv"].coef_, 3))

# 3. ElasticNetCV (L1 + L2 blend)
elastic_pipe = make_pipeline(StandardScaler(), ElasticNetCV(l1_ratio=[0.1, 0.5, 0.9], cv=5, random_state=42))
elastic_pipe.fit(X, y)
print("ElasticNet Optimal l1_ratio:", elastic_pipe.named_steps["elasticnetcv"].l1_ratio_)
print("ElasticNet Learned Weights:", np.round(elastic_pipe.named_steps["elasticnetcv"].coef_, 3))
```

## 7. Common Pitfalls & Best Practices

- **Always Scale Features First:** Regularization penalizes the raw magnitude of weights. If feature 1 ranges $0–1$ and feature 2 ranges $0–100,000$, unscaled regularization will unfairly crush feature 2. Always use `StandardScaler`.
- **Never Regularize the Bias ($b$):** The intercept represents the global baseline offset, not feature importance. Penalizing $b$ damages model calibration.
- **Tune via Cross-Validation:** Never guess $\lambda$ (or $\alpha$). Use `RidgeCV`, `LassoCV`, and `ElasticNetCV` to select optimal penalty strengths.

## Summary

- Regularization prevents overfitting by penalizing large weights in the cost function.
- **Ridge (L2)** shrinks all weights smoothly, handling multicollinearity effectively.
- **Lasso (L1)** zeroes out irrelevant features entirely, producing sparse, interpretable models.
- **Elastic Net** combines L1 and L2 penalties, providing stability for correlated feature clusters.

## FAQ

### What is the difference between Ridge (L2) and Lasso (L1) regression?

Ridge adds a penalty proportional to the square of the weights (sum w^2), shrinking all weights smoothly without zeroing them out. Lasso adds a penalty proportional to the absolute value (sum |w|), forcing irrelevant feature weights to exactly zero for automatic feature selection.

### Why does Lasso produce sparse weights with zeros while Ridge does not?

Geometrically, Lasso constraint regions are diamonds with sharp corners situated on the coordinate axes, causing elliptical error contours to intersect at the axes (w = 0). Algebraically, the derivative of |w| is a constant sign(w), maintaining a constant pull into zero.

### When should I use Elastic Net instead of Ridge or Lasso?

Use Elastic Net when working with high-dimensional data (p > n) or datasets where multiple features are strongly correlated. Elastic Net avoids Lasso's erratic selection while still producing sparse models.

### Why is feature scaling mandatory before running Ridge or Lasso regression?

Regularization penalizes weight magnitudes uniformly. Features on larger numerical scales naturally have smaller weights and are unfairly over-penalized unless all features are standardized to mean 0 and variance 1.

---

Written by Sythra — Learn machine learning by building. Practice this topic with Sythra's AI tutor: https://app.sythra.ai/pricing