Ridge, Lasso, and Elastic Net in Python: The Math of Regularization Explained

Regularization is a technique that penalizes overly large feature weights during model training, preventing linear models from overfitting to noise in the training data. Ridge (L2), Lasso (L1), and Elastic Net are the three primary regularization algorithms used in regression modeling.

Imagine grading a student's essay. If you tell them: "Use as many complex vocabulary words as you want — no penalties," they will stuff every sentence with unnecessary jargon to sound smart, resulting in a convoluted essay that confuses readers (overfitting). But if you say: "Every complex word costs you a small penalty — use them only if they are genuinely necessary," the student becomes deliberate, keeping only what truly adds meaning. That is exactly how regularization restrains regression weights.

1. Why Regularization Is Necessary (Combating Overfitting)

Standard Ordinary Least Squares (OLS) regression minimizes only the training prediction error. When a dataset contains many features, noise, or correlated columns (multicollinearity), OLS assigns gigantic positive and negative weights to cancel out noise, causing the model to collapse when making predictions on unseen test data.

Regularization adds a second objective: keep model weights as small as possible. A feature can only maintain a large weight if it improves predictions enough to justify its penalty cost.

2. Ridge vs. Lasso vs. Elastic Net: The Decision Matrix

3. The Geometric Intuition: Why Lasso Zeroes Out Weights

Why does Lasso (L1) force weights to exactly zero while Ridge (L2) only shrinks them?

• Ridge (L2 Circle Constraint $\sum w_j^2 \le C$): The constraint region is a smooth circle. The elliptical contours of the MSE loss function touch the circle at smooth points off the axes, shrinking weights evenly without landing on the axes ($w_j \neq 0$).
• Lasso (L1 Diamond Constraint $\sum |w_j| \le C$): The constraint region is a diamond with sharp corners located directly on the coordinate axes. The elliptical MSE loss contours almost always intersect the diamond at one of these sharp corners, forcing one or more weights to exactly zero ($w_j = 0$).

4. The Mathematical Formulations

1. Baseline OLS Loss Function

$$J(w) = \frac{1}{2m} \sum_{i=1}^{m} (\hat{y}^{(i)} - y^{(i)})^2$$

2. Ridge Regression (L2 Regularization)

$$J_{\text{ridge}}(w) = \frac{1}{2m} \sum_{i=1}^{m} (\hat{y}^{(i)} - y^{(i)})^2 + \lambda \sum_{j=1}^{n} w_j^2$$

Ridge Gradient Update Rule:

$$w_j := w_j - \alpha \left[ \frac{1}{m}\sum_{i=1}^m (\hat{y}^{(i)} - y^{(i)})x_j^{(i)} + 2\lambda w_j \right]$$

3. Lasso Regression (L1 Regularization)

$$J_{\text{lasso}}(w) = \frac{1}{2m} \sum_{i=1}^{m} (\hat{y}^{(i)} - y^{(i)})^2 + \lambda \sum_{j=1}^{n} |w_j|$$

Lasso Subgradient:

$$\frac{\partial J}{\partial w_j} = \frac{1}{m}\sum_{i=1}^m (\hat{y}^{(i)} - y^{(i)})x_j^{(i)} + \lambda \cdot \text{sign}(w_j)$$

4. Elastic Net (L1 + L2 Hybrid)

$$J_{\text{elastic}}(w) = \frac{1}{2m} \sum_{i=1}^{m} (\hat{y}^{(i)} - y^{(i)})^2 + \lambda \left[ r \sum_{j=1}^{n} |w_j| + (1-r) \sum_{j=1}^{n} w_j^2 \right]$$

5. Worked Example (The Mathematical Pull Toward Zero)

Let's compare the derivative penalty pull for a weight with $\lambda = 0.5$ as $w$ shrinks:

6. Code: Python Implementation From Scratch & Scikit-Learn

1. From-Scratch Regularized Gradient Descent (NumPy)

import numpy as np

np.random.seed(42)
m, n = 100, 3

# True relationship: y = 3*x1 + 0*x2 - 1.5*x3 (feature 2 is useless noise)
X = np.random.randn(m, n)
true_w = np.array([3.0, 0.0, -1.5])
y = X @ true_w + np.random.randn(m) * 0.5

def train_regularized(X, y, penalty="ridge", alpha=0.05, lam=0.1, r=0.5, epochs=2000):
    m, n = X.shape
    w = np.zeros(n)
    b = 0.0
    
    for _ in range(epochs):
        y_pred = X @ w + b
        error = y_pred - y
        
        grad_w = (1 / m) * (X.T @ error)
        grad_b = (1 / m) * np.sum(error)
        
        # Apply specific regularization penalty gradient
        if penalty == "ridge":
            grad_w += 2 * lam * w
        elif penalty == "lasso":
            grad_w += lam * np.sign(w)
        elif penalty == "elastic":
            grad_w += lam * (r * np.sign(w) + (1 - r) * 2 * w)
            
        w -= alpha * grad_w
        b -= alpha * grad_b
        
    return w, b

w_ridge, _ = train_regularized(X, y, penalty="ridge", lam=0.3)
w_lasso, _ = train_regularized(X, y, penalty="lasso", lam=0.1)
w_elastic, _ = train_regularized(X, y, penalty="elastic", lam=0.2, r=0.5)

print("Ridge weights:  ", np.round(w_ridge, 3))
print("Lasso weights:  ", np.round(w_lasso, 3))
print("Elastic weights:", np.round(w_elastic, 3))

2. Production Scikit-Learn with Automated Cross-Validation

from sklearn.linear_model import RidgeCV, LassoCV, ElasticNetCV
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

# 1. RidgeCV (L2 with automated optimal alpha search)
ridge_pipe = make_pipeline(StandardScaler(), RidgeCV(alphas=[0.01, 0.1, 1.0, 10.0]))
ridge_pipe.fit(X, y)
print("Ridge Optimal Alpha:", ridge_pipe.named_steps["ridgecv"].alpha_)
print("Ridge Learned Weights:", np.round(ridge_pipe.named_steps["ridgecv"].coef_, 3))

# 2. LassoCV (L1 feature selection)
lasso_pipe = make_pipeline(StandardScaler(), LassoCV(cv=5, random_state=42))
lasso_pipe.fit(X, y)
print("Lasso Optimal Alpha:", np.round(lasso_pipe.named_steps["lassocv"].alpha_, 4))
print("Lasso Learned Weights:", np.round(lasso_pipe.named_steps["lassocv"].coef_, 3))

# 3. ElasticNetCV (L1 + L2 blend)
elastic_pipe = make_pipeline(StandardScaler(), ElasticNetCV(l1_ratio=[0.1, 0.5, 0.9], cv=5, random_state=42))
elastic_pipe.fit(X, y)
print("ElasticNet Optimal l1_ratio:", elastic_pipe.named_steps["elasticnetcv"].l1_ratio_)
print("ElasticNet Learned Weights:", np.round(elastic_pipe.named_steps["elasticnetcv"].coef_, 3))

7. Common Pitfalls & Best Practices

• Always Scale Features First: Regularization penalizes the raw magnitude of weights. If feature 1 ranges $0–1$ and feature 2 ranges $0–100,000$, unscaled regularization will unfairly crush feature 2. Always use StandardScaler.
• Never Regularize the Bias ($b$): The intercept represents the global baseline offset, not feature importance. Penalizing $b$ damages model calibration.
• Tune via Cross-Validation: Never guess $\lambda$ (or $\alpha$). Use RidgeCV, LassoCV, and ElasticNetCV to select optimal penalty strengths.

Summary

• Regularization prevents overfitting by penalizing large weights in the cost function.
• Ridge (L2) shrinks all weights smoothly, handling multicollinearity effectively.
• Lasso (L1) zeroes out irrelevant features entirely, producing sparse, interpretable models.
• Elastic Net combines L1 and L2 penalties, providing stability for correlated feature clusters.