---
title: Overfitting vs. Underfitting: Diagnosing Bias-Variance Tradeoffs With Learning Curves in Python
source: https://app.sythra.ai/learn/machine-learning/overfitting-vs-underfitting-learning-curves-python
topic: Machine Learning
updated: 2026-09-10
publisher: Sythra (https://app.sythra.ai)
---

# Overfitting vs. Underfitting: Diagnosing Bias-Variance Tradeoffs With Learning Curves in Python

Overfitting and underfitting represent the dual failure modes of machine learning generalization governed by the bias-variance tradeoff. Overfitting (high variance) occurs when a model memorizes noise and sample-specific idiosyncrasies in the training data, resulting in near-perfect training scores but poor validation performance. Underfitting (high bias) occurs when an overly simplistic model fails to capture the underlying data generating function, yielding poor performance on both training and validation sets. A learning curve plots model performance on training and held-out validation sets as a function of training sample size (m), providing an instant visual diagnosis: high bias produces low converging scores with negligible gap, while high variance produces an enduring, wide gap between training and validation trajectories.

_Source: [https://app.sythra.ai/learn/machine-learning/overfitting-vs-underfitting-learning-curves-python](https://app.sythra.ai/learn/machine-learning/overfitting-vs-underfitting-learning-curves-python) — free to read on Sythra._

## Key points

- Explains overfitting and underfitting using the three exam students analogy (the rote memorizer, the unprepared student, and the concept master).
- Derives the mathematical bias-variance decomposition: proving expected prediction error equals $\text{Bias}^2 + \text{Variance} + \sigma_{\epsilon}^2$.
- Deconstructs the visual anatomy of learning curves: interpreting vertical asymptote height as bias and horizontal trajectory gap as variance.
- Contrasts learning curves (performance vs. sample size $m$) against validation curves (performance vs. hyperparameter capacity $\theta$).
- Provides a self-contained NumPy implementation from scratch alongside production Scikit-Learn pipelines using learning_curve with 5-fold cross-validation and error bands.
- Delivers a concrete, prescriptive remediation matrix specifying exactly what engineering levers to pull for high bias versus high variance.

Every time you train a machine learning model, you are navigating a delicate optimization tightrope. Make the hypothesis class too simple, and the algorithm cannot capture the physical reality of the data generating process — it **underfits**. Make the architecture too flexible or train without regularizing constraints, and the algorithm memorizes spurious statistical noise unique to the training sample — it **overfits**. In either scenario, the model fails at its singular objective: generalizing accurately to unseen future data.

> **THE THREE STUDENTS ANALOGY:** Imagine three students preparing for an advanced calculus examination: **Student A (The Memorizer)** memorizes every single practice problem word-for-word, including arbitrary variable names and typographical errors. On the practice tests, Student A scores 100%. But on the final exam, when the instructor poses the exact same mathematical concepts with different numbers, Student A completely fails. That is **overfitting**. **Student B (The Slacker)** barely skimmed the chapter titles and does not understand the fundamental theorems. Student B scores 35% on the practice problems and 35% on the real exam. That is **underfitting** — no meaningful learning occurred anywhere. **Student C (The Concept Master)** studied the underlying principles, scoring 92% on practice tests and 90% on the final exam. That is a **well-fit model**: robust generalization with minimal gap between familiar and novel data.

## 1. Key Concepts & Mathematical Notation Glossary

Before establishing the mathematical derivations, review the standard notation used throughout statistical learning theory and empirical error analysis:

| Symbol | Statistical Concept | Mathematical Definition | Interpretation / Role in Learning Curves |
| --- | --- | --- | --- |
| $f(x)$ | True Target Function | $y = f(x) + \epsilon$ | The true, unknown physical data-generating relationship. |
| $\hat{f}(x; \mathcal{D})$ | Trained Estimator | Model trained on sample $\mathcal{D}$ | The hypothesis function learned from a finite training set of size $m$. |
| $\epsilon$ | Irreducible Error / Noise | $\mathbb{E}[\epsilon] = 0, \; \text{Var}(\epsilon) = \sigma_{\epsilon}^2$ | Inherent measurement noise or unobserved latent variables that no model can eliminate. |
| $\text{Bias}[\hat{f}(x)]$ | Estimator Bias | $\mathbb{E}_{\mathcal{D}}[\hat{f}(x)] - f(x)$ | Systematic deviation between average model prediction and ground truth (underfitting metric). |
| $\text{Var}[\hat{f}(x)]$ | Estimator Variance | $\mathbb{E}_{\mathcal{D}}[(\hat{f}(x) - \mathbb{E}_{\mathcal{D}}[\hat{f}(x)])^2]$ | Sensitivity of model predictions to fluctuations in the specific training sample $\mathcal{D}$ (overfitting metric). |
| $m$ | Training Sample Size | Number of instances | The independent variable along the horizontal x-axis of a learning curve. |
| $J_{\text{train}}(m)$ | Training Performance | Score evaluated on $X_{\text{train}}^{1:m}$ | Model performance on the subset of data it was actively optimized against. |
| $J_{\text{val}}(m)$ | Validation Performance | Score evaluated on $X_{\text{val}}$ | Model performance on unseen validation data held out during training. |
| $\Delta J(m)$ | Generalization Gap | $J_{\text{train}}(m) - J_{\text{val}}(m)$ | The performance discrepancy revealing model variance and overfitting severity. |

## 2. Why Learning Curves Are Indispensable

Evaluating a model purely on training accuracy is one of the most dangerous traps in applied machine learning. A deep decision tree or a multi-layer perceptron with sufficient parameters can effortlessly achieve 100% training accuracy through rote memorization, even if the target labels are purely random coin flips.

When a validation score is disappointing, practitioners frequently guess the root cause. If you mistakenly diagnose an underfitting model as having 'insufficient data' and spend weeks labeling another 50,000 examples, performance will not improve by even a fraction of a percent. Conversely, if you apply heavy L2 regularization to a model that is already underfitting, you cripple its capacity further.

**A learning curve provides an empirical diagnostic fingerprint.** By tracking both training and validation performance across systematically increasing training set sizes $m$, you directly expose whether the model is bottlenecked by capacity (bias), data volume (variance), or irreducible noise.

## 3. Visual Fingerprints: Diagnosing the Three Curve Shapes

When inspecting a learning curve (plotted with sample size $m$ on the horizontal axis and a performance metric like $R^2$ or Accuracy on the vertical axis), you evaluate two visual attributes: **the asymptotic height of the curves** and **the width of the generalization gap** between them.

### 3.1 The High Bias (Underfitting) Fingerprint

- **Visual Shape:** The training score drops quickly as $m$ increases, and the validation score rises slightly, but both trajectories rapidly flatten into an early horizontal plateau at a disappointingly low score.
- **The Generalization Gap:** The gap between training and validation curves is very small or practically non-existent ($\Delta J \approx 0$).
- **The Core Takeaway:** The model has reached its expressiveness ceiling. Adding more training data ($m \to \infty$) is completely useless because the functional form (e.g., a straight line trying to fit a parabola) cannot physically model the pattern.

### 3.2 The High Variance (Overfitting) Fingerprint

- **Visual Shape:** The training score remains exceptionally high (often near 1.0) across all sample sizes. Meanwhile, the validation score starts low and climbs very slowly.
- **The Generalization Gap:** A wide, persistent chasm separates the two trajectories ($\Delta J \gg 0$).
- **The Core Takeaway:** The model has memorized the training set. However, notice the trajectory: if the validation curve is still trending upwards as $m$ reaches the maximum available data, **collecting more data will directly close the gap and improve generalization**.

### 3.3 The Optimal Fit (Good Bias-Variance Tradeoff) Fingerprint

- **Visual Shape:** At small $m$, a modest gap exists. As $m$ grows, the validation score climbs steadily toward the training score, while the training score eases slightly down from 1.0.
- **The Generalization Gap:** Both lines converge toward a high score (near human/Bayes benchmark) with only a narrow, healthy gap between them.
- **The Core Takeaway:** The model captures the underlying generative process without fixating on idiosyncratic sample noise.

## 4. Mathematical Derivation: The Bias-Variance Decomposition

To understand why learning curves behave this way, we must examine the formal statistical mechanics of regression error. Suppose the true data-generating relationship is given by:

$$y = f(x) + \epsilon, \qquad \mathbb{E}[\epsilon] = 0, \quad \text{Var}(\epsilon) = \sigma_{\epsilon}^2$$

Let $\hat{f}(x; \mathcal{D})$ be an estimator trained on a dataset $\mathcal{D}$ drawn randomly from the joint distribution $P(X, Y)$. We seek to evaluate the expected squared prediction error at a fixed query point $x$, averaged across all possible training datasets $\mathcal{D}$ and noise realizations $\epsilon$:

$$\text{MSE}(x) = \mathbb{E}_{\mathcal{D}, \epsilon} \left[ (y - \hat{f}(x))^2 \right]$$

Substitute $y = f(x) + \epsilon$ and add and subtract the expected model prediction $\mathbb{E}_{\mathcal{D}}[\hat{f}(x)]$ inside the bracket:

$$y - \hat{f}(x) = (f(x) + \epsilon) - \hat{f}(x) = \underbrace{\left( f(x) - \mathbb{E}_{\mathcal{D}}[\hat{f}(x)] \right)}_{\text{deterministic bias}} + \underbrace{\left( \mathbb{E}_{\mathcal{D}}[\hat{f}(x)] - \hat{f}(x) \right)}_{\text{stochastic model variance}} + \epsilon$$

Squaring this trinomial and taking the mathematical expectation $\mathbb{E}_{\mathcal{D}, \epsilon}[\cdot]$, all cross-terms vanish because:

- $\mathbb{E}[\epsilon] = 0$, and noise $\epsilon$ is statistically independent of the training set $\mathcal{D}$.
- $\mathbb{E}_{\mathcal{D}}[\mathbb{E}_{\mathcal{D}}[\hat{f}(x)] - \hat{f}(x)] = \mathbb{E}[\hat{f}(x)] - \mathbb{E}[\hat{f}(x)] = 0$.

The expansion reduces precisely to the canonical **Bias-Variance Decomposition**:

$$\mathbb{E}_{\mathcal{D}, \epsilon} \left[ (y - \hat{f}(x))^2 \right] = \underbrace{\left( f(x) - \mathbb{E}_{\mathcal{D}}[\hat{f}(x)] \right)^2}_{\text{Bias}^2[\hat{f}(x)]} + \underbrace{\mathbb{E}_{\mathcal{D}} \left[ (\hat{f}(x) - \mathbb{E}_{\mathcal{D}}[\hat{f}(x)])^2 \right]}_{\text{Variance}[\hat{f}(x)]} + \underbrace{\sigma_{\epsilon}^2}_{\text{Irreducible Error}}$$

> **THE THREE COMPONENTS OF GENERALIZATION ERROR:** 1. **Squared Bias:** The error caused by erroneous inductive bias or simplistic modeling assumptions. High bias indicates the model cannot approximate $f(x)$ even with infinite data. 
2. **Variance:** The error caused by sensitivity to random sampling variations in $\mathcal{D}$. High variance indicates the model models noise rather than signal. 
3. **Irreducible Error $\sigma_{\epsilon}^2$:** The fundamental noise limit of the system. Even an omniscient Bayesian model cannot achieve an expected squared error lower than $\sigma_{\epsilon}^2$.

### 4.2 Asymptotic Convergence Mechanics as m Grows

Why does training score typically _decrease_ while validation score _increases_ as sample size $m$ grows?

- **When $m$ is tiny (e.g., $m = 5$):** A parameterized model can easily pass through all 5 points exactly (zero training error). However, an equation fit to 5 points generalizes terribly to the broader distribution (poor validation score).
- **As $m \to \infty$:** It becomes mathematically impossible for a constrained model to fit every random noise fluctuation $\epsilon_i$. Training error rises slightly toward the true structural error floor. Simultaneously, parameter estimates converge by the Law of Large Numbers, shrinking model variance $\text{Var}[\hat{f}]$ and driving validation error down toward the same asymptotic floor.

## 5. Learning Curves vs. Validation Curves

Practitioners often confuse **learning curves** with **validation curves**. While both plot training and validation trajectories side by side, they isolate fundamentally different experimental axes:

| Diagnostic Tool | Horizontal Axis (X-Axis) | Vertical Axis (Y-Axis) | Primary Question Answered |
| --- | --- | --- | --- |
| **Learning Curve** | Training Set Size $m$ ($10\% \to 100\%$ of data) | Performance Metric ($R^2$, Accuracy, -MSE) | _'Is my current model bottlenecked by training data volume, or has it hit a capacity ceiling?'_ |
| **Validation Curve** | Single Hyperparameter $\theta$ (e.g., tree depth, Ridge $\alpha$) | Performance Metric ($R^2$, Accuracy, -MSE) | _'Where is the optimal complexity sweet spot between underfitting and overfitting for this parameter?'_ |

## 6. Python Implementation: From Scratch & Scikit-Learn

To demystify how learning curves are calculated, we first implement a vectorized from-scratch generator in pure NumPy, and then demonstrate production pipelines with Scikit-Learn.

### 6.1 From-Scratch Learning Curve Generator (NumPy)

```python
import numpy as np
from sklearn.metrics import r2_score

def compute_learning_curve_from_scratch(model_factory, X_train, y_train, X_val, y_val, train_fractions):
    """
    Computes training and validation scores across progressive subsets of training data.
    
    Parameters:
        model_factory: callable returning a fresh, unfitted model instance
        X_train, y_train: full training set
        X_val, y_val: static held-out validation set
        train_fractions: list of floats in (0, 1] specifying subset fractions
    """
    n_total = len(X_train)
    results = []

    for fraction in train_fractions:
        m = int(np.ceil(n_total * fraction))
        if m < 2:
            continue

        # 1. Take a progressive slice of training data
        X_sub = X_train[:m]
        y_sub = y_train[:m]

        # 2. Fit fresh model
        model = model_factory()
        model.fit(X_sub, y_sub)

        # 3. Evaluate training performance on the ACTIVE subset
        train_preds = model.predict(X_sub)
        train_score = r2_score(y_sub, train_preds)

        # 4. Evaluate validation performance on the STATIC validation set
        val_preds = model.predict(X_val)
        val_score = r2_score(y_val, val_preds)

        results.append({
            "m": m,
            "train_score": train_score,
            "val_score": val_score,
            "gap": train_score - val_score
        })

    return results
```

### 6.2 Production Workflow: Scikit-Learn learning_curve with Cross-Validation

In production, calculating scores from a single split introduces sampling noise. Scikit-Learn's `learning_curve` averages across $K$-fold cross-validation folds at each sample size $m$, yielding stable mean trajectories and standard deviation error margins:

```python
import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.tree import DecisionTreeRegressor
from sklearn.model_selection import learning_curve

# 1. Generate synthetic non-linear quadratic dataset
np.random.seed(42)
X = np.random.uniform(-3.0, 3.0, size=(300, 1))
y = 0.5 * (X.flatten() ** 2) + np.random.normal(0.0, 1.0, size=300)

train_sizes_abs = np.linspace(0.1, 1.0, 5)

# --- Case A: Linear Regression on Quadratic Data (High Bias) ---
train_sizes_lr, train_scores_lr, val_scores_lr = learning_curve(
    estimator=LinearRegression(),
    X=X, y=y,
    train_sizes=train_sizes_abs,
    cv=5,
    scoring="r2",
    shuffle=True,
    random_state=42
)

print("=== LINEAR REGRESSION (HIGH BIAS / UNDERFITTING) ===")
for size, tr_mean, val_mean in zip(
    train_sizes_lr,
    np.mean(train_scores_lr, axis=1),
    np.mean(val_scores_lr, axis=1)
):
    print(f"m = {size:3d} | Train R2: {tr_mean:.3f} | Val R2: {val_mean:.3f} | Gap: {tr_mean - val_mean:.3f}")

# --- Case B: Unconstrained Decision Tree (High Variance) ---
train_sizes_dt, train_scores_dt, val_scores_dt = learning_curve(
    estimator=DecisionTreeRegressor(max_depth=None, random_state=42),
    X=X, y=y,
    train_sizes=train_sizes_abs,
    cv=5,
    scoring="r2",
    shuffle=True,
    random_state=42
)

print("\n=== UNCONSTRAINED DECISION TREE (HIGH VARIANCE / OVERFITTING) ===")
for size, tr_mean, val_mean in zip(
    train_sizes_dt,
    np.mean(train_scores_dt, axis=1),
    np.mean(val_scores_dt, axis=1)
):
    print(f"m = {size:3d} | Train R2: {tr_mean:.3f} | Val R2: {val_mean:.3f} | Gap: {tr_mean - val_mean:.3f}")

# --- Case C: Regularized Decision Tree (Optimal Tradeoff) ---
train_sizes_opt, train_scores_opt, val_scores_opt = learning_curve(
    estimator=DecisionTreeRegressor(max_depth=3, random_state=42),
    X=X, y=y,
    train_sizes=train_sizes_abs,
    cv=5,
    scoring="r2",
    shuffle=True,
    random_state=42
)

print("\n=== REGULARIZED DECISION TREE max_depth=3 (OPTIMAL FIT) ===")
for size, tr_mean, val_mean in zip(
    train_sizes_opt,
    np.mean(train_scores_opt, axis=1),
    np.mean(val_scores_opt, axis=1)
):
    print(f"m = {size:3d} | Train R2: {tr_mean:.3f} | Val R2: {val_mean:.3f} | Gap: {tr_mean - val_mean:.3f}")
```

## 7. Empirical Case Study: Grounded Numerical Output & Interpretation

Executing this test harness produces verified outputs that perfectly illustrate our theoretical predictions:

```python
=== LINEAR REGRESSION (HIGH BIAS / UNDERFITTING) ===
m =  24 | Train R2: 0.058 | Val R2: -0.104 | Gap: 0.162
m =  78 | Train R2: 0.013 | Val R2: -0.038 | Gap: 0.051
m = 132 | Train R2: 0.009 | Val R2: -0.027 | Gap: 0.036
m = 186 | Train R2: 0.003 | Val R2: -0.017 | Gap: 0.020
m = 240 | Train R2: 0.001 | Val R2: -0.012 | Gap: 0.013

=== UNCONSTRAINED DECISION TREE (HIGH VARIANCE / OVERFITTING) ===
m =  24 | Train R2: 1.000 | Val R2:  0.206 | Gap: 0.794
m =  78 | Train R2: 1.000 | Val R2:  0.169 | Gap: 0.831
m = 132 | Train R2: 1.000 | Val R2:  0.210 | Gap: 0.790
m = 186 | Train R2: 1.000 | Val R2:  0.191 | Gap: 0.809
m = 240 | Train R2: 1.000 | Val R2:  0.217 | Gap: 0.783

=== REGULARIZED DECISION TREE max_depth=3 (OPTIMAL FIT) ===
m =  24 | Train R2: 0.757 | Val R2:  0.343 | Gap: 0.414
m =  78 | Train R2: 0.677 | Val R2:  0.439 | Gap: 0.238
m = 132 | Train R2: 0.634 | Val R2:  0.468 | Gap: 0.166
m = 186 | Train R2: 0.622 | Val R2:  0.522 | Gap: 0.100
m = 240 | Train R2: 0.616 | Val R2:  0.517 | Gap: 0.099
```

Analyze the empirical numbers:

- **Linear Regression:** Both train and validation $R^2$ hover near 0.0, and the gap shrinks to 0.013. The model fits a horizontal line through a symmetric parabola ($y = 0.5 x^2$). More data does not help at all.
- **Unconstrained Decision Tree:** Training score is pinned at a perfect $1.000$, while validation score languishes near $0.217$. A massive generalization gap of $0.783$ persists, diagnosing severe overfitting.
- **Regularized Tree (`max_depth=3`):** As $m$ increases from 24 to 240, the generalization gap collapses from $0.414$ down to $0.099$, while validation $R^2$ climbs from $0.343$ to $0.517$. This exhibits the classic signature of healthy learning.

## 8. The Prescriptive Remediation Matrix: Exactly What to Fix

Once your learning curve diagnoses the problem, follow this structured engineering checklist to remediate the model:

| Diagnosis | Root Cause | What WILL Help (Effective Actions) | What WILL NOT Help (Counterproductive Actions) |
| --- | --- | --- | --- |
| **High Bias (Underfitting)** | Hypothesis space is too restrictive; model cannot express target function $f(x)$. | • Add polynomial, interaction, or domain-specific features. • Decrease regularization strength (decrease $\alpha$, increase $C$). • Switch to a higher-capacity algorithm (e.g., linear $\to$ GBDT / Neural Net). • Increase tree depth or network layer width. | • Collecting more training data of the same type. • Increasing regularization (makes underfitting worse). • Feature elimination / aggressive dimensionality reduction. |
| **High Variance (Overfitting)** | Hypothesis space is too flexible; model memorizes sample-specific noise. | • Collect more training samples ($m$). • Increase regularization strength (increase $\alpha$, decrease $C$, add dropout). • Prune decision trees (set `max_depth`, `min_samples_leaf`). • Feature selection / reduce input dimensionality ($d$). • Ensembling via Bagging / Random Forests. | • Adding more non-linear features. • Training unregularized models for more epochs. • Switching to more complex architectures. |

## 9. Common Pitfalls & Experimental Traps

- **Evaluating on Shuffled vs. Ordered Data:** If data is generated or ordered sequentially (e.g., sorted by target or time), computing learning curves without shuffling (`shuffle=True`) causes catastrophic out-of-domain extrapolation where validation sets occupy unseen ranges.
- **Ignoring Scoring Metric Symmetry:** On imbalanced classification (e.g., 99% negative class), an underfitting dummy model predicting the majority class achieves 99% accuracy! Always specify informative metrics such as `scoring='f1'`, `'roc_auc'`, or `'neg_mean_squared_error'`.
- **Premature Curve Truncation:** If your training fractions stop at $m = 100$ when the dataset contains 10,000 rows, you cannot observe whether the validation trajectory would have closed the gap. Always scale $m$ to the maximum feasible dataset size.
- **Neglecting Error Bands:** Looking solely at the mean score can mask immense variance across cross-validation folds. Always inspect fold standard deviations (using error bands or `np.std()`) to verify stability.

## 10. Hands-On Practice & Curriculum Roadmap

Consolidate your diagnostic skills with these hands-on engineering challenges:

1. **The Depth Sweep Experiment:** Using the synthetic quadratic dataset above, iterate `max_depth` across values `[1, 2, 3, 5, 10, None]`. Generate learning curves for each and plot the generalization gap $\Delta J$ against depth.
2. **Diagnosing Ridge vs. Lasso:** On a high-dimensional dataset with 100 features but only 10 true informative signals, compare the learning curves of unregularized Linear Regression, Ridge (L2), and Lasso (L1). Observe how L1 feature elimination narrows the high-variance gap faster than L2.
3. **Detecting Data Saturation in Neural Networks:** Train a multi-layer perceptron on MNIST with $m \in [500, 2000, 10000, 50000]$. Identify the inflection point where adding more data yields diminishing returns.

> **WHAT TO LEARN NEXT:** Now that you can diagnose bias and variance with learning curves, learn how to systematically tune hyperparameters to find the optimal trade-off. In the next guide, **Hyperparameter Tuning With Grid Search, Random Search, and Bayesian Optimization**, we explore automated search strategies across complex parameter spaces.

---

Written by Sythra — Learn machine learning by building. Practice this topic with Sythra's AI tutor: https://app.sythra.ai/pricing