Handling Missing Data in Python: 5 Imputation Methods Explained

Missing data imputation is the process of replacing empty or NaN (Not a Number) values in a dataset with substituted values calculated using statistical summaries or machine learning algorithms, allowing models to train without crashing on incomplete records.

Imagine filling out a classroom attendance register where several students forgot to write down their test scores — the boxes are simply blank. You cannot average "blank," and you cannot plot "blank" on a graph. Imputation gives you smart, mathematically sound strategies to estimate what belongs in those empty cells.

Why It Is Used: The Real-World NaN Problem

Real-world data collection is messy: IoT sensors momentarily disconnect, survey respondents skip sensitive questions, web forms submit partially filled, and database merges create gaps.

Here is the challenge: almost no core machine learning algorithm in Scikit-Learn (including Linear Regression, SVMs, and Random Forests) can process a NaN value. Attempting to fit a model on data containing missing values will immediately trigger a ValueError.

You generally have two options:

• Dropping rows or columns: Deleting every row with a missing value is wasteful. If 20% of rows have just one empty feature, dropping them throws away 20% of your valuable training data.
• Imputing values: Replacing the missing gaps with defensible estimates based on surrounding patterns, preserving sample size and statistical power.

The 3 Types of Missing Data (MCAR, MAR, MNAR)

Before selecting an imputation technique, statisticians determine why the data is missing:

The Math: 5 Imputation Methods Explained

Method 1: Mean, Median, and Mode Imputation

The simplest strategy calculates a single summary statistic from observed non-missing values ($n$) and fills every empty cell in that column:

$\\\\\\\\\\\\\\\\\\\\\\\hat{x}_{\text{missing}} = \frac{1}{n}\sum_{i=1}^{n} x_i$

• Mean: Best for symmetric, normally distributed numerical data.
• Median: Best for skewed numerical data (e.g. house prices or salaries) because it is resilient against extreme outliers.
• Mode: The most frequent category; essential for categorical text columns (e.g. City or Device Type).

Method 2: Forward Fill & Backward Fill (Time Series)

For time-ordered data (such as stock prices or hourly temperatures), we copy the nearest known chronological observation:s

$$x_t = x_{t-1} \quad \text{(Forward Fill: propagate last known value forward)}$$

$$x_t = x_{t+1} \quad \text{(Backward Fill: propagate next known value backward)}$$

Method 3: K-Nearest Neighbors (KNN) Imputation

KNN Imputation finds the $k$ most similar rows based on all other non-missing features using Euclidean distance:

$$d(a, b) = \sqrt{\sum_{i=1}^{m} (a_i - b_i)^2}$$

The missing value is then imputed as the average of those $k$ nearest neighbors:

$$\hat{x} = \frac{1}{k}\sum_{j=1}^{k} x_j$$

Method 4: Regression Imputation

Treats the column with missing entries as a dependent target variable ($y$) and fits a linear model using the other features ($x_1, \dots, x_p$):

$$\hat{x} = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \dots + \beta_p x_p$$

Method 5: Multiple Imputation by Chained Equations (MICE)

MICE (implemented via Scikit-Learns IterativeImputer) is a round-robin algorithm: each missing column is modeled as a function of all other columns, and the estimates are iteratively refined across multiple successive rounds until values converge.

Code: Step-by-Step Python Implementation

1. Setting Up the Sample Dataset

import pandas as pd
import numpy as np

# A small, realistic dataset: student exam scores and study hours
data = {
    "study_hours": [5.0, 3.0, np.nan, 8.0, 6.0, 2.0, np.nan, 7.0],
    "attendance_pct": [90.0, 85.0, 70.0, np.nan, 95.0, 60.0, 75.0, np.nan],
    "exam_score": [78, 65, 55, 88, 92, 50, 60, 85]
}

df = pd.DataFrame(data)
print(df)

2. Method 1: Mean & Median Imputation

From Scratch:

def mean_impute_from_scratch(column):
    known_values = column.dropna()
    mean_value = known_values.sum() / len(known_values)
    return column.fillna(mean_value)

df_manual = df.copy()
df_manual["study_hours"] = mean_impute_from_scratch(df_manual["study_hours"])
print(df_manual["study_hours"])

Scikit-Learn (Production Library Version):

from sklearn.impute import SimpleImputer

imputer = SimpleImputer(strategy="mean")  # or "median", "most_frequent"
df_sklearn = df.copy()
df_sklearn[["study_hours", "attendance_pct"]] = imputer.fit_transform(
    df_sklearn[["study_hours", "attendance_pct"]]
)
print(df_sklearn)

3. Method 2: Forward & Backward Fill (Time Series)

df_ffill = df.copy()
df_ffill["study_hours"] = df_ffill["study_hours"].ffill()  # copy from previous row
df_ffill["study_hours"] = df_ffill["study_hours"].bfill()  # backup from next row
print(df_ffill["study_hours"])

4. Method 3: KNN Imputation

From Scratch (Simplified):

def knn_impute_scratch(df, target_col, k=2):
    d = df.copy()
    known = d.dropna()
    missing_idx = d[d[target_col].isna()].index
    other_cols = [c for c in d.columns if c != target_col]
    
    for idx in missing_idx:
        row = d.loc[idx, other_cols]
        distances = ((known[other_cols] - row) ** 2).sum(axis=1) ** 0.5
        nearest = distances.nsmallest(k).index
        dnoc[idx, target_col] = known.loc[nearest, target_col].mean()
    return d

df_knn_manual = knn_impute_scratch(df, target_col="study_hours", k=2)
print(df_knn_manual)

Scikit-Learn Version:

from sklearn.impute import KNNImputer

knn_imputer = KNNImputer(n_neighbors=2)
df_knn_sklearn = pd.DataFrame(knn_imputer.fit_transform(df), columns=df.columns)
print(df_knn_sklearn)

5. Method 4: Regression Imputation

from sklearn.linear_model import LinearRegression

def regression_impute(df, target_col, feature_cols):
    d = df.copy()
    train = d.dropna(subset=[target_col] + feature_cols)
    predict_rows = dtd[target_col].isna()]
    
    model = LinearRegression()
    model.fit(train[feature_cols], train[target_col])
    
    if len(predict_rows) > 0:
        predicted = model.predict(predict_rows[feature_cols].fillna(train[feature_cols].mean()))
        d.loc[predict_rows.index, target_col] = predicted
    return d

df_reg = regression_impute(df, target_col="study_hours", feature_cols=["exam_score"])
print(df_reg)

6. Method 5: MICE (IterativeImputer)

from sklearn.experimental import enable_iterative_imputer
from sklearn.impute import IterativeImputer

mice_imputer = IterativeImputer(max_iter=10, random_state=42)
df_mice = pd.DataFrame(mice_imputer.fit_transform(df), columns=df.columns)
print(df_mice.round(2))

Decision Matrix: Which Method Should You Pick?

Pros and Cons of Each Method

Common Pitfalls & Best Practices

• Always call imputer.fit(X_train) and then imputer.transform(X_test). Imputing on the full dataset before splitting leaks test data information into training.
• Scale Features Before KNN: Because KNN computes Euclidean distance, unscaled features (e.g. Salary in thousands vs Age in tens) will distort neighbor selection.
• Add a Missingness Indicator: For critical columns, use MissingIndicator or add a boolean flag (e.g. salary_was_missing). The fact that data was omitted often carries predictive signal.

Summary

• Missing data crashes standard Scikit-Learn models and must be resolved before training.
• Simple statistics (Mean/Median/Mode) provide quick baselines, while multivariate methods (KNN and MICE) provide superior accuracy.
• Always isolate imputation fitting to the training set to prevent data leakage.
• Use median for skewed data, forward fill for time series, and MICE for complex tabular datasets.