Data Leakage in Machine Learning: Types, Detection, and Prevention in Python

Data leakage (also known as information leakage) occurs when information from outside the training dataset is inadvertently used to create or train a machine learning model, artificially inflating validation accuracy while causing the model to catastrophically fail in production.

Think of it like a student studying for an exam who accidentally gets a copy of tomorrow's answer key mixed into their study notes. The student scores 100% on practice tests, but the moment they sit for the real, unseen exam without the leaked answers, their score collapses. In machine learning, the algorithm does not know it cheated — it simply exploited subtle clues that will not exist at real inference time.

Why Data Leakage Is the Most Dangerous ML Bug

Unlike typical software bugs that throw exceptions or crash your script, data leakage produces zero errors. In fact, it makes your metrics look phenomenal — accuracy jumps to 99%, loss drops near zero, and validation curves look perfect.

The failure only exposes itself after deployment, when the model makes expensive wrong decisions because the leaked signals are completely absent in live real-world inputs.

The 4 Main Types of Data Leakage

The Mathematical Violation

Supervised machine learning relies on the fundamental assumption that the test set $D_{\text{test}}$ is strictly independent from the training set $D_{\text{train}}$:

$$\text{Generalization Error} \approx \frac{1}{|D_{\text{test}}|}\sum_{i=1}^{|D_{\text{test}}|} L(y_i, \hat{f}(x_i))$$

When leakage occurs, a feature $x_j$ becomes an unintended mathematical proxy for target $y$ ($x_j \approx g(y)$), or the preprocessing transformation $T$ incorporates test distribution parameters:

$$T_{\text{leaky}} = \text{fit}(T, D_{\text{train}} \cup D_{\text{test}}) \quad (\text{Invalid Contamination})$$

$$T_{\text{correct}} = \text{fit}(T, D_{\text{train}}) \quad (\text{Statistically Sound})$$

Code: Reproducing and Fixing Data Leakage in Python

1. Setting Up a Leaky Loan Default Dataset

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
from sklearn.preprocessing import StandardScaler

np.random.seed(42)
n_samples = 200

# Base borrower profile
df = pd.DataFrame({
    "income": np.random.normal(50000, 15000, n_samples).round(0),
    "credit_score": np.random.normal(650, 50, n_samples).round(0),
})

# Ground truth loan default target
df["defaulted"] = (
    (df["income"] < 45000).astype(int) & (df["credit_score"] < 630).astype(int)
)

# TARGET LEAKAGE: collections_flag only occurs AFTER default happens
df["collections_flag"] = df["defaulted"].apply(
    lambda x: 1 if x == 1 and np.random.rand() < 0.95 else 0
)

print(df.head())

2. The Leaky Model vs. The Clean Model

Notice how the leaky feature inflates accuracy to an unrealistic 97.5%:

# 1. LEAKY MODEL (using collections_flag)
X_leaky = df[["income", "credit_score", "collections_flag"]]
y = df["defaulted"]

X_tr_l, X_te_l, y_tr_l, y_te_l = train_test_split(X_leaky, y, test_size=0.2, random_state=42)
model_leaky = LogisticRegression()
model_leaky.fit(X_tr_l, y_tr_l)
print(f"Accuracy WITH Target Leakage: {accuracy_score(y_te_l, model_leaky.predict(X_te_l)):.1%}")

# 2. CLEAN MODEL (removing post-outcome feature)
X_clean = df[["income", "credit_score"]]
X_tr_c, X_te_c, y_tr_c, y_te_c = train_test_split(X_clean, y, test_size=0.2, random_state=42)
model_clean = LogisticRegression()
model_clean.fit(X_tr_c, y_tr_c)
print(f"Accuracy WITHOUT Leakage (Honest Score): {accuracy_score(y_te_c, model_clean.predict(X_te_c)):.1%}")

3. Preprocessing Leakage and the Pipeline Defense

The gold standard to eliminate preprocessing leakage is wrapping transformers inside a Scikit-Learn Pipeline:

from sklearn.pipeline import Pipeline
from sklearn.model_selection import cross_val_score

# Pipeline ensures StandardScaler is fit ONLY on training folds
pipeline = Pipeline([
    ("scaler", StandardScaler()),
    ("model", LogisticRegression())
])

scores = cross_val_score(pipeline, X_clean, y, cv=5, scoring="accuracy")
print(f"Leak-Free Cross-Validated Accuracy: {scores.mean():.1%} +/- {scores.std():.1%}")

4. Automated Leakage Detection Script (The Smell Test)

def detect_potential_leakage(df, target_col, threshold=0.85):
    print("--- Automated Leakage Smell Test ---")
    correlations = df.drop(columns=[target_col]).corrwith(df[target_col]).abs()
    suspicious = correlations[correlations > threshold]
    
    if len(suspicious) > 0:
        print(f"WARNING: Found {len(suspicious)} suspicious features with |r| > {threshold}:")
        for col, val in suspicious.items():
            print(f"   -> {col}: correlation = {val:.2f}")
    else:
        print("No suspicious high-correlation features detected.")

detect_potential_leakage(df, target_col="defaulted")

The Leakage Prevention Checklist

• Timeline Verification: Map out the exact timestamp when each feature is generated. If a feature is populated at or after prediction time, drop it immediately.
• Split Before Preprocessing: Always run train_test_split before calling fit() on any scaler, imputer, or encoder.
• Use Scikit-Learn Pipelines: Encapsulate all transformations inside Pipeline or ColumnTransformer so cross-validation folds cannot leak.
• Use GroupKFold for Clustered Data: When observations come from the same patients, accounts, or physical sensors, split by Group ID.
• Investigate "Too Good to Be True" Scores: An accuracy jump from 78% to 99% upon introducing a new feature is almost always target leakage.

Summary

• Data leakage inflates evaluation metrics by letting models cheat on test data.
• Target leakage occurs when future/outcome information is included in features.
• Preprocessing leakage occurs when transformers are fit on the full dataset before splitting.
• Scikit-Learn Pipeline objects and automated correlation checks provide the best defense.