---
title: Data Leakage in Machine Learning: Types, Detection, and Prevention in Python
source: https://app.sythra.ai/learn/machine-learning/data-leakage-in-machine-learning-python
topic: Machine Learning
updated: 2026-08-28
publisher: Sythra (https://app.sythra.ai)
---

# Data Leakage in Machine Learning: Types, Detection, and Prevention in Python

Data leakage occurs when information from outside the training dataset (such as future events or test set statistics) is accidentally included in model training, artificially inflating evaluation scores while causing real-world performance to fail.

_Source: [https://app.sythra.ai/learn/machine-learning/data-leakage-in-machine-learning-python](https://app.sythra.ai/learn/machine-learning/data-leakage-in-machine-learning-python) — free to read on Sythra._

## Key points

- Covers the 4 main types: Target Leakage, Preprocessing Leakage, Temporal Leakage, and Group Leakage.
- Demonstrates the real vs. inflated accuracy gap in Python.
- Provides automated correlation smell test code to flag leaked features.
- Shows how Scikit-Learn Pipeline guarantees leak-free cross-validation.

**Data leakage** (also known as information leakage) occurs when information from outside the training dataset is inadvertently used to create or train a machine learning model, artificially inflating validation accuracy while causing the model to catastrophically fail in production.

Think of it like a student studying for an exam who accidentally gets a copy of tomorrow's answer key mixed into their study notes. The student scores 100% on practice tests, but the moment they sit for the real, unseen exam without the leaked answers, their score collapses. In machine learning, the algorithm does not know it cheated — it simply exploited subtle clues that will not exist at real inference time.

## Why Data Leakage Is the Most Dangerous ML Bug

Unlike typical software bugs that throw exceptions or crash your script, **data leakage produces zero errors**. In fact, it makes your metrics look phenomenal — accuracy jumps to 99%, loss drops near zero, and validation curves look perfect.

The failure only exposes itself after deployment, when the model makes expensive wrong decisions because the leaked signals are completely absent in live real-world inputs.

## The 4 Main Types of Data Leakage

| Leakage Type | How It Happens | Real-World Example | How to Prevent |
| --- | --- | --- | --- |
| **1. Target Leakage** | Features are created using data that only exists _after_ the target outcome has already occurred. | Including `collections_flag` when predicting loan default, or `days_in_icu` when predicting patient mortality. | Ask: _"Would this exact column exist at the very millisecond we make the prediction?"_ |
| **2. Preprocessing Leakage** | Fitting scalers, imputers, PCA, or encoders on the combined dataset before train-test splitting. | Calculating the column mean on all rows before `train_test_split`. | Always split first, or use a Scikit-Learn `Pipeline` to fit exclusively on `X_train`. |
| **3. Temporal Leakage** | Randomly shuffling sequential or time-series data, allowing past events to be predicted using future data. | Predicting stock prices on Monday using trading volume from Friday of that week. | Use `TimeSeriesSplit` or chronological cut-off dates. |
| **4. Group / Identity Leakage** | Multiple records from the same entity (patient, customer, device) appear in both train and test sets. | The model memorizes a specific patient's identity rather than learning general disease patterns. | Use `GroupKFold` or split by unique Group ID. |

## The Mathematical Violation

Supervised machine learning relies on the fundamental assumption that the test set $D_{\text{test}}$ is strictly independent from the training set $D_{\text{train}}$:

$$\text{Generalization Error} \approx \frac{1}{|D_{\text{test}}|}\sum_{i=1}^{|D_{\text{test}}|} L(y_i, \hat{f}(x_i))$$

When leakage occurs, a feature $x_j$ becomes an unintended mathematical proxy for target $y$ ($x_j \approx g(y)$), or the preprocessing transformation $T$ incorporates test distribution parameters:

$$T_{\text{leaky}} = \text{fit}(T, D_{\text{train}} \cup D_{\text{test}}) \quad (\text{Invalid Contamination})$$

$$T_{\text{correct}} = \text{fit}(T, D_{\text{train}}) \quad (\text{Statistically Sound})$$

## Code: Reproducing and Fixing Data Leakage in Python

### 1. Setting Up a Leaky Loan Default Dataset

```python
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
from sklearn.preprocessing import StandardScaler

np.random.seed(42)
n_samples = 200

# Base borrower profile
df = pd.DataFrame({
    "income": np.random.normal(50000, 15000, n_samples).round(0),
    "credit_score": np.random.normal(650, 50, n_samples).round(0),
})

# Ground truth loan default target
df["defaulted"] = (
    (df["income"] < 45000).astype(int) & (df["credit_score"] < 630).astype(int)
)

# TARGET LEAKAGE: collections_flag only occurs AFTER default happens
df["collections_flag"] = df["defaulted"].apply(
    lambda x: 1 if x == 1 and np.random.rand() < 0.95 else 0
)

print(df.head())
```

### 2. The Leaky Model vs. The Clean Model

Notice how the leaky feature inflates accuracy to an unrealistic 97.5%:

```python
# 1. LEAKY MODEL (using collections_flag)
X_leaky = df[["income", "credit_score", "collections_flag"]]
y = df["defaulted"]

X_tr_l, X_te_l, y_tr_l, y_te_l = train_test_split(X_leaky, y, test_size=0.2, random_state=42)
model_leaky = LogisticRegression()
model_leaky.fit(X_tr_l, y_tr_l)
print(f"Accuracy WITH Target Leakage: {accuracy_score(y_te_l, model_leaky.predict(X_te_l)):.1%}")

# 2. CLEAN MODEL (removing post-outcome feature)
X_clean = df[["income", "credit_score"]]
X_tr_c, X_te_c, y_tr_c, y_te_c = train_test_split(X_clean, y, test_size=0.2, random_state=42)
model_clean = LogisticRegression()
model_clean.fit(X_tr_c, y_tr_c)
print(f"Accuracy WITHOUT Leakage (Honest Score): {accuracy_score(y_te_c, model_clean.predict(X_te_c)):.1%}")
```

### 3. Preprocessing Leakage and the Pipeline Defense

The gold standard to eliminate preprocessing leakage is wrapping transformers inside a Scikit-Learn `Pipeline`:

```python
from sklearn.pipeline import Pipeline
from sklearn.model_selection import cross_val_score

# Pipeline ensures StandardScaler is fit ONLY on training folds
pipeline = Pipeline([
    ("scaler", StandardScaler()),
    ("model", LogisticRegression())
])

scores = cross_val_score(pipeline, X_clean, y, cv=5, scoring="accuracy")
print(f"Leak-Free Cross-Validated Accuracy: {scores.mean():.1%} +/- {scores.std():.1%}")
```

### 4. Automated Leakage Detection Script (The Smell Test)

```python
def detect_potential_leakage(df, target_col, threshold=0.85):
    print("--- Automated Leakage Smell Test ---")
    correlations = df.drop(columns=[target_col]).corrwith(df[target_col]).abs()
    suspicious = correlations[correlations > threshold]
    
    if len(suspicious) > 0:
        print(f"WARNING: Found {len(suspicious)} suspicious features with |r| > {threshold}:")
        for col, val in suspicious.items():
            print(f"   -> {col}: correlation = {val:.2f}")
    else:
        print("No suspicious high-correlation features detected.")

detect_potential_leakage(df, target_col="defaulted")
```

## The Leakage Prevention Checklist

- **Timeline Verification:** Map out the exact timestamp when each feature is generated. If a feature is populated at or after prediction time, drop it immediately.
- **Split Before Preprocessing:** Always run `train_test_split` before calling `fit()` on any scaler, imputer, or encoder.
- **Use Scikit-Learn Pipelines:** Encapsulate all transformations inside `Pipeline` or `ColumnTransformer` so cross-validation folds cannot leak.
- **Use GroupKFold for Clustered Data:** When observations come from the same patients, accounts, or physical sensors, split by Group ID.
- **Investigate "Too Good to Be True" Scores:** An accuracy jump from 78% to 99% upon introducing a new feature is almost always target leakage.

## Summary

- Data leakage inflates evaluation metrics by letting models cheat on test data.
- Target leakage occurs when future/outcome information is included in features.
- Preprocessing leakage occurs when transformers are fit on the full dataset before splitting.
- Scikit-Learn `Pipeline` objects and automated correlation checks provide the best defense.

## FAQ

### What is the difference between Target Leakage and Train-Test Contamination?

Target Leakage happens when a feature contains information that only becomes available after the target event occurs. Train-Test Contamination happens when preprocessing statistics (like mean or standard deviation) are calculated using the test set before splitting.

### How do I detect data leakage in a machine learning dataset?

Check for unrealistically high feature correlations with the target (|r| > 0.85), examine feature importance in tree models, check chronological timestamps, and compare validation scores with truly out-of-time test data.

### How does Scikit-Learn Pipeline prevent data leakage?

A Pipeline ensures that any preprocessor (like StandardScaler or SimpleImputer) is fit strictly on the training fold during cross-validation, applying only the learned transform to the validation fold.

### What is Group Leakage?

Group Leakage happens when multiple records belonging to the same entity (such as patient visits or customer logs) are split randomly between train and test sets, allowing the model to memorize the entity rather than learning general patterns.

---

Written by Sythra — Learn machine learning by building. Practice this topic with Sythra's AI tutor: https://app.sythra.ai/pricing