Handling Imbalanced Datasets in Python: SMOTE, Class Weights, and Math Explained

In machine learning classification, an imbalanced dataset occurs when the distribution of target classes is severely skewed — where a majority class represents the overwhelming bulk of the data (e.g., 99% legitimate transactions, 99.5% healthy patients), while the class of genuine interest (fraud, rare disease, equipment failure) constitutes a tiny fraction. When trained on skewed data, standard algorithms fall victim to the Accuracy Paradox: they maximize overall mathematical accuracy by simply predicting the majority class for every sample, quietly rendering the model useless for the exact task it was built to solve.

1. Key Concepts & Definitions

Handling class imbalance requires attacking the problem from one of three distinct architectural angles: altering the loss penalty (Class Weighting), generating synthetic data points (SMOTE), or altering the post-training decision cutoff (Threshold Moving). Before examining the mathematics, let us define the core terminology in plain English:

2. Method 1: Class Weighting & Cost-Sensitive Loss

Standard machine learning models treat every training observation identically. In standard Binary Cross-Entropy (Log Loss) for logistic regression, the cost function over $m$ training examples is:

$$J(w, b) = -\frac{1}{m} \sum_{i=1}^{m} \left[ y^{(i)} \log(\hat{y}^{(i)}) + (1 - y^{(i)}) \log(1 - \hat{y}^{(i)}) \right]$$

When 99% of samples belong to class 0 ($y^{(i)} = 0$), the right-hand term completely dominates the loss summation. The optimizer can slash total cost almost to zero by simply shifting the bias $b$ strongly negative so that $\hat{y}^{(i)} \approx 0$ everywhere, completely ignoring the 1% of minority points.

The Weighted Cost Function

Class weighting introduces a per-class loss multiplier $\omega_{y^{(i)}}$, scaling the penalty based on the ground-truth label of sample $i$:

$$J_{\text{weighted}}(w, b) = -\frac{1}{m} \sum_{i=1}^{m} \omega_{y^{(i)}} \left[ y^{(i)} \log(\hat{y}^{(i)}) + (1 - y^{(i)}) \log(1 - \hat{y}^{(i)}) \right]$$

The standard, principled approach adopted by Scikit-Learn (via class_weight='balanced') sets these weights inversely proportional to class frequencies:

$$\omega_c = \frac{m}{K \cdot m_c}$$

Where $m$ is the total dataset size, $K$ is the number of classes ($K=2$ for binary classification), and $m_c$ is the number of samples in class $c$.

When computing gradient updates during gradient descent, this scalar weight flows directly into the parameter gradient:

$$\frac{\partial J_{\text{weighted}}}{\partial w_j} = \frac{1}{m} \sum_{i=1}^{m} \omega_{y^{(i)}} \left( \hat{y}^{(i)} - y^{(i)} \right) x_j^{(i)}$$

If class 1 is 9 times rarer than class 0 (e.g., 90 healthy vs 10 sick): $\omega_{\text{sick}} = \frac{100}{2 \times 10} = 5.0$, while $\omega_{\text{healthy}} = \frac{100}{2 \times 90} \approx 0.556$. A misclassification on a sick patient pulls the weights 9 times harder ($5.0 / 0.556 = 9$) than a mistake on a healthy patient, forcefully steering the decision boundary toward protecting the rare class.

3. Method 2: SMOTE Math & Geometry

Rather than modifying the loss function, SMOTE (Synthetic Minority Oversampling Technique) balances the dataset by generating synthetic minority examples directly within the feature space. Crucially, SMOTE does not create exact duplicates of existing points (which would cause severe overfitting to individual noise points). Instead, it draws synthetic samples along the line segments connecting existing minority points to their nearest minority neighbors.

The SMOTE Algorithm Step-by-Step

1. Identify the Minority Set: Isolate all minority class observations $X_{\text{min}}$.
2. Find k-Nearest Neighbors: For each minority observation $x_i$, compute the Euclidean distance to all other points in $X_{\text{min}}$, and identify its $k$ nearest neighbors (typically $k=5$).
3. Sample a Neighbor: Randomly select one neighbor $x_{zi}$ from those $k$ nearest neighbors.
4. Sample an Interpolation Dial: Draw a random scalar $\lambda \sim \text{Uniform}(0, 1)$.
5. Synthesize the New Observation: Create the synthetic point $x_{\text{new}}$ along the connecting vector:

$$x_{\text{new}} = x_i + \lambda \cdot (x_{zi} - x_i)$$

Geometrically, $(x_{zi} - x_i)$ is the directional vector extending from $x_i$ to $x_{zi}$. Multiplying this vector by $\lambda \in [0, 1]$ scales it to an intermediate segment. Adding this back to $x_i$ produces an entirely new point that resides firmly within the plausible topological region of the minority class.

4. Method 3: Threshold Moving (The Zero-Cost Alternative)

Probabilistic binary classifiers (such as Logistic Regression, Random Forests, and Gradient Boosting) output a continuous predicted probability $\hat{p} = P(y=1 \mid x)$. By default, libraries apply a decision cutoff of $\tau = 0.5$:

$$\hat{y} = \begin{cases} 1 & \text{if } \hat{p} \ge 0.5 \\ 0 & \text{if } \hat{p} < 0.5 \end{cases}$$

The $0.5$ cutoff inherently assumes that a False Positive (false alarm) and a False Negative (missed case) carry identical costs. In imbalanced problems, this assumption is false. Missing a fraudulent transaction or a malignant tumor is orders of magnitude more expensive than flagging a legitimate user for verification.

Threshold Moving shifts $\tau$ to optimize a specific objective (such as maximizing the F1-score or minimizing asymmetric financial loss) without altering training data or model weights:

$$\tau^* = \frac{C_{\text{FP}}}{C_{\text{FP}} + C_{\text{FN}}}$$

If missing a fraud incident costs $C_{\text{FN}} = 500$ while checking a false positive costs $C_{\text{FP}} = 10$, the optimal decision threshold drops to $\tau^* = \frac{10}{10 + 500} \approx 0.0196$. Lowering the threshold to $\approx 0.02$ forces the model to flag cases whenever there is even a 2% suspicion, dramatically boosting minority recall instantly.

5. Why ROC-AUC Fails (And Why PR-AUC Is Required)

A pervasive mistake in machine learning is evaluating imbalanced models using the standard Receiver Operating Characteristic (ROC-AUC) curve. ROC curves plot the True Positive Rate (Recall) against the False Positive Rate (FPR):

$$\text{FPR} = \frac{\text{FP}}{\text{FP} + \text{TN}}, \quad \text{Precision} = \frac{\text{TP}}{\text{TP} + \text{FP}}$$

Examine the denominator of $\text{FPR}$: it contains $\text{TN}$ (True Negatives). In a dataset with 99,000 negatives and 1,000 positives, if the model generates 1,000 false alarms ($\text{FP} = 1000$) while catching 800 positives ($\text{TP} = 800$):

• $$\text{FPR} = \frac{1000}{1000 + 98000} = \frac{1000}{99000} \approx 0.0101 \quad (\text{only } 1\% \text{ false positive rate!})$$
• The ROC curve looks outstanding, yielding an AUC of $0.95+$ because the massive number of True Negatives dwarfs the False Positives.
• However, compute Precision: $$\text{Precision} = \frac{800}{800 + 1000} = \frac{800}{1800} \approx 44.4\%$$ More than half of all alerts are false alarms!

PR-AUC (Precision-Recall Area Under Curve / Average Precision) does not include True Negatives in either axis. It focuses exclusively on the minority class performance, providing an honest, uninflated evaluation of model effectiveness under heavy skew.

6. From-Scratch Python Implementation (NumPy)

Let us implement both SMOTE's geometric interpolation algorithm and a Class-Weighted Logistic Regression training loop from scratch using pure NumPy.

import numpy as np

def smote_from_scratch(X_minority, n_samples_to_generate, k=5, random_state=42):
    """
    Generates synthetic samples by interpolating between minority seed points
    and their k-nearest neighbors in Euclidean feature space.
    """
    rng = np.random.default_rng(random_state)
    n_minority, n_features = X_minority.shape
    synthetic = np.empty((n_samples_to_generate, n_features))

    for i in range(n_samples_to_generate):
        # 1. Randomly choose a minority seed point
        seed_idx = rng.integers(0, n_minority)
        x_seed = X_minority[seed_idx]

        # 2. Find Euclidean distance to all other minority observations
        distances = np.linalg.norm(X_minority - x_seed, axis=1)
        distances[seed_idx] = np.inf  # Exclude self-distance

        # 3. Identify k nearest neighbors and pick one at random
        effective_k = min(k, n_minority - 1)
        k_nearest_indices = np.argpartition(distances, effective_k)[:effective_k]
        neighbor_idx = rng.choice(k_nearest_indices)
        x_neighbor = X_minority[neighbor_idx]

        # 4. Generate new point along connecting line segment
        lam = rng.uniform(0.0, 1.0)
        synthetic[i] = x_seed + lam * (x_neighbor - x_seed)

    return synthetic


def sigmoid(z):
    return 1.0 / (1.0 + np.exp(-np.clip(z, -250, 250)))


class WeightedLogisticRegressionScratch:
    """
    Cost-sensitive Logistic Regression with sample weights inversely proportional
    to class frequencies: omega_c = m / (K * m_c).
    """
    def __init__(self, lr=0.1, n_iters=1500):
        self.lr = lr
        self.n_iters = n_iters
        self.w = None
        self.b = 0.0

    def fit(self, X, y):
        m, n = X.shape
        self.w = np.zeros(n)
        self.b = 0.0

        # Compute balanced class weights
        classes, counts = np.unique(y, return_counts=True)
        K = len(classes)
        weight_map = {c: m / (K * count) for c, count in zip(classes, counts)}
        sample_weights = np.array([weight_map[label] for label in y])

        # Gradient descent optimization
        for _ in range(self.n_iters):
            z = X @ self.w + self.b
            y_pred = sigmoid(z)

            # Weighted gradient calculation
            error = (y_pred - y) * sample_weights
            dw = (1.0 / m) * (X.T @ error)
            db = (1.0 / m) * np.sum(error)

            self.w -= self.lr * dw
            self.b -= self.lr * db

        return self

    def predict_proba(self, X):
        return sigmoid(X @ self.w + self.b)

    def predict(self, X, threshold=0.5):
        return (self.predict_proba(X) >= threshold).astype(int)


# Verification test on synthetic imbalanced data (90:10 ratio)
if __name__ == "__main__":
    rng = np.random.default_rng(42)
    # Class 0: 90 majority points around (0, 0)
    X_0 = rng.normal(loc=0.0, scale=1.0, size=(90, 2))
    # Class 1: 10 minority points around (2.5, 2.5)
    X_1 = rng.normal(loc=2.5, scale=1.0, size=(10, 2))

    X = np.vstack([X_0, X_1])
    y = np.array([0] * 90 + [1] * 10)

    # Generate 80 synthetic minority observations to achieve a 1:1 balance
    X_synthetic = smote_from_scratch(X_1, n_samples_to_generate=80, k=4)
    print(f"Original shape: {X.shape}, Synthetic points created: {X_synthetic.shape}")

    # Fit weighted model
    model = WeightedLogisticRegressionScratch(lr=0.05, n_iters=2000)
    model.fit(X, y)
    preds = model.predict(X)
    print(f"Learned weights w: {np.round(model.w, 3)}, Bias b: {round(model.b, 3)}")
    print(f"Minority Recall on Training Set: {np.sum((preds == 1) & (y == 1)) / 10:.2f}")

7. Production Scikit-Learn & Imblearn Pipeline (Preventing Data Leakage)

When implementing SMOTE in real-world systems, the most widespread bug is data leakage during cross-validation. If you apply SMOTE to the whole dataset before splitting into folds, synthetic samples generated from test-fold points end up in the training fold. The model is evaluated on test instances that it effectively already memorized during synthesis, producing dishonestly optimistic performance metrics.

Here is the production-ready script benchmarking four approaches on an imbalanced dataset (95% majority, 5% minority):

import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, average_precision_score
from imblearn.over_sampling import SMOTE
from imblearn.pipeline import Pipeline as ImbPipeline

# 1. Synthesize an imbalanced dataset (95:5 ratio, 2,000 samples)
X, y = make_classification(
    n_samples=2000,
    n_features=10,
    n_informative=6,
    weights=[0.95, 0.05],
    flip_y=0.01,
    random_state=42
)

# Always stratify split so train and test preserve the 95:5 ratio
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=42
)

print(f"Train class distribution: {np.bincount(y_train)} (95% vs 5%)")
print(f"Test class distribution:  {np.bincount(y_test)}")

# --- Approach 1: Unweighted Baseline ---
model_baseline = LogisticRegression(random_state=42)
model_baseline.fit(X_train, y_train)
y_prob_baseline = model_baseline.predict_proba(X_test)[:, 1]
y_pred_baseline = model_baseline.predict(X_test)

# --- Approach 2: Cost-Sensitive Class Weighting ---
model_weighted = LogisticRegression(class_weight="balanced", random_state=42)
model_weighted.fit(X_train, y_train)
y_prob_weighted = model_weighted.predict_proba(X_test)[:, 1]
y_pred_weighted = model_weighted.predict(X_test)

# --- Approach 3: Leak-Free SMOTE Pipeline ---
smote_pipe = ImbPipeline([
    ("smote", SMOTE(random_state=42, k_neighbors=5)),
    ("classifier", LogisticRegression(random_state=42))
])
smote_pipe.fit(X_train, y_train)
y_prob_smote = smote_pipe.predict_proba(X_test)[:, 1]
y_pred_smote = smote_pipe.predict(X_test)

# --- Approach 4: Threshold Moving (tau = 0.20 on baseline probabilities) ---
y_pred_threshold = (y_prob_baseline >= 0.20).astype(int)

# Evaluate all four strategies on the UNTOUCHED real test set
models = {
    "1. Baseline (Unweighted)": (y_pred_baseline, y_prob_baseline),
    "2. Class Weighting": (y_pred_weighted, y_prob_weighted),
    "3. SMOTE Pipeline": (y_pred_smote, y_prob_smote),
    "4. Threshold Moving (tau=0.20)": (y_pred_threshold, y_prob_baseline)
}

print("\n" + "="*70)
for name, (pred, prob) in models.items():
    report = classification_report(y_test, pred, output_dict=True, zero_division=0)
    rec_min = report["1"]["recall"]
    prec_min = report["1"]["precision"]
    f1_min = report["1"]["f1-score"]
    pr_auc = average_precision_score(y_test, prob)
    print(f"{name:30} | Minority Recall: {rec_min:.2f} | Precision: {prec_min:.2f} | F1: {f1_min:.2f} | PR-AUC: {pr_auc:.3f}")
print("="*70)

Expected Benchmark Results

Notice how the baseline unweighted model achieved poor minority recall (0.24) — missing 76% of rare cases despite having a high raw accuracy (96%). Both Class Weighting and SMOTE surged minority recall past 0.80, while Threshold Moving accomplished virtually identical gains without generating a single synthetic sample or altering the loss function.

8. When SMOTE Fails & Advanced Variants

While SMOTE is popular, it has critical structural failure modes that machine learning engineers must recognize:

• The Blind Line Problem: Standard SMOTE naively connects any minority point to its neighbors. If a minority point is an outlier trapped deep inside majority territory, SMOTE generates synthetic points straight through majority space, creating noisy, unrealistic boundary bridges.
• Categorical Features: Standard SMOTE computes Euclidean distances and continuous interpolation. You cannot interpolate 30% along the line between 'New York' and 'London'. You must use SMOTE-NC (Nominal and Continuous), which uses Hamming distance for discrete features and mode selection for categorical levels.
• High Dimensionality (Curse of Dimensionality): In 100+ dimensions, Euclidean distances concentrate (all points appear roughly equidistant), making $k$-NN lookups arbitrary and degrading synthetic quality.

Advanced SMOTE Family Variants

9. Architecture Decision Matrix: Which Method to Choose?

10. Top 5 Critical Mistakes Checklist

1. Leaking into Validation / Test Sets: Never apply SMOTE prior to splitting your data or outside of an imblearn.pipeline.Pipeline during cross-validation.
2. Evaluating with Accuracy or ROC-AUC: Always evaluate imbalanced problems using Precision, Recall, F1-score, and PR-AUC (Average Precision).
3. Applying SMOTE to Categorical Data: Standard SMOTE corrupts categorical variables. Always use SMOTE-NC or one-hot encode after proper frequency imputation.
4. Ignoring Threshold Calibration: Sticking to the default $0.5$ decision threshold is usually suboptimal. Plot the Precision-Recall curve to tune $\tau$ for business objectives.
5. Defaulting to SMOTE When Class Weighting Suffices: In modern gradient boosted trees (LightGBM, XGBoost, CatBoost), setting scale_pos_weight is faster, lighter, and almost always matches or exceeds SMOTE performance.

11. Summary & Key Takeaways

• The Accuracy Paradox: On imbalanced data, models achieve deceptively high accuracy by simply ignoring the minority class.
• Class Weighting: Penalizes errors on rare instances inversely to their frequency ($\omega_c = \frac{m}{K \cdot m_c}$), requiring zero extra data.
• SMOTE: Synthetically interpolates new observations between minority nearest neighbors ($x_{\text{new}} = x_i + \lambda(x_{zi} - x_i)$).
• Threshold Moving: Shifts the probability cutoff $\tau$ from $0.5$ toward $0.1$ or $0.2$, dramatically raising recall without retraining.
• Leak-Free Pipelines: Use imblearn.pipeline.Pipeline during cross-validation so resampling never contaminates evaluation folds.