ROC Curve and AUC in Python: How They Are Calculated Step by Step

An ROC Curve (Receiver Operating Characteristic) is a graphical plot illustrating the diagnostic ability of a binary classifier across all possible classification thresholds. The AUC (Area Under the Curve) reduces this entire curve into a single scalar value ($0.0$ to $1.0$) summarizing the model's overall class discrimination ability independent of any single threshold.

Think about an airport security guard with a sensitivity dial ranging from 'Flag Literally Everything' (threshold $0.0$) down to 'Flag Almost Nothing' (threshold $1.0$). At each dial setting, the guard achieves a different True Positive Rate (Recall) while generating a different number of False Positive Rates (False Alarms). The ROC curve maps out the complete trade-off frontier as that dial is turned.

1. Why ROC and AUC Are Essential

• Threshold Independence: Evaluates the model's intrinsic ranking ability before committing to a specific production decision boundary.
• Fair Model Comparison: Compares multiple competing classifiers across the entire spectrum of operating sensitivities.
• Clear Visual Diagnostics: Curves bulging toward the top-left corner identify superior sensitivity-to-false-alarm trade-offs.

2. ROC-AUC vs. Precision-Recall AUC (PR-AUC)

3. The Mathematical Formulations

1. The Two Axes: TPR and FPR

$$\text{TPR} = \frac{TP}{TP + FN} \quad (\text{Recall / Sensitivity})$$

$$\text{FPR} = \frac{FP}{FP + TN} \quad (1 - \text{Specificity})$$

2. AUC via the Trapezoidal Rule

Sorting the threshold points by increasing $\text{FPR}$, total AUC is computed by summing the trapezoidal slices between consecutive points:

$$\text{AUC} = \sum_{k=1}^{N-1} \frac{(\text{TPR}_k + \text{TPR}_{k+1})}{2} \times (\text{FPR}_{k+1} - \text{FPR}_k)$$

3. The Probabilistic Interpretation (Mann-Whitney U)

Mathematically, AUC equals the exact probability that a randomly chosen positive instance is assigned a higher predicted score than a randomly chosen negative instance:

$$\text{AUC} = P(\hat{y}_{\text{pos}} > \hat{y}_{\text{neg}})$$
An AUC of $0.85$ means that $85\%$ of the time, the model correctly ranks a true positive above a true negative.

4. Worked Numerical Example (By Hand)

Consider $5$ samples ($3$ actual positives: A, B, E; $2$ actual negatives: C, D) with predicted probabilities:

• Threshold $0.70$: Flagged: A $\implies TP=1, FN=2, FP=0, TN=2 \implies (\text{FPR}=0.0, \text{TPR}=0.333)$.
• Threshold $0.50$: Flagged: A, B, C $\implies TP=2, FN=1, FP=1, TN=1 \implies (\text{FPR}=0.5, \text{TPR}=0.667)$.
• Threshold $0.10$: Flagged: All $\implies TP=3, FN=0, FP=2, TN=0 \implies (\text{FPR}=1.0, \text{TPR}=1.0)$.
• Trapezoidal Sum: Slices $(0, 0) \rightarrow (0, 0.333) \rightarrow (0.5, 0.667) \rightarrow (1.0, 1.0)$ yield:
  $$\text{AUC} = 0 + \left(\frac{0.333+0.667}{2} \times 0.5\right) + \left(\frac{0.667+1.0}{2} \times 0.5\right) = 0.250 + 0.417 = 0.667$$

5. Code: Python Implementation From Scratch & Scikit-Learn

1. NumPy ROC Sweep & Trapezoidal AUC Engine

import numpy as np

y_actual = np.array([1, 1, 0, 0, 1])
y_probs  = np.array([0.90, 0.60, 0.55, 0.30, 0.20])

def compute_roc_points(y_true, y_score):
    thresholds = np.sort(np.unique(y_score))[::-1]
    thresholds = np.concatenate(([1.1], thresholds, [0.0]))
    points = []

    for t in thresholds:
        y_pred = (y_score >= t).astype(int)
        tp = np.sum((y_pred == 1) & (y_true == 1))
        fn = np.sum((y_pred == 0) & (y_true == 1))
        fp = np.sum((y_pred == 1) & (y_true == 0))
        tn = np.sum((y_pred == 0) & (y_true == 0))

        tpr = tp / (tp + fn) if (tp + fn) > 0 else 0.0
        fpr = fp / (fp + tn) if (fp + tn) > 0 else 0.0
        points.append((fpr, tpr))

    return sorted(points)

def compute_auc(points):
    auc = 0.0
    for i in range(1, len(points)):
        fpr_prev, tpr_prev = points[i - 1]
        fpr_curr, tpr_curr = points[i]
        width = fpr_curr - fpr_prev
        avg_height = (tpr_curr + tpr_prev) / 2.0
        auc += width * avg_height
    return auc

points = compute_roc_points(y_actual, y_probs)
auc_val = compute_auc(points)

print("ROC Points (FPR, TPR):", [(round(f, 3), round(t, 3)) for f, t in points])
print(f"Calculated AUC: {auc_val:.3f}")

2. Production Scikit-Learn & Optimal Threshold (Youden's J)

from sklearn.metrics import roc_curve, roc_auc_score

fpr, tpr, thresholds = roc_curve(y_actual, y_probs)
auc_score = roc_auc_score(y_actual, y_probs)

# Youden's J statistic finds the optimal threshold: max(TPR - FPR)
j_scores = tpr - fpr
best_idx = np.argmax(j_scores)
best_threshold = thresholds[best_idx]

print(f"Scikit-Learn AUC Score: {auc_score:.3f}")
print(f"Optimal Threshold (Youden's J): {best_threshold:.2f} (TPR={tpr[best_idx]:.2f}, FPR={fpr[best_idx]:.2f})")

Summary

• The ROC Curve plots True Positive Rate vs. False Positive Rate across all possible probability thresholds.
• AUC (Area Under the Curve) measures overall class separation ($1.0$ is perfect, $0.5$ is random guessing).
• AUC corresponds to the probability that a randomly chosen positive example ranks higher than a randomly chosen negative example.
• On heavily imbalanced datasets, Precision-Recall AUC (PR-AUC) provides a more reliable metric than ROC-AUC.
• Youden's J Statistic ($J = \text{TPR} - \text{FPR}$) helps identify the single most balanced operational threshold.