Confusion Matrix, Precision, Recall, and F1 Score in Python Explained

A Confusion Matrix is a cross-tabulated contingency table that evaluates the performance of a classification model by categorizing its predictions into four distinct quadrants: True Positives ($TP$), True Negatives ($TN$), False Positives ($FP$), and False Negatives ($FN$). Core evaluation metrics including Accuracy, Precision, Recall, and the $F_1$ Score are derived directly from these four counts.

Imagine an airport baggage screening scanner. Every scanned bag has exactly four possible diagnostic outcomes: correctly flagging a bag with prohibited items ($TP$), correctly passing a safe bag ($TN$), raising a false alarm on a harmless suitcase ($FP$), or mistakenly allowing a dangerous bag through ($FN$). Evaluating a classifier requires understanding the specific frequency and real-world cost of each mistake.

1. The Accuracy Paradox (Why Accuracy Fails on Imbalanced Data)

In fraud detection or rare disease diagnosis where only $0.1\%$ (1 in 1,000) of samples are positive, a trivial model that always predicts negative achieves $99.9\%$ Accuracy while catching zero fraud cases ($Recall = 0.0$). Relying on accuracy alone conceals severe model failure.

2. Classification Metric Taxonomy Matrix

3. The Mathematical Formulations

1. The 2x2 Binary Confusion Matrix

2. Why the Harmonic Mean for $F_1$?

The arithmetic average of $Precision = 1.0$ and $Recall = 0.01$ equals $0.505$, falsely indicating acceptable performance. The harmonic mean gives:

$$F_1 = 2 \times \frac{1.0 \times 0.01}{1.0 + 0.01} = \frac{0.02}{1.01} \approx 0.0198$$

The harmonic mean penalizes extreme imbalances harshly, ensuring that $F_1$ is high only when both Precision and Recall are strong.

3. The Generalized $F_\beta$ Score

When business requirements prioritize Recall over Precision (or vice versa), the $F_\beta$ Score assigns weight $\beta$ to Recall:

$$F_\beta = (1 + \beta^2) \frac{\text{Precision} \times \text{Recall}}{(\beta^2 \cdot \text{Precision}) + \text{Recall}}$$

• $F_2$ Score ($\beta = 2$): Weights Recall twice as heavily as Precision (medical diagnosis, fault detection).
• $F_{0.5}$ Score ($\beta = 0.5$): Weights Precision twice as heavily as Recall (marketing outreach, auto-spam delete).

4. Worked Numerical Example (By Hand)

Suppose a diagnostic model tests $100$ patients ($10$ actually have the disease, $90$ are healthy):

• Matrix Counts: $TP = 7, \quad FN = 3, \quad FP = 8, \quad TN = 82$.
• Accuracy: $\frac{7 + 82}{100} = 0.89$ ($89\%$ overall accuracy).
• Precision: $\frac{7}{7 + 8} = \frac{7}{15} \approx 0.467$ ($53.3\%$ of flagged patients are false alarms).
• Recall: $\frac{7}{7 + 3} = \frac{7}{10} = 0.70$ ($30\%$ of diseased patients were missed).
• $F_1$ Score: $2 \times \frac{0.467 \times 0.70}{0.467 + 0.70} = \frac{0.6538}{1.167} \approx 0.560$.

5. Multi-Class Averaging: Macro vs. Micro vs. Weighted

6. Code: Implementation From Scratch & Scikit-Learn

1. NumPy Matrix & Metric Calculations From Scratch

import numpy as np

# 10 diseased (1s), 90 healthy (0s)
y_true = np.array([1]*10 + [0]*90)
# Model catches 7 of 10 positives, makes 8 false alarms on negatives
y_pred = np.array([1]*7 + [0]*3 + [1]*8 + [0]*82)

def compute_confusion_matrix(y_true, y_pred):
    tp = np.sum((y_true == 1) & (y_pred == 1))
    tn = np.sum((y_true == 0) & (y_pred == 0))
    fp = np.sum((y_true == 0) & (y_pred == 1))
    fn = np.sum((y_true == 1) & (y_pred == 0))
    return tp, tn, fp, fn

def evaluate_classification(y_true, y_pred, beta=1.0):
    tp, tn, fp, fn = compute_confusion_matrix(y_true, y_pred)
    
    accuracy = (tp + tn) / (tp + tn + fp + fn)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    specificity = tn / (tn + fp) if (tn + fp) > 0 else 0.0
    
    # F-beta formula
    f_beta = (1 + beta**2) * (precision * recall) / ((beta**2 * precision) + recall)
    
    return {
        "TP": tp, "TN": tn, "FP": fp, "FN": fn,
        "Accuracy": round(accuracy, 3),
        "Precision": round(precision, 3),
        "Recall": round(recall, 3),
        "Specificity": round(specificity, 3),
        "F1": round(f_beta, 3)
    }

results = evaluate_classification(y_true, y_pred)
for metric, val in results.items():
    print(f"{metric}: {val}")

2. Production Scikit-Learn Report

from sklearn.metrics import confusion_matrix, classification_report

cm = confusion_matrix(y_true, y_pred)
print("Scikit-Learn Confusion Matrix [[TN, FP], [FN, TP]]:\n", cm)
print("\nClassification Report:\n", classification_report(y_true, y_pred, digits=3))

Summary

• The Confusion Matrix partitions classification outputs into True Positives, True Negatives, False Positives, and False Negatives.
• Accuracy is misleading on imbalanced datasets — always inspect Precision, Recall, and the Confusion Matrix.
• Precision measures positive prediction reliability, while Recall measures detection completeness.
• $F_1$ Score calculates the harmonic mean of Precision and Recall, while $F_\beta$ allows custom weighting based on business costs.