Naive Bayes From Scratch in Python: Math, Bayes' Theorem, and Spam Filter Code

Naive Bayes is a probabilistic supervised classification algorithm grounded in Bayes' Theorem. It computes the posterior probability of each class given an observation's features under the simplifying ('naive') assumption of conditional feature independence.

Imagine a detective investigating a crime scene. Before inspecting any physical evidence, they hold a baseline prior belief based on historical city data (e.g. 30% burglary, 20% fraud). As new clues emerge (a broken window, a missing safe, a digital transfer), each independent clue updates their probability distribution until they arrive at a final, evidence-backed verdict. Naive Bayes performs this exact evidentiary update mathematically.

1. The 3 Core Naive Bayes Variants

2. Mathematical Foundations

1. Bayes' Theorem for Classification

To classify an email based on its constituent words $w_1, w_2, \dots, w_n$, Bayes' Theorem establishes:

$$P(\text{Spam} \mid \text{Words}) = \frac{P(\text{Words} \mid \text{Spam}) \cdot P(\text{Spam})}{P(\text{Words})}$$

• Posterior $P(\text{Spam} \mid \text{Words})$: The probability the document is spam given the observed text.
• Likelihood $P(\text{Words} \mid \text{Spam})$: The probability a spam email generates this word sequence.
• Prior $P(\text{Spam})$: The baseline proportion of spam in the training dataset.
• Evidence $P(\text{Words})$: Total probability of observing these words across all classes (constant normalizer).

2. The Independence Assumption & Log-Sum Trick

Assuming words occur independently given the class transforms the joint probability into a product of individual likelihoods:

$$P(w_1, w_2, \dots, w_n \mid C) = \prod_{k=1}^{n} P(w_k \mid C)$$

Multiplying many small floating-point decimals causes arithmetic underflow. Taking the natural logarithm converts multiplication into numerically stable addition:

$$\log P(C \mid \text{Words}) \propto \log P(C) + \sum_{k=1}^{n} \log P(w_k \mid C)$$

3. Laplace Smoothing (Add-One Smoothing)

If an incoming word was never seen in training, its likelihood is $0.0$, collapsing the entire class probability to zero. Laplace Smoothing with $\alpha = 1.0$ eliminates this zero-frequency trap:

$$P(w_k \mid C) = \frac{\text{count}(w_k, C) + \alpha}{N_C + \alpha |V|}$$

4. Gaussian Naive Bayes for Continuous Features

For real-valued numeric features, the likelihood is estimated using class mean $\mu_C$ and variance $\sigma_C^2$:

$$P(x_i \mid C) = \frac{1}{\sigma_C \sqrt{2\pi}} \exp\left( -\frac{(x_i - \mu_C)^2}{2\sigma_C^2} \right)$$

3. Worked Numerical Example (By Hand)

• Priors: $P(\text{Spam}) = 0.50, \quad P(\text{NotSpam}) = 0.50$.
• Spam word count ($N_{\text{spam}} = 6$): free=2, money=1, now=1, vacation=1, offer=1.
• Not-Spam word count ($N_{\text{not}} = 6$): meeting=1, schedule=1, today=2, project=1, report=1.
• Query 'free today':
• $P(\text{free}|\text{Spam}) = \frac{2+1}{6+10} = \frac{3}{16} = 0.1875$; $P(\text{today}|\text{Spam}) = \frac{0+1}{6+10} = \frac{1}{16} = 0.0625$.
• $P(\text{free}|\text{Not}) = \frac{0+1}{6+10} = 0.0625$; $P(\text{today}|\text{Not}) = \frac{2+1}{6+10} = 0.1875$.
• Spam log-score: $\ln(0.50) + \ln(0.1875) + \ln(0.0625) = -5.140$.
• Not-Spam log-score: $\ln(0.50) + \ln(0.0625) + \ln(0.1875) = -5.140$.

4. Code: From Scratch & Scikit-Learn

1. NumPy Naive Bayes Spam Filter From Scratch

import numpy as np
from collections import defaultdict

emails = [
    ("free money now", "spam"),
    ("free vacation offer", "spam"),
    ("meeting schedule today", "not_spam"),
    ("project report today", "not_spam"),
]

def train_naive_bayes(corpus):
    word_counts = defaultdict(lambda: defaultdict(int))
    class_totals = defaultdict(int)
    class_docs = defaultdict(int)
    vocab = set()

    for text, label in corpus:
        class_docs[label] += 1
        for word in text.lower().split():
            word_counts[label][word] += 1
            class_totals[label] += 1
            vocab.add(word)

    total_docs = len(corpus)
    priors = {c: count / total_docs for c, count in class_docs.items()}
    return priors, word_counts, class_totals, vocab

def predict_naive_bayes(text, priors, word_counts, class_totals, vocab, alpha=1.0):
    v_size = len(vocab)
    log_posteriors = {}

    for c in priors:
        score = np.log(priors[c])
        for word in text.lower().split():
            count = word_counts[c][word]
            prob = (count + alpha) / (class_totals[c] + alpha * v_size)
            score += np.log(prob)
        log_posteriors[c] = score

    return max(log_posteriors, key=log_posteriors.get), log_posteriors

priors, word_counts, class_totals, vocab = train_naive_bayes(emails)
best_class, scores = predict_naive_bayes("free today", priors, word_counts, class_totals, vocab)

print("Log-Posterior Scores:", {k: round(v, 3) for k, v in scores.items()})
print("Predicted Class:", best_class)

2. Production Scikit-Learn MultinomialNB

from sklearn.naive_bayes import MultinomialNB
from sklearn.feature_extraction.text import CountVectorizer

texts = [doc[0] for doc in emails]
labels = [doc[1] for doc in emails]

vectorizer = CountVectorizer()
X = vectorizer.fit_transform(texts)

nb = MultinomialNB(alpha=1.0)
nb.fit(X, labels)

query_vec = vectorizer.transform(["free today"])
print(f"Scikit-Learn Prediction: {nb.predict(query_vec)[0]}")
print(f"Class Probabilities: {dict(zip(nb.classes_, nb.predict_proba(query_vec)[0]))}")

Summary

• Naive Bayes computes the class posterior by combining a prior class probability with independent feature likelihoods.
• The conditional independence assumption simplifies joint probability estimation into a product of per-word probabilities.
• Log-Sum Transformation: Converts probability products into summations, preventing floating-point underflow.
• Laplace Smoothing ($\alpha = 1.0$): Prevents zero probabilities for vocabulary tokens unseen during training.
• Three primary variants: MultinomialNB (counts), BernoulliNB (binary), and GaussianNB (continuous).