---
title: Naive Bayes From Scratch in Python: Math, Bayes' Theorem, and Spam Filter Code
source: https://app.sythra.ai/learn/machine-learning/naive-bayes-classifier-python-from-scratch
topic: Machine Learning
updated: 2026-09-01
publisher: Sythra (https://app.sythra.ai)
---

# Naive Bayes From Scratch in Python: Math, Bayes' Theorem, and Spam Filter Code

Naive Bayes calculates class posterior probabilities by multiplying prior beliefs by independent feature likelihoods using Bayes' Theorem, leveraging Laplace smoothing and log-sums for numerical stability.

_Source: [https://app.sythra.ai/learn/machine-learning/naive-bayes-classifier-python-from-scratch](https://app.sythra.ai/learn/machine-learning/naive-bayes-classifier-python-from-scratch) — free to read on Sythra._

## Key points

- Explains Bayes' Theorem for text classification and spam detection.
- Derives the log-sum transformation to prevent arithmetic underflow.
- Details Laplace smoothing to eliminate the zero-frequency trap.
- Compares Multinomial, Bernoulli, and Gaussian Naive Bayes variants with pure Python code.

**Naive Bayes** is a probabilistic supervised classification algorithm grounded in **Bayes' Theorem**. It computes the posterior probability of each class given an observation's features under the simplifying ('naive') assumption of **conditional feature independence**.

Imagine a detective investigating a crime scene. Before inspecting any physical evidence, they hold a baseline prior belief based on historical city data (e.g. 30% burglary, 20% fraud). As new clues emerge (a broken window, a missing safe, a digital transfer), each independent clue updates their probability distribution until they arrive at a final, evidence-backed verdict. Naive Bayes performs this exact evidentiary update mathematically.

## 1. The 3 Core Naive Bayes Variants

| Algorithm Variant | Feature Representation | Likelihood Distribution | Primary Application |
| --- | --- | --- | --- |
| **Multinomial Naive Bayes** | Discrete word counts or TF-IDF frequencies ($0, 1, 2, \dots$). | Multinomial distribution. | Full-text document classification, spam filtering, topic tagging. |
| **Bernoulli Naive Bayes** | Binary indicators ($1$ if word appears, $0$ if absent). | Multivariate Bernoulli distribution. | Short text classification, tweet sentiment analysis. |
| **Gaussian Naive Bayes** | Continuous numerical feature measurements. | Normal / Gaussian distribution $\mathcal{N}(\mu, \sigma^2)$. | Medical diagnostics, biometric sensor data. |

## 2. Mathematical Foundations

### 1. Bayes' Theorem for Classification

To classify an email based on its constituent words $w_1, w_2, \dots, w_n$, Bayes' Theorem establishes:

$$P(\text{Spam} \mid \text{Words}) = \frac{P(\text{Words} \mid \text{Spam}) \cdot P(\text{Spam})}{P(\text{Words})}$$

- **Posterior $P(\text{Spam} \mid \text{Words})$:** The probability the document is spam given the observed text.
- **Likelihood $P(\text{Words} \mid \text{Spam})$:** The probability a spam email generates this word sequence.
- **Prior $P(\text{Spam})$:** The baseline proportion of spam in the training dataset.
- **Evidence $P(\text{Words})$:** Total probability of observing these words across all classes (constant normalizer).

### 2. The Independence Assumption & Log-Sum Trick

Assuming words occur independently given the class transforms the joint probability into a product of individual likelihoods:

$$P(w_1, w_2, \dots, w_n \mid C) = \prod_{k=1}^{n} P(w_k \mid C)$$

Multiplying many small floating-point decimals causes **arithmetic underflow**. Taking the natural logarithm converts multiplication into numerically stable addition:

$$\log P(C \mid \text{Words}) \propto \log P(C) + \sum_{k=1}^{n} \log P(w_k \mid C)$$

### 3. Laplace Smoothing (Add-One Smoothing)

If an incoming word was never seen in training, its likelihood is $0.0$, collapsing the entire class probability to zero. Laplace Smoothing with $\alpha = 1.0$ eliminates this zero-frequency trap:

$$P(w_k \mid C) = \frac{\text{count}(w_k, C) + \alpha}{N_C + \alpha |V|}$$

### 4. Gaussian Naive Bayes for Continuous Features

For real-valued numeric features, the likelihood is estimated using class mean $\mu_C$ and variance $\sigma_C^2$:

$$P(x_i \mid C) = \frac{1}{\sigma_C \sqrt{2\pi}} \exp\left( -\frac{(x_i - \mu_C)^2}{2\sigma_C^2} \right)$$

## 3. Worked Numerical Example (By Hand)

| Document | Text Content | Class Label |
| --- | --- | --- |
| D1 | 'free money now' | Spam |
| D2 | 'free vacation offer' | Spam |
| D3 | 'meeting schedule today' | Not Spam |
| D4 | 'project report today' | Not Spam |

- **Priors:** $P(\text{Spam}) = 0.50, \quad P(\text{NotSpam}) = 0.50$.
- **Spam word count ($N_{\text{spam}} = 6$):** free=2, money=1, now=1, vacation=1, offer=1.
- **Not-Spam word count ($N_{\text{not}} = 6$):** meeting=1, schedule=1, today=2, project=1, report=1.
- **Query 'free today':**
- $P(\text{free}|\text{Spam}) = \frac{2+1}{6+10} = \frac{3}{16} = 0.1875$; $P(\text{today}|\text{Spam}) = \frac{0+1}{6+10} = \frac{1}{16} = 0.0625$.
- $P(\text{free}|\text{Not}) = \frac{0+1}{6+10} = 0.0625$; $P(\text{today}|\text{Not}) = \frac{2+1}{6+10} = 0.1875$.
- **Spam log-score:** $\ln(0.50) + \ln(0.1875) + \ln(0.0625) = -5.140$.
- **Not-Spam log-score:** $\ln(0.50) + \ln(0.0625) + \ln(0.1875) = -5.140$.

## 4. Code: From Scratch & Scikit-Learn

### 1. NumPy Naive Bayes Spam Filter From Scratch

```python
import numpy as np
from collections import defaultdict

emails = [
    ("free money now", "spam"),
    ("free vacation offer", "spam"),
    ("meeting schedule today", "not_spam"),
    ("project report today", "not_spam"),
]

def train_naive_bayes(corpus):
    word_counts = defaultdict(lambda: defaultdict(int))
    class_totals = defaultdict(int)
    class_docs = defaultdict(int)
    vocab = set()

    for text, label in corpus:
        class_docs[label] += 1
        for word in text.lower().split():
            word_counts[label][word] += 1
            class_totals[label] += 1
            vocab.add(word)

    total_docs = len(corpus)
    priors = {c: count / total_docs for c, count in class_docs.items()}
    return priors, word_counts, class_totals, vocab

def predict_naive_bayes(text, priors, word_counts, class_totals, vocab, alpha=1.0):
    v_size = len(vocab)
    log_posteriors = {}

    for c in priors:
        score = np.log(priors[c])
        for word in text.lower().split():
            count = word_counts[c][word]
            prob = (count + alpha) / (class_totals[c] + alpha * v_size)
            score += np.log(prob)
        log_posteriors[c] = score

    return max(log_posteriors, key=log_posteriors.get), log_posteriors

priors, word_counts, class_totals, vocab = train_naive_bayes(emails)
best_class, scores = predict_naive_bayes("free today", priors, word_counts, class_totals, vocab)

print("Log-Posterior Scores:", {k: round(v, 3) for k, v in scores.items()})
print("Predicted Class:", best_class)
```

### 2. Production Scikit-Learn MultinomialNB

```python
from sklearn.naive_bayes import MultinomialNB
from sklearn.feature_extraction.text import CountVectorizer

texts = [doc[0] for doc in emails]
labels = [doc[1] for doc in emails]

vectorizer = CountVectorizer()
X = vectorizer.fit_transform(texts)

nb = MultinomialNB(alpha=1.0)
nb.fit(X, labels)

query_vec = vectorizer.transform(["free today"])
print(f"Scikit-Learn Prediction: {nb.predict(query_vec)[0]}")
print(f"Class Probabilities: {dict(zip(nb.classes_, nb.predict_proba(query_vec)[0]))}")
```

## Summary

- Naive Bayes computes the class posterior by combining a prior class probability with independent feature likelihoods.
- The conditional independence assumption simplifies joint probability estimation into a product of per-word probabilities.
- **Log-Sum Transformation:** Converts probability products into summations, preventing floating-point underflow.
- **Laplace Smoothing ($\alpha = 1.0$):** Prevents zero probabilities for vocabulary tokens unseen during training.
- Three primary variants: **MultinomialNB** (counts), **BernoulliNB** (binary), and **GaussianNB** (continuous).

## FAQ

### Why is Naive Bayes called 'naive'?

It is called naive because it assumes that all features (e.g. words in a sentence) are mutually independent given the class label, which is rarely true in natural language but works remarkably well in practice.

### What is Laplace Smoothing and why is it necessary?

Laplace smoothing adds a constant alpha (usually 1.0) to word counts and alpha * |V| to denominators so that words unseen during training do not receive a probability of zero, which would wipe out the entire class score.

### Why do we take the log of probabilities in Naive Bayes?

Multiplying many small decimal probabilities together causes floating-point underflow where numbers collapse to 0.0. Taking the logarithm converts multiplication into addition, preserving numerical precision.

### What is the difference between MultinomialNB and GaussianNB?

MultinomialNB is designed for discrete frequency counts such as text word tokens, whereas GaussianNB is used for continuous numerical features assuming a normal bell-curve distribution.

---

Written by Sythra — Learn machine learning by building. Practice this topic with Sythra's AI tutor: https://app.sythra.ai/pricing