Logistic Regression From Scratch in Python: Deriving Sigmoid and Log Loss

Logistic Regression is a foundational supervised classification algorithm that estimates the probability of a discrete outcome (such as binary $0$ or $1$) by passing a linear combination of input features through the non-linear Sigmoid (logistic) function.

Imagine an oncologist evaluating diagnostic scans. The doctor does not deliver a blunt yes/no verdict immediately; instead, they weigh multiple risk factors (age, blood biomarkers, family history) to establish a calibrated degree of suspicion between 0% and 100%. Only when confidence crosses a defined medical threshold do they flag the case for further biopsy. Logistic Regression mirrors this exact clinical process: computing continuous probabilities first, and applying classification cutoffs second.

1. Why Linear Regression Fails for Classification

Using standard Ordinary Least Squares linear regression for binary classification presents two fatal flaws:

• Unbounded Outputs: Linear regression produces predictions anywhere on $(-\infty, +\infty)$ (e.g. $-0.35$ or $+1.42$), which cannot be interpreted as valid probabilities.
• Non-Convex Loss Surface: Pairing linear regression's Mean Squared Error (MSE) with non-linear probabilities produces a bumpy, non-convex cost landscape filled with local minima where gradient descent gets trapped.

2. The Mathematical Derivation (From Odds to Sigmoid)

1. Modeling the Log-Odds (Logit)

The ratio of the probability of an event happening ($p$) to it not happening ($1-p$) is called the Odds:

$$\text{Odds} = \frac{p}{1 - p}$$

Taking the natural logarithm yields the Log-Odds (Logit), which maps probabilities $(0, 1)$ across all real numbers $(-\infty, +\infty)$. We set this equal to our linear combination:

$$\ln\left(\frac{p}{1 - p}\right) = w \cdot x + b = z$$

Exponentiating and solving for $p$ yields the Sigmoid Function $\sigma(z)$:

$$p = \frac{1}{1 + e^{-z}} = \sigma(w \cdot x + b)$$

2. The Calculus of the Sigmoid Derivative

The derivative of the sigmoid function possesses a remarkably clean closed-form structure:

$$\sigma'(z) = \frac{d}{dz}(1 + e^{-z})^{-1} = -(1 + e^{-z})^{-2}(-e^{-z}) = \frac{1}{1 + e^{-z}} \cdot \frac{e^{-z}}{1 + e^{-z}} = \sigma(z)(1 - \sigma(z))$$

3. Binary Cross-Entropy Loss (Log Loss)

To guarantee a convex optimization landscape with a single global minimum, logistic regression minimizes Binary Cross-Entropy (derived from Maximum Likelihood Estimation):

$$J(w, b) = -\frac{1}{m} \sum_{i=1}^{m} \left[ y^{(i)} \log(\hat{y}^{(i)}) + (1 - y^{(i)}) \log(1 - \hat{y}^{(i)}) \right]$$

• When True Class $y^{(i)} = 1$: Cost $= -\log(\hat{y}^{(i)})$. If $\hat{y} \approx 1$, cost $\to 0$. If $\hat{y} \approx 0$ (confidently wrong), cost $\to +\infty$.
• When True Class $y^{(i)} = 0$: Cost $= -\log(1 - \hat{y}^{(i)})$. Punishes confident false alarms exponentially.

4. Gradient Descent Update Rules

Applying the chain rule through Cross-Entropy and the Sigmoid derivative produces the exact gradient form as linear regression:

$$\frac{\partial J}{\partial w_j} = \frac{1}{m}\sum_{i=1}^{m} (\hat{y}^{(i)} - y^{(i)}) x_j^{(i)}, \qquad \frac{\partial J}{\partial b} = \frac{1}{m}\sum_{i=1}^{m} (\hat{y}^{(i)} - y^{(i)})$$

$$w_j := w_j - \alpha \frac{\partial J}{\partial w_j}, \qquad b := b - \alpha \frac{\partial J}{\partial b}$$

4. Worked Numerical Example (By Hand)

Let's perform 1 update step for 2 samples with initial parameters $w = 0.5, b = 0.0$ and learning rate $\alpha = 0.1$:

• Sample 1: $x = 1, y = 0 \implies z = 0.5(1) + 0 = 0.5 \implies \hat{y}_1 = \sigma(0.5) \approx 0.622$. Error: $0.622 - 0 = +0.622$.
• Sample 2: $x = 3, y = 1 \implies z = 0.5(3) + 0 = 1.5 \implies \hat{y}_2 = \sigma(1.5) \approx 0.818$. Error: $0.818 - 1 = -0.182$.
• Weight Gradient: $\frac{\partial J}{\partial w} = \frac{1}{2}[(0.622)(1) + (-0.182)(3)] = \frac{0.076}{2} = 0.038$.
• Bias Gradient: $\frac{\partial J}{\partial b} = \frac{1}{2}[0.622 - 0.182] = \frac{0.440}{2} = 0.220$.
• Updated Parameters: $w := 0.5 - 0.1(0.038) = 0.4962, \quad b := 0 - 0.1(0.220) = -0.022$.

5. Decision Threshold Strategy Matrix

6. Code: Logistic Regression in Python

1. NumPy Implementation From Scratch

import numpy as np

X = np.array([1, 2, 3, 4, 5, 6], dtype=float)
y = np.array([0, 0, 0, 1, 1, 1], dtype=float)

def sigmoid(z):
    return 1.0 / (1.0 + np.exp(-np.clip(z, -250, 250)))

def train_logistic_regression(X, y, alpha=0.1, epochs=3000):
    m = len(X)
    w, b = 0.0, 0.0
    
    for _ in range(epochs):
        z = w * X + b
        y_pred = sigmoid(z)
        error = y_pred - y
        
        # Gradients
        dw = (1 / m) * np.sum(error * X)
        db = (1 / m) * np.sum(error)
        
        w -= alpha * dw
        b -= alpha * db
        
    return w, b

w_fit, b_fit = train_logistic_regression(X, y)
print(f"Learned Weight: {w_fit:.4f}, Learned Bias: {b_fit:.4f}")

# Prediction helper with customizable decision threshold
def predict(x_val, w, b, threshold=0.5):
    prob = sigmoid(w * x_val + b)
    return int(prob >= threshold), prob

label, prob = predict(3.5, w_fit, b_fit, threshold=0.5)
print(f"Prediction for x=3.5 -> Probability: {prob:.3f}, Assigned Class: {label}")

2. Production Scikit-Learn

from sklearn.linear_model import LogisticRegression

X_2d = X.reshape(-1, 1)
model = LogisticRegression()
model.fit(X_2d, y)

# Evaluating probabilities vs hard predictions
prob_sample = model.predict_proba([[3.5]])[0]
print(f"Class 0 Probability: {prob_sample[0]:.3f}, Class 1 Probability: {prob_sample[1]:.3f}")
print("Predicted Class:", model.predict([[3.5]])[0])

Summary

• Logistic regression models binary probabilities by mapping linear inputs through the Sigmoid function $\sigma(z) = \frac{1}{1 + e^{-z}}$.
• Binary Cross-Entropy (Log Loss) guarantees a convex optimization surface, penalizing confident wrong predictions exponentially.
• The gradient $\frac{1}{m}\sum(\hat{y}-y)x$ mirrors linear regression, enabling efficient gradient descent parameter updates.
• The classification decision threshold can be tuned away from $0.5$ to optimize precision or recall based on domain requirements.