---
title: Logistic Regression From Scratch in Python: Deriving Sigmoid and Log Loss
source: https://app.sythra.ai/learn/machine-learning/logistic-regression-from-scratch-sigmoid-loss-python
topic: Machine Learning
updated: 2026-08-30
publisher: Sythra (https://app.sythra.ai)
---

# Logistic Regression From Scratch in Python: Deriving Sigmoid and Log Loss

Logistic Regression models binary probabilities by passing a linear combination of features through the Sigmoid function, optimizing weights using Binary Cross-Entropy (Log Loss) gradient descent.

_Source: [https://app.sythra.ai/learn/machine-learning/logistic-regression-from-scratch-sigmoid-loss-python](https://app.sythra.ai/learn/machine-learning/logistic-regression-from-scratch-sigmoid-loss-python) — free to read on Sythra._

## Key points

- Explains why linear regression MSE fails for classification due to non-convexity.
- Derives the Sigmoid function algebraically from log-odds (logit).
- Details Binary Cross-Entropy loss and the sigmoid derivative proof.
- Provides pure NumPy gradient descent implementation and Scikit-Learn threshold tuning.

**Logistic Regression** is a foundational supervised classification algorithm that estimates the probability of a discrete outcome (such as binary $0$ or $1$) by passing a linear combination of input features through the non-linear **Sigmoid (logistic) function**.

Imagine an oncologist evaluating diagnostic scans. The doctor does not deliver a blunt yes/no verdict immediately; instead, they weigh multiple risk factors (age, blood biomarkers, family history) to establish a calibrated degree of suspicion between 0% and 100%. Only when confidence crosses a defined medical threshold do they flag the case for further biopsy. Logistic Regression mirrors this exact clinical process: computing continuous probabilities first, and applying classification cutoffs second.

## 1. Why Linear Regression Fails for Classification

Using standard Ordinary Least Squares linear regression for binary classification presents two fatal flaws:

- **Unbounded Outputs:** Linear regression produces predictions anywhere on $(-\infty, +\infty)$ (e.g. $-0.35$ or $+1.42$), which cannot be interpreted as valid probabilities.
- **Non-Convex Loss Surface:** Pairing linear regression's Mean Squared Error (MSE) with non-linear probabilities produces a bumpy, non-convex cost landscape filled with local minima where gradient descent gets trapped.

## 2. The Mathematical Derivation (From Odds to Sigmoid)

### 1. Modeling the Log-Odds (Logit)

The ratio of the probability of an event happening ($p$) to it not happening ($1-p$) is called the **Odds**:

$$\text{Odds} = \frac{p}{1 - p}$$

Taking the natural logarithm yields the **Log-Odds (Logit)**, which maps probabilities $(0, 1)$ across all real numbers $(-\infty, +\infty)$. We set this equal to our linear combination:

$$\ln\left(\frac{p}{1 - p}\right) = w \cdot x + b = z$$

Exponentiating and solving for $p$ yields the **Sigmoid Function $\sigma(z)$**:

$$p = \frac{1}{1 + e^{-z}} = \sigma(w \cdot x + b)$$

### 2. The Calculus of the Sigmoid Derivative

The derivative of the sigmoid function possesses a remarkably clean closed-form structure:

$$\sigma'(z) = \frac{d}{dz}(1 + e^{-z})^{-1} = -(1 + e^{-z})^{-2}(-e^{-z}) = \frac{1}{1 + e^{-z}} \cdot \frac{e^{-z}}{1 + e^{-z}} = \sigma(z)(1 - \sigma(z))$$

## 3. Binary Cross-Entropy Loss (Log Loss)

To guarantee a convex optimization landscape with a single global minimum, logistic regression minimizes **Binary Cross-Entropy** (derived from Maximum Likelihood Estimation):

$$J(w, b) = -\frac{1}{m} \sum_{i=1}^{m} \left[ y^{(i)} \log(\hat{y}^{(i)}) + (1 - y^{(i)}) \log(1 - \hat{y}^{(i)}) \right]$$

- **When True Class $y^{(i)} = 1$:** Cost $= -\log(\hat{y}^{(i)})$. If $\hat{y} \approx 1$, cost $\to 0$. If $\hat{y} \approx 0$ (confidently wrong), cost $\to +\infty$.
- **When True Class $y^{(i)} = 0$:** Cost $= -\log(1 - \hat{y}^{(i)})$. Punishes confident false alarms exponentially.

### 4. Gradient Descent Update Rules

Applying the chain rule through Cross-Entropy and the Sigmoid derivative produces the exact gradient form as linear regression:

$$\frac{\partial J}{\partial w_j} = \frac{1}{m}\sum_{i=1}^{m} (\hat{y}^{(i)} - y^{(i)}) x_j^{(i)}, \qquad \frac{\partial J}{\partial b} = \frac{1}{m}\sum_{i=1}^{m} (\hat{y}^{(i)} - y^{(i)})$$

$$w_j := w_j - \alpha \frac{\partial J}{\partial w_j}, \qquad b := b - \alpha \frac{\partial J}{\partial b}$$

## 4. Worked Numerical Example (By Hand)

Let's perform 1 update step for 2 samples with initial parameters $w = 0.5, b = 0.0$ and learning rate $\alpha = 0.1$:

- **Sample 1:** $x = 1, y = 0 \implies z = 0.5(1) + 0 = 0.5 \implies \hat{y}_1 = \sigma(0.5) \approx 0.622$. Error: $0.622 - 0 = +0.622$.
- **Sample 2:** $x = 3, y = 1 \implies z = 0.5(3) + 0 = 1.5 \implies \hat{y}_2 = \sigma(1.5) \approx 0.818$. Error: $0.818 - 1 = -0.182$.
- **Weight Gradient:** $\frac{\partial J}{\partial w} = \frac{1}{2}[(0.622)(1) + (-0.182)(3)] = \frac{0.076}{2} = 0.038$.
- **Bias Gradient:** $\frac{\partial J}{\partial b} = \frac{1}{2}[0.622 - 0.182] = \frac{0.440}{2} = 0.220$.
- **Updated Parameters:** $w := 0.5 - 0.1(0.038) = 0.4962, \quad b := 0 - 0.1(0.220) = -0.022$.

## 5. Decision Threshold Strategy Matrix

| Threshold | False Positive Rate | False Negative Rate | Ideal Real-World Use Case |
| --- | --- | --- | --- |
| **$0.50$ (Balanced)** | Balanced | Balanced | Standard benchmark binary classification problems. |
| **$0.20 – 0.30$ (High Recall)** | Higher (more false alarms) | **Extremely Low** (catches nearly all positives) | Cancer screening, credit card fraud detection, critical safety alerts. |
| **$0.70 – 0.80$ (High Precision)** | **Extremely Low** (rare false alarms) | Higher (misses borderline cases) | Automatic spam email deletion, high-budget targeted marketing campaigns. |

## 6. Code: Logistic Regression in Python

### 1. NumPy Implementation From Scratch

```python
import numpy as np

X = np.array([1, 2, 3, 4, 5, 6], dtype=float)
y = np.array([0, 0, 0, 1, 1, 1], dtype=float)

def sigmoid(z):
    return 1.0 / (1.0 + np.exp(-np.clip(z, -250, 250)))

def train_logistic_regression(X, y, alpha=0.1, epochs=3000):
    m = len(X)
    w, b = 0.0, 0.0
    
    for _ in range(epochs):
        z = w * X + b
        y_pred = sigmoid(z)
        error = y_pred - y
        
        # Gradients
        dw = (1 / m) * np.sum(error * X)
        db = (1 / m) * np.sum(error)
        
        w -= alpha * dw
        b -= alpha * db
        
    return w, b

w_fit, b_fit = train_logistic_regression(X, y)
print(f"Learned Weight: {w_fit:.4f}, Learned Bias: {b_fit:.4f}")

# Prediction helper with customizable decision threshold
def predict(x_val, w, b, threshold=0.5):
    prob = sigmoid(w * x_val + b)
    return int(prob >= threshold), prob

label, prob = predict(3.5, w_fit, b_fit, threshold=0.5)
print(f"Prediction for x=3.5 -> Probability: {prob:.3f}, Assigned Class: {label}")
```

### 2. Production Scikit-Learn

```python
from sklearn.linear_model import LogisticRegression

X_2d = X.reshape(-1, 1)
model = LogisticRegression()
model.fit(X_2d, y)

# Evaluating probabilities vs hard predictions
prob_sample = model.predict_proba([[3.5]])[0]
print(f"Class 0 Probability: {prob_sample[0]:.3f}, Class 1 Probability: {prob_sample[1]:.3f}")
print("Predicted Class:", model.predict([[3.5]])[0])
```

## Summary

- Logistic regression models binary probabilities by mapping linear inputs through the **Sigmoid function** $\sigma(z) = \frac{1}{1 + e^{-z}}$.
- **Binary Cross-Entropy (Log Loss)** guarantees a convex optimization surface, penalizing confident wrong predictions exponentially.
- The gradient $\frac{1}{m}\sum(\hat{y}-y)x$ mirrors linear regression, enabling efficient gradient descent parameter updates.
- The classification decision threshold can be tuned away from $0.5$ to optimize precision or recall based on domain requirements.

## FAQ

### Why is Logistic Regression called 'regression' if it is used for classification?

It is named logistic regression because it computes a continuous weighted sum of features (linear regression) before passing the log-odds through the sigmoid function to output calibrated probabilities.

### Why is Binary Cross-Entropy used instead of Mean Squared Error (MSE)?

Combining MSE with the non-linear Sigmoid creates a non-convex loss function with multiple local minima. Binary Cross-Entropy produces a perfectly convex surface, guaranteeing that gradient descent finds the global minimum.

### What is the derivative of the Sigmoid function?

The derivative of the sigmoid function sigma(z) is sigma'(z) = sigma(z) * (1 - sigma(z)), which simplifies the gradient of binary cross-entropy into the clean form (y_pred - y) * x.

### How do you choose the decision threshold in Logistic Regression?

While 0.5 is the default threshold, you lower the threshold (0.2-0.3) in high-risk settings like medical diagnosis to maximize recall, or raise it (0.7-0.8) in spam detection to maximize precision.

---

Written by Sythra — Learn machine learning by building. Practice this topic with Sythra's AI tutor: https://app.sythra.ai/pricing