---
title: Linear Regression From Scratch in Python: Deriving Gradient Descent Step by Step
source: https://app.sythra.ai/learn/machine-learning/linear-regression-from-scratch-gradient-descent-python
topic: Machine Learning
updated: 2026-08-28
publisher: Sythra (https://app.sythra.ai)
---

# Linear Regression From Scratch in Python: Deriving Gradient Descent Step by Step

Linear Regression models the linear relationship between features and continuous targets (y = wx + b) by minimizing Mean Squared Error using Gradient Descent, an optimization algorithm that iteratively adjusts slope and intercept in the opposite direction of the error gradient until convergence.

_Source: [https://app.sythra.ai/learn/machine-learning/linear-regression-from-scratch-gradient-descent-python](https://app.sythra.ai/learn/machine-learning/linear-regression-from-scratch-gradient-descent-python) — free to read on Sythra._

## Key points

- Comprehensive explanation of Linear Regression concepts, line of best fit, slope, and intercept.
- Deep dive into Gradient Descent intuition, the gradient vector, and learning rate dynamics.
- Calculus chain-rule derivation of partial derivatives for weights and bias.
- NumPy implementations from scratch (univariate and multi-feature) plus Scikit-Learn benchmarks.

**Linear Regression** is a foundational supervised learning algorithm that models the linear relationship between input features ($x$) and a continuous target ($y$) by fitting a straight line. **Gradient Descent** is the general-purpose optimization algorithm that iteratively searches for the model parameters (weights and biases) that minimize prediction error.

Imagine you have a scatter plot of study hours versus test scores. Linear regression is the concept of placing a straight ruler through those dots to capture the overall trend. But because points are scattered with natural variance, no line will pass through every single point. **Gradient Descent** is the mathematical mechanism that continuously rotates and shifts the ruler until the total error across all data points reaches the absolute lowest possible value.

## 1. What Is Linear Regression? (The Foundation)

Linear regression models the relationship between an independent variable $x$ and dependent target $y$ using the standard slope-intercept form:

$$\hat{y} = wx + b$$

- **$x$ (Input Feature):** The independent variable (e.g. house square footage, advertising budget, or study hours).
- **$\hat{y}$ (Predicted Value / Hypothesis):** The model's predicted continuous output (e.g. house price, revenue, or exam score).
- **$w$ (Weight / Slope):** Controls the steepness and direction of the line — how much $\hat{y}$ increases or decreases per 1-unit increase in $x$.
- **$b$ (Bias / Intercept):** Where the line crosses the y-axis (the baseline prediction when $x = 0$).

## 2. The Cost Function: Measuring Model Error (MSE)

For every training sample $i$, the difference between the true label $y^{(i)}$ and the line's prediction $\hat{y}^{(i)}$ is the **residual error** ($e_i = \hat{y}^{(i)} - y^{(i)}$).

To evaluate how well our line fits the entire dataset of $m$ observations, we calculate the **Mean Squared Error (MSE)** cost function $J(w, b)$:

$$J(w, b) = \frac{1}{2m} \sum_{i=1}^{m} (\hat{y}^{(i)} - y^{(i)})^2 = \frac{1}{2m} \sum_{i=1}^{m} (wx^{(i)} + b - y^{(i)})^2$$

- **Why square the residuals?** Squaring eliminates negative signs (preventing errors from canceling out to zero) and heavily penalizes large outlier mistakes.
- **Why the $\frac{1}{2}$ multiplier?** When taking the derivative of a squared term ($u^2$), the power rule brings down a factor of 2. Multiplying by $\frac{1}{2}$ neatly cancels the 2, leaving clean derivatives.
- **Strict Convexity:** The MSE loss surface forms a smooth 3D paraboloid (bowl shape). It is mathematically convex, guaranteeing that any local minimum is simultaneously the global minimum.

## 3. What Is Gradient Descent? (Deep Dive & Intuition)

**Gradient Descent** is an iterative first-order optimization algorithm used to find the local (or global) minimum of a differentiable function. In machine learning, it is the universal engine that adjusts weights to minimize the cost function $J$.

### The Blindfolded Fog Valley Metaphor

Imagine you are standing on the hillside of a fog-covered mountain bowl, blindfolded. Your mission is to reach the absolute lowest point in the valley floor (where error is 0). Because you cannot see the landscape, you feel the slope of the ground directly under your boots:

- **1. Feel the slope (The Gradient):** You measure whether the terrain is sloping up or down along the north-south ($w$) and east-west ($b$) directions.
- **2. Step downhill (Opposite to the Gradient):** The gradient points uphill in the direction of steepest incline. To decrease elevation, you step in the exact opposite direction ($-\nabla J$).
- **3. Step size (The Learning Rate $\alpha$):** When the hill is very steep (far from minimum), you take larger strides. As the terrain levels out near the bottom, your strides naturally shrink to micro-steps until you stand on completely flat ground (gradient = 0).

### Why Not Just Solve the Normal Equation?

For simple linear regression, the closed-form **Normal Equation** ($(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y}$) can calculate the exact optimal parameters in a single step. However, computing the inverse of an $(p \times p)$ matrix has a time complexity of $O(p^3)$.

If your dataset has 50,000 features (like text n-grams or image pixels), inverting that matrix requires trillions of operations. **Gradient Descent scales linearly ($O(m \cdot p)$)** per epoch, works on datasets with billions of rows, and is the exact backbone used to train modern Deep Neural Networks and LLMs.

## 4. Mathematical Derivation of the Gradient Updates

The gradient vector $\nabla J$ is the vector of partial derivatives with respect to each parameter:

$$\nabla J = \begin{bmatrix} \frac{\partial J}{\partial w} \\[6pt] \frac{\partial J}{\partial b} \end{bmatrix}$$

### Deriving Partial Derivative w.r.t Slope $w$

Let $u = (wx^{(i)} + b - y^{(i)})$. Using the calculus chain rule $\frac{\partial J}{\partial w} = \frac{\partial J}{\partial u} \cdot \frac{\partial u}{\partial w}$:

$$\frac{\partial J}{\partial w} = \frac{1}{2m} \sum_{i=1}^{m} 2(wx^{(i)} + b - y^{(i)}) \cdot x^{(i)} = \frac{1}{m} \sum_{i=1}^{m} (\hat{y}^{(i)} - y^{(i)}) x^{(i)}$$

### Deriving Partial Derivative w.r.t Bias $b$

Similarly, differentiating with respect to $b$ (where $\frac{\partial u}{\partial b} = 1$):

$$\frac{\partial J}{\partial b} = \frac{1}{m} \sum_{i=1}^{m} (\hat{y}^{(i)} - y^{(i)})$$

### The Simultaneous Parameter Update Rule

In every epoch, both $w$ and $b$ are updated simultaneously using learning rate $\alpha$:

$$w := w - \alpha \frac{\partial J}{\partial w}, \quad b := b - \alpha \frac{\partial J}{\partial b}$$

### Vectorized Matrix Update (Multi-Feature Regression)

For a dataset with $p$ features $\mathbf{X} \in \mathbb{R}^{m \times p}$ and weight vector $\mathbf{w} \in \mathbb{R}^p$:

$$\hat{\mathbf{y}} = \mathbf{X}\mathbf{w} + b, \quad \nabla_{\mathbf{w}} J = \frac{1}{m} \mathbf{X}^T (\hat{\mathbf{y}} - \mathbf{y}), \quad \mathbf{w} := \mathbf{w} - \alpha \nabla_{\mathbf{w}} J$$

## 5. The 3 Variants of Gradient Descent

| Variant | Data Used per Update | Convergence Trajectory | Pros & Cons |
| --- | --- | --- | --- |
| **Batch Gradient Descent** | Entire dataset of $m$ rows | Smooth, direct path straight to the global minimum. | Stable convergence, but very slow on massive datasets. |
| **Stochastic Gradient Descent (SGD)** | 1 single randomly sampled row | Noisy, oscillating path that jumps around the minimum. | Super fast updates and low memory; escapes shallow saddle points. |
| **Mini-Batch Gradient Descent** | Small batches (32, 64, 128, 256 rows) | Stable, fast path combining the benefits of Batch and SGD. | Industry gold standard; maximizes GPU parallel matrix acceleration. |

## 6. Worked Numerical Example (By Hand)

Let's trace 1 iteration by hand on 3 data points: $(1, 3), (2, 5), (3, 7)$ (where the true ground truth line is $y = 2x + 1$):

- **Initial state:** Initialize $w = 0.0, b = 0.0$, learning rate $\alpha = 0.1$.
- **Forward predictions $\hat{y}$:** $\hat{y}_1 = 0(1)+0=0, \quad \hat{y}_2 = 0(2)+0=0, \quad \hat{y}_3 = 0(3)+0=0$.
- **Residual errors $(\hat{y} - y)$:** $0-3 = -3, \quad 0-5 = -5, \quad 0-7 = -7$.
- **Gradient for slope $w$:** $\frac{\partial J}{\partial w} = \frac{1}{3}[(-3)(1) + (-5)(2) + (-7)(3)] = \frac{-34}{3} \approx -11.33$.
- **Gradient for intercept $b$:** $\frac{\partial J}{\partial b} = \frac{1}{3}[-3 + -5 + -7] = \frac{-15}{3} = -5.00$.
- **Update $w$:** $w := 0.0 - (0.1)(-11.33) = 1.133$.
- **Update $b$:** $b := 0.0 - (0.1)(-5.00) = 0.500$.

In just a single iteration, the line shifted from $(w=0, b=0)$ to $(w=1.133, b=0.500)$, immediately moving closer to the true line ($w=2, b=1$).

## 7. Python Implementation: From Scratch to Scikit-Learn

### 1. Univariate Linear Regression From Scratch (NumPy)

```python
import numpy as np

# Synthetic dataset: true relationship y = 2x + 1
X = np.array([1.0, 2.0, 3.0, 4.0, 5.0])
y = np.array([3.0, 5.0, 7.0, 9.0, 11.0])

def linear_regression_gradient_descent(X, y, alpha=0.05, epochs=1000):
    m = len(X)
    w = 0.0
    b = 0.0
    cost_history = []
    
    for epoch in range(epochs):
        # 1. Forward prediction
        y_hat = w * X + b
        error = y_hat - y
        
        # 2. Compute gradients
        dw = (1 / m) * np.sum(error * X)
        db = (1 / m) * np.sum(error)
        
        # 3. Update parameters
        w = w - alpha * dw
        b = b - alpha * db
        
        # 4. Record MSE cost
        cost = (1 / (2 * m)) * np.sum(error ** 2)
        cost_history.append(cost)
        
    return w, b, cost_history

w_opt, b_opt, history = linear_regression_gradient_descent(X, y, alpha=0.05, epochs=1000)
print(f"Optimal Slope (w): {w_opt:.4f}")
print(f"Optimal Intercept (b): {b_opt:.4f}")
print(f"Final MSE Loss: {history[-1]:.6e}")
```

### 2. Vectorized Multi-Feature Regression Class

```python
class LinearRegressionScratch:
    def __init__(self, lr=0.01, n_iters=1000):
        self.lr = lr
        self.n_iters = n_iters
        self.weights = None
        self.bias = None
        
    def fit(self, X, y):
        m, n_features = X.shape
        self.weights = np.zeros(n_features)
        self.bias = 0.0
        
        for _ in range(self.n_iters):
            y_predicted = np.dot(X, self.weights) + self.bias
            error = y_predicted - y
            
            # Vectorized gradient calculation
            dw = (1 / m) * np.dot(X.T, error)
            db = (1 / m) * np.sum(error)
            
            self.weights -= self.lr * dw
            self.bias -= self.lr * db
            
    def predict(self, X):
        return np.dot(X, self.weights) + self.bias

# Testing multi-feature regression: y = 1*x1 + 2*x2 + 0
X_multi = np.array([[1, 2], [2, 3], [3, 4], [4, 5], [5, 6]], dtype=float)
y_multi = np.array([5, 8, 11, 14, 17], dtype=float)

model = LinearRegressionScratch(lr=0.01, n_iters=2000)
model.fit(X_multi, y_multi)
print("Learned Multi-Feature Weights:", model.weights.round(2))
print("Learned Bias:", round(model.bias, 2))
```

### 3. Scikit-Learn Production Benchmarks

```python
from sklearn.linear_model import LinearRegression, SGDRegressor
from sklearn.preprocessing import StandardScaler

# 1. Closed-Form Normal Equation (sklearn LinearRegression)
lr_closed = LinearRegression()
lr_closed.fit(X.reshape(-1, 1), y)
print(f"Normal Equation -> w: {lr_closed.coef_[0]:.2f}, b: {lr_closed.intercept_:.2f}")

# 2. Iterative Gradient Descent (sklearn SGDRegressor)
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X.reshape(-1, 1))

sgd = SGDRegressor(max_iter=1000, learning_rate="constant", eta0=0.05, random_state=42)
sgd.fit(X_scaled, y)
print("SGDRegressor Fitted Successfully (Iterative)")
```

## 8. Pitfalls & Tuning the Learning Rate (\alpha)

- **Learning Rate Too High ($\alpha > 1.0$):** Overshoots the valley floor and bounces out of the bowl, causing loss to explode toward infinity (NaN).
- **Learning Rate Too Low ($\alpha = 10^{-6}$):** Requires millions of iterations to converge, consuming massive compute unnecessarily.
- **Unscaled Features:** Distorts the circular loss bowl into an elongated canyon, causing the gradient to bounce back and forth instead of moving directly downhill. Always apply `StandardScaler`.

## Summary

- Linear regression finds the line of best fit ($y = wx + b$) by minimizing Mean Squared Error (MSE).
- Gradient descent iteratively computes partial derivatives and steps opposite to the slope: $w := w - \alpha \nabla J$.
- Because MSE is strictly convex, gradient descent is mathematically guaranteed to find the global minimum.
- Gradient descent scales to massive datasets where matrix inversion ($O(p^3)$) fails.

## FAQ

### What is the difference between Linear Regression and Gradient Descent?

Linear Regression is the machine learning model that assumes a straight-line relationship (y = wx + b) between inputs and outputs. Gradient Descent is the optimization algorithm used to train the model by finding the weight and bias values that minimize prediction error.

### How does Gradient Descent work in simple terms?

Gradient Descent starts with initial guesses for parameters, calculates the slope (derivative) of the error curve, and takes small steps in the downhill direction until it reaches the lowest point (minimum error).

### Why use Gradient Descent instead of the Normal Equation for Linear Regression?

The Normal Equation requires inverting an (X^T X) matrix with O(p^3) time complexity. For datasets with thousands or millions of features, matrix inversion is computationally impossible, whereas Gradient Descent scales efficiently via O(m * p) matrix-vector operations.

### Can Linear Regression get stuck in a local minimum during Gradient Descent?

No. The Mean Squared Error cost function for linear regression is strictly convex (a smooth bowl shape), which guarantees that any local minimum is also the global minimum.

---

Written by Sythra — Learn machine learning by building. Practice this topic with Sythra's AI tutor: https://app.sythra.ai/pricing