Random Forest vs. Gradient Boosting in Python: Ensemble Algorithms From Scratch

Random Forest and Gradient Boosting are the two most dominant tree-based ensemble learning paradigms in modern machine learning. While Random Forest builds dozens of deep, independent decision trees in parallel to reduce variance, Gradient Boosting trains a sequential chain of shallow decision trees where each new tree directly corrects the residual errors of its predecessors.

Imagine an investment firm trying to predict company valuations. In a Random Forest approach, you hire 100 independent analysts, each examining a random slice of financial indicators without consulting one another, and average their final price estimates (wisdom of crowds). In a Gradient Boosting approach, you hire a single analyst, identify their exact prediction mistakes, and hire a second specialist whose sole job is to correct those specific errors. You repeat this relay until residual error is minimized.

1. Head-to-Head Comparison: Bagging vs. Boosting

2. The Mathematical Foundations

1. Random Forest: Bagging & Variance Reduction

For regression, the ensemble prediction is the arithmetic average across $B$ bootstrap-trained trees:

$$\hat{y} = \frac{1}{B} \sum_{b=1}^{B} T_b(x)$$

The variance of the average of $B$ random variables with individual variance $\sigma^2$ and pairwise correlation $\rho$ is:

$$\text{Var}(\hat{y}) = \rho\sigma^2 + \frac{1-\rho}{B}\sigma^2$$

As $B \to \infty$, the second term vanishes, leaving $\rho\sigma^2$. Random Forests deliberately use feature subsampling (e.g. considering only $\sqrt{n}$ features per split) to drive $\rho$ toward zero, minimizing overall model variance.

2. Gradient Boosting: Sequential Residual Minimization

Step 1 (Initialize): Start with a baseline constant prediction $F_0(x) = \bar{y}$.

Step 2 (Compute Residuals): For iteration $t$, compute the negative gradient of the loss function. For Squared Error loss $L(y, \hat{y}) = \frac{1}{2}(y - \hat{y})^2$:

$$-\frac{\partial L}{\partial F_{t-1}(x_i)} = y_i - F_{t-1}(x_i) = r_i^{(t)}$$

Step 3 (Fit Weak Learner): Train a shallow tree $h_t(x)$ to predict residuals $r_i^{(t)}$.

Step 4 (Update Model): Update the ensemble prediction with shrinkage learning rate $\eta$:

$$F_t(x) = F_{t-1}(x) + \eta \cdot h_t(x)$$

Step 5 (Final Model): Summing over $T$ boosting rounds gives:

$$F_T(x) = F_0(x) + \eta \sum_{t=1}^{T} h_t(x)$$

3. Worked Numerical Trace (Gradient Boosting Step by Step)

Let's trace 2 iterations on $y = [10, 20, 30, 40]$ with learning rate $\eta = 0.5$:

• Initial Baseline $F_0$: $\bar{y} = \frac{10+20+30+40}{4} = 25.0$ for all samples.
• Iteration 1 Residuals: $r^{(1)} = y - F_0 = [-15.0, -5.0, +5.0, +15.0]$.
• Tree 1 Update $h_1$: Fits $r^{(1)} \implies F_1 = 25.0 + 0.5 \times [-15, -5, 5, 15] = [17.5, 22.5, 27.5, 32.5]$.
• Iteration 2 Residuals: $r^{(2)} = y - F_1 = [-7.5, -2.5, +2.5, +7.5]$.
• Tree 2 Update $h_2$: Fits $r^{(2)} \implies F_2 = F_1 + 0.5 \times [-7.5, -2.5, 2.5, 7.5] = [13.75, 21.25, 28.75, 36.25]$.

Notice how predictions steadily converge toward the true target values $[10, 20, 30, 40]$ without overshooting.

4. Python Code: Implementing Both From Scratch & Scikit-Learn

1. NumPy Ensemble Engine From Scratch

import numpy as np
from sklearn.tree import DecisionTreeRegressor

# 1. Random Forest (Bagging + Feature Subsampling)
class SimpleRandomForest:
    def __init__(self, n_trees=20, max_depth=4, max_features="sqrt"):
        self.n_trees = n_trees
        self.max_depth = max_depth
        self.max_features = max_features
        self.trees = []

    def fit(self, X, y):
        self.trees = []
        n_samples = X.shape[0]

        for _ in range(self.n_trees):
            # Bootstrap sample: sampling rows with replacement
            idx = np.random.choice(n_samples, n_samples, replace=True)
            tree = DecisionTreeRegressor(max_depth=self.max_depth, max_features=self.max_features)
            tree.fit(X[idx], y[idx])
            self.trees.append(tree)

    def predict(self, X):
        predictions = np.array([tree.predict(X) for tree in self.trees])
        return np.mean(predictions, axis=0)

# 2. Gradient Boosting (Sequential Residual Fitting)
class SimpleGradientBoosting:
    def __init__(self, n_trees=100, max_depth=2, learning_rate=0.1):
        self.n_trees = n_trees
        self.max_depth = max_depth
        self.learning_rate = learning_rate
        self.trees = []
        self.f0 = None

    def fit(self, X, y):
        self.f0 = np.mean(y)
        current_pred = np.full(y.shape, self.f0)
        self.trees = []

        for _ in range(self.n_trees):
            residuals = y - current_pred
            tree = DecisionTreeRegressor(max_depth=self.max_depth)
            tree.fit(X, residuals)
            current_pred += self.learning_rate * tree.predict(X)
            self.trees.append(tree)

    def predict(self, X):
        pred = np.full(X.shape[0], self.f0)
        for tree in self.trees:
            pred += self.learning_rate * tree.predict(X)
        return pred

# Synthetic test dataset
np.random.seed(42)
X = np.random.rand(200, 3) * 10
y = 3 * X[:, 0] - 2 * X[:, 1] + X[:, 2] + np.random.randn(200) * 1.5

rf = SimpleRandomForest(n_trees=30, max_depth=4)
rf.fit(X, y)

gb = SimpleGradientBoosting(n_trees=80, max_depth=2, learning_rate=0.1)
gb.fit(X, y)

print("Actual Targets (first 3):     ", np.round(y[:3], 2))
print("Random Forest Predictions:    ", np.round(rf.predict(X[:3]), 2))
print("Gradient Boosting Predictions:", np.round(gb.predict(X[:3]), 2))

2. Production Scikit-Learn with Early Stopping

from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor

# Production Random Forest (Parallel CPU execution)
rf_prod = RandomForestRegressor(n_estimators=100, max_depth=6, n_jobs=-1, random_state=42)
rf_prod.fit(X, y)

# Production Gradient Boosting (with early stopping to prevent overfitting)
gb_prod = GradientBoostingRegressor(
    n_estimators=200,
    learning_rate=0.05,
    max_depth=3,
    validation_fraction=0.15,
    n_iter_no_change=10,
    random_state=42
)
gb_prod.fit(X, y)

print(f"Gradient Boosting stopped early at tree #{gb_prod.n_estimators_}")

5. Common Pitfalls & Practical Advice

• Setting High Learning Rates in Boosting: Using $\eta > 0.3$ causes Gradient Boosting to aggressively fit noise, producing jagged decision boundaries. Always pair a lower learning rate ($\eta \in [0.01, 0.1]$) with early stopping.
• Assuming More Trees Always Fixes Underfitting: Adding more trees in Random Forest reduces variance until a plateau, but will never cure a high-bias model. If a Random Forest underfits, increase max_depth or reduce min_samples_split.
• Ignoring Outliers in Boosting: Because Gradient Boosting focuses on the largest residuals, extreme outliers receive enormous weight in subsequent iterations. Robust losses (e.g. Huber loss) or Random Forests handle heavy-tailed outliers better.

Summary

• Random Forest (Bagging): Combines deep, independent trees trained on bootstrap samples and random feature subsets to eliminate model variance.
• Gradient Boosting (Boosting): Chains shallow weak learners sequentially, with each tree performing gradient descent on the residuals to eliminate bias.
• Random Forests are robust, low-maintenance, and parallelizable, while Gradient Boosters (like XGBoost/LightGBM) deliver state-of-the-art accuracy when properly tuned with early stopping.