RMSE, MAE, and R-Squared in Python: Regression Evaluation Metrics Explained

RMSE (Root Mean Squared Error), MAE (Mean Absolute Error), and $R^2$ (R-squared / Coefficient of Determination) are the three primary statistical metrics used to evaluate the accuracy and predictive power of regression models.

Imagine you are a daily temperature forecaster. At the end of the season, your station manager asks: "How accurate were your forecasts overall?" You cannot hand over a 90-row spreadsheet — you need a single, rigorous score to summarize performance. MAE tells you the raw average error in degrees, RMSE penalizes the days you were wildly off, and $R^2$ measures how much better your model is compared to just guessing the seasonal average every single day.

1. The Regression Evaluation Decision Matrix

2. Mathematical Formulations & Derivations

1. Mean Absolute Error (MAE)

Calculates the average absolute magnitude of the residual errors across all $m$ samples:

$$\text{MAE} = \frac{1}{m} \sum_{i=1}^{m} | \hat{y}^{(i)} - y^{(i)} |$$

2. Root Mean Squared Error (RMSE)

Squares residuals before averaging, then takes the square root to return to the target's original units:

$$\text{RMSE} = \sqrt{\frac{1}{m} \sum_{i=1}^{m} (\hat{y}^{(i)} - y^{(i)})^2}$$

Why $\text{RMSE} \ge \text{MAE}$ always: Because squaring magnifies larger numbers disproportionately ($20^2 = 400$ vs $10^2 = 100$), RMSE is always greater than or equal to MAE. A large gap between RMSE and MAE indicates that your model occasionally produces extreme outlier errors.

3. R-Squared ($R^2$) Formulation

Compares the model's Sum of Squared Residuals ($\text{SSR}$) against the Total Sum of Squares ($\text{SST}$) of a naive baseline predicting the sample mean $\bar{y}$:

$$R^2 = 1 - \frac{\text{SSR}}{\text{SST}} = 1 - \frac{\sum_{i=1}^{m} (y^{(i)} - \hat{y}^{(i)})^2}{\sum_{i=1}^{m} (y^{(i)} - \bar{y})^2}$$

• $R^2 = 1.0$: Perfect prediction ($\text{SSR} = 0$).
• $R^2 = 0.0$: The model performs no better than guessing the constant mean $\bar{y}$.
• $R^2 < 0.0$: The model performs worse than guessing the baseline average (common with severe overfitting or improper constraints).

4. Adjusted $R^2$ (Penalizing Feature Inflation)

Standard $R^2$ has a flaw: adding more feature columns will never decrease $R^2$, even if the features are random noise. Adjusted $R^2$ penalizes adding useless predictors $p$:

$$R^2_{\text{adj}} = 1 - \left[ \frac{(1 - R^2)(m - 1)}{m - p - 1} \right]$$

3. Worked Numerical Example (By Hand)

Let's evaluate a 4-sample house price dataset (in Lakhs ₹):

• $\text{MAE}$: $\frac{2 + 5 + 5 + 2}{4} = \frac{14}{4} = 3.50$ (Average error is ₹3.5 Lakh).
• $\text{RMSE}$: $\sqrt{\frac{4 + 25 + 25 + 4}{4}} = \sqrt{14.5} \approx 3.81$ Lakh.
• Mean $\bar{y}$: $\frac{50 + 60 + 55 + 70}{4} = 58.75$.
• $\text{SST}$: $(50-58.75)^2 + (60-58.75)^2 + (55-58.75)^2 + (70-58.75)^2 = 218.75$.
• $R^2$: $1 - \frac{58}{218.75} = 1 - 0.265 = 0.735$ (Model explains 73.5% of target variance).

4. Python Implementation: From Scratch & Scikit-Learn

1. NumPy Implementation From Scratch

import numpy as np

y_actual = np.array([50.0, 60.0, 55.0, 70.0])
y_pred   = np.array([48.0, 65.0, 50.0, 72.0])

def calculate_mae(y_true, y_pred):
    return np.mean(np.abs(y_true - y_pred))

def calculate_rmse(y_true, y_pred):
    return np.sqrt(np.mean((y_true - y_pred) ** 2))

def calculate_r2(y_true, y_pred):
    ssr = np.sum((y_true - y_pred) ** 2)
    sst = np.sum((y_true - np.mean(y_true)) ** 2)
    return 1 - (ssr / sst)

def calculate_adjusted_r2(y_true, y_pred, n_features):
    r2 = calculate_r2(y_true, y_pred)
    m = len(y_true)
    return 1 - ((1 - r2) * (m - 1) / (m - n_features - 1))

print(f"MAE:         {calculate_mae(y_actual, y_pred):.3f}")
print(f"RMSE:        {calculate_rmse(y_actual, y_pred):.3f}")
print(f"R²:          {calculate_r2(y_actual, y_pred):.3f}")
print(f"Adjusted R²: {calculate_adjusted_r2(y_actual, y_pred, n_features=1):.3f}")

2. Production Scikit-Learn Metrics

from sklearn.metrics import mean_absolute_error, root_mean_squared_error, r2_score

mae = mean_absolute_error(y_actual, y_pred)
rmse = root_mean_squared_error(y_actual, y_pred)
r2 = r2_score(y_actual, y_pred)

print(f"Scikit-Learn MAE:  {mae:.3f}")
print(f"Scikit-Learn RMSE: {rmse:.3f}")
print(f"Scikit-Learn R²:   {r2:.3f}")

3. The Outlier Stress Test (MAE vs RMSE)

# Introducing one extreme prediction mistake (150 instead of 65)
y_pred_outlier = np.array([48.0, 150.0, 50.0, 72.0])

print("--- Outlier Impact Comparison ---")
print(f"MAE  shifted from 3.50  -> {mean_absolute_error(y_actual, y_pred_outlier):.2f}")
print(f"RMSE shifted from 3.81  -> {root_mean_squared_error(y_actual, y_pred_outlier):.2f} (Exploded!)")
print(f"R²   shifted from 0.735 -> {r2_score(y_actual, y_pred_outlier):.3f} (Collapsed Negative!)")

5. Common Pitfalls When Evaluating Models

• Comparing RMSE Across Different Datasets: Because RMSE has units of the target variable, an RMSE of 5 on housing (lakhs) is fundamentally different from 5 on weather (°C). Use $R^2$ for cross-dataset comparisons.
• Relying on $R^2$ Alone: An $R^2$ of 0.92 sounds impressive, but if target variance is massive, your predictions could still be off by thousands of dollars. Always report $R^2$ alongside MAE or RMSE.
• Ignoring Negative $R^2$: A negative $R^2$ is not a bug; it proves your model performs worse than a horizontal line predicting the sample mean.

Summary

• MAE averages absolute residuals — intuitive, robust to outliers, and directly interpretable.
• RMSE squares residuals before taking the root — heavily penalizes large errors.
• $R^2$ measures percentage of variance explained relative to a mean baseline ($0–1$, can be negative).
• Adjusted $R^2$ prevents feature inflation by penalizing the addition of irrelevant predictor columns.