Parameters vs. Hyperparameters in Machine Learning: Differences, Math, and Python Examples

When building machine learning systems, few concepts create more initial confusion for practitioners than the distinction between model parameters and hyperparameters. Both are numeric values that dictate how a model behaves, and both directly control predictive performance. Yet, they sit on opposite sides of the mathematical boundary separating training from tuning.

Conflating the two leads to severe architectural bugs: attempting to optimize hyperparameters via ordinary gradient descent on training data causes catastrophic overfitting, while failing to identify learned parameters makes it impossible to inspect, interpret, or deploy production weights. Understanding this distinction unlocks the mechanics behind automated tuning algorithms like Grid Search, Random Search, and Bayesian Optimization.

1. Key Concepts & Mathematical Notation Glossary

Review the core mathematical notation defining the dual-level optimization hierarchy:

2. Mathematical Foundations: The Bilevel Optimization Problem

In statistical learning theory, training and tuning are not arbitrary conventions — they represent a formal bilevel optimization problem where an outer optimization problem embeds an inner optimization problem as a constraint:

Let us unpack this mathematical structure:

• The Inner Optimization (Parameter Learning): For a fixed set of hyperparameters $\lambda$, the model finds the optimal internal parameters $\theta^*(\lambda)$ by minimizing empirical training loss over $\mathcal{D}_{\text{train}}$. This is solved analytically (via OLS Normal Equations) or iteratively (via Gradient Descent).
• The Outer Optimization (Hyperparameter Tuning): The outer loop evaluates how well the fitted parameters $\theta^*(\lambda)$ generalize to unseen validation data $\mathcal{D}_{\text{val}}$. It systematically navigates the hyperparameter space $\Lambda$ to identify $\lambda^*$ that minimizes validation loss.

2.1 Why Hyperparameters Cannot Be Learned From Training Data

A fundamental question in machine learning is: Why can we not simply treat $\lambda$ as another parameter and minimize training loss with respect to both $\theta$ and $\lambda$ simultaneously?

Consider Ridge Regression with $L_2$ penalty $\alpha \ge 0$:

If the optimization algorithm is allowed to minimize this loss with respect to $\alpha$ during training, the partial derivative is:

Because $\sum w_j^2$ is strictly non-negative, any gradient descent step will continually push $\alpha$ toward its lower bound: $\alpha \to 0$. The optimizer would completely discard regularization, reverting to unconstrained Ordinary Least Squares! Similarly, for a Decision Tree, minimizing training loss with respect to max_depth would trivially drive $\text{depth} \to \infty$, memorizing training noise into leaf nodes of size 1. Hyperparameters regulate model capacity; evaluating them on training data destroys their regulatory function.

3. Master Comparison Matrix Across 6 Algorithm Families

Every machine learning algorithm exhibits a strict boundary between its internal parameters and external hyperparameters. Review this comprehensive comparative reference:

4. The Bilevel Optimization Pipeline Architecture

The relationship between hyperparameters and parameters is structured as a nested control loop:

5. Python Implementation: Scikit-Learn Trailing Underscore Convention

In Python's Scikit-Learn library, the distinction between parameters and hyperparameters is enforced through a strict naming convention:

• Hyperparameters are arguments supplied to the model's constructor (__init__()) prior to training, and can be inspected via model.get_params().
• Parameters do not exist when the estimator is created. They are populated strictly after calling .fit() and always end with a trailing underscore (e.g., model.coef_, model.intercept_, model.cluster_centers_).

import numpy as np
from sklearn.linear_model import Ridge
from sklearn.tree import DecisionTreeRegressor
from sklearn.cluster import KMeans

# Generate reproducible synthetic feature matrix
np.random.seed(42)
X = np.random.normal(0, 1, (100, 3))
y = 2.5 * X[:, 0] - 1.8 * X[:, 1] + 0.5 * X[:, 2] + np.random.normal(0, 0.2, 100)

print("="*65)
print("1. RIDGE REGRESSION: HYPERPARAMETER VS LEARNED PARAMETERS")
print("="*65)
# alpha is a HYPERPARAMETER (passed into constructor)
ridge = Ridge(alpha=10.0)
print("Before .fit(): Has coef_ attribute?", hasattr(ridge, "coef_"))

# Fitting discovers the PARAMETERS (coef_, intercept_)
ridge.fit(X, y)
print("After .fit():  Has coef_ attribute?", hasattr(ridge, "coef_"))
print(f"Configured Hyperparameter (alpha):     {ridge.alpha}")
print(f"Learned Weight Parameters (coef_):       {ridge.coef_.round(3)}")
print(f"Learned Bias Parameter (intercept_):     {ridge.intercept_:.3f}")

print("\n" + "="*65)
print("2. DECISION TREE: HYPERPARAMETER VS LEARNED TREE TOPOLOGY")
print("="*65)
# max_depth is a HYPERPARAMETER
tree = DecisionTreeRegressor(max_depth=3, random_state=42)
tree.fit(X, y)
print(f"Configured Hyperparameter (max_depth): {tree.max_depth}")
print(f"Learned Parameter: Node Count:          {tree.tree_.node_count}")
print(f"Learned Parameter: First 5 Split Feats: {tree.tree_.feature[:5]}")

print("\n" + "="*65)
print("3. K-MEANS: HYPERPARAMETER VS LEARNED CENTROID COORDINATES")
print("="*65)
# n_clusters is a HYPERPARAMETER
kmeans = KMeans(n_clusters=3, random_state=42, n_init=10)
kmeans.fit(X)
print(f"Configured Hyperparameter (n_clusters):  {kmeans.n_clusters}")
print(f"Learned Parameters (cluster_centers_):   Shape {kmeans.cluster_centers_.shape}")
print(f"Centroid Coordinates:\n{kmeans.cluster_centers_.round(2)}")

print("\n" + "="*65)
print("4. EXPERIMENT: HOW HYPERPARAMETERS GOVERN LEARNED PARAMETERS")
print("="*65)
# Varying alpha forces the optimization algorithm to find different weights
for alpha in [0.1, 1.0, 10.0, 100.0]:
    model = Ridge(alpha=alpha)
    model.fit(X, y)
    print(f"Hyperparameter alpha={alpha:5.1f} -> Learned coef_: {model.coef_.round(3)} | intercept: {model.intercept_:.3f}")

6. Empirical Results: How Hyperparameters Dictate Parameter Convergence

Executing the script demonstrates the mathematical coupling between the two tiers. Inspect the exact output produced across varying values of the hyperparameter $\alpha$:

Notice what occurs: the underlying dataset $X, y$ remained identical throughout all four runs. Yet, by altering a single hyperparameter ($\alpha$), we fundamentally transformed the parameters ($\mathbf{w}, b$) that the algorithm converged upon. When $\alpha = 100.0$, the parameter norm $\|\mathbf{w}\|_2$ collapsed by more than 55% relative to $\alpha = 0.1$.

7. Hand-Worked Trace: 1D Gradient Descent Parameter Updates

To visualize the distinction step-by-step, trace simple 1D linear regression: $\hat{y} = w x + b$. We have a single observation $(x = 2.0, y = 5.0)$.

• Hyperparameters (Fixed in Advance): Learning rate $\eta = 0.10$, total iterations $T = 2$.
• Initial Parameters: $w^{(0)} = 0.0$, $b^{(0)} = 0.0$.

The Mean Squared Error loss for a single point is $\mathcal{L} = \frac{1}{2}(\hat{y} - y)^2$. The analytical gradients with respect to the parameters are:

Throughout this trace, the learning rate $\eta = 0.10$ never changed — it is a hyperparameter dictated prior to execution. Meanwhile, the slope $w$ ($0.0 \to 1.0 \to 1.5$) and intercept $b$ ($0.0 \to 0.5 \to 0.75$) adapted dynamically by learning from the error signal.

8. Subtle Edge Cases: When the Boundary Blurs

While the conceptual distinction is sharp, advanced machine learning architectures introduce nuanced edge cases:

• Adaptive Learning Rate Optimizers (Adam, RMSprop): Standard gradient descent treats learning rate $\eta$ as a rigid hyperparameter. Adam calculates running averages of past gradients ($m_t, v_t$) to dynamically scale parameter-specific step sizes during training. However, the base learning rate $\eta_0$ and decay constants $\beta_1, \beta_2$ remain external hyperparameters.
• Learnable Hyperparameters (Bilevel Meta-Learning): In meta-learning ('learning to learn'), an inner network learns task weights while an outer meta-network updates regularization or architecture parameters across hundreds of tasks. Here, hyperparameters for the inner task become parameters for the outer meta-learner.
• Early Stopping: When training deep neural networks, monitoring validation loss to halt training after $E$ epochs converts the training duration — traditionally a hyperparameter — into a data-driven stopping condition.

9. Common Pitfalls & Misconceptions

• Confusing Predictions with Parameters: Output predictions (e.g., $\hat{y} = 345,000\text{ USD}$) are neither parameters nor hyperparameters. They are ephemeral mathematical outputs resulting from multiplying input features $X$ by learned parameters $\theta$.
• Hyperparameter Non-Transferability: A hyperparameter setting that yields top performance on one dataset (e.g., max_depth=5 on Tabular Churn) will frequently fail on a different dataset with different noise characteristics, dimensionality, or sample size. Hyperparameters must be re-tuned per dataset.
• The More Hyperparameters Fallacy: Models with dozens of tunable knobs (such as XGBoost or multi-layer neural networks) are not inherently superior to simpler models. Excessive hyperparameters expand the search space exponentially, increasing the risk of over-tuning to validation noise.

10. Summary & Practice Exercises

Consolidate your understanding of the training vs. tuning duality with these three hands-on challenges:

1. Inspect Scikit-Learn Estimator Internals: Initialize a RandomForestClassifier(). Use model.get_params() to list all hyperparameters before training. Fit it on any dataset and print all attributes in dir(model) that end with an underscore (_).
2. Regularization Shrinkage Curve: Take the Ridge script from Section 5. Create a logarithmic sweep of $\alpha \in [10^{-3}, 10^3]$. Plot $\alpha$ on the x-axis (log scale) against the $L_2$ norm of the learned coefficients $\|\mathbf{w}\|_2$ on the y-axis.
3. GridSearchCV Inspection: Run GridSearchCV on a Decision Tree. Inspect cv.best_params_ (the optimal hyperparameters discovered) versus cv.best_estimator_.tree_.node_count (the internal parameters resulting from that hyperparameter choice).