---
title: Parameters vs. Hyperparameters in Machine Learning: Differences, Math, and Python Examples
source: https://app.sythra.ai/learn/machine-learning/parameters-vs-hyperparameters-machine-learning-python
topic: Machine Learning
updated: 2026-09-10
publisher: Sythra (https://app.sythra.ai)
---

# Parameters vs. Hyperparameters in Machine Learning: Differences, Math, and Python Examples

In machine learning, the fundamental distinction between a parameter and a hyperparameter lies in whether the value is learned automatically from training data or configured externally prior to model fitting. A parameter (such as a linear regression slope, decision tree split threshold, or neural network connection weight) is internal to the model and iteratively discovered through an optimization algorithm like gradient descent or the normal equation. In contrast, a hyperparameter (such as regularization strength lambda, maximum tree depth, learning rate eta, or cluster count k) is external to the model, cannot be directly learned from single-dataset training loss without causing catastrophic overfitting, and must be selected through validation techniques like cross-validation, grid search, or Bayesian optimization.

_Source: [https://app.sythra.ai/learn/machine-learning/parameters-vs-hyperparameters-machine-learning-python](https://app.sythra.ai/learn/machine-learning/parameters-vs-hyperparameters-machine-learning-python) — free to read on Sythra._

## Key points

- Contrasts model parameters (learned internally from data) against hyperparameters (configured externally prior to training) using the Cake Baking mental model.
- Formalizes machine learning as a Bilevel Optimization problem, mathematically proving why optimizing hyperparameters on training data collapses generalization.
- Establishes Scikit-Learn's universal programmatic convention: hyperparameters are set in class constructors, while learned parameters end with a trailing underscore (model.coef_, model.cluster_centers_).
- Provides an exhaustive reference matrix comparing parameters versus hyperparameters across 6 core algorithm families: Linear Models, Regularized Regressors, Decision Trees, SVMs, K-Means, and Deep Neural Networks.
- Features a complete, runnable Python implementation verifying how sweeping the hyperparameter alpha in Ridge regression directly shrinks learned weight parameters.
- Traces 1D Gradient Descent step-by-step, illustrating how internal parameters evolve across iterations while the learning rate hyperparameter remains fixed.

When building machine learning systems, few concepts create more initial confusion for practitioners than the distinction between **model parameters** and **hyperparameters**. Both are numeric values that dictate how a model behaves, and both directly control predictive performance. Yet, they sit on opposite sides of the mathematical boundary separating _training_ from _tuning_.

Conflating the two leads to severe architectural bugs: attempting to optimize hyperparameters via ordinary gradient descent on training data causes catastrophic overfitting, while failing to identify learned parameters makes it impossible to inspect, interpret, or deploy production weights. Understanding this distinction unlocks the mechanics behind automated tuning algorithms like Grid Search, Random Search, and Bayesian Optimization.

> **THE CAKE BAKING & OVEN MENTAL MODEL:** Imagine baking a cake. Before placing the batter in the oven, you must configure external settings: the oven temperature (e.g., 350°F), the baking duration (e.g., 35 minutes), and the rack height. You set these knobs based on the recipe or prior experience — the batter has no say in them. **These are hyperparameters**. Once inside the oven, thermal energy triggers complex internal chemical reactions: gluten proteins cross-link, leavening gases expand, and surface sugars caramelize. The resulting crumb density, moisture level, and rise height emerge naturally from the process itself. **These are parameters**. You configure the hyperparameters; the system learns its parameters.

## 1. Key Concepts & Mathematical Notation Glossary

Review the core mathematical notation defining the dual-level optimization hierarchy:

| Symbol | Category | Mathematical Role | Concrete Domain Example |
| --- | --- | --- | --- |
| $\theta \in \mathbb{R}^p$ | Model Parameter | Internal model weights learned by minimizing empirical training loss $\mathcal{L}_{\text{train}}(\theta)$ | Linear regression coefficients $\mathbf{w}$, intercept $b$, neural net synaptic weights $W$. |
| $\lambda \in \Lambda$ | Hyperparameter | External configuration setting chosen prior to training that constrains the model capacity | Ridge regularization penalty $\alpha$, decision tree `max_depth`, cluster count $k$. |
| $\mathcal{D}_{\text{train}}$ | Training Dataset | Data split utilized strictly by the optimization algorithm to update $\theta$ | Historical feature-label pairs $(X_{\text{train}}, y_{\text{train}})$. |
| $\mathcal{D}_{\text{val}}$ | Validation Dataset | Holdout data split utilized strictly to evaluate generalization and search for $\lambda^*$ | Cross-validation folds $(X_{\text{val}}, y_{\text{val}})$. |
| $\mathcal{L}(\theta; \mathcal{D}, \lambda)$ | Training Loss | Differentiable objective function minimized by the inner optimization loop | Mean Squared Error (MSE), Binary Cross-Entropy, Margin Slack Penalty. |
| $\mathcal{V}(\theta^*(\lambda); \mathcal{D}_{\text{val}})$ | Validation Objective | Generalization metric evaluated across candidates in the outer hyperparameter loop | Validation RMSE, Classification $F_1$ Score, ROC-AUC, Silhouette Score. |

## 2. Mathematical Foundations: The Bilevel Optimization Problem

In statistical learning theory, training and tuning are not arbitrary conventions — they represent a formal **bilevel optimization problem** where an outer optimization problem embeds an inner optimization problem as a constraint:

Let us unpack this mathematical structure:

- **The Inner Optimization (Parameter Learning):** For a fixed set of hyperparameters $\lambda$, the model finds the optimal internal parameters $\theta^*(\lambda)$ by minimizing empirical training loss over $\mathcal{D}_{\text{train}}$. This is solved analytically (via OLS Normal Equations) or iteratively (via Gradient Descent).
- **The Outer Optimization (Hyperparameter Tuning):** The outer loop evaluates how well the fitted parameters $\theta^*(\lambda)$ generalize to unseen validation data $\mathcal{D}_{\text{val}}$. It systematically navigates the hyperparameter space $\Lambda$ to identify $\lambda^*$ that minimizes validation loss.

### 2.1 Why Hyperparameters Cannot Be Learned From Training Data

A fundamental question in machine learning is: _Why can we not simply treat $\lambda$ as another parameter and minimize training loss with respect to both $\theta$ and $\lambda$ simultaneously?_

Consider Ridge Regression with $L_2$ penalty $\alpha \ge 0$:

If the optimization algorithm is allowed to minimize this loss with respect to $\alpha$ during training, the partial derivative is:

Because $\sum w_j^2$ is strictly non-negative, any gradient descent step will continually push $\alpha$ toward its lower bound: **$\alpha \to 0$**. The optimizer would completely discard regularization, reverting to unconstrained Ordinary Least Squares! Similarly, for a Decision Tree, minimizing training loss with respect to `max_depth` would trivially drive $\text{depth} \to \infty$, memorizing training noise into leaf nodes of size 1. **Hyperparameters regulate model capacity; evaluating them on training data destroys their regulatory function.**

## 3. Master Comparison Matrix Across 6 Algorithm Families

Every machine learning algorithm exhibits a strict boundary between its internal parameters and external hyperparameters. Review this comprehensive comparative reference:

| Algorithm Family | Learned Parameters (Internal $\theta$) | Configured Hyperparameters (External $\lambda$) | Optimization Mechanism |
| --- | --- | --- | --- |
| **Linear / Logistic Regression** | Feature weights $\mathbf{w} \in \mathbb{R}^p$, intercept $b \in \mathbb{R}$ | Fit intercept boolean, solver algorithm (`'lbfgs'`, `'saga'`), tolerance $\epsilon$, max iterations | Normal Equations: $(X^TX)^{-1}X^Ty$ or Iterative Gradient Descent |
| **Ridge / Lasso Regression** | Shrunk or sparse weight vector $\mathbf{w}$, bias $b$ | Regularization strength $\alpha$ (or $\lambda$), $L_1$ ratio (ElasticNet) | Coordinate Descent or Ridge closed-form: $(X^TX + \alpha I)^{-1}X^Ty$ |
| **Decision Trees / Random Forests** | Internal tree topology: split features, split thresholds $\tau_m$, leaf values $\hat{y}_m$ | `max_depth`, `min_samples_split`, `n_estimators`, `max_features`, splitting criterion | Greedy recursive binary splitting (CART algorithm maximizing Gini or MSE drop) |
| **Support Vector Machines (SVM)** | Dual Lagrange multipliers $\alpha_i$, support vectors $x_i$, bias $b$ | Box constraint $C$, kernel type (RBF, Polynomial), kernel bandwidth $\gamma$, degree $d$ | Sequential Minimal Optimization (SMO) solving convex quadratic programming |
| **K-Means Clustering** | Centroid coordinate vectors $\mu_1, \dots, \mu_k \in \mathbb{R}^d$ | Cluster count $k$, initialization scheme (`'k-means++'`), `n_init`, max iterations | Lloyd's alternating expectation-maximization (Assign $\to$ Update) |
| **Deep Neural Networks (MLP / CNN)** | Synaptic weight matrices $W^{[l]}$, layer bias vectors $\mathbf{b}^{[l]}$ | Learning rate $\eta$, batch size $B$, layer depth $L$, hidden units, dropout rate $p$, optimizer, weight decay | Backpropagation with stochastic optimizers (SGD, Adam, RMSprop) |

## 4. The Bilevel Optimization Pipeline Architecture

The relationship between hyperparameters and parameters is structured as a nested control loop:

## 5. Python Implementation: Scikit-Learn Trailing Underscore Convention

In Python's Scikit-Learn library, the distinction between parameters and hyperparameters is enforced through a strict **naming convention**:

- **Hyperparameters** are arguments supplied to the model's constructor (`__init__()`) prior to training, and can be inspected via `model.get_params()`.
- **Parameters** do not exist when the estimator is created. They are populated strictly after calling `.fit()` and always end with a **trailing underscore** (e.g., `model.coef_`, `model.intercept_`, `model.cluster_centers_`).

```python
import numpy as np
from sklearn.linear_model import Ridge
from sklearn.tree import DecisionTreeRegressor
from sklearn.cluster import KMeans

# Generate reproducible synthetic feature matrix
np.random.seed(42)
X = np.random.normal(0, 1, (100, 3))
y = 2.5 * X[:, 0] - 1.8 * X[:, 1] + 0.5 * X[:, 2] + np.random.normal(0, 0.2, 100)

print("="*65)
print("1. RIDGE REGRESSION: HYPERPARAMETER VS LEARNED PARAMETERS")
print("="*65)
# alpha is a HYPERPARAMETER (passed into constructor)
ridge = Ridge(alpha=10.0)
print("Before .fit(): Has coef_ attribute?", hasattr(ridge, "coef_"))

# Fitting discovers the PARAMETERS (coef_, intercept_)
ridge.fit(X, y)
print("After .fit():  Has coef_ attribute?", hasattr(ridge, "coef_"))
print(f"Configured Hyperparameter (alpha):     {ridge.alpha}")
print(f"Learned Weight Parameters (coef_):       {ridge.coef_.round(3)}")
print(f"Learned Bias Parameter (intercept_):     {ridge.intercept_:.3f}")

print("\n" + "="*65)
print("2. DECISION TREE: HYPERPARAMETER VS LEARNED TREE TOPOLOGY")
print("="*65)
# max_depth is a HYPERPARAMETER
tree = DecisionTreeRegressor(max_depth=3, random_state=42)
tree.fit(X, y)
print(f"Configured Hyperparameter (max_depth): {tree.max_depth}")
print(f"Learned Parameter: Node Count:          {tree.tree_.node_count}")
print(f"Learned Parameter: First 5 Split Feats: {tree.tree_.feature[:5]}")

print("\n" + "="*65)
print("3. K-MEANS: HYPERPARAMETER VS LEARNED CENTROID COORDINATES")
print("="*65)
# n_clusters is a HYPERPARAMETER
kmeans = KMeans(n_clusters=3, random_state=42, n_init=10)
kmeans.fit(X)
print(f"Configured Hyperparameter (n_clusters):  {kmeans.n_clusters}")
print(f"Learned Parameters (cluster_centers_):   Shape {kmeans.cluster_centers_.shape}")
print(f"Centroid Coordinates:\n{kmeans.cluster_centers_.round(2)}")

print("\n" + "="*65)
print("4. EXPERIMENT: HOW HYPERPARAMETERS GOVERN LEARNED PARAMETERS")
print("="*65)
# Varying alpha forces the optimization algorithm to find different weights
for alpha in [0.1, 1.0, 10.0, 100.0]:
    model = Ridge(alpha=alpha)
    model.fit(X, y)
    print(f"Hyperparameter alpha={alpha:5.1f} -> Learned coef_: {model.coef_.round(3)} | intercept: {model.intercept_:.3f}")
```

## 6. Empirical Results: How Hyperparameters Dictate Parameter Convergence

Executing the script demonstrates the mathematical coupling between the two tiers. Inspect the exact output produced across varying values of the hyperparameter $\alpha$:

| Hyperparameter $\alpha$ | Learned $w_1$ | Learned $w_2$ | Learned $w_3$ | Learned Intercept $b$ | Parameter Norm $\\|\mathbf{w}\\|_2$ |
| --- | --- | --- | --- | --- | --- |
| $\alpha = 0.1$ | $+2.481$ | $-1.808$ | $+0.478$ | $+0.023$ | $3.107$ (Near OLS truth) |
| $\alpha = 1.0$ | $+2.448$ | $-1.793$ | $+0.473$ | $+0.029$ | $3.071$ |
| $\alpha = 10.0$ | $+2.164$ | $-1.655$ | $+0.431$ | $+0.084$ | $2.758$ (Noticeable shrinkage) |
| $\alpha = 100.0$ | $+1.005$ | $-0.927$ | $+0.244$ | $+0.338$ | $1.389$ (Heavy penalty shrinkage) |

Notice what occurs: the underlying dataset $X, y$ remained identical throughout all four runs. Yet, by altering a single **hyperparameter** ($\alpha$), we fundamentally transformed the **parameters** ($\mathbf{w}, b$) that the algorithm converged upon. When $\alpha = 100.0$, the parameter norm $\|\mathbf{w}\|_2$ collapsed by more than 55% relative to $\alpha = 0.1$.

## 7. Hand-Worked Trace: 1D Gradient Descent Parameter Updates

To visualize the distinction step-by-step, trace simple 1D linear regression: $\hat{y} = w x + b$. We have a single observation $(x = 2.0, y = 5.0)$.

- **Hyperparameters (Fixed in Advance):** Learning rate $\eta = 0.10$, total iterations $T = 2$.
- **Initial Parameters:** $w^{(0)} = 0.0$, $b^{(0)} = 0.0$.

The Mean Squared Error loss for a single point is $\mathcal{L} = \frac{1}{2}(\hat{y} - y)^2$. The analytical gradients with respect to the parameters are:

| Iteration $t$ | Parameters $(w, b)$ | Prediction $\hat{y}$ | Error $(\hat{y} - y)$ | Gradients $(\nabla_w, \nabla_b)$ | Parameter Update ($w - \eta \nabla_w, b - \eta \nabla_b$) |
| --- | --- | --- | --- | --- | --- |
| $t = 0$ | $w=0.00, \; b=0.00$ | $0.00 \cdot 2 + 0 = 0.0$ | $0.0 - 5.0 = -5.0$ | $\nabla_w = -10.0, \; \nabla_b = -5.0$ | $w^{(1)} = 0 - 0.1(-10) = \mathbf{1.00}, \; b^{(1)} = 0 - 0.1(-5) = \mathbf{0.50}$ |
| $t = 1$ | $w=1.00, \; b=0.50$ | $1.00 \cdot 2 + 0.5 = 2.5$ | $2.5 - 5.0 = -2.5$ | $\nabla_w = -5.0, \; \nabla_b = -2.5$ | $w^{(2)} = 1.0 - 0.1(-5) = \mathbf{1.50}, \; b^{(2)} = 0.5 - 0.1(-2.5) = \mathbf{0.75}$ |

Throughout this trace, the **learning rate $\eta = 0.10$ never changed** — it is a hyperparameter dictated prior to execution. Meanwhile, the slope $w$ ($0.0 \to 1.0 \to 1.5$) and intercept $b$ ($0.0 \to 0.5 \to 0.75$) adapted dynamically by learning from the error signal.

## 8. Subtle Edge Cases: When the Boundary Blurs

While the conceptual distinction is sharp, advanced machine learning architectures introduce nuanced edge cases:

- **Adaptive Learning Rate Optimizers (Adam, RMSprop):** Standard gradient descent treats learning rate $\eta$ as a rigid hyperparameter. Adam calculates running averages of past gradients ($m_t, v_t$) to dynamically scale parameter-specific step sizes during training. However, the _base learning rate_ $\eta_0$ and decay constants $\beta_1, \beta_2$ remain external hyperparameters.
- **Learnable Hyperparameters (Bilevel Meta-Learning):** In meta-learning ('learning to learn'), an inner network learns task weights while an outer meta-network updates regularization or architecture parameters across hundreds of tasks. Here, hyperparameters for the inner task become parameters for the outer meta-learner.
- **Early Stopping:** When training deep neural networks, monitoring validation loss to halt training after $E$ epochs converts the training duration — traditionally a hyperparameter — into a data-driven stopping condition.

## 9. Common Pitfalls & Misconceptions

- **Confusing Predictions with Parameters:** Output predictions (e.g., $\hat{y} = 345,000\text{ USD}$) are neither parameters nor hyperparameters. They are ephemeral mathematical outputs resulting from multiplying input features $X$ by learned parameters $\theta$.
- **Hyperparameter Non-Transferability:** A hyperparameter setting that yields top performance on one dataset (e.g., `max_depth=5` on Tabular Churn) will frequently fail on a different dataset with different noise characteristics, dimensionality, or sample size. Hyperparameters must be re-tuned per dataset.
- **The More Hyperparameters Fallacy:** Models with dozens of tunable knobs (such as XGBoost or multi-layer neural networks) are not inherently superior to simpler models. Excessive hyperparameters expand the search space exponentially, increasing the risk of over-tuning to validation noise.

## 10. Summary & Practice Exercises

Consolidate your understanding of the training vs. tuning duality with these three hands-on challenges:

1. **Inspect Scikit-Learn Estimator Internals:** Initialize a `RandomForestClassifier()`. Use `model.get_params()` to list all hyperparameters before training. Fit it on any dataset and print all attributes in `dir(model)` that end with an underscore (`_`).
2. **Regularization Shrinkage Curve:** Take the Ridge script from Section 5. Create a logarithmic sweep of $\alpha \in [10^{-3}, 10^3]$. Plot $\alpha$ on the x-axis (log scale) against the $L_2$ norm of the learned coefficients $\|\mathbf{w}\|_2$ on the y-axis.
3. **GridSearchCV Inspection:** Run `GridSearchCV` on a Decision Tree. Inspect `cv.best_params_` (the optimal hyperparameters discovered) versus `cv.best_estimator_.tree_.node_count` (the internal parameters resulting from that hyperparameter choice).

> **WHAT TO LEARN NEXT:** Now that you understand what hyperparameters are and why they cannot be learned on training data, explore how automated search algorithms systematically discover optimal settings. Read our companion guide: **Grid Search vs Random Search vs Bayesian Optimization: Implemented and Compared**.

---

Written by Sythra — Learn machine learning by building. Practice this topic with Sythra's AI tutor: https://app.sythra.ai/pricing