# Parameters vs hyperparameters

> Parameters are learned by gradient descent; hyperparameters are chosen by you before training and tuned on validation data.

Source: https://www.mlfoundry.in/learn/parameters-vs-hyperparameters  
Section: Deep Learning › Deep Learning Fundamentals  
Level: core · 7 min read · updated 2026-10-09

**Parameters vs hyperparameters:** **Parameters** $\theta$ are the numbers inside the model (weights, biases) that the optimizer changes, step by step, to lower the training loss. **Hyperparameters** $\lambda$ are the settings that define the model and the training procedure (learning rate, batch size, depth, dropout). You fix them before a run starts, and gradient descent never touches them.

## Two nested loops

Training a network is really two optimization problems, one inside the other. The **inner loop** is ordinary training: for a fixed $\lambda$, gradient descent finds $\theta$. The **outer loop** is you (or a search algorithm) trying different $\lambda$ and keeping the one whose trained model scores best on **validation** data.

$$
\theta^*(\lambda)=\arg\min_\theta\;\mathcal L_{\text{train}}(\theta;\lambda),\qquad \lambda^*=\arg\min_\lambda\;\mathcal L_{\text{val}}\big(\theta^*(\lambda)\big)
$$

- $\theta$: parameters: weights, biases, BatchNorm γ and β, embedding tables
- $\lambda$: hyperparameters: learning rate, batch size, architecture, regularization strength, …
- $\mathcal L_{\text{train}}$: training loss, which gradient descent minimizes
- $\mathcal L_{\text{val}}$: loss (or 1 − accuracy) on held-out validation data

Why can't gradient descent learn $\lambda$ too? Many hyperparameters are discrete (number of layers, optimizer choice), and the rest have no useful training-loss gradient. If dropout $p$ or weight decay were tuned on the training loss, the optimizer would set them to zero, because regularization exists precisely to make the training loss worse in exchange for better generalization.

Side by side

| Aspect | Parameters | Hyperparameters |
| --- | --- | --- |
| Set by | the optimizer (gradient descent) | you, or a search procedure |
| When | updated every iteration during training | fixed before the run (or a fixed schedule) |
| Judged on | training loss | validation performance |
| How many | thousands to billions | usually 3 to 20 that matter |
| Examples | weights $W$, biases $b$, BatchNorm $\gamma,\beta$, embeddings | learning rate, batch size, epochs, depth, width, dropout $p$, weight decay, optimizer, init scheme |
| Saved in | the checkpoint (`state_dict`) | your config file or experiment log |

## Counting parameters

$$
\#\theta_{\text{dense}}=(n_{\text{in}}+1)\,n_{\text{out}},\qquad \#\theta_{\text{BatchNorm}}=2\,C
$$

- $n_{\text{in}}, n_{\text{out}}$: input and output width of a fully connected layer (the $+1$ is the bias)
- $C$: number of features or channels normalized; each gets one $\gamma$ and one $\beta$

**MNIST MLP: 784 → 256 (BatchNorm, ReLU) → 10**

1. First layer: $784\cdot256+256=200{,}960$.
2. BatchNorm on 256 features: $2\cdot256=512$ learnable ($\gamma,\beta$).
3. Output layer: $256\cdot10+10=2{,}570$. ReLU has none.
4. Total: $200{,}960+512+2{,}570=204{,}042$ parameters. The width 256, the depth (one hidden layer) and the choice of ReLU are hyperparameters: they decide how many parameters there are, but are not among them.

```python
import torch.nn as nn

model = nn.Sequential(nn.Linear(784, 256), nn.BatchNorm1d(256), nn.ReLU(), nn.Linear(256, 10))
print(sum(p.numel() for p in model.parameters()))  # 204042  learned by the optimizer
print(sum(b.numel() for b in model.buffers()))     # 513     running mean/var (256 + 256) + a step counter
```

Buffers are estimated during training but not by gradient descent, so they are neither parameters nor hyperparameters.

## Tuning hyperparameters

**Grid search** tries every combination of a few values per hyperparameter. **Random search** draws each trial independently from a range. With the same budget, random search is usually better, as Bergstra & Bengio (2012) showed.

The reason is that usually only one or two hyperparameters matter much. A $3\times3$ grid spends 9 runs but tries only **3** learning rates; the other 6 runs just repeat those learning rates with a different, unimportant setting. Nine random trials try **9** different learning rates.

There is a budget rule too. If a "good" setting fills the top 5% of the search space, $n$ random trials miss it with probability $0.95^n$. With 60 trials, $1-0.95^{60}\approx95.4\%$ of searches land at least one run in that top 5%, however many hyperparameters there are.

**Search scale-sensitive hyperparameters on a log scale.** Learning rate and weight decay matter by ratios: going from $10^{-4}$ to $10^{-3}$ is as big a change as going from $10^{-3}$ to $10^{-2}$. Sample the exponent uniformly, $\eta=10^{u}$ with $u\sim U(-5,-1)$. Sampling $\eta$ uniformly on $[10^{-5},10^{-1}]$ instead puts 90% of trials above $10^{-2}$.

**Bayesian optimization** (with a Gaussian-process or tree-structured Parzen estimator surrogate; TPE is Optuna's default) fits a surrogate of validation score as a function of $\lambda$ and picks the next trial where improvement looks likely. It pays off when each run is expensive. Methods such as Hyperband and ASHA add early stopping, abandoning runs whose learning curves look poor.

> Figure: Validation accuracy (darker = higher) depends sharply on learning rate and barely on dropout. At the default 9-run budget, grid search tries 3 learning rates and misses the narrow good band; random search tries 9. Drag the budget, redraw the random trials to see that random is better on average rather than every time, and switch to 'uniform in lr' to watch nearly every trial pile up at large learning rates.

Numbers from the figure (log-scale sampling). Random search averaged over 200 independent draws.

| Budget | Grid: distinct LRs | Grid best | Random: distinct LRs | Random best (mean) |
| --- | --- | --- | --- | --- |
| 9 (3×3) | 3 | 67.2% | 9 | 84.5% |
| 25 (5×5) | 5 | 71.8% | 25 | 90.0% |
| 49 (7×7) | 7 | 89.9% | 49 | 90.9% |

The optimum is 92.0%. The 3×3 grid only tries $10^{-5}$, $10^{-3}$ and $10^{-1}$, and the good band around $3.5\times10^{-3}$ falls between them until the grid is dense enough. Random search gets many more distinct looks at the dimension that matters.

```python
import numpy as np
rng = np.random.default_rng(0)

def sample():
    return {
        "lr": 10 ** rng.uniform(-5, -1),            # log-uniform
        "weight_decay": 10 ** rng.uniform(-6, -2),  # log-uniform
        "dropout": rng.uniform(0.0, 0.5),            # linear is fine
        "batch_size": int(rng.choice([32, 64, 128, 256])),
    }

best_acc, best_hp = -1.0, None
for trial in range(30):
    hp = sample()
    model = build_model(hp)                  # fresh parameters every trial
    train(model, train_loader, hp)           # inner loop: gradient descent learns parameters
    acc = evaluate(model, val_loader)        # outer loop: hyperparameters judged on validation
    if acc > best_acc:
        best_acc, best_hp = acc, hp

# Only after the search is finished: evaluate best_hp's model on the test set, once.
```

Random search. build_model, train and evaluate stand for your own code.

> **Never tune on the test set:** Every comparison you make on a dataset leaks information about it into your choices. Pick the learning rate, architecture or epoch count by test accuracy and the test score becomes an optimistically biased estimate, because you selected the configuration that got lucky on those particular examples. Tune on validation data (or cross-validation), and touch the test set once at the end.

> **Some things blur the line:** The number of epochs is a hyperparameter, but **early stopping** picks it automatically by watching validation loss. A learning-rate schedule is a hyperparameter even though $\eta$ changes during training, because the rule is fixed in advance. Methods that learn hyperparameters by gradient (on validation loss) exist, but they are the exception.

**Question:** Which of these is a parameter (learned by gradient descent)?

- A. The learning rate
- B. BatchNorm's scale γ
- C. The dropout probability p
- D. The number of hidden layers

**Answer:** B. BatchNorm's scale γ — $\gamma$ (and the shift $\beta$) receive gradients and are updated by the optimizer like any weight. The learning rate, dropout $p$ and depth are fixed before training and chosen on validation data.

**Question:** How many parameters does `nn.Linear(100, 50)` have?

- A. 5,000
- B. 5,050
- C. 5,100
- D. 150

**Answer:** B. 5,050 — $100\cdot50$ weights plus 50 biases: $(100+1)\cdot50=5{,}050$.

**Question:** You tune learning rate and momentum with a 16-run budget. How many distinct learning rates do a 4×4 grid and 16 random trials try?

- A. 4 and 4
- B. 16 and 16
- C. 4 and 16
- D. 16 and 4

**Answer:** C. 4 and 16 — The grid reuses each of its 4 learning rates with 4 momentum values. Every random trial draws its own learning rate, so you get 16 distinct values, which matters most if the learning rate is the hyperparameter that counts.

**Key points**

- Parameters $\theta$ are learned by the optimizer on training data; hyperparameters $\lambda$ are fixed before the run and chosen on validation data.
- A dense layer has $(n_{\text{in}}+1)\,n_{\text{out}}$ parameters; width and depth are hyperparameters that decide that count.
- With the same budget, random search beats grid search when only a few hyperparameters matter, because it tries more distinct values of each.
- Search learning rate and weight decay on a log scale: $\eta=10^{u}$, with $u$ uniform.
- Tune on validation data; evaluate on the test set once, at the end.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
