Parameters vs hyperparameters

Parameters are learned by gradient descent; hyperparameters are chosen by you before training and tuned on validation data.

7 min read683 wordscore0 viewsupdated 2026-10-09

Definition

Parameters vs hyperparameters Parameters θ\theta are the numbers inside the model (weights, biases) that the optimizer changes, step by step, to lower the training loss. Hyperparameters λ\lambda are the settings that define the model and the training procedure (learning rate, batch size, depth, dropout). You fix them before a run starts, and gradient descent never touches them.

Two nested loops

Training a network is really two optimization problems, one inside the other. The inner loop is ordinary training: for a fixed λ\lambda, gradient descent finds θ\theta. The outer loop is you (or a search algorithm) trying different λ\lambda and keeping the one whose trained model scores best on validation data.

θ∗(λ)=arg⁡min⁡θ  Ltrain(θ;λ),λ∗=arg⁡min⁡λ  Lval(θ∗(λ))\theta^*(\lambda)=\arg\min_\theta\;\mathcal L_{\text{train}}(\theta;\lambda),\qquad \lambda^*=\arg\min_\lambda\;\mathcal L_{\text{val}}\big(\theta^*(\lambda)\big)
θ\theta
parameters: weights, biases, BatchNorm γ and β, embedding tables
λ\lambda
hyperparameters: learning rate, batch size, architecture, regularization strength, …
Ltrain\mathcal L_{\text{train}}
training loss, which gradient descent minimizes
Lval\mathcal L_{\text{val}}
loss (or 1 − accuracy) on held-out validation data

Why can't gradient descent learn λ\lambda too? Many hyperparameters are discrete (number of layers, optimizer choice), and the rest have no useful training-loss gradient. If dropout pp or weight decay were tuned on the training loss, the optimizer would set them to zero, because regularization exists precisely to make the training loss worse in exchange for better generalization.

AspectParametersHyperparameters
Set bythe optimizer (gradient descent)you, or a search procedure
Whenupdated every iteration during trainingfixed before the run (or a fixed schedule)
Judged ontraining lossvalidation performance
How manythousands to billionsusually 3 to 20 that matter
Examplesweights WW, biases bb, BatchNorm γ,β\gamma,\beta, embeddingslearning rate, batch size, epochs, depth, width, dropout pp, weight decay, optimizer, init scheme
Saved inthe checkpoint (state_dict)your config file or experiment log
Side by side

Counting parameters

#θdense=(nin+1) nout,#θBatchNorm=2 C\#\theta_{\text{dense}}=(n_{\text{in}}+1)\,n_{\text{out}},\qquad \#\theta_{\text{BatchNorm}}=2\,C
nin,noutn_{\text{in}}, n_{\text{out}}
input and output width of a fully connected layer (the +1+1 is the bias)
CC
number of features or channels normalized; each gets one γ\gamma and one β\beta

MNIST MLP: 784 → 256 (BatchNorm, ReLU) → 10

  1. 1First layer: 784⋅256+256=200,960784\cdot256+256=200{,}960.
  2. 2BatchNorm on 256 features: 2⋅256=5122\cdot256=512 learnable (γ,β\gamma,\beta).
  3. 3Output layer: 256⋅10+10=2,570256\cdot10+10=2{,}570. ReLU has none.
  4. 4Total: 200,960+512+2,570=204,042200{,}960+512+2{,}570=204{,}042 parameters. The width 256, the depth (one hidden layer) and the choice of ReLU are hyperparameters: they decide how many parameters there are, but are not among them.
python
1import torch.nn as nn
2
3model = nn.Sequential(nn.Linear(784, 256), nn.BatchNorm1d(256), nn.ReLU(), nn.Linear(256, 10))
4print(sum(p.numel() for p in model.parameters())) # 204042 learned by the optimizer
5print(sum(b.numel() for b in model.buffers())) # 513 running mean/var (256 + 256) + a step counter
Buffers are estimated during training but not by gradient descent, so they are neither parameters nor hyperparameters.

Tuning hyperparameters

Grid search tries every combination of a few values per hyperparameter. Random search draws each trial independently from a range. With the same budget, random search is usually better, as Bergstra & Bengio (2012) showed.

The reason is that usually only one or two hyperparameters matter much. A 3×33\times3 grid spends 9 runs but tries only 3 learning rates; the other 6 runs just repeat those learning rates with a different, unimportant setting. Nine random trials try 9 different learning rates.

There is a budget rule too. If a "good" setting fills the top 5% of the search space, nn random trials miss it with probability 0.95n0.95^n. With 60 trials, 1−0.9560≈95.4%1-0.95^{60}\approx95.4\% of searches land at least one run in that top 5%, however many hyperparameters there are.

Search scale-sensitive hyperparameters on a log scale. Learning rate and weight decay matter by ratios: going from 10−410^{-4} to 10−310^{-3} is as big a change as going from 10−310^{-3} to 10−210^{-2}. Sample the exponent uniformly, η=10u\eta=10^{u} with u∼U(−5,−1)u\sim U(-5,-1). Sampling η\eta uniformly on [10−5,10−1][10^{-5},10^{-1}] instead puts 90% of trials above 10−210^{-2}.

Bayesian optimization (with a Gaussian-process or tree-structured Parzen estimator surrogate; TPE is Optuna's default) fits a surrogate of validation score as a function of λ\lambda and picks the next trial where improvement looks likely. It pays off when each run is expensive. Methods such as Hyperband and ASHA add early stopping, abandoning runs whose learning curves look poor.

grid search · 9 runs3 distinct learning rates · best 67.2%1e-51e-41e-31e-21e-100.250.5dropout plearning rate (log axis) →on the lr axis: best acc at each lrrandom search · 9 runs9 distinct learning rates · best 90.9%1e-51e-41e-31e-21e-100.250.5learning rate (log axis) →on the lr axis: best acc at each lr
sample learning rate|
grid: 3 learning rates, best 67.2% at lr 1.0e-3
random: 9 learning rates, best 90.9% at lr 3.9e-3
random, mean best over 200 draws: 84.5%
optimum 92.0% at lr 3.5e-3, p = 0.2 · dropout moves acc by ≤ 4 pts
fig.Validation accuracy (darker = higher) depends sharply on learning rate and barely on dropout. At the default 9-run budget, grid search tries 3 learning rates and misses the narrow good band; random search tries 9. Drag the budget, redraw the random trials to see that random is better on average rather than every time, and switch to 'uniform in lr' to watch nearly every trial pile up at large learning rates.
BudgetGrid: distinct LRsGrid bestRandom: distinct LRsRandom best (mean)
9 (3×3)367.2%984.5%
25 (5×5)571.8%2590.0%
49 (7×7)789.9%4990.9%
Numbers from the figure (log-scale sampling). Random search averaged over 200 independent draws.

The optimum is 92.0%. The 3×3 grid only tries 10−510^{-5}, 10−310^{-3} and 10−110^{-1}, and the good band around 3.5×10−33.5\times10^{-3} falls between them until the grid is dense enough. Random search gets many more distinct looks at the dimension that matters.

python
1import numpy as np
2rng = np.random.default_rng(0)
3
4def sample():
5 return {
6 "lr": 10 ** rng.uniform(-5, -1), # log-uniform
7 "weight_decay": 10 ** rng.uniform(-6, -2), # log-uniform
8 "dropout": rng.uniform(0.0, 0.5), # linear is fine
9 "batch_size": int(rng.choice([32, 64, 128, 256])),
10 }
11
12best_acc, best_hp = -1.0, None
13for trial in range(30):
14 hp = sample()
15 model = build_model(hp) # fresh parameters every trial
16 train(model, train_loader, hp) # inner loop: gradient descent learns parameters
17 acc = evaluate(model, val_loader) # outer loop: hyperparameters judged on validation
18 if acc > best_acc:
19 best_acc, best_hp = acc, hp
20
21# Only after the search is finished: evaluate best_hp's model on the test set, once.
Random search. build_model, train and evaluate stand for your own code.

Check yourself

Which of these is a parameter (learned by gradient descent)?

Check yourself

How many parameters does nn.Linear(100, 50) have?

Check yourself

You tune learning rate and momentum with a 16-run budget. How many distinct learning rates do a 4×4 grid and 16 random trials try?

Remember

  • ✓Parameters θ\theta are learned by the optimizer on training data; hyperparameters λ\lambda are fixed before the run and chosen on validation data.
  • ✓A dense layer has (nin+1) nout(n_{\text{in}}+1)\,n_{\text{out}} parameters; width and depth are hyperparameters that decide that count.
  • ✓With the same budget, random search beats grid search when only a few hyperparameters matter, because it tries more distinct values of each.
  • ✓Search learning rate and weight decay on a log scale: η=10u\eta=10^{u}, with uu uniform.
  • ✓Tune on validation data; evaluate on the test set once, at the end.
← back to Deep Learning roadmap