Definition
Parameters vs hyperparameters Parameters are the numbers inside the model (weights, biases) that the optimizer changes, step by step, to lower the training loss. Hyperparameters are the settings that define the model and the training procedure (learning rate, batch size, depth, dropout). You fix them before a run starts, and gradient descent never touches them.
Two nested loops
Training a network is really two optimization problems, one inside the other. The inner loop is ordinary training: for a fixed , gradient descent finds . The outer loop is you (or a search algorithm) trying different and keeping the one whose trained model scores best on validation data.
- parameters: weights, biases, BatchNorm γ and β, embedding tables
- hyperparameters: learning rate, batch size, architecture, regularization strength, …
- training loss, which gradient descent minimizes
- loss (or 1 − accuracy) on held-out validation data
Why can't gradient descent learn too? Many hyperparameters are discrete (number of layers, optimizer choice), and the rest have no useful training-loss gradient. If dropout or weight decay were tuned on the training loss, the optimizer would set them to zero, because regularization exists precisely to make the training loss worse in exchange for better generalization.
| Aspect | Parameters | Hyperparameters |
|---|---|---|
| Set by | the optimizer (gradient descent) | you, or a search procedure |
| When | updated every iteration during training | fixed before the run (or a fixed schedule) |
| Judged on | training loss | validation performance |
| How many | thousands to billions | usually 3 to 20 that matter |
| Examples | weights , biases , BatchNorm , embeddings | learning rate, batch size, epochs, depth, width, dropout , weight decay, optimizer, init scheme |
| Saved in | the checkpoint (state_dict) | your config file or experiment log |
Counting parameters
- input and output width of a fully connected layer (the is the bias)
- number of features or channels normalized; each gets one and one
MNIST MLP: 784 → 256 (BatchNorm, ReLU) → 10
- 1First layer: .
- 2BatchNorm on 256 features: learnable ().
- 3Output layer: . ReLU has none.
- 4Total: parameters. The width 256, the depth (one hidden layer) and the choice of ReLU are hyperparameters: they decide how many parameters there are, but are not among them.
1import torch.nn as nn23model = nn.Sequential(nn.Linear(784, 256), nn.BatchNorm1d(256), nn.ReLU(), nn.Linear(256, 10))4print(sum(p.numel() for p in model.parameters())) # 204042 learned by the optimizer5print(sum(b.numel() for b in model.buffers())) # 513 running mean/var (256 + 256) + a step counterTuning hyperparameters
Grid search tries every combination of a few values per hyperparameter. Random search draws each trial independently from a range. With the same budget, random search is usually better, as Bergstra & Bengio (2012) showed.
The reason is that usually only one or two hyperparameters matter much. A grid spends 9 runs but tries only 3 learning rates; the other 6 runs just repeat those learning rates with a different, unimportant setting. Nine random trials try 9 different learning rates.
There is a budget rule too. If a "good" setting fills the top 5% of the search space, random trials miss it with probability . With 60 trials, of searches land at least one run in that top 5%, however many hyperparameters there are.
Search scale-sensitive hyperparameters on a log scale. Learning rate and weight decay matter by ratios: going from to is as big a change as going from to . Sample the exponent uniformly, with . Sampling uniformly on instead puts 90% of trials above .
Bayesian optimization (with a Gaussian-process or tree-structured Parzen estimator surrogate; TPE is Optuna's default) fits a surrogate of validation score as a function of and picks the next trial where improvement looks likely. It pays off when each run is expensive. Methods such as Hyperband and ASHA add early stopping, abandoning runs whose learning curves look poor.
| Budget | Grid: distinct LRs | Grid best | Random: distinct LRs | Random best (mean) |
|---|---|---|---|---|
| 9 (3×3) | 3 | 67.2% | 9 | 84.5% |
| 25 (5×5) | 5 | 71.8% | 25 | 90.0% |
| 49 (7×7) | 7 | 89.9% | 49 | 90.9% |
The optimum is 92.0%. The 3×3 grid only tries , and , and the good band around falls between them until the grid is dense enough. Random search gets many more distinct looks at the dimension that matters.
1import numpy as np2rng = np.random.default_rng(0)34def sample():5 return {6 "lr": 10 ** rng.uniform(-5, -1), # log-uniform7 "weight_decay": 10 ** rng.uniform(-6, -2), # log-uniform8 "dropout": rng.uniform(0.0, 0.5), # linear is fine9 "batch_size": int(rng.choice([32, 64, 128, 256])),10 }1112best_acc, best_hp = -1.0, None13for trial in range(30):14 hp = sample()15 model = build_model(hp) # fresh parameters every trial16 train(model, train_loader, hp) # inner loop: gradient descent learns parameters17 acc = evaluate(model, val_loader) # outer loop: hyperparameters judged on validation18 if acc > best_acc:19 best_acc, best_hp = acc, hp2021# Only after the search is finished: evaluate best_hp's model on the test set, once.Check yourself
Which of these is a parameter (learned by gradient descent)?
Check yourself
How many parameters does nn.Linear(100, 50) have?
Check yourself
You tune learning rate and momentum with a 16-run budget. How many distinct learning rates do a 4×4 grid and 16 random trials try?
Remember
- ✓Parameters are learned by the optimizer on training data; hyperparameters are fixed before the run and chosen on validation data.
- ✓A dense layer has parameters; width and depth are hyperparameters that decide that count.
- ✓With the same budget, random search beats grid search when only a few hyperparameters matter, because it tries more distinct values of each.
- ✓Search learning rate and weight decay on a log scale: , with uniform.
- ✓Tune on validation data; evaluate on the test set once, at the end.