Definition
Input, hidden and output layers The input layer is the feature vector itself. Hidden layers are the layers in between, whose activations are never given targets directly. The output layer produces the prediction, and its width and activation are chosen to match what the task asks for.
The three roles are not equally free. You choose the hidden layers. The data fixes the input layer, and the task fixes the output layer. Getting the two ends right is mostly bookkeeping, but getting them wrong silently breaks training.
Input layer: the features
The input layer has no weights, no biases and no activation. It just holds . Its width is the number of features: 784 for a flattened image, 13 for a table with 13 numeric columns.
Since nothing in it is learned, whatever happens to the features happens before the network: standardize numeric columns to mean 0 and variance 1 (using training-set statistics), one-hot encode or embed categorical ones, and scale pixels to or standardize them. Features on wildly different scales make some weights need huge updates and others tiny ones, which slows gradient descent.
Hidden layers: learned representations
Each hidden layer computes , a new description of the example. No one tells hidden unit 3 what to detect; the loss on the output only says whether the final prediction was good, and backpropagation shapes the hidden units into whatever features make the output layer's job easy.
A useful way to read a network: the hidden layers transform the inputs into a representation , and the output layer is then just a linear model (linear regression, logistic regression or softmax regression) on top of that representation. Hidden layers need a nonlinearity such as ReLU, otherwise the whole stack collapses into one linear map.
Output layer: set by the task
The output layer computes logits and then an output activation. The task decides the width , the activation, and the loss that pairs with it:
| Task | Units | Activation | Output means | Loss |
|---|---|---|---|---|
| Regression | 1 (or one per target) | none (linear) | the predicted value, any real number | MSE (or MAE / Huber) |
| Binary classification | 1 | sigmoid | binary cross-entropy | |
| Multiclass (one of K) | K | softmax | a distribution over K classes, sums to 1 | cross-entropy |
| Multi-label (any of K) | K | K independent sigmoids | K separate probabilities, need not sum to 1 | binary cross-entropy per label |
- logits: the output layer's pre-activations
- sigmoid, squashes one logit into (0, 1)
- number of output units (classes or labels)
- probability of class ; all entries are positive and sum to 1
Worked example: softmax on three logits
Logits $z=(2.0,\ 1.0,\ 0.1)$ for classes cat, dog, bird
- 1Exponentiate: , , .
- 2Sum: .
- 3Divide: , which sums to 1.
- 4If the true class is cat, the cross-entropy loss is . If it is dog, the loss is .
- 5For contrast, three independent sigmoids on the same logits give , summing to : fine for multi-label, wrong for "pick one class".
Softmax depends only on differences between logits: adding the same constant to every leaves unchanged. Implementations subtract first ( here) so the exponentials cannot overflow.
1import torch, torch.nn as nn23hidden = nn.Sequential(nn.Linear(13, 64), nn.ReLU(), nn.Linear(64, 64), nn.ReLU())45regression = nn.Sequential(hidden, nn.Linear(64, 1)); loss = nn.MSELoss()6binary = nn.Sequential(hidden, nn.Linear(64, 1)); loss = nn.BCEWithLogitsLoss()7multiclass = nn.Sequential(hidden, nn.Linear(64, 10)); loss = nn.CrossEntropyLoss() # targets: class indices8multilabel = nn.Sequential(hidden, nn.Linear(64, 5)); loss = nn.BCEWithLogitsLoss() # targets: 0/1 vectors910torch.softmax(torch.tensor([2.0, 1.0, 0.1]), dim=0) # tensor([0.6590, 0.2424, 0.0986])Check yourself
A photo can contain any combination of {dog, car, tree}. Which output layer fits?
Check yourself
Logits (2.0, 1.0, 0.1) give softmax (0.659, 0.242, 0.099). What does softmax give for (5.0, 4.0, 3.1)?
Remember
- ✓The input layer is the feature vector: no parameters, so scaling and encoding happen before it.
- ✓Hidden layers learn representations without direct targets; the output layer is a linear model on top of them.
- ✓Regression: 1 linear unit + MSE. Binary: 1 sigmoid + BCE. Multiclass: K softmax + cross-entropy. Multi-label: K sigmoids + BCE.
- ✓softmax(2.0, 1.0, 0.1) = (0.659, 0.242, 0.099); it only depends on logit differences.
- ✓In PyTorch, end with a plain Linear layer and let the loss apply sigmoid or softmax.