Input, hidden & output layers

The input layer holds the features, hidden layers learn intermediate representations, and the output layer's size and activation are set by the task.

7 min read564 wordscore0 viewsupdated 2026-10-09

Definition

Input, hidden and output layers The input layer is the feature vector xx itself. Hidden layers are the layers in between, whose activations are never given targets directly. The output layer produces the prediction, and its width and activation are chosen to match what the task asks for.

The three roles are not equally free. You choose the hidden layers. The data fixes the input layer, and the task fixes the output layer. Getting the two ends right is mostly bookkeeping, but getting them wrong silently breaks training.

Input layer: the features

The input layer has no weights, no biases and no activation. It just holds h(0)=xh^{(0)}=x. Its width is the number of features: 784 for a flattened 28×2828\times28 image, 13 for a table with 13 numeric columns.

Since nothing in it is learned, whatever happens to the features happens before the network: standardize numeric columns to mean 0 and variance 1 (using training-set statistics), one-hot encode or embed categorical ones, and scale pixels to [0,1][0,1] or standardize them. Features on wildly different scales make some weights need huge updates and others tiny ones, which slows gradient descent.

Hidden layers: learned representations

Each hidden layer computes h(ℓ)=ϕ(W(ℓ)h(ℓ−1)+b(ℓ))h^{(\ell)}=\phi(W^{(\ell)}h^{(\ell-1)}+b^{(\ell)}), a new description of the example. No one tells hidden unit 3 what to detect; the loss on the output only says whether the final prediction was good, and backpropagation shapes the hidden units into whatever features make the output layer's job easy.

A useful way to read a network: the hidden layers transform the inputs into a representation h(L−1)h^{(L-1)}, and the output layer is then just a linear model (linear regression, logistic regression or softmax regression) on top of that representation. Hidden layers need a nonlinearity such as ReLU, otherwise the whole stack collapses into one linear map.

Output layer: set by the task

The output layer computes logits z=W(L)h(L−1)+b(L)z=W^{(L)}h^{(L-1)}+b^{(L)} and then an output activation. The task decides the width KK, the activation, and the loss that pairs with it:

TaskUnitsActivationOutput meansLoss
Regression1 (or one per target)none (linear)the predicted value, any real numberMSE (or MAE / Huber)
Binary classification1sigmoidP(y=1∣x)∈(0,1)P(y=1\mid x)\in(0,1)binary cross-entropy
Multiclass (one of K)Ksoftmaxa distribution over K classes, sums to 1cross-entropy
Multi-label (any of K)KK independent sigmoidsK separate probabilities, need not sum to 1binary cross-entropy per label
Output layer and loss by task
σ(z)=11+e−z,softmax⁡(z)k=ezk∑j=1Kezj\sigma(z)=\frac{1}{1+e^{-z}},\qquad \operatorname{softmax}(z)_k=\frac{e^{z_k}}{\sum_{j=1}^{K}e^{z_j}}
zz
logits: the output layer's pre-activations W(L)h(L−1)+b(L)W^{(L)}h^{(L-1)}+b^{(L)}
σ\sigma
sigmoid, squashes one logit into (0, 1)
KK
number of output units (classes or labels)
softmax⁡(z)k\operatorname{softmax}(z)_k
probability of class kk; all KK entries are positive and sum to 1
1.2x1−0.4x20.5x31.410.000.290.710.65z=1.780.05z=−0.890.30z=1.00softmaxinput3 featureshidden4 · ReLUoutput3 · softmaxmulticlass (K = 3)units 3 · act softmaxloss cross-entropy−log p_trueoutput vector (bar) vs target (tick)cat0.655dog0.045bird0.300Σ outputs = 1.000 (always 1)loss = 0.423example A, target cat
task
example
fig.A fixed 3-4-K network. Pick a task: the output layer's width, activation and loss change, while the input values and the hidden ReLU activations stay the same. Switch examples A–C to see the hidden representation change. Drag the nudge slider to push the first logit up: with softmax the other probabilities fall so the sum stays 1, while with multi-label sigmoids only the first output moves.

Worked example: softmax on three logits

Logits $z=(2.0,\ 1.0,\ 0.1)$ for classes cat, dog, bird

  1. 1Exponentiate: e2.0=7.389e^{2.0}=7.389, e1.0=2.718e^{1.0}=2.718, e0.1=1.105e^{0.1}=1.105.
  2. 2Sum: 7.389+2.718+1.105=11.2137.389+2.718+1.105=11.213.
  3. 3Divide: p=(0.659, 0.242, 0.099)p=(0.659,\ 0.242,\ 0.099), which sums to 1.
  4. 4If the true class is cat, the cross-entropy loss is −ln⁡0.659=0.417-\ln 0.659=0.417. If it is dog, the loss is −ln⁡0.242=1.417-\ln 0.242=1.417.
  5. 5For contrast, three independent sigmoids on the same logits give (0.881, 0.731, 0.525)(0.881,\ 0.731,\ 0.525), summing to 2.1372.137: fine for multi-label, wrong for "pick one class".

Softmax depends only on differences between logits: adding the same constant to every zkz_k leaves pp unchanged. Implementations subtract max⁡kzk\max_k z_k first (e0,e−1,e−1.9e^{0}, e^{-1}, e^{-1.9} here) so the exponentials cannot overflow.

python
1import torch, torch.nn as nn
2
3hidden = nn.Sequential(nn.Linear(13, 64), nn.ReLU(), nn.Linear(64, 64), nn.ReLU())
4
5regression = nn.Sequential(hidden, nn.Linear(64, 1)); loss = nn.MSELoss()
6binary = nn.Sequential(hidden, nn.Linear(64, 1)); loss = nn.BCEWithLogitsLoss()
7multiclass = nn.Sequential(hidden, nn.Linear(64, 10)); loss = nn.CrossEntropyLoss() # targets: class indices
8multilabel = nn.Sequential(hidden, nn.Linear(64, 5)); loss = nn.BCEWithLogitsLoss() # targets: 0/1 vectors
9
10torch.softmax(torch.tensor([2.0, 1.0, 0.1]), dim=0) # tensor([0.6590, 0.2424, 0.0986])
Same hidden stack, four heads. The models end in a plain Linear layer; the loss applies the activation.

Check yourself

A photo can contain any combination of {dog, car, tree}. Which output layer fits?

Check yourself

Logits (2.0, 1.0, 0.1) give softmax (0.659, 0.242, 0.099). What does softmax give for (5.0, 4.0, 3.1)?

Remember

  • ✓The input layer is the feature vector: no parameters, so scaling and encoding happen before it.
  • ✓Hidden layers learn representations without direct targets; the output layer is a linear model on top of them.
  • ✓Regression: 1 linear unit + MSE. Binary: 1 sigmoid + BCE. Multiclass: K softmax + cross-entropy. Multi-label: K sigmoids + BCE.
  • ✓softmax(2.0, 1.0, 0.1) = (0.659, 0.242, 0.099); it only depends on logit differences.
  • ✓In PyTorch, end with a plain Linear layer and let the loss apply sigmoid or softmax.
← back to Deep Learning roadmap