# Input, hidden & output layers

> The input layer holds the features, hidden layers learn intermediate representations, and the output layer's size and activation are set by the task.

Source: https://www.mlfoundry.in/learn/input-hidden-output-layers  
Section: Deep Learning › Deep Learning Fundamentals  
Level: core · 7 min read · updated 2026-10-09

**Input, hidden and output layers:** The **input layer** is the feature vector $x$ itself. **Hidden layers** are the layers in between, whose activations are never given targets directly. The **output layer** produces the prediction, and its width and activation are chosen to match what the task asks for.

The three roles are not equally free. You choose the hidden layers. The data fixes the input layer, and the task fixes the output layer. Getting the two ends right is mostly bookkeeping, but getting them wrong silently breaks training.

## Input layer: the features

The input layer has no weights, no biases and no activation. It just holds $h^{(0)}=x$. Its width is the number of features: 784 for a flattened $28\times28$ image, 13 for a table with 13 numeric columns.

Since nothing in it is learned, whatever happens to the features happens **before** the network: standardize numeric columns to mean 0 and variance 1 (using training-set statistics), one-hot encode or embed categorical ones, and scale pixels to $[0,1]$ or standardize them. Features on wildly different scales make some weights need huge updates and others tiny ones, which slows gradient descent.

## Hidden layers: learned representations

Each hidden layer computes $h^{(\ell)}=\phi(W^{(\ell)}h^{(\ell-1)}+b^{(\ell)})$, a new description of the example. No one tells hidden unit 3 what to detect; the loss on the output only says whether the final prediction was good, and backpropagation shapes the hidden units into whatever features make the output layer's job easy.

A useful way to read a network: the hidden layers transform the inputs into a representation $h^{(L-1)}$, and the output layer is then just a linear model (linear regression, logistic regression or softmax regression) on top of that representation. Hidden layers need a nonlinearity such as ReLU, otherwise the whole stack collapses into one linear map.

## Output layer: set by the task

The output layer computes logits $z=W^{(L)}h^{(L-1)}+b^{(L)}$ and then an output activation. The task decides the width $K$, the activation, and the loss that pairs with it:

Output layer and loss by task

| Task | Units | Activation | Output means | Loss |
| --- | --- | --- | --- | --- |
| Regression | 1 (or one per target) | none (linear) | the predicted value, any real number | MSE (or MAE / Huber) |
| Binary classification | 1 | sigmoid | $P(y=1\mid x)\in(0,1)$ | binary cross-entropy |
| Multiclass (one of K) | K | softmax | a distribution over K classes, sums to 1 | cross-entropy |
| Multi-label (any of K) | K | K independent sigmoids | K separate probabilities, need not sum to 1 | binary cross-entropy per label |

$$
\sigma(z)=\frac{1}{1+e^{-z}},\qquad \operatorname{softmax}(z)_k=\frac{e^{z_k}}{\sum_{j=1}^{K}e^{z_j}}
$$

- $z$: logits: the output layer's pre-activations $W^{(L)}h^{(L-1)}+b^{(L)}$
- $\sigma$: sigmoid, squashes one logit into (0, 1)
- $K$: number of output units (classes or labels)
- $\operatorname{softmax}(z)_k$: probability of class $k$; all $K$ entries are positive and sum to 1

> Figure: A fixed 3-4-K network. Pick a task: the output layer's width, activation and loss change, while the input values and the hidden ReLU activations stay the same. Switch examples A–C to see the hidden representation change. Drag the nudge slider to push the first logit up: with softmax the other probabilities fall so the sum stays 1, while with multi-label sigmoids only the first output moves.

## Worked example: softmax on three logits

**Logits $z=(2.0,\ 1.0,\ 0.1)$ for classes cat, dog, bird**

1. Exponentiate: $e^{2.0}=7.389$, $e^{1.0}=2.718$, $e^{0.1}=1.105$.
2. Sum: $7.389+2.718+1.105=11.213$.
3. Divide: $p=(0.659,\ 0.242,\ 0.099)$, which sums to 1.
4. If the true class is cat, the cross-entropy loss is $-\ln 0.659=0.417$. If it is dog, the loss is $-\ln 0.242=1.417$.
5. For contrast, three independent sigmoids on the same logits give $(0.881,\ 0.731,\ 0.525)$, summing to $2.137$: fine for multi-label, wrong for "pick one class".

Softmax depends only on differences between logits: adding the same constant to every $z_k$ leaves $p$ unchanged. Implementations subtract $\max_k z_k$ first ($e^{0}, e^{-1}, e^{-1.9}$ here) so the exponentials cannot overflow.

```python
import torch, torch.nn as nn

hidden = nn.Sequential(nn.Linear(13, 64), nn.ReLU(), nn.Linear(64, 64), nn.ReLU())

regression = nn.Sequential(hidden, nn.Linear(64, 1));  loss = nn.MSELoss()
binary     = nn.Sequential(hidden, nn.Linear(64, 1));  loss = nn.BCEWithLogitsLoss()
multiclass = nn.Sequential(hidden, nn.Linear(64, 10)); loss = nn.CrossEntropyLoss()  # targets: class indices
multilabel = nn.Sequential(hidden, nn.Linear(64, 5));  loss = nn.BCEWithLogitsLoss() # targets: 0/1 vectors

torch.softmax(torch.tensor([2.0, 1.0, 0.1]), dim=0)  # tensor([0.6590, 0.2424, 0.0986])
```

Same hidden stack, four heads. The models end in a plain Linear layer; the loss applies the activation.

> **Don't apply softmax twice:** `nn.CrossEntropyLoss` expects raw logits and applies log-softmax internally, and `nn.BCEWithLogitsLoss` applies the sigmoid itself. Adding `nn.Softmax` or `nn.Sigmoid` before them squashes the outputs twice, which shrinks gradients and stalls training. Apply the activation only at inference, when you need probabilities.

> **The loss and the activation are a pair:** Sigmoid + BCE and softmax + cross-entropy are matched: the gradient of the loss with respect to the logits is simply $p-y$. Pairing sigmoid with MSE instead gives a gradient multiplied by $\sigma'(z)$, which vanishes when a confident prediction is wrong.

**Question:** A photo can contain any combination of {dog, car, tree}. Which output layer fits?

- A. 1 unit, sigmoid
- B. 3 units, softmax
- C. 3 units, independent sigmoids
- D. 3 units, linear with MSE

**Answer:** C. 3 units, independent sigmoids — This is multi-label: each label is its own yes/no question, so each unit gets its own sigmoid and binary cross-entropy. Softmax would force the three probabilities to compete and sum to 1, so a photo with both a dog and a car could not get high probability for both.

**Question:** Logits (2.0, 1.0, 0.1) give softmax (0.659, 0.242, 0.099). What does softmax give for (5.0, 4.0, 3.1)?

- A. (0.659, 0.242, 0.099)
- B. (0.857, 0.116, 0.027)
- C. (0.333, 0.333, 0.333)
- D. It depends on the hidden layers

**Answer:** A. (0.659, 0.242, 0.099) — Every logit increased by 3. Softmax is invariant to adding a constant to all logits, because $e^{z_k+c}/\sum_j e^{z_j+c}=e^{z_k}/\sum_j e^{z_j}$.

**Key points**

- The input layer is the feature vector: no parameters, so scaling and encoding happen before it.
- Hidden layers learn representations without direct targets; the output layer is a linear model on top of them.
- Regression: 1 linear unit + MSE. Binary: 1 sigmoid + BCE. Multiclass: K softmax + cross-entropy. Multi-label: K sigmoids + BCE.
- softmax(2.0, 1.0, 0.1) = (0.659, 0.242, 0.099); it only depends on logit differences.
- In PyTorch, end with a plain Linear layer and let the loss apply sigmoid or softmax.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
