Definition
A deep feedforward network is a feedforward network with several hidden layers, trained to approximate some target function . "Deep" refers to the length of that chain. Goodfellow, Bengio and Courville (2016, ch. 6) treat it as the quintessential deep-learning model: everything else (convolutions, attention, residuals) changes which layers are in the chain.
One hidden layer is already a universal approximator. So why stack many? This article covers the three questions a deep feedforward design has to answer: why depth (what it buys in expressiveness), what it costs (signals and gradients across many layers), and how to finish the network (output units and the cost function that matches them).
Why depth: composition multiplies pieces
A ReLU network computes a continuous piecewise-linear function. Its complexity can be measured by the number of linear pieces (in 1-D) or linear regions (in -D). Width and depth add pieces in very different ways.
- the tent map: two ReLU units folding [0, 1] onto itself, two-to-one
- a width-2, depth- ReLU network ( units, parameters) with exactly linear pieces, a sawtooth with teeth
Each layer folds the interval in half, so whatever the layers above draw on is drawn twice, once on each half. That gives pieces from units. A single hidden layer with units on a 1-D input has at most kinks, so at most pieces. To even represent (1024 pieces) a shallow net needs at least 1023 units, against 20 in the deep one.
Depth separation: shallow nets cannot even approximate it
Counting pieces shows that a shallow net cannot represent exactly. Telgarsky (2016) showed more: it cannot get close. The argument fits in a few lines.
Every missed tooth costs a fixed amount of error
- 1 has teeth, each on an interval of length , rising from 0 to 1 and back. On each tooth it is above on the middle half.
- 2A piecewise-linear with pieces crosses at most times, so at most teeth contain a crossing of .
- 3On a tooth with no crossing, stays on one side of . Then over that tooth is at least the area of on the other side of , which is either way (one triangle above, or two below).
- 4So . For and a shallow net with 20 units () that is . Ignoring and predicting the constant already gets .
In dimensions the same effect is exponential in depth. Montúfar, Pascanu, Cho and Bengio (2014) constructed ReLU nets with hidden layers of width that have at least linear regions. With inputs and four layers of 8 units that is regions. One layer of the same 32 units is a hyperplane arrangement, with at most regions.
The practical case for depth is therefore partly empirical. Deep nets build features compositionally (edges, then parts, then objects in vision), they reuse intermediate features for many outputs, and in practice they generalise better per parameter than very wide shallow ones on structured data such as images, audio and text.
What depth costs: signals across many layers
Every layer multiplies the forward signal and the backward gradient by a random-looking matrix and a diagonal of activation slopes. For a ReLU layer of width with weight variance , the second moment of the activations is multiplied by about per layer. The same factor applies to the squared gradient on the way back:
- per-layer gain: the ½ is the fraction of ReLU units active on a symmetric input
- PyTorch's default
nn.Linearinit, , which gives gain per layer for ReLU
| Quantity | PyTorch default init | He init (σ² = 2/n) |
|---|---|---|
| activation RMS after layer 1 / 10 / 50 | 0.41 / 0.030 / 0.026 | 1.00 / 1.09 / 0.84 |
| gradient RMS at the top / at the input | / | 0.98 / 1.18 |
| cosine similarity of two random inputs at layer 50 (0.03–0.04 at the input) | 1.0000 | 0.9934 |
With the default init the gradient shrinks by per layer, so after 50 layers it is about : the first layers simply do not train. The activations do not vanish completely only because the biases put a floor under them. He initialisation (He et al., 2015) fixes the scale. It does not fix the last row: even at the right scale, a deep plain ReLU stack maps different inputs to nearly parallel vectors, so the network's output barely depends on which input it got. Residual connections and normalisation layers (Ioffe and Szegedy 2015; He et al. 2016) are the standard remedies. They are why depths of 50 to 1000 layers became trainable at all.
Output units and the cost function
The last layer decides what the network predicts, and the right loss follows from treating the output as the parameters of a distribution and minimising the negative log-likelihood. Each pairing below has the same property: the gradient with respect to the output logits is prediction minus target, with no extra factors that could vanish.
| Task | Output unit | Distribution | Loss (NLL) | ∂L/∂z |
|---|---|---|---|---|
| regression | linear, | Gaussian mean | MSE (up to constants) | |
| binary | sigmoid, | Bernoulli | binary cross-entropy | |
| k classes | softmax, | categorical | cross-entropy | |
| heteroscedastic regression | two heads: mean and | Gaussian with learned variance | Gaussian NLL | both heads get gradients |
| multimodal targets | mixture weights, means, variances | mixture of Gaussians | mixture NLL | mixture density network |
Architecture design
Use when
- +More depth: compositional features and exponentially more expressible pieces for the same number of units
- +More width: easier optimisation and more over-parameterisation, so fewer bad seeds and dead units; parallelises well
- +Skip connections + normalisation: keep the signal and gradients well scaled, which makes great depth trainable
Avoid when
- −Depth without care: vanishing/exploding signals and inputs collapsing to one direction, as in the table above
- −Width alone: the number of units can need to grow exponentially to match a deep net
- −Both: more parameters to regularise, and depth adds sequential latency that width does not
A sensible default for a plain deep MLP on tabular or embedding inputs is a few layers (2 to 8) of equal width, ReLU or GELU, He or Xavier initialisation, normalisation and residual connections once you go deeper, and an output layer and loss chosen from the table. Then tune width, depth and learning rate on validation data.
1import torch, torch.nn as nn23def tent_net(k):4 layers, inp = [], nn.Linear(1, 2)5 with torch.no_grad():6 inp.weight.copy_(torch.tensor([[1.], [1.]])); inp.bias.copy_(torch.tensor([0., -0.5]))7 layers += [inp, nn.ReLU()]8 for _ in range(k - 1): # next layer reads T = 2·h1 − 4·h2 and folds again9 l = nn.Linear(2, 2)10 with torch.no_grad():11 l.weight.copy_(torch.tensor([[2., -4.], [2., -4.]])); l.bias.copy_(torch.tensor([0., -0.5]))12 layers += [l, nn.ReLU()]13 out = nn.Linear(2, 1)14 with torch.no_grad():15 out.weight.copy_(torch.tensor([[2., -4.]])); out.bias.zero_()16 return nn.Sequential(*layers, out)1718k = 1019net = tent_net(k)20print(sum(p.numel() for p in net.parameters())) # 61 = 6k + 121x = torch.arange(2**k + 1).float().unsqueeze(1) / 2**k22print(net(x).squeeze()[:6]) # tensor([0., 1., 0., 1., 0., 1.]): 1024 pieces23print(round(net(torch.tensor([[0.3]])).item(), 4)) # 0.8: 0.3 → 0.6 → 0.8 → 0.4 → 0.8 → …Check yourself
A ReLU net with one input and one hidden layer of 30 units must match the depth-6 tent map T⁶ exactly on [0, 1]. Is that possible?
Check yourself
A binary classifier with a sigmoid output is trained with squared error, and learning stalls on examples it gets confidently wrong. What is the best fix?
Remember
- ✓Depth composes folds: the width-2, depth- tent net has pieces from units; one hidden layer of units has at most .
- ✓Depth separation (Telgarsky): with pieces, the error to is at least ; Montúfar et al. give exponentially many regions in -D.
- ✓Constructed, not typical: random nets have roughly as many regions as neurons, and SGD is not guaranteed to find the exponential ones.
- ✓Depth multiplies signal scale by per ReLU layer: the default init gives gradients at depth 50, He keeps them near 1.
- ✓Pick the output unit and the NLL loss together (linear+MSE, sigmoid+BCE, softmax+CE) so the logit gradient is prediction − target.