Deep feedforward network

Depth composes folds, so a deep ReLU net can have exponentially more linear pieces than a shallow one of the same size; making that pay off needs careful init, outputs and losses.

11 min read1036 wordsadvanced0 viewsupdated 2026-10-11

Definition

A deep feedforward network is a feedforward network f(x;θ)=f(L)(⋯f(2)(f(1)(x)))f(x;\theta)=f^{(L)}(\cdots f^{(2)}(f^{(1)}(x))) with several hidden layers, trained to approximate some target function f\*f^\*. "Deep" refers to the length LL of that chain. Goodfellow, Bengio and Courville (2016, ch. 6) treat it as the quintessential deep-learning model: everything else (convolutions, attention, residuals) changes which layers are in the chain.

One hidden layer is already a universal approximator. So why stack many? This article covers the three questions a deep feedforward design has to answer: why depth (what it buys in expressiveness), what it costs (signals and gradients across many layers), and how to finish the network (output units and the cost function that matches them).

Why depth: composition multiplies pieces

A ReLU network computes a continuous piecewise-linear function. Its complexity can be measured by the number of linear pieces (in 1-D) or linear regions (in dd-D). Width and depth add pieces in very different ways.

T(x)=2 ReLU(x)−4 ReLU ⁣(x−12)={2x0≤x≤122−2x12≤x≤1,Tk=T∘⋯∘T⏟kT(x)=2\,\mathrm{ReLU}(x)-4\,\mathrm{ReLU}\!\big(x-\tfrac12\big)=\begin{cases}2x & 0\le x\le\frac12\\ 2-2x & \frac12\le x\le1\end{cases},\qquad T^{k}=\underbrace{T\circ\cdots\circ T}_{k}
TT
the tent map: two ReLU units folding [0, 1] onto itself, two-to-one
TkT^{k}
a width-2, depth-kk ReLU network (2k2k units, 6k+16k+1 parameters) with exactly 2k2^k linear pieces, a sawtooth with 2k−12^{k-1} teeth

Each layer folds the interval in half, so whatever the layers above draw on [0,1][0,1] is drawn twice, once on each half. That gives 2k2^k pieces from 2k2k units. A single hidden layer with mm units on a 1-D input has at most mm kinks, so at most m+1m+1 pieces. To even represent T10T^{10} (1024 pieces) a shallow net needs at least 1023 units, against 20 in the deep one.

f = T∘…∘T (3×), T(x) = 2·ReLU(x) − 4·ReLU(x − ½)000.50.5118 pieces · 8 crossings of ½output after layer j: each layer folds [0, 1] in two12345678910depth 3 · width 2 · 6 ReLU units · 19 paramsone hidden layer with 6 units: at most 7 pieceslinear pieces on [0, 1] (log₂)141664256102412345678910depth k (width-2 layers) →● deep tent net: 2ᵏ = 8● shallow, same units: ≤ 7● same shape, random init: 2.10(PyTorch default init, mean of 30)
fig.Drag the depth slider. The sawtooth is the exact output of the depth-k tent network; the thumbnails show the output after each layer, so you can watch each fold double the teeth. Every piece crosses the level ½ once (orange dots), so T^k has 2^k crossings. Right: pieces against depth for the tent net (2^k), the most a one-hidden-layer net with the same 2k units can have (2k + 1), and what 30 randomly initialised nets of the tent net's exact shape actually have: between 1.67 and 2.10 at every depth from 1 to 10.

Depth separation: shallow nets cannot even approximate it

Counting pieces shows that a shallow net cannot represent TkT^k exactly. Telgarsky (2016) showed more: it cannot get close. The argument fits in a few lines.

Every missed tooth costs a fixed amount of error

  1. 1TkT^k has 2k−12^{k-1} teeth, each on an interval of length ℓ=21−k\ell=2^{1-k}, rising from 0 to 1 and back. On each tooth it is above 12\tfrac12 on the middle half.
  2. 2A piecewise-linear gg with pp pieces crosses 12\tfrac12 at most pp times, so at most pp teeth contain a crossing of gg.
  3. 3On a tooth with no crossing, gg stays on one side of 12\tfrac12. Then ∫∣Tk−g∣\int|T^k-g| over that tooth is at least the area of TkT^k on the other side of 12\tfrac12, which is ℓ/8\ell/8 either way (one triangle above, or two below).
  4. 4So ∫01∣Tk−g∣≥(2k−1−p) ℓ/8=18(1−p 21−k)\int_0^1|T^k-g|\ge(2^{k-1}-p)\,\ell/8=\tfrac18\big(1-p\,2^{1-k}\big). For k=10k=10 and a shallow net with 20 units (p≤21p\le21) that is ≥0.1199\ge0.1199. Ignoring xx and predicting the constant 12\tfrac12 already gets 0.250.25.

In dd dimensions the same effect is exponential in depth. Montúfar, Pascanu, Cho and Bengio (2014) constructed ReLU nets with LL hidden layers of width n≥dn\ge d that have at least (∏l=1L−1⌊n/d⌋d)∑j=0d(nj)\big(\prod_{l=1}^{L-1}\lfloor n/d\rfloor^{d}\big)\sum_{j=0}^{d}\binom{n}{j} linear regions. With d=2d=2 inputs and four layers of 8 units that is 163⋅37=151,55216^3\cdot37=151{,}552 regions. One layer of the same 32 units is a hyperplane arrangement, with at most ∑j≤2(32j)=529\sum_{j\le2}\binom{32}{j}=529 regions.

The practical case for depth is therefore partly empirical. Deep nets build features compositionally (edges, then parts, then objects in vision), they reuse intermediate features for many outputs, and in practice they generalise better per parameter than very wide shallow ones on structured data such as images, audio and text.

What depth costs: signals across many layers

Every layer multiplies the forward signal and the backward gradient by a random-looking matrix and a diagonal of activation slopes. For a ReLU layer of width nn with weight variance σw2\sigma_w^2, the second moment of the activations is multiplied by about nσw2/2n\sigma_w^2/2 per layer. The same factor applies to the squared gradient on the way back:

E[∥h(l)∥2]≈(n σw22)l E[∥x∥2],σw2=2n (He) ⇒ factor 1\mathbb{E}\big[\lVert h^{(l)}\rVert^2\big]\approx\Big(\frac{n\,\sigma_w^2}{2}\Big)^{l}\,\mathbb{E}\big[\lVert x\rVert^2\big],\qquad \sigma_w^2=\frac{2}{n}\ (\text{He})\ \Rightarrow\ \text{factor }1
nσw2/2n\sigma_w^2/2
per-layer gain: the ½ is the fraction of ReLU units active on a symmetric input
σw2=1/(3n)\sigma_w^2=1/(3n)
PyTorch's default nn.Linear init, U(±1/n)U(\pm1/\sqrt n), which gives gain 1/61/6 per layer for ReLU
QuantityPyTorch default initHe init (σ² = 2/n)
activation RMS after layer 1 / 10 / 500.41 / 0.030 / 0.0261.00 / 1.09 / 0.84
gradient RMS at the top / at the input0.390.39 / 2.2×10−202.2\times10^{-20}0.98 / 1.18
cosine similarity of two random inputs at layer 50 (0.03–0.04 at the input)1.00000.9934
A 50-layer ReLU MLP of width 256 on a batch of 64 Gaussian inputs, one seeded draw of each init. RMS values are per unit; the gradient is that of a random unit-variance signal injected at the top.

With the default init the gradient shrinks by 1/6≈0.41\sqrt{1/6}\approx0.41 per layer, so after 50 layers it is about 10−2010^{-20}: the first layers simply do not train. The activations do not vanish completely only because the biases put a floor under them. He initialisation (He et al., 2015) fixes the scale. It does not fix the last row: even at the right scale, a deep plain ReLU stack maps different inputs to nearly parallel vectors, so the network's output barely depends on which input it got. Residual connections and normalisation layers (Ioffe and Szegedy 2015; He et al. 2016) are the standard remedies. They are why depths of 50 to 1000 layers became trainable at all.

Output units and the cost function

The last layer decides what the network predicts, and the right loss follows from treating the output as the parameters of a distribution p(y∣x;θ)p(y\mid x;\theta) and minimising the negative log-likelihood. Each pairing below has the same property: the gradient with respect to the output logits is prediction minus target, with no extra factors that could vanish.

TaskOutput unitDistributionLoss (NLL)∂L/∂z
regressionlinear, y^=z\hat y=zGaussian meanMSE (up to constants)y^−y\hat y-y
binarysigmoid, p^=σ(z)\hat p=\sigma(z)Bernoullibinary cross-entropyp^−y\hat p-y
k classessoftmax, p^=softmax(z)\hat p=\mathrm{softmax}(z)categoricalcross-entropyp^−onehot(y)\hat p-\mathrm{onehot}(y)
heteroscedastic regressiontwo heads: mean and log⁡σ2\log\sigma^2Gaussian with learned varianceGaussian NLLboth heads get gradients
multimodal targetsmixture weights, means, variancesmixture of Gaussiansmixture NLLmixture density network
Standard output units (Goodfellow et al., §6.2). z is the vector of output logits.

Architecture design

Use when

  • +More depth: compositional features and exponentially more expressible pieces for the same number of units
  • +More width: easier optimisation and more over-parameterisation, so fewer bad seeds and dead units; parallelises well
  • +Skip connections + normalisation: keep the signal and gradients well scaled, which makes great depth trainable

Avoid when

  • −Depth without care: vanishing/exploding signals and inputs collapsing to one direction, as in the table above
  • −Width alone: the number of units can need to grow exponentially to match a deep net
  • −Both: more parameters to regularise, and depth adds sequential latency that width does not

A sensible default for a plain deep MLP on tabular or embedding inputs is a few layers (2 to 8) of equal width, ReLU or GELU, He or Xavier initialisation, normalisation and residual connections once you go deeper, and an output layer and loss chosen from the table. Then tune width, depth and learning rate on validation data.

python
1import torch, torch.nn as nn
2
3def tent_net(k):
4 layers, inp = [], nn.Linear(1, 2)
5 with torch.no_grad():
6 inp.weight.copy_(torch.tensor([[1.], [1.]])); inp.bias.copy_(torch.tensor([0., -0.5]))
7 layers += [inp, nn.ReLU()]
8 for _ in range(k - 1): # next layer reads T = 2·h1 − 4·h2 and folds again
9 l = nn.Linear(2, 2)
10 with torch.no_grad():
11 l.weight.copy_(torch.tensor([[2., -4.], [2., -4.]])); l.bias.copy_(torch.tensor([0., -0.5]))
12 layers += [l, nn.ReLU()]
13 out = nn.Linear(2, 1)
14 with torch.no_grad():
15 out.weight.copy_(torch.tensor([[2., -4.]])); out.bias.zero_()
16 return nn.Sequential(*layers, out)
17
18k = 10
19net = tent_net(k)
20print(sum(p.numel() for p in net.parameters())) # 61 = 6k + 1
21x = torch.arange(2**k + 1).float().unsqueeze(1) / 2**k
22print(net(x).squeeze()[:6]) # tensor([0., 1., 0., 1., 0., 1.]): 1024 pieces
23print(round(net(torch.tensor([[0.3]])).item(), 4)) # 0.8: 0.3 → 0.6 → 0.8 → 0.4 → 0.8 → …
The tent network built from nn.Linear layers, checked to have 2^k pieces

Check yourself

A ReLU net with one input and one hidden layer of 30 units must match the depth-6 tent map T⁶ exactly on [0, 1]. Is that possible?

Check yourself

A binary classifier with a sigmoid output is trained with squared error, and learning stalls on examples it gets confidently wrong. What is the best fix?

Remember

  • ✓Depth composes folds: the width-2, depth-kk tent net has 2k2^k pieces from 2k2k units; one hidden layer of mm units has at most m+1m+1.
  • ✓Depth separation (Telgarsky): with pp pieces, the L1L^1 error to TkT^k is at least 18(1−p 21−k)\tfrac18(1-p\,2^{1-k}); Montúfar et al. give exponentially many regions in dd-D.
  • ✓Constructed, not typical: random nets have roughly as many regions as neurons, and SGD is not guaranteed to find the exponential ones.
  • ✓Depth multiplies signal scale by nσw2/2n\sigma_w^2/2 per ReLU layer: the default init gives 10−2010^{-20} gradients at depth 50, He keeps them near 1.
  • ✓Pick the output unit and the NLL loss together (linear+MSE, sigmoid+BCE, softmax+CE) so the logit gradient is prediction − target.
← back to Deep Learning roadmap