# Deep feedforward network

> Depth composes folds, so a deep ReLU net can have exponentially more linear pieces than a shallow one of the same size; making that pay off needs careful init, outputs and losses.

Source: https://www.mlfoundry.in/learn/deep-feedforward-network  
Section: Deep Learning › Neural Network Algorithms  
Level: advanced · 11 min read · updated 2026-10-11

**A deep feedforward network:** is a feedforward network $f(x;\theta)=f^{(L)}(\cdots f^{(2)}(f^{(1)}(x)))$ with several hidden layers, trained to approximate some target function $f^\*$. "Deep" refers to the length $L$ of that chain. Goodfellow, Bengio and Courville (2016, ch. 6) treat it as the quintessential deep-learning model: everything else (convolutions, attention, residuals) changes which layers are in the chain.

One hidden layer is already a universal approximator. So why stack many? This article covers the three questions a deep feedforward design has to answer: **why depth** (what it buys in expressiveness), **what it costs** (signals and gradients across many layers), and **how to finish the network** (output units and the cost function that matches them).

## Why depth: composition multiplies pieces

A ReLU network computes a continuous **piecewise-linear** function. Its complexity can be measured by the number of linear pieces (in 1-D) or linear regions (in $d$-D). Width and depth add pieces in very different ways.

$$
T(x)=2\,\mathrm{ReLU}(x)-4\,\mathrm{ReLU}\!\big(x-\tfrac12\big)=\begin{cases}2x & 0\le x\le\frac12\\ 2-2x & \frac12\le x\le1\end{cases},\qquad T^{k}=\underbrace{T\circ\cdots\circ T}_{k}
$$

- $T$: the tent map: two ReLU units folding [0, 1] onto itself, two-to-one
- $T^{k}$: a width-2, depth-$k$ ReLU network ($2k$ units, $6k+1$ parameters) with exactly $2^k$ linear pieces, a sawtooth with $2^{k-1}$ teeth

Each layer folds the interval in half, so whatever the layers above draw on $[0,1]$ is drawn twice, once on each half. That gives $2^k$ pieces from $2k$ units. A single hidden layer with $m$ units on a 1-D input has at most $m$ kinks, so at most $m+1$ pieces. To even represent $T^{10}$ (1024 pieces) a shallow net needs at least 1023 units, against 20 in the deep one.

> Figure: Drag the depth slider. The sawtooth is the exact output of the depth-k tent network; the thumbnails show the output after each layer, so you can watch each fold double the teeth. Every piece crosses the level ½ once (orange dots), so T^k has 2^k crossings. Right: pieces against depth for the tent net (2^k), the most a one-hidden-layer net with the same 2k units can have (2k + 1), and what 30 randomly initialised nets of the tent net's exact shape actually have: between 1.67 and 2.10 at every depth from 1 to 10.

## Depth separation: shallow nets cannot even approximate it

Counting pieces shows that a shallow net cannot represent $T^k$ exactly. Telgarsky (2016) showed more: it cannot get close. The argument fits in a few lines.

**Every missed tooth costs a fixed amount of error**

1. $T^k$ has $2^{k-1}$ teeth, each on an interval of length $\ell=2^{1-k}$, rising from 0 to 1 and back. On each tooth it is above $\tfrac12$ on the middle half.
2. A piecewise-linear $g$ with $p$ pieces crosses $\tfrac12$ at most $p$ times, so at most $p$ teeth contain a crossing of $g$.
3. On a tooth with no crossing, $g$ stays on one side of $\tfrac12$. Then $\int|T^k-g|$ over that tooth is at least the area of $T^k$ on the other side of $\tfrac12$, which is $\ell/8$ either way (one triangle above, or two below).
4. So $\int_0^1|T^k-g|\ge(2^{k-1}-p)\,\ell/8=\tfrac18\big(1-p\,2^{1-k}\big)$. For $k=10$ and a shallow net with 20 units ($p\le21$) that is $\ge0.1199$. Ignoring $x$ and predicting the constant $\tfrac12$ already gets $0.25$.

In $d$ dimensions the same effect is exponential in depth. Montúfar, Pascanu, Cho and Bengio (2014) constructed ReLU nets with $L$ hidden layers of width $n\ge d$ that have at least $\big(\prod_{l=1}^{L-1}\lfloor n/d\rfloor^{d}\big)\sum_{j=0}^{d}\binom{n}{j}$ linear regions. With $d=2$ inputs and four layers of 8 units that is $16^3\cdot37=151{,}552$ regions. One layer of the same 32 units is a hyperplane arrangement, with at most $\sum_{j\le2}\binom{32}{j}=529$ regions.

> **Possible is not typical:** The exponential count is a property of carefully *constructed* weights. At random initialisation, the number of regions along a line grows roughly with the **total number of neurons**, not exponentially with depth (Hanin and Rolnick, 2019). In the figure, random nets of the tent net's exact shape average at most 2.1 pieces at every depth up to 10, against 1024 for the constructed weights at depth 10. Many of their units are dead or nearly linear on [0, 1]. Training can create more pieces where the data need them, but nothing guarantees it finds the exponential solutions. Depth separation results describe **what can be represented**, not what SGD will learn.

The practical case for depth is therefore partly empirical. Deep nets build features compositionally (edges, then parts, then objects in vision), they reuse intermediate features for many outputs, and in practice they generalise better per parameter than very wide shallow ones on structured data such as images, audio and text.

## What depth costs: signals across many layers

Every layer multiplies the forward signal and the backward gradient by a random-looking matrix and a diagonal of activation slopes. For a ReLU layer of width $n$ with weight variance $\sigma_w^2$, the second moment of the activations is multiplied by about $n\sigma_w^2/2$ per layer. The same factor applies to the squared gradient on the way back:

$$
\mathbb{E}\big[\lVert h^{(l)}\rVert^2\big]\approx\Big(\frac{n\,\sigma_w^2}{2}\Big)^{l}\,\mathbb{E}\big[\lVert x\rVert^2\big],\qquad \sigma_w^2=\frac{2}{n}\ (\text{He})\ \Rightarrow\ \text{factor }1
$$

- $n\sigma_w^2/2$: per-layer gain: the ½ is the fraction of ReLU units active on a symmetric input
- $\sigma_w^2=1/(3n)$: PyTorch's default `nn.Linear` init, $U(\pm1/\sqrt n)$, which gives gain $1/6$ per layer for ReLU

A 50-layer ReLU MLP of width 256 on a batch of 64 Gaussian inputs, one seeded draw of each init. RMS values are per unit; the gradient is that of a random unit-variance signal injected at the top.

| Quantity | PyTorch default init | He init (σ² = 2/n) |
| --- | --- | --- |
| activation RMS after layer 1 / 10 / 50 | 0.41 / 0.030 / 0.026 | 1.00 / 1.09 / 0.84 |
| gradient RMS at the top / at the input | $0.39$ / $2.2\times10^{-20}$ | 0.98 / 1.18 |
| cosine similarity of two random inputs at layer 50 (0.03–0.04 at the input) | 1.0000 | 0.9934 |

With the default init the gradient shrinks by $\sqrt{1/6}\approx0.41$ per layer, so after 50 layers it is about $10^{-20}$: the first layers simply do not train. The activations do not vanish completely only because the biases put a floor under them. He initialisation (He et al., 2015) fixes the scale. It does not fix the last row: even at the right scale, a deep plain ReLU stack maps different inputs to nearly parallel vectors, so the network's output barely depends on which input it got. Residual connections and normalisation layers (Ioffe and Szegedy 2015; He et al. 2016) are the standard remedies. They are why depths of 50 to 1000 layers became trainable at all.

## Output units and the cost function

The last layer decides what the network predicts, and the right loss follows from treating the output as the parameters of a distribution $p(y\mid x;\theta)$ and minimising the negative log-likelihood. Each pairing below has the same property: the gradient with respect to the output logits is **prediction minus target**, with no extra factors that could vanish.

Standard output units (Goodfellow et al., §6.2). z is the vector of output logits.

| Task | Output unit | Distribution | Loss (NLL) | ∂L/∂z |
| --- | --- | --- | --- | --- |
| regression | linear, $\hat y=z$ | Gaussian mean | MSE (up to constants) | $\hat y-y$ |
| binary | sigmoid, $\hat p=\sigma(z)$ | Bernoulli | binary cross-entropy | $\hat p-y$ |
| k classes | softmax, $\hat p=\mathrm{softmax}(z)$ | categorical | cross-entropy | $\hat p-\mathrm{onehot}(y)$ |
| heteroscedastic regression | two heads: mean and $\log\sigma^2$ | Gaussian with learned variance | Gaussian NLL | both heads get gradients |
| multimodal targets | mixture weights, means, variances | mixture of Gaussians | mixture NLL | mixture density network |

> **Sigmoid + squared error saturates:** Take a confident wrong prediction: $z=-5$, so $\sigma(z)=0.0067$, with $y=1$. With cross-entropy, $\partial\mathcal{L}/\partial z=\sigma(z)-y=-0.9933$, a strong push. With $\tfrac12(\sigma(z)-y)^2$ it is $(\sigma(z)-y)\,\sigma(z)(1-\sigma(z))=-0.0066$, about 150× weaker, and it weakens further the more wrong the prediction is. The log in the likelihood undoes the exp in the sigmoid. That is the reason to pair saturating outputs with log-likelihood losses.

## Architecture design

**Strengths**

- More depth: compositional features and exponentially more expressible pieces for the same number of units
- More width: easier optimisation and more over-parameterisation, so fewer bad seeds and dead units; parallelises well
- Skip connections + normalisation: keep the signal and gradients well scaled, which makes great depth trainable

**Limitations**

- Depth without care: vanishing/exploding signals and inputs collapsing to one direction, as in the table above
- Width alone: the number of units can need to grow exponentially to match a deep net
- Both: more parameters to regularise, and depth adds sequential latency that width does not

A sensible default for a plain deep MLP on tabular or embedding inputs is a few layers (2 to 8) of equal width, ReLU or GELU, He or Xavier initialisation, normalisation and residual connections once you go deeper, and an output layer and loss chosen from the table. Then tune width, depth and learning rate on validation data.

```python
import torch, torch.nn as nn

def tent_net(k):
    layers, inp = [], nn.Linear(1, 2)
    with torch.no_grad():
        inp.weight.copy_(torch.tensor([[1.], [1.]])); inp.bias.copy_(torch.tensor([0., -0.5]))
    layers += [inp, nn.ReLU()]
    for _ in range(k - 1):                       # next layer reads T = 2·h1 − 4·h2 and folds again
        l = nn.Linear(2, 2)
        with torch.no_grad():
            l.weight.copy_(torch.tensor([[2., -4.], [2., -4.]])); l.bias.copy_(torch.tensor([0., -0.5]))
        layers += [l, nn.ReLU()]
    out = nn.Linear(2, 1)
    with torch.no_grad():
        out.weight.copy_(torch.tensor([[2., -4.]])); out.bias.zero_()
    return nn.Sequential(*layers, out)

k = 10
net = tent_net(k)
print(sum(p.numel() for p in net.parameters()))   # 61 = 6k + 1
x = torch.arange(2**k + 1).float().unsqueeze(1) / 2**k
print(net(x).squeeze()[:6])                        # tensor([0., 1., 0., 1., 0., 1.]): 1024 pieces
print(round(net(torch.tensor([[0.3]])).item(), 4)) # 0.8: 0.3 → 0.6 → 0.8 → 0.4 → 0.8 → …
```

The tent network built from nn.Linear layers, checked to have 2^k pieces

**Question:** A ReLU net with one input and one hidden layer of 30 units must match the depth-6 tent map T⁶ exactly on [0, 1]. Is that possible?

- A. Yes, 30 units give up to 31 pieces, and T⁶ has 12
- B. No, T⁶ has 64 pieces and 30 units give at most 31
- C. Yes, because one hidden layer is a universal approximator
- D. Only with sigmoid units

**Answer:** B. No, T⁶ has 64 pieces and 30 units give at most 31 — $T^k$ has $2^k$ pieces, so $T^6$ has 64. A single hidden layer of $m$ ReLU units on a 1-D input has at most $m$ kinks, so at most $m+1=31$ pieces. Exact representation needs at least 63 units. Universal approximation only says that *enough* units get close, and Telgarsky's argument shows that with too few the $L^1$ error stays bounded away from 0.

**Question:** A binary classifier with a sigmoid output is trained with squared error, and learning stalls on examples it gets confidently wrong. What is the best fix?

- A. Lower the learning rate
- B. Switch to binary cross-entropy, whose gradient with respect to the logit is σ(z) − y
- C. Remove the sigmoid and keep squared error on the logit
- D. Add more layers

**Answer:** B. Switch to binary cross-entropy, whose gradient with respect to the logit is σ(z) − y — With squared error the logit gradient carries the factor $\sigma(z)(1-\sigma(z))$, which is near 0 exactly when the prediction is confidently wrong (e.g. $-0.0066$ at $z=-5$, $y=1$). Cross-entropy cancels it and gives $\sigma(z)-y=-0.9933$. Squared error on the raw logit would treat a classification target as a regression target.

**Key points**

- Depth composes folds: the width-2, depth-$k$ tent net has $2^k$ pieces from $2k$ units; one hidden layer of $m$ units has at most $m+1$.
- Depth separation (Telgarsky): with $p$ pieces, the $L^1$ error to $T^k$ is at least $\tfrac18(1-p\,2^{1-k})$; Montúfar et al. give exponentially many regions in $d$-D.
- Constructed, not typical: random nets have roughly as many regions as neurons, and SGD is not guaranteed to find the exponential ones.
- Depth multiplies signal scale by $n\sigma_w^2/2$ per ReLU layer: the default init gives $10^{-20}$ gradients at depth 50, He keeps them near 1.
- Pick the output unit and the NLL loss together (linear+MSE, sigmoid+BCE, softmax+CE) so the logit gradient is prediction − target.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
