# Feedforward neural network (FNN)

> A network whose computation graph has no cycles: the output is a fixed composition of layers, each one a coordinate change that bends the input space until a line can split it.

Source: https://www.mlfoundry.in/learn/feedforward-neural-network-fnn  
Section: Deep Learning › Neural Network Algorithms  
Level: core · 9 min read · updated 2026-10-11

**A feedforward neural network:** is a neural network whose computation graph is a **directed acyclic graph** (DAG): information flows from the inputs to the outputs and never returns to an earlier node. With layers, it is the composition $f=f^{(L)}\circ\cdots\circ f^{(1)}$, and computing $f(x)$ takes exactly one pass through the graph, with no state carried between inputs.

"Feedforward" describes **graph shape**, not layer type. The fully connected MLP is the textbook case, but convolutional nets, residual nets with skip connections, and a Transformer applied to a fixed input sequence are feedforward too: each is a DAG of operations evaluated once per input. What is excluded is **feedback**: a node whose output, directly or through other nodes, becomes its own input.

## Acyclic means evaluable in one pass

Because a DAG has a **topological order** (an ordering in which every node comes after all its inputs), the forward pass is just "evaluate the nodes in that order". Each node runs once, as soon as its inputs exist. Backpropagation walks the same order in reverse. Every network a framework can run with a single `forward` call is, at that moment, a DAG.

$$
h^{(0)}=x,\qquad h^{(l)}=f^{(l)}\big(h^{(l-1)}\big)=\varphi\big(W^{(l)}h^{(l-1)}+b^{(l)}\big),\qquad \hat y=f^{(L)}\big(h^{(L-1)}\big)
$$

- $f^{(l)}$: layer l: an affine map followed by an elementwise nonlinearity φ (the last layer usually has none, or a softmax/sigmoid)
- $h^{(l)}$: the representation after layer l; it depends only on h⁽ˡ⁻¹⁾, never on later layers or past inputs

A general DAG allows more than a chain. A node may read from several earlier nodes, for example $h^{(l)}=\varphi(Wh^{(l-1)})+h^{(l-2)}$ in a residual block or a concatenation of branches in Inception. All that is required is that no path loops back.

## Feedforward versus feedback

Where the line is drawn

| Network | Graph | Consequence |
| --- | --- | --- |
| MLP, CNN, ResNet, Transformer encoder | DAG | output is a fixed function of the current input; one forward pass |
| RNN: $h_t=\varphi(Wh_{t-1}+Ux_t)$ | cycle $h\to h$ through time | has memory; the output depends on the whole input history |
| RNN unrolled over T steps | DAG of depth T with shared weights | trained as a deep feedforward net (backpropagation through time) |
| Hopfield net, Boltzmann machine | symmetric feedback connections | computes by iterating to a fixed point or sampling, not in one pass |
| Autoregressive generation with a Transformer | the network is a DAG; the loop feeding outputs back as inputs is outside it | every token is a fresh feedforward pass |

The unrolling row is worth remembering. Feedback over a finite horizon can always be rewritten as a deeper feedforward graph whose layers share weights. That is how recurrent nets are trained, and why everything about depth (vanishing gradients, initialization) carries over to them.

## Each layer is a change of coordinates

Read a layer geometrically. The affine part $z=Wh+b$ rotates, scales, shears and shifts the space: straight lines stay straight and parallel lines stay parallel. The nonlinearity then acts coordinate by coordinate. $\tanh$ squashes every axis into $(-1,1)$, so far-away regions get pressed against the walls of a square and straight grid lines bend. Composing layers is composing these warps.

The classifier at the end is just a line in the last hidden space. A feedforward net solves a non-linear problem by **moving the data**, layer by layer, into coordinates where a linear readout works. Watch that happen for a 2-2-2-1 tanh network trained on two moons. Every hidden layer has width 2, so every intermediate representation is itself a plane you can draw.

> Figure: Slide the stage, or press play, to carry the input grid, its shading (the network's final prediction) and the 160 data points through z⁽¹⁾ = W⁽¹⁾x + b⁽¹⁾, h⁽¹⁾ = tanh z⁽¹⁾, z⁽²⁾ and h⁽²⁾. Affine stages keep grid lines straight; tanh stages bend them into the square (−1, 1)². In input space, logistic regression gets 86.3% of the points. After layer 1 the two moons are already linearly separable, and in h⁽²⁾ they sit at opposite ends of a thin curve, split by the output line (100%). Switch to random weights: the grid is still warped, but the classes stay entangled, and logistic regression on that h⁽²⁾ gets only 87.5%.

Two things to notice. First, the whole plane is transformed, not just the data. Every point of input space has a fate, and the shading shows where it ends up. Second, nothing is ever torn: each layer is a continuous map. With width 2 (no more than the input dimension) and monotone activations like tanh, the network cannot wrap one region around another, so it cannot classify the inside of a circle against its outside at any depth (Johnson, 2019). Some width beyond the input dimension is necessary, not just convenient.

Third, look at stage $z^{(2)}$: the plane collapses almost onto a line. The rows of the trained $W^{(2)}$ are nearly parallel ($\det W^{(2)}=0.29$, while each row has norm above 3.4). The network has learned that it needs only one direction to separate the classes, and it spends the second layer stretching the data along it. A trained network discards the directions the task does not need. That is the geometric side of the low-dimensional features studied under representation learning.

## Worked example: one input through the trained network

x = (1, 0) through the figure's network, rounded to 4 decimals. Stages match the figure's buttons.

| Stage | Computation | Value |
| --- | --- | --- |
| $z^{(1)}$ | $W^{(1)}x+b^{(1)}$ with $W^{(1)}=\begin{bmatrix}-4.3605&-0.9736\\-0.9221&0.5472\end{bmatrix}$, $b^{(1)}=(-0.1283,-0.0690)$ | (−4.4888, −0.9911) |
| $h^{(1)}$ | $\tanh z^{(1)}$ | (−0.9997, −0.7578) |
| $z^{(2)}$ | $W^{(2)}h^{(1)}+b^{(2)}$ with $W^{(2)}=\begin{bmatrix}-2.5538&4.3706\\1.6918&-3.0089\end{bmatrix}$, $b^{(2)}=(0.1997,-0.1311)$ | (−0.5593, 0.4578) |
| $h^{(2)}$ | $\tanh z^{(2)}$ | (−0.5075, 0.4283) |
| logit | $(-5.5317,\ 3.6284)\cdot h^{(2)}+0.1353$ | 4.4964 |
| $\hat y$ | $\sigma(4.4964)$ | 0.9890 (class 1) |

The first unit saturates: $\tanh(-4.49)=-0.9997$, so moving $x$ further right changes almost nothing. Saturation is the tanh layer pressing regions against the wall of the square. It is useful for classification and, as the vanishing-gradients article shows, costly for learning.

## Counting parameters and compute

A fully connected layer from $n_{l-1}$ to $n_l$ units has $n_l(n_{l-1}+1)$ parameters and needs $n_ln_{l-1}$ multiply-adds (MACs) per input. Since one MAC is two floating-point operations, a forward pass costs about $2\sum_l n_ln_{l-1}$ FLOPs per example, and training (forward plus backward) about three times that.

An MNIST-sized MLP, 784 → 512 → 512 → 10

| Layer | Parameters | MACs per example |
| --- | --- | --- |
| 784 → 512 | 401,920 | 401,408 |
| 512 → 512 | 262,656 | 262,144 |
| 512 → 10 | 5,130 | 5,120 |
| **total** | **669,706** | **668,672** (≈ 1.34 MFLOPs) |

Weight sharing changes this balance, and it is the main reason CNNs scale to images. A $3\times3$ convolution from 64 to 64 channels has only $3\cdot3\cdot64\cdot64+64=36{,}928$ parameters. Applied at every position of a $56\times56$ feature map it costs $56\cdot56\cdot3\cdot3\cdot64\cdot64\approx1.16\times10^8$ MACs. A fully connected layer between the same two $56\times56\times64$ tensors would need about $4\times10^{10}$ weights. Both are feedforward. They differ in which edges of the DAG exist and which weights are tied.

> **Feedforward is a statement about time, not power:** An FNN has no memory: give it the same input twice and you get the same output twice. Anything that must depend on history, such as the previous words of a sentence, has to be **put into the input** (a window of tokens, a whole sequence at once) or carried by recurrence. Transformers take the first route: a feedforward graph over the entire context window.

> **Skip connections do not make a network recurrent:** A residual edge $h^{(l)}=h^{(l-2)}+F(h^{(l-1)})$ jumps forward over layers. The graph is still acyclic, and the network is still evaluated in one pass. Recurrence needs an edge pointing back to an earlier time step or layer. "Has skip connections" and "is feedforward" are independent properties.

```python
import torch, torch.nn as nn

mlp = nn.Sequential(nn.Linear(784, 512), nn.ReLU(),
                    nn.Linear(512, 512), nn.ReLU(),
                    nn.Linear(512, 10))
print(sum(p.numel() for p in mlp.parameters()))         # 669706

class SkipNet(nn.Module):                                # still a DAG: the skip edge points forward
    def __init__(self, d=64):
        super().__init__()
        self.a, self.b, self.out = nn.Linear(d, d), nn.Linear(d, d), nn.Linear(d, 1)
    def forward(self, x):
        h1 = torch.relu(self.a(x))
        h2 = torch.relu(self.b(h1)) + x                  # reads from two earlier nodes
        return self.out(h2)

net = SkipNet()
print(sum(p.numel() for p in net.parameters()))         # 8385 = 2·(64·65) + 65
y1, y2 = net(torch.ones(1, 64)), net(torch.ones(1, 64))
print(torch.equal(y1, y2))                              # True: no state between calls
```

Two feedforward nets: a plain chain and a DAG with a skip connection, with parameter counts

**Question:** Which of these is NOT a feedforward network in the graph sense?

- A. A ResNet with skip connections every two layers
- B. A CNN with max-pooling and a softmax head
- C. An RNN that carries $h_t=\tanh(Wh_{t-1}+Ux_t)$ across time steps
- D. A Transformer encoder applied to a fixed 128-token input

**Answer:** C. An RNN that carries $h_t=\tanh(Wh_{t-1}+Ux_t)$ across time steps — The RNN's hidden state feeds back into itself, which is a cycle in the graph (through time). The others are DAGs: skip connections, pooling and attention over a fixed input all point forward. Unrolling the RNN over a finite number of steps turns it into a DAG with shared weights.

**Question:** A 2-2-2-1 tanh network (every hidden layer of width 2) is trained on points inside a circle (class 1) versus a ring around it (class 0). What happens as you add more width-2 layers?

- A. Accuracy reaches 100% once there are enough layers
- B. It can never classify the inside against the ring perfectly, at any depth, because width ≤ input dimension cannot enclose a bounded region
- C. It needs ReLU instead of tanh, then 3 layers suffice
- D. Depth does not matter, but a larger learning rate solves it

**Answer:** B. It can never classify the inside against the ring perfectly, at any depth, because width ≤ input dimension cannot enclose a bounded region — Each layer is a continuous map through a 2-D bottleneck with monotone activations. Johnson (2019) shows that such skinny networks produce decision regions that are unbounded, so the bounded disk cannot be one class against its surroundings. One extra hidden unit (width 3) removes the obstruction.

**Key points**

- Feedforward = the computation graph is a DAG: one pass in topological order, no feedback, no memory between inputs.
- MLPs, CNNs, ResNets and Transformers on a fixed input are all FNNs; RNNs and Hopfield nets are not (until unrolled).
- Each layer is a coordinate change: an affine map keeps lines straight, the activation bends them; the readout is a line in the last space.
- Dense layer: $n_l(n_{l-1}+1)$ parameters and $n_ln_{l-1}$ MACs; a 784-512-512-10 MLP has 669,706 parameters.
- Width ≤ input dimension is a hard limit: such nets cannot enclose a bounded class region at any depth.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
