Feedforward neural network (FNN)

A network whose computation graph has no cycles: the output is a fixed composition of layers, each one a coordinate change that bends the input space until a line can split it.

9 min read916 wordscore0 viewsupdated 2026-10-11

Definition

A feedforward neural network is a neural network whose computation graph is a directed acyclic graph (DAG): information flows from the inputs to the outputs and never returns to an earlier node. With layers, it is the composition f=f(L)∘⋯∘f(1)f=f^{(L)}\circ\cdots\circ f^{(1)}, and computing f(x)f(x) takes exactly one pass through the graph, with no state carried between inputs.

"Feedforward" describes graph shape, not layer type. The fully connected MLP is the textbook case, but convolutional nets, residual nets with skip connections, and a Transformer applied to a fixed input sequence are feedforward too: each is a DAG of operations evaluated once per input. What is excluded is feedback: a node whose output, directly or through other nodes, becomes its own input.

Acyclic means evaluable in one pass

Because a DAG has a topological order (an ordering in which every node comes after all its inputs), the forward pass is just "evaluate the nodes in that order". Each node runs once, as soon as its inputs exist. Backpropagation walks the same order in reverse. Every network a framework can run with a single forward call is, at that moment, a DAG.

h(0)=x,h(l)=f(l)(h(l−1))=φ(W(l)h(l−1)+b(l)),y^=f(L)(h(L−1))h^{(0)}=x,\qquad h^{(l)}=f^{(l)}\big(h^{(l-1)}\big)=\varphi\big(W^{(l)}h^{(l-1)}+b^{(l)}\big),\qquad \hat y=f^{(L)}\big(h^{(L-1)}\big)
f(l)f^{(l)}
layer l: an affine map followed by an elementwise nonlinearity φ (the last layer usually has none, or a softmax/sigmoid)
h(l)h^{(l)}
the representation after layer l; it depends only on h⁽ˡ⁻¹⁾, never on later layers or past inputs

A general DAG allows more than a chain. A node may read from several earlier nodes, for example h(l)=φ(Wh(l−1))+h(l−2)h^{(l)}=\varphi(Wh^{(l-1)})+h^{(l-2)} in a residual block or a concatenation of branches in Inception. All that is required is that no path loops back.

Feedforward versus feedback

NetworkGraphConsequence
MLP, CNN, ResNet, Transformer encoderDAGoutput is a fixed function of the current input; one forward pass
RNN: ht=φ(Wht−1+Uxt)h_t=\varphi(Wh_{t-1}+Ux_t)cycle h→hh\to h through timehas memory; the output depends on the whole input history
RNN unrolled over T stepsDAG of depth T with shared weightstrained as a deep feedforward net (backpropagation through time)
Hopfield net, Boltzmann machinesymmetric feedback connectionscomputes by iterating to a fixed point or sampling, not in one pass
Autoregressive generation with a Transformerthe network is a DAG; the loop feeding outputs back as inputs is outside itevery token is a fresh feedforward pass
Where the line is drawn

The unrolling row is worth remembering. Feedback over a finite horizon can always be rewritten as a deeper feedforward graph whose layers share weights. That is how recurrent nets are trained, and why everything about depth (vanishing gradients, initialization) carries over to them.

Each layer is a change of coordinates

Read a layer geometrically. The affine part z=Wh+bz=Wh+b rotates, scales, shears and shifts the space: straight lines stay straight and parallel lines stay parallel. The nonlinearity then acts coordinate by coordinate. tanh⁡\tanh squashes every axis into (−1,1)(-1,1), so far-away regions get pressed against the walls of a square and straight grid lines bend. Composing layers is composing these warps.

The classifier at the end is just a line in the last hidden space. A feedforward net solves a non-linear problem by moving the data, layer by layer, into coordinates where a linear readout works. Watch that happen for a 2-2-2-1 tanh network trained on two moons. Every hidden layer has width 2, so every intermediate representation is itself a plane you can draw.

input xgrid + shading = input space, carried along · shade = final predictionview span 3.24 unitsthe DAG: edges only point forwardxh⁽¹⁾h⁽²⁾ŷlayer · params · multiply-adds1: 2→2 · 2·(2+1) = 6 · 42: 2→2 · 2·(2+1) = 6 · 43: 2→1 · 1·(2+1) = 3 · 2total 15 params · 10 MACs per inputaccuracy of this net: 100%affine steps: lines stay straighttanh steps: lines bend, plane foldsinto the square (−1, 1)²
fig.Slide the stage, or press play, to carry the input grid, its shading (the network's final prediction) and the 160 data points through z⁽¹⁾ = W⁽¹⁾x + b⁽¹⁾, h⁽¹⁾ = tanh z⁽¹⁾, z⁽²⁾ and h⁽²⁾. Affine stages keep grid lines straight; tanh stages bend them into the square (−1, 1)². In input space, logistic regression gets 86.3% of the points. After layer 1 the two moons are already linearly separable, and in h⁽²⁾ they sit at opposite ends of a thin curve, split by the output line (100%). Switch to random weights: the grid is still warped, but the classes stay entangled, and logistic regression on that h⁽²⁾ gets only 87.5%.

Two things to notice. First, the whole plane is transformed, not just the data. Every point of input space has a fate, and the shading shows where it ends up. Second, nothing is ever torn: each layer is a continuous map. With width 2 (no more than the input dimension) and monotone activations like tanh, the network cannot wrap one region around another, so it cannot classify the inside of a circle against its outside at any depth (Johnson, 2019). Some width beyond the input dimension is necessary, not just convenient.

Third, look at stage z(2)z^{(2)}: the plane collapses almost onto a line. The rows of the trained W(2)W^{(2)} are nearly parallel (det⁡W(2)=0.29\det W^{(2)}=0.29, while each row has norm above 3.4). The network has learned that it needs only one direction to separate the classes, and it spends the second layer stretching the data along it. A trained network discards the directions the task does not need. That is the geometric side of the low-dimensional features studied under representation learning.

Worked example: one input through the trained network

StageComputationValue
z(1)z^{(1)}W(1)x+b(1)W^{(1)}x+b^{(1)} with W(1)=[−4.3605−0.9736−0.92210.5472]W^{(1)}=\begin{bmatrix}-4.3605&-0.9736\\-0.9221&0.5472\end{bmatrix}, b(1)=(−0.1283,−0.0690)b^{(1)}=(-0.1283,-0.0690)(−4.4888, −0.9911)
h(1)h^{(1)}tanh⁡z(1)\tanh z^{(1)}(−0.9997, −0.7578)
z(2)z^{(2)}W(2)h(1)+b(2)W^{(2)}h^{(1)}+b^{(2)} with W(2)=[−2.55384.37061.6918−3.0089]W^{(2)}=\begin{bmatrix}-2.5538&4.3706\\1.6918&-3.0089\end{bmatrix}, b(2)=(0.1997,−0.1311)b^{(2)}=(0.1997,-0.1311)(−0.5593, 0.4578)
h(2)h^{(2)}tanh⁡z(2)\tanh z^{(2)}(−0.5075, 0.4283)
logit(−5.5317, 3.6284)⋅h(2)+0.1353(-5.5317,\ 3.6284)\cdot h^{(2)}+0.13534.4964
y^\hat yσ(4.4964)\sigma(4.4964)0.9890 (class 1)
x = (1, 0) through the figure's network, rounded to 4 decimals. Stages match the figure's buttons.

The first unit saturates: tanh⁡(−4.49)=−0.9997\tanh(-4.49)=-0.9997, so moving xx further right changes almost nothing. Saturation is the tanh layer pressing regions against the wall of the square. It is useful for classification and, as the vanishing-gradients article shows, costly for learning.

Counting parameters and compute

A fully connected layer from nl−1n_{l-1} to nln_l units has nl(nl−1+1)n_l(n_{l-1}+1) parameters and needs nlnl−1n_ln_{l-1} multiply-adds (MACs) per input. Since one MAC is two floating-point operations, a forward pass costs about 2∑lnlnl−12\sum_l n_ln_{l-1} FLOPs per example, and training (forward plus backward) about three times that.

LayerParametersMACs per example
784 → 512401,920401,408
512 → 512262,656262,144
512 → 105,1305,120
total669,706668,672 (≈ 1.34 MFLOPs)
An MNIST-sized MLP, 784 → 512 → 512 → 10

Weight sharing changes this balance, and it is the main reason CNNs scale to images. A 3×33\times3 convolution from 64 to 64 channels has only 3⋅3⋅64⋅64+64=36,9283\cdot3\cdot64\cdot64+64=36{,}928 parameters. Applied at every position of a 56×5656\times56 feature map it costs 56⋅56⋅3⋅3⋅64⋅64≈1.16×10856\cdot56\cdot3\cdot3\cdot64\cdot64\approx1.16\times10^8 MACs. A fully connected layer between the same two 56×56×6456\times56\times64 tensors would need about 4×10104\times10^{10} weights. Both are feedforward. They differ in which edges of the DAG exist and which weights are tied.

python
1import torch, torch.nn as nn
2
3mlp = nn.Sequential(nn.Linear(784, 512), nn.ReLU(),
4 nn.Linear(512, 512), nn.ReLU(),
5 nn.Linear(512, 10))
6print(sum(p.numel() for p in mlp.parameters())) # 669706
7
8class SkipNet(nn.Module): # still a DAG: the skip edge points forward
9 def __init__(self, d=64):
10 super().__init__()
11 self.a, self.b, self.out = nn.Linear(d, d), nn.Linear(d, d), nn.Linear(d, 1)
12 def forward(self, x):
13 h1 = torch.relu(self.a(x))
14 h2 = torch.relu(self.b(h1)) + x # reads from two earlier nodes
15 return self.out(h2)
16
17net = SkipNet()
18print(sum(p.numel() for p in net.parameters())) # 8385 = 2·(64·65) + 65
19y1, y2 = net(torch.ones(1, 64)), net(torch.ones(1, 64))
20print(torch.equal(y1, y2)) # True: no state between calls
Two feedforward nets: a plain chain and a DAG with a skip connection, with parameter counts

Check yourself

Which of these is NOT a feedforward network in the graph sense?

Check yourself

A 2-2-2-1 tanh network (every hidden layer of width 2) is trained on points inside a circle (class 1) versus a ring around it (class 0). What happens as you add more width-2 layers?

Remember

  • ✓Feedforward = the computation graph is a DAG: one pass in topological order, no feedback, no memory between inputs.
  • ✓MLPs, CNNs, ResNets and Transformers on a fixed input are all FNNs; RNNs and Hopfield nets are not (until unrolled).
  • ✓Each layer is a coordinate change: an affine map keeps lines straight, the activation bends them; the readout is a line in the last space.
  • ✓Dense layer: nl(nl−1+1)n_l(n_{l-1}+1) parameters and nlnl−1n_ln_{l-1} MACs; a 784-512-512-10 MLP has 669,706 parameters.
  • ✓Width ≤ input dimension is a hard limit: such nets cannot enclose a bounded class region at any depth.
← back to Deep Learning roadmap