Definition
A feedforward neural network is a neural network whose computation graph is a directed acyclic graph (DAG): information flows from the inputs to the outputs and never returns to an earlier node. With layers, it is the composition , and computing takes exactly one pass through the graph, with no state carried between inputs.
"Feedforward" describes graph shape, not layer type. The fully connected MLP is the textbook case, but convolutional nets, residual nets with skip connections, and a Transformer applied to a fixed input sequence are feedforward too: each is a DAG of operations evaluated once per input. What is excluded is feedback: a node whose output, directly or through other nodes, becomes its own input.
Acyclic means evaluable in one pass
Because a DAG has a topological order (an ordering in which every node comes after all its inputs), the forward pass is just "evaluate the nodes in that order". Each node runs once, as soon as its inputs exist. Backpropagation walks the same order in reverse. Every network a framework can run with a single forward call is, at that moment, a DAG.
- layer l: an affine map followed by an elementwise nonlinearity φ (the last layer usually has none, or a softmax/sigmoid)
- the representation after layer l; it depends only on h⁽ˡ⁻¹⁾, never on later layers or past inputs
A general DAG allows more than a chain. A node may read from several earlier nodes, for example in a residual block or a concatenation of branches in Inception. All that is required is that no path loops back.
Feedforward versus feedback
| Network | Graph | Consequence |
|---|---|---|
| MLP, CNN, ResNet, Transformer encoder | DAG | output is a fixed function of the current input; one forward pass |
| RNN: | cycle through time | has memory; the output depends on the whole input history |
| RNN unrolled over T steps | DAG of depth T with shared weights | trained as a deep feedforward net (backpropagation through time) |
| Hopfield net, Boltzmann machine | symmetric feedback connections | computes by iterating to a fixed point or sampling, not in one pass |
| Autoregressive generation with a Transformer | the network is a DAG; the loop feeding outputs back as inputs is outside it | every token is a fresh feedforward pass |
The unrolling row is worth remembering. Feedback over a finite horizon can always be rewritten as a deeper feedforward graph whose layers share weights. That is how recurrent nets are trained, and why everything about depth (vanishing gradients, initialization) carries over to them.
Each layer is a change of coordinates
Read a layer geometrically. The affine part rotates, scales, shears and shifts the space: straight lines stay straight and parallel lines stay parallel. The nonlinearity then acts coordinate by coordinate. squashes every axis into , so far-away regions get pressed against the walls of a square and straight grid lines bend. Composing layers is composing these warps.
The classifier at the end is just a line in the last hidden space. A feedforward net solves a non-linear problem by moving the data, layer by layer, into coordinates where a linear readout works. Watch that happen for a 2-2-2-1 tanh network trained on two moons. Every hidden layer has width 2, so every intermediate representation is itself a plane you can draw.
Two things to notice. First, the whole plane is transformed, not just the data. Every point of input space has a fate, and the shading shows where it ends up. Second, nothing is ever torn: each layer is a continuous map. With width 2 (no more than the input dimension) and monotone activations like tanh, the network cannot wrap one region around another, so it cannot classify the inside of a circle against its outside at any depth (Johnson, 2019). Some width beyond the input dimension is necessary, not just convenient.
Third, look at stage : the plane collapses almost onto a line. The rows of the trained are nearly parallel (, while each row has norm above 3.4). The network has learned that it needs only one direction to separate the classes, and it spends the second layer stretching the data along it. A trained network discards the directions the task does not need. That is the geometric side of the low-dimensional features studied under representation learning.
Worked example: one input through the trained network
| Stage | Computation | Value |
|---|---|---|
| with , | (−4.4888, −0.9911) | |
| (−0.9997, −0.7578) | ||
| with , | (−0.5593, 0.4578) | |
| (−0.5075, 0.4283) | ||
| logit | 4.4964 | |
| 0.9890 (class 1) |
The first unit saturates: , so moving further right changes almost nothing. Saturation is the tanh layer pressing regions against the wall of the square. It is useful for classification and, as the vanishing-gradients article shows, costly for learning.
Counting parameters and compute
A fully connected layer from to units has parameters and needs multiply-adds (MACs) per input. Since one MAC is two floating-point operations, a forward pass costs about FLOPs per example, and training (forward plus backward) about three times that.
| Layer | Parameters | MACs per example |
|---|---|---|
| 784 → 512 | 401,920 | 401,408 |
| 512 → 512 | 262,656 | 262,144 |
| 512 → 10 | 5,130 | 5,120 |
| total | 669,706 | 668,672 (≈ 1.34 MFLOPs) |
Weight sharing changes this balance, and it is the main reason CNNs scale to images. A convolution from 64 to 64 channels has only parameters. Applied at every position of a feature map it costs MACs. A fully connected layer between the same two tensors would need about weights. Both are feedforward. They differ in which edges of the DAG exist and which weights are tied.
1import torch, torch.nn as nn23mlp = nn.Sequential(nn.Linear(784, 512), nn.ReLU(),4 nn.Linear(512, 512), nn.ReLU(),5 nn.Linear(512, 10))6print(sum(p.numel() for p in mlp.parameters())) # 66970678class SkipNet(nn.Module): # still a DAG: the skip edge points forward9 def __init__(self, d=64):10 super().__init__()11 self.a, self.b, self.out = nn.Linear(d, d), nn.Linear(d, d), nn.Linear(d, 1)12 def forward(self, x):13 h1 = torch.relu(self.a(x))14 h2 = torch.relu(self.b(h1)) + x # reads from two earlier nodes15 return self.out(h2)1617net = SkipNet()18print(sum(p.numel() for p in net.parameters())) # 8385 = 2·(64·65) + 6519y1, y2 = net(torch.ones(1, 64)), net(torch.ones(1, 64))20print(torch.equal(y1, y2)) # True: no state between callsCheck yourself
Which of these is NOT a feedforward network in the graph sense?
Check yourself
A 2-2-2-1 tanh network (every hidden layer of width 2) is trained on points inside a circle (class 1) versus a ring around it (class 0). What happens as you add more width-2 layers?
Remember
- ✓Feedforward = the computation graph is a DAG: one pass in topological order, no feedback, no memory between inputs.
- ✓MLPs, CNNs, ResNets and Transformers on a fixed input are all FNNs; RNNs and Hopfield nets are not (until unrolled).
- ✓Each layer is a coordinate change: an affine map keeps lines straight, the activation bends them; the readout is a line in the last space.
- ✓Dense layer: parameters and MACs; a 784-512-512-10 MLP has 669,706 parameters.
- ✓Width ≤ input dimension is a hard limit: such nets cannot enclose a bounded class region at any depth.