Definition
Highway network is a feedforward network whose layers compute : a transform gate , learned per unit, decides how much of the layer's nonlinear transform to use and how much of the input to carry through unchanged.
Srivastava, Greff and Schmidhuber proposed highway networks in May 2015 ("Highway Networks", then "Training Very Deep Networks", NeurIPS 2015). That was seven months before ResNet, and it was one of the first architectures to train feedforward nets with 100 layers directly with SGD. Plain nets of that depth were far harder to optimise.
The idea comes from the LSTM. An LSTM carries its cell state across time steps through a multiplicative gate, so the gradient is not forced through a squashing nonlinearity at every step. A highway layer gives the same kind of gated, mostly linear path to information moving across depth.
The highway layer
- the usual layer, e.g. or a convolution
- transform gate: a sigmoid per unit, its own weights and bias
- carry gate; the paper couples it as C = 1 − T, so each unit is a convex mix
- gate bias, initialised negative so T starts small and the layer starts close to the identity
Each unit interpolates between two behaviours. At it copies its input, . At it is an ordinary layer, . Because the gate depends on , the choice can differ per example and per unit. A layer needs as many outputs as inputs for the carry to work. When the width changes, the paper inserts a plain layer or uses subsampling and zero-padding.
Backward pass: three paths
- slope of the transform's activation at
- slope of the sigmoid gate,
When the Jacobian is close to the identity, so the gradient passes through the layer almost unchanged, whatever and look like. Stack layers and the carry path alone contributes . That is still a product. With the gates half open it is , which vanishes faster than many plain nets. This is why the gate bias matters so much.
| b_T | T = σ(b_T) | 1 − T | D = 10 | D = 50 | D = 100 |
|---|---|---|---|---|---|
| 0 | 0.5000 | 0.5000 | |||
| −1 | 0.2689 | 0.7311 | |||
| −2 | 0.1192 | 0.8808 | 0.281 | ||
| −3 | 0.0474 | 0.9526 | 0.615 | ||
| −5 | 0.0067 | 0.9933 | 0.935 | 0.715 | 0.511 |
The paper suggests negative initial biases such as or , and the table shows why deeper stacks need the more negative end. Those values put the network in the middle rows. The layers start mostly carrying, with the transform slightly open so still gets gradient (the transform term is scaled by ). Training then opens the gates where transforming helps.
Worked example: one scalar unit
Take , , :
| Quantity | Computation | Value |
|---|---|---|
| 0.8337 | ||
| 0.1680 | ||
| 0.8057 | ||
| carry term | 0.8320 | |
| transform term | 0.0769 | |
| gate term | 0.0024 | |
| sum of the three | 0.9112 |
The plain layer has slope at this point, so ten such layers would scale the gradient by roughly . The highway unit's slope is , and 91% of it comes from the carry. The output, , is almost the input, : the unit starts near the identity, as intended.
Building and training a highway net
- 1Embed the input to a fixed width with one plain layer (the carry needs matching shapes).
- 2Stack highway layers . Every layer has two weight matrices, and .
- 3Initialise as usual (e.g. He) and negative, around to , choosing the more negative end for deeper stacks.
- 4Train end to end with SGD. The gates learn per example which layers transform and which just carry.
1import torch, torch.nn as nn23class Highway(nn.Module):4 def __init__(self, d, gate_bias=-3.0):5 super().__init__()6 self.H = nn.Linear(d, d)7 self.T = nn.Linear(d, d)8 nn.init.kaiming_normal_(self.H.weight, nonlinearity="relu")9 nn.init.constant_(self.T.bias, gate_bias) # T ≈ σ(-3) = 0.0474: start near identity1011 def forward(self, x):12 t = torch.sigmoid(self.T(x))13 return torch.relu(self.H(x)) * t + x * (1 - t) # carry gate C = 1 - T1415net = nn.Sequential(nn.Linear(784, 50), nn.ReLU(), *[Highway(50) for _ in range(100)], nn.Linear(50, 10))16# carry-only gradient factor at init: (1 - 0.0474) ** 100 ≈ 7.8e-3Highway vs ResNet
Fix both gates open, , and drop the coupling: . That is a residual block. The two designs differ in what multiplies the shortcut: a learned, data-dependent in a highway layer, and exactly 1 in a ResNet.
| Property | Highway layer | Residual block |
|---|---|---|
| Output | ||
| Shortcut factor | , learned | exactly 1 |
| Extra parameters | (doubles the layer) | none |
| Carry gradient over D layers | : shrinks unless the gates stay shut | identity term always present |
| Can switch a layer off | yes, gives | yes, gives |
What do trained gates do? In the authors' analysis, most transform gates were closed on average, yet they opened selectively, depending on the input, so different examples are routed through different layers. In lesioning experiments (forcing one layer's gates shut), many layers of an MNIST model could be removed with little loss, while a CIFAR-100 model degraded much more: the network spends its depth where the task needs it.
Where it's used
Character-level NLP
A few highway layers on top of a character CNN produce word embeddings (Kim et al. 2016; ELMo's character encoder).
Speech synthesis
Tacotron's CBHG module stacks highway layers after its convolution bank.
Recurrent depth
Recurrent Highway Networks (Zilly et al. 2017) stack highway layers inside each time step of an RNN.
Check yourself
A 50-layer highway net is initialised with b_T = 0. Ignoring the transform path, by what factor does the carry path scale the gradient from output to input?
Check yourself
Which change turns a highway layer into a residual block?
Remember
- ✓Highway layer: , with a sigmoid transform gate and a coupled carry gate .
- ✓The carry path's gradient is , so a negative gate bias ( to ) is essential: it starts every layer near the identity.
- ✓Srivastava, Greff & Schmidhuber (2015) trained 100-layer feedforward nets with plain SGD, before ResNet. The design was inspired by LSTM gating.
- ✓ResNet is the gate-free special case . Its exact identity shortcut adds no parameters and never shrinks, which is why it won at extreme depth.
- ✓Trained gates are mostly closed on average but open selectively per input, so each example uses its own subset of layers.