Highway network

A learned sigmoid gate mixes each layer's transform with its unchanged input; biasing the gate shut at init lets 100-layer nets train with plain SGD.

9 min read796 wordsadvanced0 viewsupdated 2026-10-11

Definition

Highway network is a feedforward network whose layers compute y=H(x) T(x)+x (1−T(x))y=H(x)\,T(x)+x\,(1-T(x)): a transform gate T(x)∈(0,1)T(x)\in(0,1), learned per unit, decides how much of the layer's nonlinear transform H(x)H(x) to use and how much of the input to carry through unchanged.

Srivastava, Greff and Schmidhuber proposed highway networks in May 2015 ("Highway Networks", then "Training Very Deep Networks", NeurIPS 2015). That was seven months before ResNet, and it was one of the first architectures to train feedforward nets with 100 layers directly with SGD. Plain nets of that depth were far harder to optimise.

The idea comes from the LSTM. An LSTM carries its cell state across time steps through a multiplicative gate, so the gradient is not forced through a squashing nonlinearity at every step. A highway layer gives the same kind of gated, mostly linear path to information moving across depth.

The highway layer

y=H(x,WH)⊙T(x,WT)+x⊙C(x,WC),T(x)=σ(WTx+bT),C=1−Ty=H(x,W_H)\odot T(x,W_T)+x\odot C(x,W_C),\qquad T(x)=\sigma\big(W_Tx+b_T\big),\quad C=1-T
HH
the usual layer, e.g. ReLU(WHx+bH)\mathrm{ReLU}(W_Hx+b_H) or a convolution
TT
transform gate: a sigmoid per unit, its own weights and bias
CC
carry gate; the paper couples it as C = 1 − T, so each unit is a convex mix
bTb_T
gate bias, initialised negative so T starts small and the layer starts close to the identity

Each unit interpolates between two behaviours. At T=0T=0 it copies its input, y=xy=x. At T=1T=1 it is an ordinary layer, y=H(x)y=H(x). Because the gate depends on xx, the choice can differ per example and per unit. A layer needs as many outputs as inputs for the carry to work. When the width changes, the paper inserts a plain layer or uses subsampling and zero-padding.

Backward pass: three paths

∂y∂x=diag(1−T)⏟carry+diag(T⊙H′)WH⏟transform+diag((H−x)⊙T⊙(1−T))WT⏟through the gate\frac{\partial y}{\partial x}=\underbrace{\mathrm{diag}(1-T)}_{\text{carry}}+\underbrace{\mathrm{diag}\big(T\odot H'\big)W_H}_{\text{transform}}+\underbrace{\mathrm{diag}\big((H-x)\odot T\odot(1-T)\big)W_T}_{\text{through the gate}}
H′H'
slope of the transform's activation at WHx+bHW_Hx+b_H
T(1−T)T(1-T)
slope of the sigmoid gate, ≤0.25\le 0.25

When T≈0T\approx0 the Jacobian is close to the identity, so the gradient passes through the layer almost unchanged, whatever WHW_H and H′H' look like. Stack DD layers and the carry path alone contributes ∏ldiag(1−Tl)\prod_l\mathrm{diag}(1-T_l). That is still a product. With the gates half open it is 0.5D0.5^{D}, which vanishes faster than many plain nets. This is why the gate bias matters so much.

b_TT = σ(b_T)1 − TD = 10D = 50D = 100
00.50000.50009.8×10−49.8\times10^{-4}8.9×10−168.9\times10^{-16}7.9×10−317.9\times10^{-31}
−10.26890.73114.4×10−24.4\times10^{-2}1.6×10−71.6\times10^{-7}2.5×10−142.5\times10^{-14}
−20.11920.88080.2811.8×10−31.8\times10^{-3}3.1×10−63.1\times10^{-6}
−30.04740.95260.6158.8×10−28.8\times10^{-2}7.8×10−37.8\times10^{-3}
−50.00670.99330.9350.7150.511
Carry factor per layer at x-independent gates T = σ(b_T), and the carry product after D layers (computed in Node)

The paper suggests negative initial biases such as −1-1 or −3-3, and the table shows why deeper stacks need the more negative end. Those values put the network in the middle rows. The layers start mostly carrying, with the transform slightly open so WHW_H still gets gradient (the transform term is scaled by TT). Training then opens the gates where transforming helps.

one highway layer, stacked 50× · width 32, batch 16xH = relu(W_H x)·TT = σ(W_T x + -3.0)carry: x passes unchanged, weighted by 1 − T·(1−T)+y = H·T + x·(1−T)backward: ‖∂L/∂hₗ‖ (log)1e-121e-81e-411e4l = 0 (input)l = 50forward: RMS(hₗ) (log)1e-121e-81e-411e4l = 0 (input)l = 50gate T per unit (rows) × layer (cols), one example · darker = more transformTT̄mean over batch and units, per layer · empty = 0, full colour = 1━ plain (same W_H, no gate) ━ highway ╌ carry-only prediction Πₖ(1 − T̄ₖ)
activation
W_H scale 1.00
presets
mean T̄ 0.051‖∂L/∂h₀‖ / ‖∂L/∂h_D‖: plain 3.4e-9 · highway 0.0785 · Π(1 − T̄) 0.0716
fig.One real forward and backward pass through a 32-wide highway stack (accent) and the plain stack with the same W_H (rose). At the defaults (ReLU, s = 1, 50 layers, b_T = −3), the plain gradient falls to 3.4e−9 of its top value and the highway gradient to 0.079. The dashed mint line, the carry-only product Π(1 − T̄), predicts it (0.072). Press sweep: at b_T = 0 the gates sit at 0.5 and the highway net vanishes like the plain one (5.4e−8); by b_T = −5 it keeps 0.63. The heatmap shows that each unit's gate varies with the input even at initialization.

Worked example: one scalar unit

Take x=0.8x=0.8, H(x)=tanh⁡(1.5x)H(x)=\tanh(1.5x), T(x)=σ(0.5x−2)T(x)=\sigma(0.5x-2):

QuantityComputationValue
HHtanh⁡(1.2)\tanh(1.2)0.8337
TTσ(0.4−2)=σ(−1.6)\sigma(0.4-2)=\sigma(-1.6)0.1680
yy0.8337⋅0.1680+0.8⋅0.83200.8337\cdot0.1680+0.8\cdot0.83200.8057
carry term1−T1-T0.8320
transform termT⋅(1−H2)⋅1.5T\cdot(1-H^2)\cdot1.50.0769
gate term(H−x)⋅T(1−T)⋅0.5(H-x)\cdot T(1-T)\cdot0.50.0024
dy/dxdy/dxsum of the three0.9112
Forward and backward through one highway unit (dy/dx matches a central finite difference to 1e−10)

The plain layer y=tanh⁡(1.5x)y=\tanh(1.5x) has slope 0.45750.4575 at this point, so ten such layers would scale the gradient by roughly 0.457510≈4×10−40.4575^{10}\approx4\times10^{-4}. The highway unit's slope is 0.910.91, and 91% of it comes from the carry. The output, 0.8060.806, is almost the input, 0.80.8: the unit starts near the identity, as intended.

Building and training a highway net

  1. 1Embed the input to a fixed width with one plain layer (the carry needs matching shapes).
  2. 2Stack highway layers x←H(x)⊙T(x)+x⊙(1−T(x))x\leftarrow H(x)\odot T(x)+x\odot(1-T(x)). Every layer has two weight matrices, WHW_H and WTW_T.
  3. 3Initialise WHW_H as usual (e.g. He) and bTb_T negative, around −1-1 to −3-3, choosing the more negative end for deeper stacks.
  4. 4Train end to end with SGD. The gates learn per example which layers transform and which just carry.
python
1import torch, torch.nn as nn
2
3class Highway(nn.Module):
4 def __init__(self, d, gate_bias=-3.0):
5 super().__init__()
6 self.H = nn.Linear(d, d)
7 self.T = nn.Linear(d, d)
8 nn.init.kaiming_normal_(self.H.weight, nonlinearity="relu")
9 nn.init.constant_(self.T.bias, gate_bias) # T ≈ σ(-3) = 0.0474: start near identity
10
11 def forward(self, x):
12 t = torch.sigmoid(self.T(x))
13 return torch.relu(self.H(x)) * t + x * (1 - t) # carry gate C = 1 - T
14
15net = nn.Sequential(nn.Linear(784, 50), nn.ReLU(), *[Highway(50) for _ in range(100)], nn.Linear(50, 10))
16# carry-only gradient factor at init: (1 - 0.0474) ** 100 ≈ 7.8e-3
A highway layer with a negatively biased transform gate

Highway vs ResNet

Fix both gates open, T=C=1T=C=1, and drop the coupling: y=H(x)+xy=H(x)+x. That is a residual block. The two designs differ in what multiplies the shortcut: a learned, data-dependent 1−T(x)1-T(x) in a highway layer, and exactly 1 in a ResNet.

PropertyHighway layerResidual block
OutputH⊙T+x⊙(1−T)H\odot T+x\odot(1-T)F(x)+xF(x)+x
Shortcut factor1−T(x)∈(0,1)1-T(x)\in(0,1), learnedexactly 1
Extra parametersWT, bTW_T,\,b_T (doubles the layer)none
Carry gradient over D layers∏l(1−Tl)\prod_l(1-T_l): shrinks unless the gates stay shutidentity term always present
Can switch a layer offyes, T→0T\to0 gives y=xy=xyes, F→0F\to0 gives y=xy=x

What do trained gates do? In the authors' analysis, most transform gates were closed on average, yet they opened selectively, depending on the input, so different examples are routed through different layers. In lesioning experiments (forcing one layer's gates shut), many layers of an MNIST model could be removed with little loss, while a CIFAR-100 model degraded much more: the network spends its depth where the task needs it.

Where it's used

Character-level NLP

A few highway layers on top of a character CNN produce word embeddings (Kim et al. 2016; ELMo's character encoder).

Speech synthesis

Tacotron's CBHG module stacks highway layers after its convolution bank.

Recurrent depth

Recurrent Highway Networks (Zilly et al. 2017) stack highway layers inside each time step of an RNN.

Check yourself

A 50-layer highway net is initialised with b_T = 0. Ignoring the transform path, by what factor does the carry path scale the gradient from output to input?

Check yourself

Which change turns a highway layer into a residual block?

Remember

  • ✓Highway layer: y=H(x) T(x)+x (1−T(x))y=H(x)\,T(x)+x\,(1-T(x)), with a sigmoid transform gate T=σ(WTx+bT)T=\sigma(W_Tx+b_T) and a coupled carry gate 1−T1-T.
  • ✓The carry path's gradient is ∏l(1−Tl)\prod_l(1-T_l), so a negative gate bias (−1-1 to −3-3) is essential: it starts every layer near the identity.
  • ✓Srivastava, Greff & Schmidhuber (2015) trained 100-layer feedforward nets with plain SGD, before ResNet. The design was inspired by LSTM gating.
  • ✓ResNet is the gate-free special case T=C=1T=C=1. Its exact identity shortcut adds no parameters and never shrinks, which is why it won at extreme depth.
  • ✓Trained gates are mostly closed on average but open selectively per input, so each example uses its own subset of layers.
← back to Deep Learning roadmap