# Highway network

> A learned sigmoid gate mixes each layer's transform with its unchanged input; biasing the gate shut at init lets 100-layer nets train with plain SGD.

Source: https://www.mlfoundry.in/learn/highway-network  
Section: Deep Learning › Neural Network Algorithms  
Level: advanced · 9 min read · updated 2026-10-11

**Highway network:** is a feedforward network whose layers compute $y=H(x)\,T(x)+x\,(1-T(x))$: a **transform gate** $T(x)\in(0,1)$, learned per unit, decides how much of the layer's nonlinear transform $H(x)$ to use and how much of the input to **carry** through unchanged.

Srivastava, Greff and Schmidhuber proposed highway networks in May 2015 ("Highway Networks", then "Training Very Deep Networks", NeurIPS 2015). That was seven months before ResNet, and it was one of the first architectures to train feedforward nets with **100 layers** directly with SGD. Plain nets of that depth were far harder to optimise.

The idea comes from the LSTM. An LSTM carries its cell state across time steps through a multiplicative gate, so the gradient is not forced through a squashing nonlinearity at every step. A highway layer gives the same kind of gated, mostly linear path to information moving across **depth**.

## The highway layer

$$
y=H(x,W_H)\odot T(x,W_T)+x\odot C(x,W_C),\qquad T(x)=\sigma\big(W_Tx+b_T\big),\quad C=1-T
$$

- $H$: the usual layer, e.g. $\mathrm{ReLU}(W_Hx+b_H)$ or a convolution
- $T$: transform gate: a sigmoid per unit, its own weights and bias
- $C$: carry gate; the paper couples it as C = 1 − T, so each unit is a convex mix
- $b_T$: gate bias, initialised negative so T starts small and the layer starts close to the identity

Each unit interpolates between two behaviours. At $T=0$ it copies its input, $y=x$. At $T=1$ it is an ordinary layer, $y=H(x)$. Because the gate depends on $x$, the choice can differ per example and per unit. A layer needs as many outputs as inputs for the carry to work. When the width changes, the paper inserts a plain layer or uses subsampling and zero-padding.

## Backward pass: three paths

$$
\frac{\partial y}{\partial x}=\underbrace{\mathrm{diag}(1-T)}_{\text{carry}}+\underbrace{\mathrm{diag}\big(T\odot H'\big)W_H}_{\text{transform}}+\underbrace{\mathrm{diag}\big((H-x)\odot T\odot(1-T)\big)W_T}_{\text{through the gate}}
$$

- $H'$: slope of the transform's activation at $W_Hx+b_H$
- $T(1-T)$: slope of the sigmoid gate, $\le 0.25$

When $T\approx0$ the Jacobian is close to the identity, so the gradient passes through the layer almost unchanged, whatever $W_H$ and $H'$ look like. Stack $D$ layers and the carry path alone contributes $\prod_l\mathrm{diag}(1-T_l)$. That is still a **product**. With the gates half open it is $0.5^{D}$, which vanishes faster than many plain nets. This is why the gate bias matters so much.

Carry factor per layer at x-independent gates T = σ(b_T), and the carry product after D layers (computed in Node)

| b_T | T = σ(b_T) | 1 − T | D = 10 | D = 50 | D = 100 |
| --- | --- | --- | --- | --- | --- |
| 0 | 0.5000 | 0.5000 | $9.8\times10^{-4}$ | $8.9\times10^{-16}$ | $7.9\times10^{-31}$ |
| −1 | 0.2689 | 0.7311 | $4.4\times10^{-2}$ | $1.6\times10^{-7}$ | $2.5\times10^{-14}$ |
| −2 | 0.1192 | 0.8808 | 0.281 | $1.8\times10^{-3}$ | $3.1\times10^{-6}$ |
| −3 | 0.0474 | 0.9526 | 0.615 | $8.8\times10^{-2}$ | $7.8\times10^{-3}$ |
| −5 | 0.0067 | 0.9933 | 0.935 | 0.715 | 0.511 |

The paper suggests negative initial biases such as $-1$ or $-3$, and the table shows why deeper stacks need the more negative end. Those values put the network in the middle rows. The layers start mostly carrying, with the transform slightly open so $W_H$ still gets gradient (the transform term is scaled by $T$). Training then opens the gates where transforming helps.

> Figure: One real forward and backward pass through a 32-wide highway stack (accent) and the plain stack with the same W_H (rose). At the defaults (ReLU, s = 1, 50 layers, b_T = −3), the plain gradient falls to 3.4e−9 of its top value and the highway gradient to 0.079. The dashed mint line, the carry-only product Π(1 − T̄), predicts it (0.072). Press sweep: at b_T = 0 the gates sit at 0.5 and the highway net vanishes like the plain one (5.4e−8); by b_T = −5 it keeps 0.63. The heatmap shows that each unit's gate varies with the input even at initialization.

## Worked example: one scalar unit

Take $x=0.8$, $H(x)=\tanh(1.5x)$, $T(x)=\sigma(0.5x-2)$:

Forward and backward through one highway unit (dy/dx matches a central finite difference to 1e−10)

| Quantity | Computation | Value |
| --- | --- | --- |
| $H$ | $\tanh(1.2)$ | 0.8337 |
| $T$ | $\sigma(0.4-2)=\sigma(-1.6)$ | 0.1680 |
| $y$ | $0.8337\cdot0.1680+0.8\cdot0.8320$ | 0.8057 |
| carry term | $1-T$ | 0.8320 |
| transform term | $T\cdot(1-H^2)\cdot1.5$ | 0.0769 |
| gate term | $(H-x)\cdot T(1-T)\cdot0.5$ | 0.0024 |
| $dy/dx$ | sum of the three | 0.9112 |

The plain layer $y=\tanh(1.5x)$ has slope $0.4575$ at this point, so ten such layers would scale the gradient by roughly $0.4575^{10}\approx4\times10^{-4}$. The highway unit's slope is $0.91$, and 91% of it comes from the carry. The output, $0.806$, is almost the input, $0.8$: the unit starts near the identity, as intended.

**Building and training a highway net**

1. Embed the input to a fixed width with one plain layer (the carry needs matching shapes).
2. Stack highway layers $x\leftarrow H(x)\odot T(x)+x\odot(1-T(x))$. Every layer has two weight matrices, $W_H$ and $W_T$.
3. Initialise $W_H$ as usual (e.g. He) and $b_T$ negative, around $-1$ to $-3$, choosing the more negative end for deeper stacks.
4. Train end to end with SGD. The gates learn per example which layers transform and which just carry.

```python
import torch, torch.nn as nn

class Highway(nn.Module):
    def __init__(self, d, gate_bias=-3.0):
        super().__init__()
        self.H = nn.Linear(d, d)
        self.T = nn.Linear(d, d)
        nn.init.kaiming_normal_(self.H.weight, nonlinearity="relu")
        nn.init.constant_(self.T.bias, gate_bias)      # T ≈ σ(-3) = 0.0474: start near identity

    def forward(self, x):
        t = torch.sigmoid(self.T(x))
        return torch.relu(self.H(x)) * t + x * (1 - t)   # carry gate C = 1 - T

net = nn.Sequential(nn.Linear(784, 50), nn.ReLU(), *[Highway(50) for _ in range(100)], nn.Linear(50, 10))
# carry-only gradient factor at init: (1 - 0.0474) ** 100 ≈ 7.8e-3
```

A highway layer with a negatively biased transform gate

## Highway vs ResNet

Fix both gates open, $T=C=1$, and drop the coupling: $y=H(x)+x$. That is a residual block. The two designs differ in what multiplies the shortcut: a learned, data-dependent $1-T(x)$ in a highway layer, and exactly 1 in a ResNet.

| Property | Highway layer | Residual block |
| --- | --- | --- |
| Output | $H\odot T+x\odot(1-T)$ | $F(x)+x$ |
| Shortcut factor | $1-T(x)\in(0,1)$, learned | exactly 1 |
| Extra parameters | $W_T,\,b_T$ (doubles the layer) | none |
| Carry gradient over D layers | $\prod_l(1-T_l)$: shrinks unless the gates stay shut | identity term always present |
| Can switch a layer off | yes, $T\to0$ gives $y=x$ | yes, $F\to0$ gives $y=x$ |

> **Why the parameter-free identity won:** A coupled gate cannot keep the shortcut fully open and use the transform at the same time: turning on $H$ means scaling $x$ down. Over 100 layers those factors multiply. He et al. (2016) tested highway-style gating on ResNet-110 for CIFAR-10. It trained worse than the plain identity shortcut (6.61% error). They noted earlier that highway nets had shown no accuracy gains beyond about 100 layers, while ResNets kept improving at 152 layers on ImageNet and over 1000 on CIFAR-10.

> **A zero gate bias defeats the purpose:** With $b_T=0$ (the PyTorch default for a fresh `nn.Linear`) every gate starts near 0.5. The carry then halves the signal at every layer, and a 50-layer stack starts with a carry gradient of $0.5^{50}\approx9\times10^{-16}$, no better than a plain net. Always set the gate bias explicitly.

What do trained gates do? In the authors' analysis, most transform gates were closed on average, yet they opened **selectively**, depending on the input, so different examples are routed through different layers. In lesioning experiments (forcing one layer's gates shut), many layers of an MNIST model could be removed with little loss, while a CIFAR-100 model degraded much more: the network spends its depth where the task needs it.

- **Character-level NLP:** A few highway layers on top of a character CNN produce word embeddings (Kim et al. 2016; ELMo's character encoder).
- **Speech synthesis:** Tacotron's CBHG module stacks highway layers after its convolution bank.
- **Recurrent depth:** Recurrent Highway Networks (Zilly et al. 2017) stack highway layers inside each time step of an RNN.

**Question:** A 50-layer highway net is initialised with b_T = 0. Ignoring the transform path, by what factor does the carry path scale the gradient from output to input?

- A. About 1
- B. About 0.5
- C. About 0.5⁵⁰ ≈ 9e−16
- D. About 50 × 0.5 = 25

**Answer:** C. About 0.5⁵⁰ ≈ 9e−16 — With $b_T=0$, $T\approx\sigma(0)=0.5$, so each layer's carry contributes $1-T=0.5$, and the factors multiply over 50 layers: $0.5^{50}\approx8.9\times10^{-16}$. A bias of $-3$ gives $0.9526^{50}\approx0.088$ instead.

**Question:** Which change turns a highway layer into a residual block?

- A. Set b_T → +∞ so T = 1
- B. Replace T and C = 1 − T by constants T = C = 1
- C. Remove the nonlinearity from H
- D. Set W_T = W_H

**Answer:** B. Replace T and C = 1 − T by constants T = C = 1 — With both gates fixed at 1 the layer computes $H(x)+x$, which is a residual block. Setting $T=1$ while keeping $C=1-T$ gives $C=0$: a plain layer with no shortcut at all.

**Key points**

- Highway layer: $y=H(x)\,T(x)+x\,(1-T(x))$, with a sigmoid transform gate $T=\sigma(W_Tx+b_T)$ and a coupled carry gate $1-T$.
- The carry path's gradient is $\prod_l(1-T_l)$, so a negative gate bias ($-1$ to $-3$) is essential: it starts every layer near the identity.
- Srivastava, Greff & Schmidhuber (2015) trained 100-layer feedforward nets with plain SGD, before ResNet. The design was inspired by LSTM gating.
- ResNet is the gate-free special case $T=C=1$. Its exact identity shortcut adds no parameters and never shrinks, which is why it won at extreme depth.
- Trained gates are mostly closed on average but open selectively per input, so each example uses its own subset of layers.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
