Definition
Temporal convolutional network (TCN) is a sequence-to-sequence model made of stacked causal 1-D convolutions with exponentially increasing dilation, wrapped in residual blocks. Causal means the output at time depends only on inputs , so a TCN can replace an RNN anywhere an RNN reads a sequence left to right, including autoregressive prediction.
For years "sequence model" meant "recurrent network": an RNN, LSTM or GRU that carries a hidden state from step to step (see Recurrent neural network (RNN)). Recurrence has two costs. Step cannot be computed before step , so training is sequential in time, and the gradient to a distant input passes through a product of many Jacobians, which vanishes or explodes. Convolutions have neither problem: all outputs are computed at once, and the path from any input to any output has a fixed length set by the depth, not by the time lag.
The ingredients came from audio and translation. WaveNet (van den Oord et al., 2016) generated raw audio with stacks of causal dilated convolutions, dilations per stack; ByteNet (Kalchbrenner et al., 2016) used dilated convolutions for character-level translation; Dauphin et al. (2017) built gated convolutional language models. Bai, Kolter and Koltun ("An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling", arXiv 2018) distilled these into a deliberately plain architecture they called the TCN (the name had been used before by Lea et al. for action segmentation) and compared it with LSTMs and GRUs on the recurrent networks' own benchmarks. Their conclusion: the simple convolutional model wins on most of them and should be "a natural starting point" for sequence modelling.
The three ingredients
- a filter of size ; multiplies the current input , the input steps back
- dilation: the gap between taps. Block uses (Bai et al.: at level )
- only past and present inputs: causal. Implemented by padding zeros on the left only
- the residual branch: dilated causal conv → weight norm → ReLU → spatial dropout, twice
- the skip path; a convolution replaces the identity when the channel counts differ
Causal convolution is an ordinary 1-D convolution shifted so that it never looks ahead: pad zeros on the left, none on the right, and the output keeps the input's length . Dilation spreads the taps steps apart (see Dilated convolution), so one layer's history grows from to steps at no extra cost. Residual blocks (as in Residual neural network (ResNet)) keep a network that is dozens of convolutions deep trainable. Each block of the generic TCN holds two convolutions with the same dilation, each followed by weight normalization, ReLU and spatial dropout (a whole channel is zeroed at a time).
Receptive field
With stride 1, a causal layer with kernel and dilation extends the history by steps, and the identity skip adds nothing (it connects to , offset 0). Summing over blocks of two convolutions each, with :
- receptive field: output depends on and on nothing earlier
- number of residual blocks (dilation doubles from block to block)
- two convolutions per block; with one per block (WaveNet style) the factor is 1:
The field grows exponentially in depth but the parameter count only linearly. Without dilation the same layers would reach only steps. For and blocks the dilated stack reaches 61 steps and the undilated one 17, with identical weights: 24 per channel pair, 24,832 parameters at 32 channels. To cover a sequence of length you need blocks. The models in Bai et al.'s Figure 2 (adding problem) use 7 levels with and 8 levels with : and , both longer than the task's sequences.
| blocks L | k = 2 | k = 3 | k = 5 | k = 3, no dilation |
|---|---|---|---|---|
| 1 | 3 | 5 | 9 | 5 |
| 2 | 7 | 13 | 25 | 9 |
| 4 | 31 | 61 | 121 | 17 |
| 6 | 127 | 253 | 505 | 25 |
| 8 | 511 | 1021 | 2041 | 33 |
Why it trains well, and what it costs
In an RNN, the gradient from to passes through recurrent Jacobians. In a TCN it passes through at most layers whatever is, as long as , and the residual paths shorten it further. Bai et al. list this as "stable gradients": the backpropagation path runs through depth, not along time. Training is also parallel: the whole sequence of length is one batched convolution, so a TCN uses the GPU like a CNN instead of looping over steps.
The price is paid at inference and in flexibility. An RNN summarises the whole past in a fixed-size state and needs only and for the next step; a TCN has to keep the last inputs (or, with caching, the last activations of every layer, as fast WaveNet implementations do). And the receptive field is a hard wall: an output simply cannot depend on anything older than steps, so a model trained with a short history can fail when moved to a task that needs a longer one. Recurrent models and State space models (SSMs) have no such wall, at least in principle.
Worked example: two causal layers by hand
Take for . Layer 1 has , , ; layer 2 has , , . Values before are the zero padding.
Forward pass (checked in Node with the figure's causal-convolution routine)
- 1Layer 1: gives . For example .
- 2Layer 2: gives . For example .
- 3Receptive field: . Expanding, : the derivatives are , nonzero for exactly the last 4 inputs.
- 4Causality: changing changes only ; are untouched, so the layer can be trained on all positions at once with no masking.
Against recurrent networks
| task (metric) | LSTM | GRU | RNN | TCN |
|---|---|---|---|---|
| Sequential MNIST (accuracy %) | 87.2 | 96.2 | 21.5 | 99.0 |
| Permuted MNIST (accuracy %) | 85.7 | 87.3 | 25.3 | 97.2 |
| Copy memory, T = 1000 (loss) | 0.0204 | 0.0197 | 0.0202 | 3.5e-5 |
| JSB Chorales (NLL) | 8.45 | 8.43 | 8.91 | 8.10 |
| Char-level PTB (bpc) | 1.36 | 1.37 | 1.48 | 1.31 |
| Word-level PTB (perplexity) | 78.93 | 92.48 | 114.50 | 88.68 |
| LAMBADA (perplexity) | 4186 | – | 14725 | 1279 |
The TCN wins every row except word-level Penn Treebank, where a well-tuned LSTM (78.93) beats it (88.68). The paper is explicit that these are generic baselines and that specialised recurrent models (AWD-LSTM and others) hold the state of the art on several of these tasks. The striking result is the copy-memory task: the recurrent models converge to the loss of predicting all zeros, while the TCN recalls the 10 digits after 1000 steps almost perfectly. LAMBADA, which needs long context, tells the same story.
1import torch, torch.nn as nn, torch.nn.functional as F2from torch.nn.utils.parametrizations import weight_norm34class CausalConv1d(nn.Conv1d):5 """Conv1d that pads (k-1)*d zeros on the left only: y[t] never sees x[t+1:]."""6 def __init__(self, c_in, c_out, k, d):7 super().__init__(c_in, c_out, k, dilation=d)8 self.left = (k - 1) * d9 def forward(self, x):10 return super().forward(F.pad(x, (self.left, 0)))1112class TemporalBlock(nn.Module):13 def __init__(self, c_in, c_out, k, d, p=0.1):14 super().__init__()15 self.branch = nn.Sequential(16 weight_norm(CausalConv1d(c_in, c_out, k, d)), nn.ReLU(), nn.Dropout1d(p),17 weight_norm(CausalConv1d(c_out, c_out, k, d)), nn.ReLU(), nn.Dropout1d(p))18 self.skip = nn.Conv1d(c_in, c_out, 1) if c_in != c_out else nn.Identity()19 def forward(self, x):20 return F.relu(self.branch(x) + self.skip(x))2122class TCN(nn.Module):23 def __init__(self, c_in, channels, k=3, p=0.1):24 super().__init__()25 ins = [c_in] + channels[:-1]26 self.blocks = nn.Sequential(*[TemporalBlock(a, b, k, 2 ** i, p)27 for i, (a, b) in enumerate(zip(ins, channels))])28 self.k, self.L = k, len(channels)29 def receptive_field(self):30 return 1 + 2 * (self.k - 1) * (2 ** self.L - 1)31 def forward(self, x): # x: (N, C_in, T) -> (N, C_out, T)32 return self.blocks(x)3334tcn = TCN(c_in=1, channels=[32, 32, 32, 32], k=3).eval()35print(tcn.receptive_field()) # 613637x = torch.randn(2, 1, 100)38y = tcn(x)39print(y.shape) # torch.Size([2, 32, 100]), same length as the input4041x2 = x.clone(); x2[..., 70] += 1.0 # perturb one time step42print(torch.allclose(tcn(x2)[..., :70], y[..., :70])) # True: outputs before t = 70 are unchangedUse when
- +Fully parallel over time in training: one convolution per layer for the whole sequence
- +Receptive field grows exponentially with depth at linear parameter cost
- +Gradient path length set by depth, not by the time lag: no vanishing through time
- +Simple and robust: ReLU, weight norm, dropout, residuals; few hyperparameters
- +Lower training memory than gated RNNs on long sequences (no per-step gate activations)
Avoid when
- −Hard memory limit: nothing older than steps can matter
- −Inference needs the last inputs (or per-layer caches), not a single fixed-size state
- −Changing the required history length means changing k or depth, i.e. the architecture
- −Lost to tuned LSTMs on word-level PTB and later to Transformers and SSMs on long-context tasks
Check yourself
A TCN has 5 residual blocks with two causal convolutions each, kernel size 3 and dilations 1, 2, 4, 8, 16. What is its receptive field?
Check yourself
Why can a TCN be trained on all T positions of a sequence in parallel, while an RNN cannot?
Remember
- ✓TCN = causal 1-D convolutions (left padding ) + dilation + residual blocks of two convs (weight norm, ReLU, spatial dropout).
- ✓Receptive field : , gives 61 versus 17 without dilation, with the same 24 weights per channel pair.
- ✓Parallel training and gradient paths of length O(depth) avoid the sequential cost and the vanishing gradients of RNNs.
- ✓Bai, Kolter & Koltun (2018): the generic TCN beats generic LSTM/GRU/RNN on most of their benchmarks (99.0% seq. MNIST, copy memory solved), but not on word-level PTB.
- ✓Costs: a hard memory limit at R steps and a raw-history buffer at inference instead of a fixed-size state.