Temporal convolutional network (TCN)

A sequence model built only from causal, dilated 1-D convolutions in residual blocks. Dilations 1, 2, 4, … give a receptive field of 1 + 2(k−1)(2^L−1) steps after L blocks, every time step trains in parallel, and gradients never travel through a recurrence.

12 min read1221 wordscore0 viewsupdated 2026-10-12

Definition

Temporal convolutional network (TCN) is a sequence-to-sequence model made of stacked causal 1-D convolutions with exponentially increasing dilation, wrapped in residual blocks. Causal means the output at time tt depends only on inputs x0,…,xtx_0,\dots,x_t, so a TCN can replace an RNN anywhere an RNN reads a sequence left to right, including autoregressive prediction.

For years "sequence model" meant "recurrent network": an RNN, LSTM or GRU that carries a hidden state from step to step (see Recurrent neural network (RNN)). Recurrence has two costs. Step tt cannot be computed before step t−1t-1, so training is sequential in time, and the gradient to a distant input passes through a product of many Jacobians, which vanishes or explodes. Convolutions have neither problem: all outputs are computed at once, and the path from any input to any output has a fixed length set by the depth, not by the time lag.

The ingredients came from audio and translation. WaveNet (van den Oord et al., 2016) generated raw audio with stacks of causal dilated convolutions, dilations 1,2,…,5121,2,\dots,512 per stack; ByteNet (Kalchbrenner et al., 2016) used dilated convolutions for character-level translation; Dauphin et al. (2017) built gated convolutional language models. Bai, Kolter and Koltun ("An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling", arXiv 2018) distilled these into a deliberately plain architecture they called the TCN (the name had been used before by Lea et al. for action segmentation) and compared it with LSTMs and GRUs on the recurrent networks' own benchmarks. Their conclusion: the simple convolutional model wins on most of them and should be "a natural starting point" for sequence modelling.

The three ingredients

F(s)=(x∗df)(s)=∑i=0k−1f(i) xs−d⋅i,o=ReLU(x+F(x))F(s)=(x*_d f)(s)=\sum_{i=0}^{k-1} f(i)\,x_{s-d\cdot i},\qquad o=\mathrm{ReLU}\big(x+\mathcal F(x)\big)
ff
a filter of size kk; f(0)f(0) multiplies the current input xsx_s, f(i)f(i) the input d⋅id\cdot i steps back
dd
dilation: the gap between taps. Block ii uses d=2id=2^i (Bai et al.: d=O(2i)d=O(2^i) at level ii)
xs−dix_{s-d i}
only past and present inputs: causal. Implemented by padding (k−1)d(k-1)d zeros on the left only
F\mathcal F
the residual branch: dilated causal conv → weight norm → ReLU → spatial dropout, twice
x+F(x)x+\mathcal F(x)
the skip path; a 1×11\times1 convolution replaces the identity when the channel counts differ

Causal convolution is an ordinary 1-D convolution shifted so that it never looks ahead: pad (k−1)d(k-1)d zeros on the left, none on the right, and the output keeps the input's length TT. Dilation spreads the kk taps dd steps apart (see Dilated convolution), so one layer's history grows from k−1k-1 to (k−1)d(k-1)d steps at no extra cost. Residual blocks (as in Residual neural network (ResNet)) keep a network that is dozens of convolutions deep trainable. Each block of the generic TCN holds two convolutions with the same dilation, each followed by weight normalization, ReLU and spatial dropout (a whole channel is zeroed at a time).

Receptive field

With stride 1, a causal layer with kernel kk and dilation dd extends the history by (k−1)d(k-1)d steps, and the identity skip adds nothing (it connects xtx_t to oto_t, offset 0). Summing over LL blocks of two convolutions each, with d=2id=2^i:

R=1+∑layers(k−1) dℓ=1+2(k−1)∑i=0L−12i=1+2(k−1)(2L−1)R=1+\sum_{\text{layers}}(k-1)\,d_\ell=1+2(k-1)\sum_{i=0}^{L-1}2^i=1+2(k-1)\big(2^L-1\big)
RR
receptive field: output yty_t depends on xt−R+1,…,xtx_{t-R+1},\dots,x_t and on nothing earlier
LL
number of residual blocks (dilation doubles from block to block)
22
two convolutions per block; with one per block (WaveNet style) the factor is 1: R=1+(k−1)(2L−1)R=1+(k-1)(2^L-1)

The field grows exponentially in depth but the parameter count only linearly. Without dilation the same 2L2L layers would reach only 1+2L(k−1)1+2L(k-1) steps. For k=3k=3 and L=4L=4 blocks the dilated stack reaches 61 steps and the undilated one 17, with identical weights: 24 per channel pair, 24,832 parameters at 32 channels. To cover a sequence of length TT you need L≥log⁡2 ⁣(1+T−12(k−1))L\ge\log_2\!\big(1+\tfrac{T-1}{2(k-1)}\big) blocks. The models in Bai et al.'s Figure 2 (adding problem) use 7 levels with k=6k=6 and 8 levels with k=8k=8: R=1271R=1271 and 35713571, both longer than the task's sequences.

blocks Lk = 2k = 3k = 5k = 3, no dilation
13595
2713259
4316112117
612725350525
85111021204133
Receptive field of a TCN with two convolutions per block, d = 2^i (computed by the figure's library, which also counts the reachable inputs)
causal cone of output t = 63 · 4 blocks × 2 convs, k = 3, d = 2^ixd=1d=1d=2d=2d=4d=4d=8d=8++++receptive field 61RF = 1 + Σ(k−1)d61dilated formula 1+2(k−1)(2^4−1)61same stack, no dilation17inputs in cone / max paths61 / 269lag 40: x[t−40]reachable (175 paths)conv layers8weights / ch. pair24params (C = 32)24,832● output (click top row) · ● in cone, shade = paths · ]+ residual block: identity adds no reach
kernel kconvs / block
fig.A TCN over 64 time steps: the bottom row is the input, each row above is one dilated causal convolution, and violet brackets mark the residual blocks. The cone of nodes feeding the orange output is highlighted; input shading is the number of distinct paths. The default (k = 3, 4 blocks of 2 convs, d = 1, 2, 4, 8) has receptive field 61, so the output at t = 63 reads inputs 3 to 63 with up to 269 paths to a single input, and a dependency at lag 40 is reachable. Switch dilation off: the same 8 layers reach only 17 steps and lag 40 turns red. k = 2 with 4 blocks gives 31; one conv per block halves the growth. Click the top row to move the output: near t = 0 the cone is clipped because there is no earlier input, which is exactly what causal zero-padding means.

Why it trains well, and what it costs

In an RNN, the gradient from yty_t to xt−τx_{t-\tau} passes through τ\tau recurrent Jacobians. In a TCN it passes through at most 2L2L layers whatever τ\tau is, as long as τ<R\tau<R, and the residual paths shorten it further. Bai et al. list this as "stable gradients": the backpropagation path runs through depth, not along time. Training is also parallel: the whole sequence of length TT is one batched convolution, so a TCN uses the GPU like a CNN instead of looping over TT steps.

The price is paid at inference and in flexibility. An RNN summarises the whole past in a fixed-size state and needs only ht−1h_{t-1} and xtx_t for the next step; a TCN has to keep the last RR inputs (or, with caching, the last (k−1)d(k-1)d activations of every layer, as fast WaveNet implementations do). And the receptive field is a hard wall: an output simply cannot depend on anything older than R−1R-1 steps, so a model trained with a short history can fail when moved to a task that needs a longer one. Recurrent models and State space models (SSMs) have no such wall, at least in principle.

Worked example: two causal layers by hand

Take x=(1,2,0,−1,3,1,0,2)x=(1,2,0,-1,3,1,0,2) for t=0,…,7t=0,\dots,7. Layer 1 has k=2k=2, d=1d=1, f=(1, 0.5)f=(1,\,0.5); layer 2 has k=2k=2, d=2d=2, f=(1, −1)f=(1,\,-1). Values before t=0t=0 are the zero padding.

Forward pass (checked in Node with the figure's causal-convolution routine)

  1. 1Layer 1: zt=xt+0.5 xt−1z_t=x_t+0.5\,x_{t-1} gives z=(1,  2.5,  1,  −1,  2.5,  2.5,  0.5,  2)z=(1,\;2.5,\;1,\;-1,\;2.5,\;2.5,\;0.5,\;2). For example z4=3+0.5⋅(−1)=2.5z_4=3+0.5\cdot(-1)=2.5.
  2. 2Layer 2: yt=zt−zt−2y_t=z_t-z_{t-2} gives y=(1,  2.5,  0,  −3.5,  1.5,  3.5,  −2,  −0.5)y=(1,\;2.5,\;0,\;-3.5,\;1.5,\;3.5,\;-2,\;-0.5). For example y7=z7−z5=2−2.5=−0.5y_7=z_7-z_5=2-2.5=-0.5.
  3. 3Receptive field: R=1+(2−1)⋅1+(2−1)⋅2=4R=1+(2-1)\cdot1+(2-1)\cdot2=4. Expanding, y7=x7+0.5x6−x5−0.5x4y_7=x_7+0.5x_6-x_5-0.5x_4: the derivatives ∂y7/∂xj\partial y_7/\partial x_j are (0,0,0,0,−0.5,−1,0.5,1)(0,0,0,0,-0.5,-1,0.5,1), nonzero for exactly the last 4 inputs.
  4. 4Causality: changing x7x_7 changes only y7y_7; y0,…,y6y_0,\dots,y_6 are untouched, so the layer can be trained on all positions at once with no masking.

Against recurrent networks

task (metric)LSTMGRURNNTCN
Sequential MNIST (accuracy %)87.296.221.599.0
Permuted MNIST (accuracy %)85.787.325.397.2
Copy memory, T = 1000 (loss)0.02040.01970.02023.5e-5
JSB Chorales (NLL)8.458.438.918.10
Char-level PTB (bpc)1.361.371.481.31
Word-level PTB (perplexity)78.9392.48114.5088.68
LAMBADA (perplexity)4186–147251279
Bai, Kolter & Koltun (2018), Table 1. Generic models of roughly equal size; higher is better for accuracy, lower for loss, perplexity and bits per character

The TCN wins every row except word-level Penn Treebank, where a well-tuned LSTM (78.93) beats it (88.68). The paper is explicit that these are generic baselines and that specialised recurrent models (AWD-LSTM and others) hold the state of the art on several of these tasks. The striking result is the copy-memory task: the recurrent models converge to the loss of predicting all zeros, while the TCN recalls the 10 digits after 1000 steps almost perfectly. LAMBADA, which needs long context, tells the same story.

python
1import torch, torch.nn as nn, torch.nn.functional as F
2from torch.nn.utils.parametrizations import weight_norm
3
4class CausalConv1d(nn.Conv1d):
5 """Conv1d that pads (k-1)*d zeros on the left only: y[t] never sees x[t+1:]."""
6 def __init__(self, c_in, c_out, k, d):
7 super().__init__(c_in, c_out, k, dilation=d)
8 self.left = (k - 1) * d
9 def forward(self, x):
10 return super().forward(F.pad(x, (self.left, 0)))
11
12class TemporalBlock(nn.Module):
13 def __init__(self, c_in, c_out, k, d, p=0.1):
14 super().__init__()
15 self.branch = nn.Sequential(
16 weight_norm(CausalConv1d(c_in, c_out, k, d)), nn.ReLU(), nn.Dropout1d(p),
17 weight_norm(CausalConv1d(c_out, c_out, k, d)), nn.ReLU(), nn.Dropout1d(p))
18 self.skip = nn.Conv1d(c_in, c_out, 1) if c_in != c_out else nn.Identity()
19 def forward(self, x):
20 return F.relu(self.branch(x) + self.skip(x))
21
22class TCN(nn.Module):
23 def __init__(self, c_in, channels, k=3, p=0.1):
24 super().__init__()
25 ins = [c_in] + channels[:-1]
26 self.blocks = nn.Sequential(*[TemporalBlock(a, b, k, 2 ** i, p)
27 for i, (a, b) in enumerate(zip(ins, channels))])
28 self.k, self.L = k, len(channels)
29 def receptive_field(self):
30 return 1 + 2 * (self.k - 1) * (2 ** self.L - 1)
31 def forward(self, x): # x: (N, C_in, T) -> (N, C_out, T)
32 return self.blocks(x)
33
34tcn = TCN(c_in=1, channels=[32, 32, 32, 32], k=3).eval()
35print(tcn.receptive_field()) # 61
36
37x = torch.randn(2, 1, 100)
38y = tcn(x)
39print(y.shape) # torch.Size([2, 32, 100]), same length as the input
40
41x2 = x.clone(); x2[..., 70] += 1.0 # perturb one time step
42print(torch.allclose(tcn(x2)[..., :70], y[..., :70])) # True: outputs before t = 70 are unchanged
A generic TCN in PyTorch (layout (N, C, T), as in nn.Conv1d)

Use when

  • +Fully parallel over time in training: one convolution per layer for the whole sequence
  • +Receptive field 1+2(k−1)(2L−1)1+2(k-1)(2^L-1) grows exponentially with depth at linear parameter cost
  • +Gradient path length set by depth, not by the time lag: no vanishing through time
  • +Simple and robust: ReLU, weight norm, dropout, residuals; few hyperparameters
  • +Lower training memory than gated RNNs on long sequences (no per-step gate activations)

Avoid when

  • −Hard memory limit: nothing older than R−1R-1 steps can matter
  • −Inference needs the last RR inputs (or per-layer caches), not a single fixed-size state
  • −Changing the required history length means changing k or depth, i.e. the architecture
  • −Lost to tuned LSTMs on word-level PTB and later to Transformers and SSMs on long-context tasks

Check yourself

A TCN has 5 residual blocks with two causal convolutions each, kernel size 3 and dilations 1, 2, 4, 8, 16. What is its receptive field?

Check yourself

Why can a TCN be trained on all T positions of a sequence in parallel, while an RNN cannot?

Remember

  • ✓TCN = causal 1-D convolutions (left padding (k−1)d(k-1)d) + dilation d=2id=2^i + residual blocks of two convs (weight norm, ReLU, spatial dropout).
  • ✓Receptive field R=1+2(k−1)(2L−1)R=1+2(k-1)(2^L-1): k=3k=3, L=4L=4 gives 61 versus 17 without dilation, with the same 24 weights per channel pair.
  • ✓Parallel training and gradient paths of length O(depth) avoid the sequential cost and the vanishing gradients of RNNs.
  • ✓Bai, Kolter & Koltun (2018): the generic TCN beats generic LSTM/GRU/RNN on most of their benchmarks (99.0% seq. MNIST, copy memory solved), but not on word-level PTB.
  • ✓Costs: a hard memory limit at R steps and a raw-history buffer at inference instead of a fixed-size state.
← back to Deep Learning roadmap