# Temporal convolutional network (TCN)

> A sequence model built only from causal, dilated 1-D convolutions in residual blocks. Dilations 1, 2, 4, … give a receptive field of 1 + 2(k−1)(2^L−1) steps after L blocks, every time step trains in parallel, and gradients never travel through a recurrence.

Source: https://www.mlfoundry.in/learn/temporal-convolutional-network-tcn  
Section: Deep Learning › Recurrent Neural Networks and Sequence Models  
Level: core · 12 min read · updated 2026-10-12

**Temporal convolutional network:** (TCN) is a sequence-to-sequence model made of stacked **causal** 1-D convolutions with exponentially increasing **dilation**, wrapped in **residual blocks**. Causal means the output at time $t$ depends only on inputs $x_0,\dots,x_t$, so a TCN can replace an RNN anywhere an RNN reads a sequence left to right, including autoregressive prediction.

For years "sequence model" meant "recurrent network": an RNN, LSTM or GRU that carries a hidden state from step to step (see **Recurrent neural network (RNN)**). Recurrence has two costs. Step $t$ cannot be computed before step $t-1$, so training is sequential in time, and the gradient to a distant input passes through a product of many Jacobians, which vanishes or explodes. Convolutions have neither problem: all outputs are computed at once, and the path from any input to any output has a fixed length set by the depth, not by the time lag.

The ingredients came from audio and translation. **WaveNet** (van den Oord et al., 2016) generated raw audio with stacks of causal dilated convolutions, dilations $1,2,\dots,512$ per stack; **ByteNet** (Kalchbrenner et al., 2016) used dilated convolutions for character-level translation; Dauphin et al. (2017) built gated convolutional language models. Bai, Kolter and Koltun ("An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling", arXiv 2018) distilled these into a deliberately plain architecture they called the **TCN** (the name had been used before by Lea et al. for action segmentation) and compared it with LSTMs and GRUs on the recurrent networks' own benchmarks. Their conclusion: the simple convolutional model wins on most of them and should be "a natural starting point" for sequence modelling.

## The three ingredients

$$
F(s)=(x*_d f)(s)=\sum_{i=0}^{k-1} f(i)\,x_{s-d\cdot i},\qquad o=\mathrm{ReLU}\big(x+\mathcal F(x)\big)
$$

- $f$: a filter of size $k$; $f(0)$ multiplies the current input $x_s$, $f(i)$ the input $d\cdot i$ steps back
- $d$: dilation: the gap between taps. Block $i$ uses $d=2^i$ (Bai et al.: $d=O(2^i)$ at level $i$)
- $x_{s-d i}$: only past and present inputs: **causal**. Implemented by padding $(k-1)d$ zeros on the left only
- $\mathcal F$: the residual branch: dilated causal conv → weight norm → ReLU → spatial dropout, twice
- $x+\mathcal F(x)$: the skip path; a $1\times1$ convolution replaces the identity when the channel counts differ

**Causal convolution** is an ordinary 1-D convolution shifted so that it never looks ahead: pad $(k-1)d$ zeros on the left, none on the right, and the output keeps the input's length $T$. **Dilation** spreads the $k$ taps $d$ steps apart (see **Dilated convolution**), so one layer's history grows from $k-1$ to $(k-1)d$ steps at no extra cost. **Residual blocks** (as in **Residual neural network (ResNet)**) keep a network that is dozens of convolutions deep trainable. Each block of the generic TCN holds two convolutions with the same dilation, each followed by weight normalization, ReLU and **spatial dropout** (a whole channel is zeroed at a time).

## Receptive field

With stride 1, a causal layer with kernel $k$ and dilation $d$ extends the history by $(k-1)d$ steps, and the identity skip adds nothing (it connects $x_t$ to $o_t$, offset 0). Summing over $L$ blocks of two convolutions each, with $d=2^i$:

$$
R=1+\sum_{\text{layers}}(k-1)\,d_\ell=1+2(k-1)\sum_{i=0}^{L-1}2^i=1+2(k-1)\big(2^L-1\big)
$$

- $R$: receptive field: output $y_t$ depends on $x_{t-R+1},\dots,x_t$ and on nothing earlier
- $L$: number of residual blocks (dilation doubles from block to block)
- $2$: two convolutions per block; with one per block (WaveNet style) the factor is 1: $R=1+(k-1)(2^L-1)$

The field grows **exponentially in depth** but the parameter count only **linearly**. Without dilation the same $2L$ layers would reach only $1+2L(k-1)$ steps. For $k=3$ and $L=4$ blocks the dilated stack reaches 61 steps and the undilated one 17, with identical weights: 24 per channel pair, 24,832 parameters at 32 channels. To cover a sequence of length $T$ you need $L\ge\log_2\!\big(1+\tfrac{T-1}{2(k-1)}\big)$ blocks. The models in Bai et al.'s Figure 2 (adding problem) use 7 levels with $k=6$ and 8 levels with $k=8$: $R=1271$ and $3571$, both longer than the task's sequences.

Receptive field of a TCN with two convolutions per block, d = 2^i (computed by the figure's library, which also counts the reachable inputs)

| blocks L | k = 2 | k = 3 | k = 5 | k = 3, no dilation |
| --- | --- | --- | --- | --- |
| 1 | 3 | 5 | 9 | 5 |
| 2 | 7 | 13 | 25 | 9 |
| 4 | 31 | 61 | 121 | 17 |
| 6 | 127 | 253 | 505 | 25 |
| 8 | 511 | 1021 | 2041 | 33 |

> Figure: A TCN over 64 time steps: the bottom row is the input, each row above is one dilated causal convolution, and violet brackets mark the residual blocks. The cone of nodes feeding the orange output is highlighted; input shading is the number of distinct paths. The default (k = 3, 4 blocks of 2 convs, d = 1, 2, 4, 8) has receptive field 61, so the output at t = 63 reads inputs 3 to 63 with up to 269 paths to a single input, and a dependency at lag 40 is reachable. Switch dilation off: the same 8 layers reach only 17 steps and lag 40 turns red. k = 2 with 4 blocks gives 31; one conv per block halves the growth. Click the top row to move the output: near t = 0 the cone is clipped because there is no earlier input, which is exactly what causal zero-padding means.

## Why it trains well, and what it costs

In an RNN, the gradient from $y_t$ to $x_{t-\tau}$ passes through $\tau$ recurrent Jacobians. In a TCN it passes through at most $2L$ layers whatever $\tau$ is, as long as $\tau<R$, and the residual paths shorten it further. Bai et al. list this as "stable gradients": the backpropagation path runs through depth, not along time. Training is also parallel: the whole sequence of length $T$ is one batched convolution, so a TCN uses the GPU like a CNN instead of looping over $T$ steps.

The price is paid at inference and in flexibility. An RNN summarises the whole past in a fixed-size state and needs only $h_{t-1}$ and $x_t$ for the next step; a TCN has to keep the last $R$ inputs (or, with caching, the last $(k-1)d$ activations of every layer, as fast WaveNet implementations do). And the receptive field is a hard wall: an output simply cannot depend on anything older than $R-1$ steps, so a model trained with a short history can fail when moved to a task that needs a longer one. Recurrent models and **State space models (SSMs)** have no such wall, at least in principle.

## Worked example: two causal layers by hand

Take $x=(1,2,0,-1,3,1,0,2)$ for $t=0,\dots,7$. Layer 1 has $k=2$, $d=1$, $f=(1,\,0.5)$; layer 2 has $k=2$, $d=2$, $f=(1,\,-1)$. Values before $t=0$ are the zero padding.

**Forward pass (checked in Node with the figure's causal-convolution routine)**

1. Layer 1: $z_t=x_t+0.5\,x_{t-1}$ gives $z=(1,\;2.5,\;1,\;-1,\;2.5,\;2.5,\;0.5,\;2)$. For example $z_4=3+0.5\cdot(-1)=2.5$.
2. Layer 2: $y_t=z_t-z_{t-2}$ gives $y=(1,\;2.5,\;0,\;-3.5,\;1.5,\;3.5,\;-2,\;-0.5)$. For example $y_7=z_7-z_5=2-2.5=-0.5$.
3. Receptive field: $R=1+(2-1)\cdot1+(2-1)\cdot2=4$. Expanding, $y_7=x_7+0.5x_6-x_5-0.5x_4$: the derivatives $\partial y_7/\partial x_j$ are $(0,0,0,0,-0.5,-1,0.5,1)$, nonzero for exactly the last 4 inputs.
4. Causality: changing $x_7$ changes only $y_7$; $y_0,\dots,y_6$ are untouched, so the layer can be trained on all positions at once with no masking.

## Against recurrent networks

Bai, Kolter & Koltun (2018), Table 1. Generic models of roughly equal size; higher is better for accuracy, lower for loss, perplexity and bits per character

| task (metric) | LSTM | GRU | RNN | TCN |
| --- | --- | --- | --- | --- |
| Sequential MNIST (accuracy %) | 87.2 | 96.2 | 21.5 | 99.0 |
| Permuted MNIST (accuracy %) | 85.7 | 87.3 | 25.3 | 97.2 |
| Copy memory, T = 1000 (loss) | 0.0204 | 0.0197 | 0.0202 | 3.5e-5 |
| JSB Chorales (NLL) | 8.45 | 8.43 | 8.91 | 8.10 |
| Char-level PTB (bpc) | 1.36 | 1.37 | 1.48 | 1.31 |
| Word-level PTB (perplexity) | 78.93 | 92.48 | 114.50 | 88.68 |
| LAMBADA (perplexity) | 4186 | – | 14725 | 1279 |

The TCN wins every row except word-level Penn Treebank, where a well-tuned LSTM (78.93) beats it (88.68). The paper is explicit that these are *generic* baselines and that specialised recurrent models (AWD-LSTM and others) hold the state of the art on several of these tasks. The striking result is the copy-memory task: the recurrent models converge to the loss of predicting all zeros, while the TCN recalls the 10 digits after 1000 steps almost perfectly. LAMBADA, which needs long context, tells the same story.

```python
import torch, torch.nn as nn, torch.nn.functional as F
from torch.nn.utils.parametrizations import weight_norm

class CausalConv1d(nn.Conv1d):
    """Conv1d that pads (k-1)*d zeros on the left only: y[t] never sees x[t+1:]."""
    def __init__(self, c_in, c_out, k, d):
        super().__init__(c_in, c_out, k, dilation=d)
        self.left = (k - 1) * d
    def forward(self, x):
        return super().forward(F.pad(x, (self.left, 0)))

class TemporalBlock(nn.Module):
    def __init__(self, c_in, c_out, k, d, p=0.1):
        super().__init__()
        self.branch = nn.Sequential(
            weight_norm(CausalConv1d(c_in, c_out, k, d)), nn.ReLU(), nn.Dropout1d(p),
            weight_norm(CausalConv1d(c_out, c_out, k, d)), nn.ReLU(), nn.Dropout1d(p))
        self.skip = nn.Conv1d(c_in, c_out, 1) if c_in != c_out else nn.Identity()
    def forward(self, x):
        return F.relu(self.branch(x) + self.skip(x))

class TCN(nn.Module):
    def __init__(self, c_in, channels, k=3, p=0.1):
        super().__init__()
        ins = [c_in] + channels[:-1]
        self.blocks = nn.Sequential(*[TemporalBlock(a, b, k, 2 ** i, p)
                                      for i, (a, b) in enumerate(zip(ins, channels))])
        self.k, self.L = k, len(channels)
    def receptive_field(self):
        return 1 + 2 * (self.k - 1) * (2 ** self.L - 1)
    def forward(self, x):                      # x: (N, C_in, T) -> (N, C_out, T)
        return self.blocks(x)

tcn = TCN(c_in=1, channels=[32, 32, 32, 32], k=3).eval()
print(tcn.receptive_field())                   # 61

x = torch.randn(2, 1, 100)
y = tcn(x)
print(y.shape)                                 # torch.Size([2, 32, 100]), same length as the input

x2 = x.clone(); x2[..., 70] += 1.0             # perturb one time step
print(torch.allclose(tcn(x2)[..., :70], y[..., :70]))   # True: outputs before t = 70 are unchanged
```

A generic TCN in PyTorch (layout (N, C, T), as in nn.Conv1d)

> **Symmetric padding leaks the future:** `nn.Conv1d(c, c, k, dilation=d, padding="same")` pads both sides, so $y_t$ reads up to $(k-1)d/2$ future steps. The model trains beautifully (it can copy the answer) and fails at generation time, when the future does not exist yet. Pad $(k-1)d$ on the left only, or pad symmetrically and drop the last $(k-1)d$ outputs (the "chomp" in the reference implementation). Test causality as in the code: perturb $x_t$ and check that earlier outputs do not move.

> **Receptive field shorter than the dependency:** Nothing beyond $R-1$ steps can influence an output, however long you train. Compute $R$ before training and make it exceed the longest dependency you care about (sequence length for classification tasks such as sequential MNIST, $T=784$). Adding one block doubles the reach at the cost of two convolutions; increasing $k$ scales it linearly.

> **Effective history is not the receptive field:** Inside the field the inputs are not equally reachable. In the default figure the input 30 steps back has 269 paths to the output, while the oldest one has a single path. Bai et al. measure "effective memory" empirically (copy memory, LAMBADA) for this reason: the edge of the field contributes weakly, much like the effective receptive field of a CNN.

> **Where TCNs sit today:** TCN-style causal dilated stacks are common in forecasting, audio, keyword spotting and sensor data, and as encoders in speech systems. For long-range language and sequence benchmarks, attention (see **Self-Attention**) and **State space models (SSMs)** such as S4 and **Mamba** took over: they reach the whole sequence without a fixed receptive field, and SSMs can also be run as a convolution with a kernel as long as the sequence.

**Strengths**

- Fully parallel over time in training: one convolution per layer for the whole sequence
- Receptive field $1+2(k-1)(2^L-1)$ grows exponentially with depth at linear parameter cost
- Gradient path length set by depth, not by the time lag: no vanishing through time
- Simple and robust: ReLU, weight norm, dropout, residuals; few hyperparameters
- Lower training memory than gated RNNs on long sequences (no per-step gate activations)

**Limitations**

- Hard memory limit: nothing older than $R-1$ steps can matter
- Inference needs the last $R$ inputs (or per-layer caches), not a single fixed-size state
- Changing the required history length means changing k or depth, i.e. the architecture
- Lost to tuned LSTMs on word-level PTB and later to Transformers and SSMs on long-context tasks

**Question:** A TCN has 5 residual blocks with two causal convolutions each, kernel size 3 and dilations 1, 2, 4, 8, 16. What is its receptive field?

- A. 21
- B. 63
- C. 125
- D. 253

**Answer:** C. 125 — $R=1+2(k-1)(2^L-1)=1+2\cdot2\cdot31=125$. With one convolution per block it would be 63, and without dilation $1+10\cdot2=21$.

**Question:** Why can a TCN be trained on all T positions of a sequence in parallel, while an RNN cannot?

- A. Each TCN output is a fixed function of a window of inputs, so all outputs are one convolution; an RNN output needs the previous hidden state first
- B. TCNs have fewer parameters
- C. TCNs use teacher forcing and RNNs do not
- D. The residual connections remove the dependence between layers

**Answer:** A. Each TCN output is a fixed function of a window of inputs, so all outputs are one convolution; an RNN output needs the previous hidden state first — Causal convolution computes $y_t$ from $x_{t-R+1..t}$ directly, with no dependence on $y_{t-1}$. An RNN's $h_t=f(h_{t-1},x_t)$ forces a loop over time (TBPTT, see **Truncated backpropagation through time (TBPTT)**). Both use teacher forcing when used as language models.

**Key points**

- TCN = causal 1-D convolutions (left padding $(k-1)d$) + dilation $d=2^i$ + residual blocks of two convs (weight norm, ReLU, spatial dropout).
- Receptive field $R=1+2(k-1)(2^L-1)$: $k=3$, $L=4$ gives 61 versus 17 without dilation, with the same 24 weights per channel pair.
- Parallel training and gradient paths of length O(depth) avoid the sequential cost and the vanishing gradients of RNNs.
- Bai, Kolter & Koltun (2018): the generic TCN beats generic LSTM/GRU/RNN on most of their benchmarks (99.0% seq. MNIST, copy memory solved), but not on word-level PTB.
- Costs: a hard memory limit at R steps and a raw-history buffer at inference instead of a fixed-size state.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
