Definition
1D, 2D and 3D convolutions are the same operation applied over a different number of spatial axes: a kernel of size slides along axes of the input and takes a dot product with every input channel at each position. The dimensionality names the axes the kernel moves along, not the rank of the tensor: a colour image is 3-D data, but it is processed by a 2-D convolution because the kernel only slides over height and width.
Choose the dimensionality by asking along which axes the data is ordered and roughly translation-invariant. An audio waveform or a sensor trace has one such axis, time. An image has two, and a video or a CT scan has three. Every other axis is a channel: the 3 colours of a pixel, the 12 leads of an ECG, the 300 dimensions of a word embedding. The kernel always covers all channels at once. It slides only along the ordered axes.
All three forms are old. Waibel et al.'s time-delay neural network (1989) was a 1-D convolutional network for phoneme recognition, and LeCun et al.'s networks for digits (1989, 1998) made 2-D convolution standard for images. Ji, Xu, Yang and Yu introduced 3-D CNNs for human action recognition (ICML 2010, TPAMI 2013). Tran et al.'s C3D (2015) then showed that a plain VGG-style network of kernels learns good generic video features.
One formula, D axes
- output position: for 1-D, for 2-D, for 3-D
- a kernel slice; the weight tensor has shape with trailing 's
- stride, dilation and padding offset, set per axis (a 3-D kernel can be )
- each extra axis multiplies the weights by , and the multiply-adds by times that axis's output length
The backward pass is the same in every dimension. The gradient for correlates the output error with input channel over all positions and the batch, and the gradient for is a transposed convolution of with the same weights. Frameworks implement all three with the same kernels, and nn.Conv1d, nn.Conv2d and nn.Conv3d take the same arguments.
| Property | Conv1d | Conv2d | Conv3d |
|---|---|---|---|
| Input layout | (N, C, L) | (N, C, H, W) | (N, C, T, H, W) or (N, C, D, H, W) |
| Weight shape | (128, 64, 3) | (128, 64, 3, 3) | (128, 64, 3, 3, 3) |
| Parameters | 24,704 | 73,856 | 221,312 |
| Typical input | 1 s of 16 kHz audio, L = 16,000 | ImageNet crop, 224 × 224 | C3D clip, 16 × 112 × 112 |
| Input values (C = 64) | 1.02M | 3.21M | 12.85M |
| Multiply-adds | 393.2M | 3.70G | 44.39G |
| Typical data | audio, ECG / sensors, text tokens, genomes | images, spectrograms | video, CT / MRI volumes, point clouds as voxels |
Conv1d: sequences
A 1-D CNN slides along time (or position in a sequence). For raw audio the input is , for a 12-lead ECG . For text, the embedding dimension becomes the channel axis: Kim (2014) ran filters of widths 3, 4 and 5 over word2vec embeddings, took the max over time of each feature map, and got strong sentence classifiers with a single conv layer. In each case the network is translation-equivariant in time: a pattern is detected wherever it occurs.
Two variants matter for sequences. A causal convolution only reads the present and the past, , so it can be used for prediction and generation. A dilated convolution spaces its taps apart. WaveNet (van den Oord et al., 2016) stacks causal convolutions with kernel 2 and dilations , so ten layers cover 1024 samples, and repeats that stack several times. Temporal convolutional networks (Bai, Kolter and Koltun, 2018) use the same recipe with residual blocks and match or beat recurrent networks on many sequence benchmarks.
Worked example on the figure's signal (edge kernel w = (−1, 0, 1))
- 1Centred, dilation 1: . The pulse ends between samples 13 and 14, so the output is strongly negative there.
- 2Causal, dilation 1: the window for output is . The same three samples now produce : a causal conv is a centred one delayed by step.
- 3Causal, dilation 4: output 30 reads , a span of samples with only three weights.
- 4Length bookkeeping, : on 16,000 samples, with no padding gives 15,998, keeps 16,000, and stride 2 gives 8,000. A kernel-2 layer with dilation 512 and no padding gives 15,488.
Conv2d: images and image-like data
The 2-D case is the subject of most of this module. The kernel slides over , and colour or feature maps are channels. Data that is not a photograph also uses Conv2d when it has two ordered axes. Speech models convolve log-mel spectrograms (frequency × time), and board-game networks treat the board as an image with one channel per piece type.
Conv3d: video and volumes
A 3-D kernel slides over time (or depth) as well as height and width, so a single filter can respond to motion: a change between frames at a given place. In the figure, the motion kernel subtracts the previous frame from the next one. It gives exactly 0 on the static square in the inner frames and responds only at the leading and trailing edges of the moving square. A 2-D network applied frame by frame (the spatial kernel) sees both squares the same way and has to infer motion later from features alone.
Three designs shaped video CNNs. C3D (Tran et al., 2015) used kernels everywhere on 16-frame clips and found that kernel to be the best of the depths it tried. I3D (Carreira and Zisserman, 2017) inflated a 2-D ImageNet Inception network: every filter becomes by repeating the 2-D weights along time and dividing by the number of repeats. A video made of one repeated frame then produces the same response as the image did. R(2+1)D (Tran et al., 2018) factorises each conv into a spatial conv and a temporal conv, with an extra ReLU in between.
- width of the intermediate (spatial) output, chosen so both versions have about the same number of weights
- temporal and spatial kernel size
- for , : and weights versus ; for , and both have
At equal parameter count, the factorised network has twice as many nonlinearities. Tran et al. also reported that it was easier to optimise, with lower training error than the full 3-D network of the same size. In medical imaging, the third axis is depth rather than time: 3D U-Net (Çiçek et al., 2016) and V-Net (Milletari et al., 2016) segment CT and MRI volumes of shape with kernels.
1import torch, torch.nn as nn, torch.nn.functional as F23for x, conv in [4 (torch.randn(2, 64, 16000), nn.Conv1d(64, 128, 3, padding=1)), # (N, C, L)5 (torch.randn(2, 64, 224, 224), nn.Conv2d(64, 128, 3, padding=1)), # (N, C, H, W)6 (torch.randn(2, 64, 16, 112, 112), nn.Conv3d(64, 128, 3, padding=1)), # (N, C, T, H, W)7]:8 n = sum(p.numel() for p in conv.parameters())9 print(tuple(conv(x).shape), tuple(conv.weight.shape), n)10# (2, 128, 16000) (128, 64, 3) 2470411# (2, 128, 224, 224) (128, 64, 3, 3) 7385612# (2, 128, 16, 112, 112) (128, 64, 3, 3, 3) 2213121314class CausalConv1d(nn.Conv1d):15 """Output t sees only x[t], x[t-d], ..., x[t-(k-1)d]: pad on the left only."""16 def __init__(self, cin, cout, k, dilation=1):17 super().__init__(cin, cout, k, dilation=dilation)18 self.left = (k - 1) * dilation19 def forward(self, x):20 return super().forward(F.pad(x, (self.left, 0)))2122wavenet_like = nn.Sequential(*[CausalConv1d(32, 32, 2, dilation=2**i) for i in range(10)])23print(wavenet_like(torch.randn(1, 32, 4000)).shape) # (1, 32, 4000); receptive field 10242425def conv2plus1d(cin, cout, t=3, d=3):26 m = (t * d * d * cin * cout) // (d * d * cin + t * cout) # 230 for 64 -> 12827 return nn.Sequential(28 nn.Conv3d(cin, m, (1, d, d), padding=(0, d // 2, d // 2), bias=False), # space29 nn.BatchNorm3d(m), nn.ReLU(inplace=True),30 nn.Conv3d(m, cout, (t, 1, 1), padding=(t // 2, 0, 0), bias=False), # time31 )3233# I3D inflation: a 2-D kernel repeated over time and divided by t34w2d = torch.randn(64, 3, 7, 7)35w3d = w2d.unsqueeze(2).repeat(1, 1, 7, 1, 1) / 7 # (64, 3, 7, 7, 7)3637# Text: embeddings are (N, L, E); Conv1d wants (N, E, L)38emb = nn.Embedding(10000, 300)(torch.randint(0, 10000, (8, 50))) # (8, 50, 300)39feat = nn.Conv1d(300, 100, 3)(emb.transpose(1, 2)).amax(dim=2) # max over time: (8, 100)4041# Pretrained video backbones in torchvision42from torchvision.models.video import r2plus1d_18, R2Plus1D_18_Weights43video_net = r2plus1d_18(weights=R2Plus1D_18_Weights.KINETICS400_V1) # input (N, 3, T, H, W)Use when
- +One operation and one API for sequences, images and volumes
- +Conv1d is fast and parallel over time (unlike RNNs), and causal + dilated stacks reach long contexts cheaply
- +Conv3d models motion and volumetric structure directly
- +2-D weights can be inflated to initialise 3-D networks (I3D)
Avoid when
- −Parameters grow as and compute as times the input volume: Conv3d is expensive
- −3-D activations make memory the bottleneck for video and medical volumes
- −Fixed receptive field: long-range dependencies need depth, dilation, pooling or attention
- −Assumes translation invariance along every convolved axis, which is wrong for e.g. the frequency axis of some audio tasks
Check yourself
A video batch has shape (4, 3, 16, 112, 112). Which layer applies a 3×3 spatial and 3-frame temporal kernel to it, and how many weights does a 3 → 64 version have (no bias)?
Check yourself
A causal Conv1d stack uses kernel size 2 and dilations 1, 2, 4, …, 512 (ten layers). How many past samples, including the current one, can the last output see?
Remember
- ✓The dimensionality is the number of axes the kernel slides along; everything else (colours, embeddings, sensors) is channels.
- ✓Layouts: Conv1d , Conv2d , Conv3d ; weights , params .
- ✓Conv1d for audio, time series and text; causal and dilated variants (WaveNet, TCN) give long contexts for generation and forecasting.
- ✓Conv3d for video and medical volumes: C3D (3×3×3), I3D (inflated 2-D weights), R(2+1)D (spatial then temporal conv).
- ✓Each extra axis multiplies weights by and compute by times its length: a layer costs 0.39, 3.70 and 44.4 GMACs on typical 1-D, 2-D and 3-D inputs.