1D, 2D and 3D CNNs

The same sliding dot product over 1, 2 or 3 axes: Conv1d for audio, sensors and text (N, C, L), Conv2d for images (N, C, H, W), Conv3d for video and medical volumes (N, C, T, H, W). Each extra axis multiplies the weights by k and the cost by k times that axis's length.

11 min read1156 wordscore0 viewsupdated 2026-10-11

Definition

1D, 2D and 3D convolutions are the same operation applied over a different number of spatial axes: a kernel of size kDk^D slides along DD axes of the input and takes a dot product with every input channel at each position. The dimensionality names the axes the kernel moves along, not the rank of the tensor: a colour image is 3-D data, but it is processed by a 2-D convolution because the kernel only slides over height and width.

Choose the dimensionality by asking along which axes the data is ordered and roughly translation-invariant. An audio waveform or a sensor trace has one such axis, time. An image has two, and a video or a CT scan has three. Every other axis is a channel: the 3 colours of a pixel, the 12 leads of an ECG, the 300 dimensions of a word embedding. The kernel always covers all channels at once. It slides only along the ordered axes.

All three forms are old. Waibel et al.'s time-delay neural network (1989) was a 1-D convolutional network for phoneme recognition, and LeCun et al.'s networks for digits (1989, 1998) made 2-D convolution standard for images. Ji, Xu, Yang and Yu introduced 3-D CNNs for human action recognition (ICML 2010, TPAMI 2013). Tran et al.'s C3D (2015) then showed that a plain VGG-style network of 3×3×33\times3\times3 kernels learns good generic video features.

One formula, D axes

yo[p]=bo+∑c=1Cin ∑u∈{0,…,k−1}DWo,c[u]  xc[ sp+d u−q ],#params=Cout Cin kD+Couty_{o}[\mathbf p]=b_o+\sum_{c=1}^{C_{in}}\ \sum_{\mathbf u\in\{0,\dots,k-1\}^D} W_{o,c}[\mathbf u]\;x_c[\,s\mathbf p+d\,\mathbf u-\mathbf q\,],\qquad \#\text{params}=C_{out}\,C_{in}\,k^D+C_{out}
p\mathbf p
output position: tt for 1-D, (i,j)(i,j) for 2-D, (t,i,j)(t,i,j) for 3-D
Wo,cW_{o,c}
a kDk^D kernel slice; the weight tensor has shape (Cout,Cin,k,…,k)(C_{out},C_{in},k,\dots,k) with DD trailing kk's
s, d, qs,\ d,\ \mathbf q
stride, dilation and padding offset, set per axis (a 3-D kernel can be 3×7×73\times7\times7)
kDk^D
each extra axis multiplies the weights by kk, and the multiply-adds by kk times that axis's output length

The backward pass is the same in every dimension. The gradient for Wo,cW_{o,c} correlates the output error δo\delta_o with input channel cc over all positions and the batch, and the gradient for xx is a transposed convolution of δ\delta with the same weights. Frameworks implement all three with the same kernels, and nn.Conv1d, nn.Conv2d and nn.Conv3d take the same arguments.

PropertyConv1dConv2dConv3d
Input layout(N, C, L)(N, C, H, W)(N, C, T, H, W) or (N, C, D, H, W)
Weight shape(128, 64, 3)(128, 64, 3, 3)(128, 64, 3, 3, 3)
Parameters24,70473,856221,312
Typical input1 s of 16 kHz audio, L = 16,000ImageNet crop, 224 × 224C3D clip, 16 × 112 × 112
Input values (C = 64)1.02M3.21M12.85M
Multiply-adds393.2M3.70G44.39G
Typical dataaudio, ECG / sensors, text tokens, genomesimages, spectrogramsvideo, CT / MRI volumes, point clouds as voxels
Layouts and costs of a layer with 64 → 128 channels, k = 3, stride 1, same padding (counted by the figure's lib)
1-D convolution · signal (audio, time series, text) · click to probeinput x[t], one channel, L = 40output y[t] (zero padding, same length; dilation 1)y[13] = −1.00·x[12] + 0.00·x[13] + 1.00·x[14] = −1.15a real layer: nn.Conv1d(64, 128, kernel_size=3, padding=1)input layout (N, C, L) · weight (128, 64, 3)params 128·64·3 + 128 = 24,704on 1 s of 16 kHz audio (L = 16000):393.2M multiply-adds per example · input 1.0M numbersConv1d: 24,704 paramsConv2d: 73,856 paramsConv3d: 221,312 params
kernel
dilationcentred window
fig.The same 3-tap-per-axis cross-correlation on a 40-sample signal, a 16×16 image and a 6-frame clip of 10×10 frames. Click to see which inputs feed one output. In 1-D the edge kernel gives y[13] = x[14] − x[12] = −1.145 at the falling edge of the pulse; switch on causal and the same value appears one step later, at t = 14, because the window now ends at t; dilation 4 spreads the three taps over 9 samples. In 3-D the motion kernel (next frame minus previous frame, each averaged over 3×3) is exactly 0 on the static square in the inner frames and ±0.67 at the leading and trailing edges of the moving one. The readout counts a real 64 → 128 layer: 24,704, 73,856 and 221,312 parameters, and 0.39, 3.70 and 44.4 billion multiply-adds on typical inputs.

Conv1d: sequences

A 1-D CNN slides along time (or position in a sequence). For raw audio the input is (N,1,L)(N,1,L), for a 12-lead ECG (N,12,L)(N,12,L). For text, the embedding dimension becomes the channel axis: Kim (2014) ran filters of widths 3, 4 and 5 over word2vec embeddings, took the max over time of each feature map, and got strong sentence classifiers with a single conv layer. In each case the network is translation-equivariant in time: a pattern is detected wherever it occurs.

Two variants matter for sequences. A causal convolution only reads the present and the past, y[t]=∑uwu x[t−u d]y[t]=\sum_u w_u\,x[t-u\,d], so it can be used for prediction and generation. A dilated convolution spaces its taps dd apart. WaveNet (van den Oord et al., 2016) stacks causal convolutions with kernel 2 and dilations 1,2,4,…,5121,2,4,\dots,512, so ten layers cover 1024 samples, and repeats that stack several times. Temporal convolutional networks (Bai, Kolter and Koltun, 2018) use the same recipe with residual blocks and match or beat recurrent networks on many sequence benchmarks.

Worked example on the figure's signal (edge kernel w = (−1, 0, 1))

  1. 1Centred, dilation 1: y[13]=−x[12]+0⋅x[13]+x[14]=−1.104+(−0.041)=−1.145y[13]=-x[12]+0\cdot x[13]+x[14]=-1.104+(-0.041)=-1.145. The pulse ends between samples 13 and 14, so the output is strongly negative there.
  2. 2Causal, dilation 1: the window for output tt is x[t−2],x[t−1],x[t]x[t-2],x[t-1],x[t]. The same three samples now produce y[14]=−1.145y[14]=-1.145: a causal conv is a centred one delayed by (k−1)/2⋅d=1(k-1)/2\cdot d=1 step.
  3. 3Causal, dilation 4: output 30 reads x[22],x[26],x[30]x[22],x[26],x[30], a span of 2⋅4+1=92\cdot4+1=9 samples with only three weights.
  4. 4Length bookkeeping, Lout=⌊(L+2p−d(k−1)−1)/s⌋+1L_{out}=\lfloor(L+2p-d(k-1)-1)/s\rfloor+1: on 16,000 samples, k=3k=3 with no padding gives 15,998, p=1p=1 keeps 16,000, and stride 2 gives 8,000. A kernel-2 layer with dilation 512 and no padding gives 15,488.

Conv2d: images and image-like data

The 2-D case is the subject of most of this module. The kernel slides over (H,W)(H,W), and colour or feature maps are channels. Data that is not a photograph also uses Conv2d when it has two ordered axes. Speech models convolve log-mel spectrograms (frequency × time), and board-game networks treat the board as an image with one channel per piece type.

Conv3d: video and volumes

A 3-D kernel slides over time (or depth) as well as height and width, so a single filter can respond to motion: a change between frames at a given place. In the figure, the motion kernel subtracts the previous frame from the next one. It gives exactly 0 on the static square in the inner frames and responds only at the leading and trailing edges of the moving square. A 2-D network applied frame by frame (the spatial kernel) sees both squares the same way and has to infer motion later from features alone.

Three designs shaped video CNNs. C3D (Tran et al., 2015) used 3×3×33\times3\times3 kernels everywhere on 16-frame 112×112112\times112 clips and found that kernel to be the best of the depths it tried. I3D (Carreira and Zisserman, 2017) inflated a 2-D ImageNet Inception network: every N×NN\times N filter becomes N×N×NN\times N\times N by repeating the 2-D weights along time and dividing by the number of repeats. A video made of one repeated frame then produces the same response as the image did. R(2+1)D (Tran et al., 2018) factorises each t×d×dt\times d\times d conv into a 1×d×d1\times d\times d spatial conv and a t×1×1t\times1\times1 temporal conv, with an extra ReLU in between.

M=⌊t d2 CinCoutd2Cin+t Cout⌋,d2CinM+t M Cout⏟(2+1)D≈t d2CinCout⏟full 3DM=\left\lfloor\frac{t\,d^2\,C_{in}C_{out}}{d^2C_{in}+t\,C_{out}}\right\rfloor,\qquad \underbrace{d^2C_{in}M+t\,M\,C_{out}}_{(2+1)\mathrm D}\approx \underbrace{t\,d^2C_{in}C_{out}}_{\text{full 3D}}
MM
width of the intermediate (spatial) output, chosen so both versions have about the same number of weights
t, dt,\ d
temporal and spatial kernel size
(2+1)D(2+1)\mathrm D
for 64→12864\to128, t=d=3t=d=3: M=230M=230 and 220,800220{,}800 weights versus 221,184221{,}184; for 64→6464\to64, M=144M=144 and both have 110,592110{,}592

At equal parameter count, the factorised network has twice as many nonlinearities. Tran et al. also reported that it was easier to optimise, with lower training error than the full 3-D network of the same size. In medical imaging, the third axis is depth rather than time: 3D U-Net (Çiçek et al., 2016) and V-Net (Milletari et al., 2016) segment CT and MRI volumes of shape (N,1,D,H,W)(N,1,D,H,W) with 3×3×33\times3\times3 kernels.

python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3for x, conv in [
4 (torch.randn(2, 64, 16000), nn.Conv1d(64, 128, 3, padding=1)), # (N, C, L)
5 (torch.randn(2, 64, 224, 224), nn.Conv2d(64, 128, 3, padding=1)), # (N, C, H, W)
6 (torch.randn(2, 64, 16, 112, 112), nn.Conv3d(64, 128, 3, padding=1)), # (N, C, T, H, W)
7]:
8 n = sum(p.numel() for p in conv.parameters())
9 print(tuple(conv(x).shape), tuple(conv.weight.shape), n)
10# (2, 128, 16000) (128, 64, 3) 24704
11# (2, 128, 224, 224) (128, 64, 3, 3) 73856
12# (2, 128, 16, 112, 112) (128, 64, 3, 3, 3) 221312
13
14class CausalConv1d(nn.Conv1d):
15 """Output t sees only x[t], x[t-d], ..., x[t-(k-1)d]: pad on the left only."""
16 def __init__(self, cin, cout, k, dilation=1):
17 super().__init__(cin, cout, k, dilation=dilation)
18 self.left = (k - 1) * dilation
19 def forward(self, x):
20 return super().forward(F.pad(x, (self.left, 0)))
21
22wavenet_like = nn.Sequential(*[CausalConv1d(32, 32, 2, dilation=2**i) for i in range(10)])
23print(wavenet_like(torch.randn(1, 32, 4000)).shape) # (1, 32, 4000); receptive field 1024
24
25def conv2plus1d(cin, cout, t=3, d=3):
26 m = (t * d * d * cin * cout) // (d * d * cin + t * cout) # 230 for 64 -> 128
27 return nn.Sequential(
28 nn.Conv3d(cin, m, (1, d, d), padding=(0, d // 2, d // 2), bias=False), # space
29 nn.BatchNorm3d(m), nn.ReLU(inplace=True),
30 nn.Conv3d(m, cout, (t, 1, 1), padding=(t // 2, 0, 0), bias=False), # time
31 )
32
33# I3D inflation: a 2-D kernel repeated over time and divided by t
34w2d = torch.randn(64, 3, 7, 7)
35w3d = w2d.unsqueeze(2).repeat(1, 1, 7, 1, 1) / 7 # (64, 3, 7, 7, 7)
36
37# Text: embeddings are (N, L, E); Conv1d wants (N, E, L)
38emb = nn.Embedding(10000, 300)(torch.randint(0, 10000, (8, 50))) # (8, 50, 300)
39feat = nn.Conv1d(300, 100, 3)(emb.transpose(1, 2)).amax(dim=2) # max over time: (8, 100)
40
41# Pretrained video backbones in torchvision
42from torchvision.models.video import r2plus1d_18, R2Plus1D_18_Weights
43video_net = r2plus1d_18(weights=R2Plus1D_18_Weights.KINETICS400_V1) # input (N, 3, T, H, W)
Conv1d / 2d / 3d side by side, a causal dilated Conv1d, a (2+1)D block and I3D-style inflation

Use when

  • +One operation and one API for sequences, images and volumes
  • +Conv1d is fast and parallel over time (unlike RNNs), and causal + dilated stacks reach long contexts cheaply
  • +Conv3d models motion and volumetric structure directly
  • +2-D weights can be inflated to initialise 3-D networks (I3D)

Avoid when

  • −Parameters grow as kDk^D and compute as kDk^D times the input volume: Conv3d is expensive
  • −3-D activations make memory the bottleneck for video and medical volumes
  • −Fixed receptive field: long-range dependencies need depth, dilation, pooling or attention
  • −Assumes translation invariance along every convolved axis, which is wrong for e.g. the frequency axis of some audio tasks

Check yourself

A video batch has shape (4, 3, 16, 112, 112). Which layer applies a 3×3 spatial and 3-frame temporal kernel to it, and how many weights does a 3 → 64 version have (no bias)?

Check yourself

A causal Conv1d stack uses kernel size 2 and dilations 1, 2, 4, …, 512 (ten layers). How many past samples, including the current one, can the last output see?

Remember

  • ✓The dimensionality is the number of axes the kernel slides along; everything else (colours, embeddings, sensors) is channels.
  • ✓Layouts: Conv1d (N,C,L)(N,C,L), Conv2d (N,C,H,W)(N,C,H,W), Conv3d (N,C,T,H,W)(N,C,T,H,W); weights (Cout,Cin,kD)(C_{out},C_{in},k^D), params CoutCinkD+CoutC_{out}C_{in}k^D+C_{out}.
  • ✓Conv1d for audio, time series and text; causal and dilated variants (WaveNet, TCN) give long contexts for generation and forecasting.
  • ✓Conv3d for video and medical volumes: C3D (3×3×3), I3D (inflated 2-D weights), R(2+1)D (spatial then temporal conv).
  • ✓Each extra axis multiplies weights by kk and compute by kk times its length: a 64→12864\to128 layer costs 0.39, 3.70 and 44.4 GMACs on typical 1-D, 2-D and 3-D inputs.
← back to Deep Learning roadmap