Video classification

Video classification gives one label per video. Models differ in where they fuse time (late fusion, optical-flow streams, 3-D CNNs such as I3D, divided space-time attention), and test scores are averaged over sampled clips.

8 min read876 wordscore0 viewsupdated 2026-10-15

Definition

Video classification maps a video V∈RT×H×W×3V\in\mathbb R^{T\times H\times W\times 3} to one of KK classes. The main design question is where frames are fused. Karpathy et al. ("Large-scale Video Classification with Convolutional Neural Networks", CVPR 2014) compared late, early and slow fusion. Simonyan and Zisserman (NeurIPS 2014) added a second stream that reads optical flow. Carreira and Zisserman's I3D (CVPR 2017) inflated ImageNet CNNs to 3-D and introduced the Kinetics-400 benchmark.

A video is a stack of images, so the obvious baseline is to run an image classifier (see Image classification) on each frame and average the scores. That works surprisingly well, because the scene and the objects often give the class away: a pool means swimming. It fails when the label is in the motion itself, such as opening versus closing a door. It also wastes compute, since neighbouring frames are nearly identical.

Cost is the other constraint. A 10-second clip at 30 fps has 300 frames. Models therefore read short clips, typically 8–64 frames sampled with a stride. At test time they score several clips and crops and average the predictions:

p(y∣V)=1Kc∑k=1Kcsoftmax⁡ ⁣(fθ(ck))y,ck=(xtk, xtk+s, …, xtk+(T−1)s)p(y\mid V)=\frac{1}{K_c}\sum_{k=1}^{K_c}\operatorname{softmax}\!\big(f_\theta(c_k)\big)_y,\qquad c_k=\big(x_{t_k},\,x_{t_k+s},\,\dots,\,x_{t_k+(T-1)s}\big)
ckc_k
clip kk: TT frames with stride ss starting at tkt_k; it spans (T−1)s+1(T-1)s+1 frames
KcK_c
number of test views, e.g. 10 temporal clips × 3 spatial crops = 30
fθf_\theta
the clip model; training uses one random clip per video per step

Where to fuse time

Karpathy et al. trained on Sports-1M, 1 million YouTube videos in 487 sports classes. Late fusion runs two single-frame towers on frames 15 apart and merges them at the first fully connected layer. Early fusion extends the first convolution over T=10T=10 frames, so its filters are 11×11×3×1011\times11\times3\times10. Slow fusion widens the temporal extent layer by layer. Slow fusion was best, but the single-frame model was only slightly worse. On that data, appearance carried most of the signal.

one label per video: sample clips, score each, average the softmax0 s2 s4 s6 s8 s10 sclip 1clip 2clip 38 frames, stride 8: 57 frames = 1.9 s per clipf(clip)mean softmaxtest: K clips × 3 cropslate fusionKarpathy et al., CVPR 2014two frames 15 apart, sharedtowers; time meets at the FCtt+152-D CNN2-D CNNsharedFCyearly / slow fusionKarpathy et al., CVPR 2014first filter spans T = 10frames; slow fusion: gradualT = 10conv111×11×3×10CNNtwo-streamSimonyan & Zisserman, 2014flow of L = 10 frames, x and ylate fusion of class scoresRGB ×3flow ×20spatialtemporalavg3-D CNNC3D 2015 · I3D 2017C3D: 3×3×3 kernels throughoutI3D: inflate k×k → k×k×k, w/k3×16×112×1128 conv3×3×3fc64096(2+1)DTran et al., CVPR 2018split each t×d×d conv intospatial then temporal, + ReLU1×3×3spaceσ3×1×1timeM mid channels: same # paramsdivided attentionTimeSformer, ICML 20218 × 196 tokens (16² patches)keys/query: 8 + 196, not 1568■ query■ time■ spacetime attn, then space attn
fig.Top: the test pipeline. Three clips of 8 frames at stride 8 (57 frames, 1.9 s each) are taken from a 10 s video, scored by the same model and averaged. Below: six places to fuse time. Late fusion merges two frames at the FC layer. Early fusion stacks 10 frames into conv1. Two-stream adds a flow ConvNet. C3D uses 3×3×3 kernels throughout, and I3D inflates 2-D filters. (2+1)D splits each 3-D kernel into space then time. TimeSformer's divided attention gives each patch 8 temporal and 196 spatial keys.

The two-stream network (Simonyan and Zisserman, "Two-Stream Convolutional Networks for Action Recognition in Videos") gives motion its own input. A spatial ConvNet sees one RGB frame. A temporal ConvNet sees the horizontal and vertical optical flow of L=10L=10 consecutive frames stacked as 2L=202L=20 channels. The two softmax outputs are fused by averaging or by a linear SVM. On UCF101 the flow stream alone (83.7%) beat the RGB stream (73.0%), and fusion reached 88.0%.

3-D convolutions and inflation

A 3-D convolution slides a t×d×dt\times d\times d kernel over time as well as space, so the network learns motion features end to end without precomputed flow (see 1D, 2D and 3D CNNs for the operation and its cost). C3D (Tran et al., ICCV 2015) is a VGG-style network with eight 3×3×33\times3\times3 conv layers on 16-frame 112×112112\times112 clips. Its fc6 activations (4096-d) became a generic video feature. Trained from scratch, though, 3-D networks lagged behind ImageNet-pretrained 2-D ones.

Wτ,i,j3D=1k Wi,j2D,τ=1,…,k⟹W3D∗[x,x,…,x]⏟"boring" video=W2D∗xW^{3\mathrm D}_{\tau,i,j}=\tfrac{1}{k}\,W^{2\mathrm D}_{i,j},\quad \tau=1,\dots,k\qquad\Longrightarrow\qquad W^{3\mathrm D}*\underbrace{[x,x,\dots,x]}_{\text{"boring" video}}=W^{2\mathrm D}*x
W2DW^{2\mathrm D}
an ImageNet-trained 2-D filter (I3D inflates Inception-v1, i.e. GoogLeNet with batch norm)
kk
temporal kernel size; the 1/k1/k rescaling keeps activations unchanged on a video made of one repeated frame

I3D used this inflation to start from ImageNet weights, then trained on Kinetics. Two I3D networks, one on RGB and one on flow, still beat either alone. (2+1)D (Tran et al., "A Closer Look at Spatiotemporal Convolutions for Action Recognition", CVPR 2018) factorises each t×d×dt\times d\times d kernel into a 1×d×d1\times d\times d spatial and a t×1×1t\times1\times1 temporal convolution. The intermediate width is chosen to match the parameter count, so the gain comes from the extra ReLU and easier optimisation.

Video transformers

A Vision Transformer (ViT) extends naturally to video: cut every frame into 16×1616\times16 patches and attend over all of them. With 8 frames of 2242224^2 that is 8×196=15688\times196=1568 tokens, and joint space-time attention is quadratic in that number. TimeSformer (Bertasius, Wang and Torresani, "Is Space-Time Attention All You Need for Video Understanding?", ICML 2021) uses divided attention: in each block, a patch first attends to the patches at the same location in the other frames, then to the patches of its own frame.

(NT)2⏟jointvsNT (T+N)⏟dividedquery–key pairs per layer,N=HWP2\underbrace{(NT)^2}_{\text{joint}}\quad\text{vs}\quad\underbrace{NT\,(T+N)}_{\text{divided}}\quad\text{query–key pairs per layer},\qquad N=\frac{HW}{P^2}
NN
patches per frame: 196 for 2242224^2 input and P=16P=16
TT
frames in the clip; the ratio of the two costs is NT / (N + T)

ViViT (Arnab et al., ICCV 2021) makes the same trade-off in other ways. Its tubelet embedding extends the patch over a few frames, and its factorised encoder runs a spatial transformer per frame and then a temporal transformer over the frame embeddings.

Datasets and reported results

Kinetics-400
Kay et al. 2017: 400 human-action classes, at least 400 YouTube clips per class, each about 10 s. Clips disappear from YouTube over time, so exact test sets differ between papers.
UCF101 / HMDB51
13,320 clips in 101 classes / about 6.8k clips in 51 classes, each evaluated as the mean over 3 train/test splits. Now used mainly for transfer.
Two-stream (2014)
UCF101 88.0%, HMDB51 59.4% (SVM fusion, no large-scale video pre-training)
Two-stream I3D (2017)
Kinetics-400 test top-1 74.2% (ImageNet init); after Kinetics pre-training, UCF101 98.0% and HMDB51 80.7%
TimeSformer (2021)
8 × 224² input, ImageNet-21K pre-training: 78.0% top-1 on Kinetics-400, RGB only
dot moves +1 px/frame · T = 8 frames, stride 2 · d = v·s = +2 px per samplesampled clip: x = 0…19, strip wrapsf0f2f4f6f8f10f12f143-D conv: 2 × 5 kernel per δ, ReLUδ = −20.000δ = −10.000δ = 00.034δ = +10.721δ = +21.000→ moving right (δ = +2)late fusion: average the per-frame features over timecliprevmax |clip − reversed| = 0.000same input to the classifier3-D conv: this clip → moving right; time-reversed → moving leftkernel width 5 reaches ±2 px; the clip shows +2 px per samplethe clip spans (T − 1)·s + 1 = 15 frames = 0.50 s at 30 fpsthe kernel matching the displacement fires on every frame pair
kernel width
fig.A dot on a 20-pixel strip that wraps around, sampled as 8 frames. Each 3-D kernel (2 frames × k pixels, bias −1, ReLU) detects one displacement δ. Default: speed +1, stride 2, so the dot moves d = +2 px per sample. The δ = +2 kernel scores 1.000 (δ = +1 gets 0.721) and the clip is classified "moving right". Played in reverse it becomes "moving left". The late-fusion feature, a time average, differs between the two by 0.000, so no classifier on top of it can tell them apart. At stride 4 (d = 4) the best width-5 kernel scores only 0.034, so nothing fires. Width 7 recovers "right" at 0.721. Speed +3 with stride 6 gives d = 18, which looks like −2 on the 20-pixel strip, so the clip reads "moving left" (temporal aliasing).

Worked example: attention cost per layer

AttentionKeys per query8 frames (1,568 tokens)16 frames (3,136 tokens)
Space only (per frame)N = 196307,328614,656
Divided: time + spaceT + N319,872664,832
Joint space-timeNT2,458,6249,834,496
Joint / divided7.69×14.79×
Query–key pairs per attention layer for 224² frames, 16² patches (196 per frame), ignoring the class token. Computed in Python.

Divided attention costs barely more than attention within single frames, and it scales linearly in TT for fixed NN. Joint attention grows as T2T^2: doubling the clip from 8 to 16 frames multiplies its cost by 4, while divided attention grows by about 2.1×.

python
1import numpy as np, torch
2from decord import VideoReader # any decoder that returns frames works
3from transformers import AutoImageProcessor, TimesformerForVideoClassification
4
5ckpt = "facebook/timesformer-base-finetuned-k400"
6proc = AutoImageProcessor.from_pretrained(ckpt)
7model = TimesformerForVideoClassification.from_pretrained(ckpt).eval()
8
9vr = VideoReader("video.mp4")
10T, stride, n_clips = 8, 8, 5
11span = (T - 1) * stride + 1 # 57 frames per clip
12starts = np.linspace(0, max(len(vr) - span, 0), n_clips).astype(int)
13
14probs = []
15for t0 in starts:
16 idx = np.minimum(t0 + stride * np.arange(T), len(vr) - 1)
17 frames = list(vr.get_batch(idx).asnumpy()) # T x H x W x 3, uint8
18 inputs = proc(frames, return_tensors="pt") # resize, crop, normalise -> (1, T, 3, 224, 224)
19 with torch.no_grad():
20 probs.append(model(**inputs).logits.softmax(-1))
21p = torch.cat(probs).mean(0) # average the clip-level softmax
22print(model.config.id2label[int(p.argmax())], float(p.max()))
Multi-clip inference with a Kinetics-400 TimeSformer from Hugging Face (not run here)
ApproachHow time entersMain costWeakness
Frame CNN + averagingnot at all (order-invariant)T × image CNNblind to direction and order
Two-streamoptical flow inputcomputing optical flow offlinetwo networks, flow is slow
3-D CNN (C3D, I3D)t × d × d kernelsabout t× the 2-D FLOPs per layershort temporal reach per layer
(2+1)Dseparate temporal conv≈ same as 3-Dstill local in time
Video transformerattention across framestokens grow with T × Nneeds heavy pre-training
Design choices at a glance

Check yourself

A model averages per-frame CNN features over the clip and then classifies. Which pair of classes can it never distinguish?

Check yourself

I3D inflates a 3 × 3 ImageNet filter to 3 × 3 × 3 and divides it by 3. Why divide?

Remember

  • ✓Video classifiers read short sampled clips (T frames, stride s) and average the softmax over several clips and crops at test time.
  • ✓Late fusion of per-frame features ignores order. Early or slow fusion, flow streams, 3-D kernels and temporal attention can see motion.
  • ✓Two-stream: RGB frame plus a 20-channel stack of 10 flow frames; on UCF101, flow alone 83.7% beats RGB alone 73.0%, and fusion reaches 88.0%.
  • ✓I3D inflates 2-D filters as W2D/kW^{2\mathrm D}/k repeated kk times. (2+1)D splits each kernel into spatial and temporal parts.
  • ✓TimeSformer's divided attention gives each patch T + N keys instead of NT: 7.69× fewer pairs for 8 × 196 tokens.
  • ✓Kinetics-400 is the standard benchmark. Appearance bias means good accuracy does not prove motion modelling.
← back to Deep Learning roadmap