Definition
Video classification maps a video to one of classes. The main design question is where frames are fused. Karpathy et al. ("Large-scale Video Classification with Convolutional Neural Networks", CVPR 2014) compared late, early and slow fusion. Simonyan and Zisserman (NeurIPS 2014) added a second stream that reads optical flow. Carreira and Zisserman's I3D (CVPR 2017) inflated ImageNet CNNs to 3-D and introduced the Kinetics-400 benchmark.
A video is a stack of images, so the obvious baseline is to run an image classifier (see Image classification) on each frame and average the scores. That works surprisingly well, because the scene and the objects often give the class away: a pool means swimming. It fails when the label is in the motion itself, such as opening versus closing a door. It also wastes compute, since neighbouring frames are nearly identical.
Cost is the other constraint. A 10-second clip at 30 fps has 300 frames. Models therefore read short clips, typically 8–64 frames sampled with a stride. At test time they score several clips and crops and average the predictions:
- clip : frames with stride starting at ; it spans frames
- number of test views, e.g. 10 temporal clips × 3 spatial crops = 30
- the clip model; training uses one random clip per video per step
Where to fuse time
Karpathy et al. trained on Sports-1M, 1 million YouTube videos in 487 sports classes. Late fusion runs two single-frame towers on frames 15 apart and merges them at the first fully connected layer. Early fusion extends the first convolution over frames, so its filters are . Slow fusion widens the temporal extent layer by layer. Slow fusion was best, but the single-frame model was only slightly worse. On that data, appearance carried most of the signal.
The two-stream network (Simonyan and Zisserman, "Two-Stream Convolutional Networks for Action Recognition in Videos") gives motion its own input. A spatial ConvNet sees one RGB frame. A temporal ConvNet sees the horizontal and vertical optical flow of consecutive frames stacked as channels. The two softmax outputs are fused by averaging or by a linear SVM. On UCF101 the flow stream alone (83.7%) beat the RGB stream (73.0%), and fusion reached 88.0%.
3-D convolutions and inflation
A 3-D convolution slides a kernel over time as well as space, so the network learns motion features end to end without precomputed flow (see 1D, 2D and 3D CNNs for the operation and its cost). C3D (Tran et al., ICCV 2015) is a VGG-style network with eight conv layers on 16-frame clips. Its fc6 activations (4096-d) became a generic video feature. Trained from scratch, though, 3-D networks lagged behind ImageNet-pretrained 2-D ones.
- an ImageNet-trained 2-D filter (I3D inflates Inception-v1, i.e. GoogLeNet with batch norm)
- temporal kernel size; the rescaling keeps activations unchanged on a video made of one repeated frame
I3D used this inflation to start from ImageNet weights, then trained on Kinetics. Two I3D networks, one on RGB and one on flow, still beat either alone. (2+1)D (Tran et al., "A Closer Look at Spatiotemporal Convolutions for Action Recognition", CVPR 2018) factorises each kernel into a spatial and a temporal convolution. The intermediate width is chosen to match the parameter count, so the gain comes from the extra ReLU and easier optimisation.
Video transformers
A Vision Transformer (ViT) extends naturally to video: cut every frame into patches and attend over all of them. With 8 frames of that is tokens, and joint space-time attention is quadratic in that number. TimeSformer (Bertasius, Wang and Torresani, "Is Space-Time Attention All You Need for Video Understanding?", ICML 2021) uses divided attention: in each block, a patch first attends to the patches at the same location in the other frames, then to the patches of its own frame.
- patches per frame: 196 for input and
- frames in the clip; the ratio of the two costs is NT / (N + T)
ViViT (Arnab et al., ICCV 2021) makes the same trade-off in other ways. Its tubelet embedding extends the patch over a few frames, and its factorised encoder runs a spatial transformer per frame and then a temporal transformer over the frame embeddings.
Datasets and reported results
- Kinetics-400
- Kay et al. 2017: 400 human-action classes, at least 400 YouTube clips per class, each about 10 s. Clips disappear from YouTube over time, so exact test sets differ between papers.
- UCF101 / HMDB51
- 13,320 clips in 101 classes / about 6.8k clips in 51 classes, each evaluated as the mean over 3 train/test splits. Now used mainly for transfer.
- Two-stream (2014)
- UCF101 88.0%, HMDB51 59.4% (SVM fusion, no large-scale video pre-training)
- Two-stream I3D (2017)
- Kinetics-400 test top-1 74.2% (ImageNet init); after Kinetics pre-training, UCF101 98.0% and HMDB51 80.7%
- TimeSformer (2021)
- 8 × 224² input, ImageNet-21K pre-training: 78.0% top-1 on Kinetics-400, RGB only
Worked example: attention cost per layer
| Attention | Keys per query | 8 frames (1,568 tokens) | 16 frames (3,136 tokens) |
|---|---|---|---|
| Space only (per frame) | N = 196 | 307,328 | 614,656 |
| Divided: time + space | T + N | 319,872 | 664,832 |
| Joint space-time | NT | 2,458,624 | 9,834,496 |
| Joint / divided | 7.69× | 14.79× |
Divided attention costs barely more than attention within single frames, and it scales linearly in for fixed . Joint attention grows as : doubling the clip from 8 to 16 frames multiplies its cost by 4, while divided attention grows by about 2.1×.
1import numpy as np, torch2from decord import VideoReader # any decoder that returns frames works3from transformers import AutoImageProcessor, TimesformerForVideoClassification45ckpt = "facebook/timesformer-base-finetuned-k400"6proc = AutoImageProcessor.from_pretrained(ckpt)7model = TimesformerForVideoClassification.from_pretrained(ckpt).eval()89vr = VideoReader("video.mp4")10T, stride, n_clips = 8, 8, 511span = (T - 1) * stride + 1 # 57 frames per clip12starts = np.linspace(0, max(len(vr) - span, 0), n_clips).astype(int)1314probs = []15for t0 in starts:16 idx = np.minimum(t0 + stride * np.arange(T), len(vr) - 1)17 frames = list(vr.get_batch(idx).asnumpy()) # T x H x W x 3, uint818 inputs = proc(frames, return_tensors="pt") # resize, crop, normalise -> (1, T, 3, 224, 224)19 with torch.no_grad():20 probs.append(model(**inputs).logits.softmax(-1))21p = torch.cat(probs).mean(0) # average the clip-level softmax22print(model.config.id2label[int(p.argmax())], float(p.max()))| Approach | How time enters | Main cost | Weakness |
|---|---|---|---|
| Frame CNN + averaging | not at all (order-invariant) | T × image CNN | blind to direction and order |
| Two-stream | optical flow input | computing optical flow offline | two networks, flow is slow |
| 3-D CNN (C3D, I3D) | t × d × d kernels | about t× the 2-D FLOPs per layer | short temporal reach per layer |
| (2+1)D | separate temporal conv | ≈ same as 3-D | still local in time |
| Video transformer | attention across frames | tokens grow with T × N | needs heavy pre-training |
Check yourself
A model averages per-frame CNN features over the clip and then classifies. Which pair of classes can it never distinguish?
Check yourself
I3D inflates a 3 × 3 ImageNet filter to 3 × 3 × 3 and divides it by 3. Why divide?
Remember
- ✓Video classifiers read short sampled clips (T frames, stride s) and average the softmax over several clips and crops at test time.
- ✓Late fusion of per-frame features ignores order. Early or slow fusion, flow streams, 3-D kernels and temporal attention can see motion.
- ✓Two-stream: RGB frame plus a 20-channel stack of 10 flow frames; on UCF101, flow alone 83.7% beats RGB alone 73.0%, and fusion reaches 88.0%.
- ✓I3D inflates 2-D filters as repeated times. (2+1)D splits each kernel into spatial and temporal parts.
- ✓TimeSformer's divided attention gives each patch T + N keys instead of NT: 7.69× fewer pairs for 8 × 196 tokens.
- ✓Kinetics-400 is the standard benchmark. Appearance bias means good accuracy does not prove motion modelling.