# Video classification

> Video classification gives one label per video. Models differ in where they fuse time (late fusion, optical-flow streams, 3-D CNNs such as I3D, divided space-time attention), and test scores are averaged over sampled clips.

Source: https://www.mlfoundry.in/learn/video-classification  
Section: Deep Learning › Deep Learning for Computer Vision  
Level: core · 8 min read · updated 2026-10-15

**Video classification:** maps a video $V\in\mathbb R^{T\times H\times W\times 3}$ to one of $K$ classes. The main design question is where frames are fused. Karpathy et al. ("Large-scale Video Classification with Convolutional Neural Networks", CVPR 2014) compared late, early and slow fusion. Simonyan and Zisserman (NeurIPS 2014) added a second stream that reads optical flow. Carreira and Zisserman's I3D (CVPR 2017) inflated ImageNet CNNs to 3-D and introduced the **Kinetics-400** benchmark.

A video is a stack of images, so the obvious baseline is to run an image classifier (see **Image classification**) on each frame and average the scores. That works surprisingly well, because the scene and the objects often give the class away: a pool means swimming. It fails when the label is in the motion itself, such as opening versus closing a door. It also wastes compute, since neighbouring frames are nearly identical.

Cost is the other constraint. A 10-second clip at 30 fps has 300 frames. Models therefore read short **clips**, typically 8–64 frames sampled with a stride. At test time they score several clips and crops and average the predictions:

$$
p(y\mid V)=\frac{1}{K_c}\sum_{k=1}^{K_c}\operatorname{softmax}\!\big(f_\theta(c_k)\big)_y,\qquad c_k=\big(x_{t_k},\,x_{t_k+s},\,\dots,\,x_{t_k+(T-1)s}\big)
$$

- $c_k$: clip $k$: $T$ frames with stride $s$ starting at $t_k$; it spans $(T-1)s+1$ frames
- $K_c$: number of test views, e.g. 10 temporal clips × 3 spatial crops = 30
- $f_\theta$: the clip model; training uses one random clip per video per step

## Where to fuse time

Karpathy et al. trained on **Sports-1M**, 1 million YouTube videos in 487 sports classes. **Late fusion** runs two single-frame towers on frames 15 apart and merges them at the first fully connected layer. **Early fusion** extends the first convolution over $T=10$ frames, so its filters are $11\times11\times3\times10$. **Slow fusion** widens the temporal extent layer by layer. Slow fusion was best, but the single-frame model was only slightly worse. On that data, appearance carried most of the signal.

> Figure: Top: the test pipeline. Three clips of 8 frames at stride 8 (57 frames, 1.9 s each) are taken from a 10 s video, scored by the same model and averaged. Below: six places to fuse time. Late fusion merges two frames at the FC layer. Early fusion stacks 10 frames into conv1. Two-stream adds a flow ConvNet. C3D uses 3×3×3 kernels throughout, and I3D inflates 2-D filters. (2+1)D splits each 3-D kernel into space then time. TimeSformer's divided attention gives each patch 8 temporal and 196 spatial keys.

The **two-stream** network (Simonyan and Zisserman, "Two-Stream Convolutional Networks for Action Recognition in Videos") gives motion its own input. A spatial ConvNet sees one RGB frame. A temporal ConvNet sees the horizontal and vertical optical flow of $L=10$ consecutive frames stacked as $2L=20$ channels. The two softmax outputs are fused by averaging or by a linear SVM. On UCF101 the flow stream alone (83.7%) beat the RGB stream (73.0%), and fusion reached 88.0%.

## 3-D convolutions and inflation

A 3-D convolution slides a $t\times d\times d$ kernel over time as well as space, so the network learns motion features end to end without precomputed flow (see **1D, 2D and 3D CNNs** for the operation and its cost). C3D (Tran et al., ICCV 2015) is a VGG-style network with eight $3\times3\times3$ conv layers on 16-frame $112\times112$ clips. Its fc6 activations (4096-d) became a generic video feature. Trained from scratch, though, 3-D networks lagged behind ImageNet-pretrained 2-D ones.

$$
W^{3\mathrm D}_{\tau,i,j}=\tfrac{1}{k}\,W^{2\mathrm D}_{i,j},\quad \tau=1,\dots,k\qquad\Longrightarrow\qquad W^{3\mathrm D}*\underbrace{[x,x,\dots,x]}_{\text{"boring" video}}=W^{2\mathrm D}*x
$$

- $W^{2\mathrm D}$: an ImageNet-trained 2-D filter (I3D inflates Inception-v1, i.e. GoogLeNet with batch norm)
- $k$: temporal kernel size; the $1/k$ rescaling keeps activations unchanged on a video made of one repeated frame

I3D used this inflation to start from ImageNet weights, then trained on Kinetics. Two I3D networks, one on RGB and one on flow, still beat either alone. **(2+1)D** (Tran et al., "A Closer Look at Spatiotemporal Convolutions for Action Recognition", CVPR 2018) factorises each $t\times d\times d$ kernel into a $1\times d\times d$ spatial and a $t\times1\times1$ temporal convolution. The intermediate width is chosen to match the parameter count, so the gain comes from the extra ReLU and easier optimisation.

## Video transformers

A **Vision Transformer (ViT)** extends naturally to video: cut every frame into $16\times16$ patches and attend over all of them. With 8 frames of $224^2$ that is $8\times196=1568$ tokens, and joint space-time attention is quadratic in that number. TimeSformer (Bertasius, Wang and Torresani, "Is Space-Time Attention All You Need for Video Understanding?", ICML 2021) uses **divided attention**: in each block, a patch first attends to the patches at the same location in the other frames, then to the patches of its own frame.

$$
\underbrace{(NT)^2}_{\text{joint}}\quad\text{vs}\quad\underbrace{NT\,(T+N)}_{\text{divided}}\quad\text{query–key pairs per layer},\qquad N=\frac{HW}{P^2}
$$

- $N$: patches per frame: 196 for $224^2$ input and $P=16$
- $T$: frames in the clip; the ratio of the two costs is NT / (N + T)

ViViT (Arnab et al., ICCV 2021) makes the same trade-off in other ways. Its **tubelet embedding** extends the patch over a few frames, and its factorised encoder runs a spatial transformer per frame and then a temporal transformer over the frame embeddings.

**Datasets and reported results**

- **Kinetics-400:** Kay et al. 2017: 400 human-action classes, at least 400 YouTube clips per class, each about 10 s. Clips disappear from YouTube over time, so exact test sets differ between papers.
- **UCF101 / HMDB51:** 13,320 clips in 101 classes / about 6.8k clips in 51 classes, each evaluated as the mean over 3 train/test splits. Now used mainly for transfer.
- **Two-stream (2014):** UCF101 88.0%, HMDB51 59.4% (SVM fusion, no large-scale video pre-training)
- **Two-stream I3D (2017):** Kinetics-400 test top-1 74.2% (ImageNet init); after Kinetics pre-training, UCF101 98.0% and HMDB51 80.7%
- **TimeSformer (2021):** 8 × 224² input, ImageNet-21K pre-training: 78.0% top-1 on Kinetics-400, RGB only

> Figure: A dot on a 20-pixel strip that wraps around, sampled as 8 frames. Each 3-D kernel (2 frames × k pixels, bias −1, ReLU) detects one displacement δ. Default: speed +1, stride 2, so the dot moves d = +2 px per sample. The δ = +2 kernel scores 1.000 (δ = +1 gets 0.721) and the clip is classified "moving right". Played in reverse it becomes "moving left". The late-fusion feature, a time average, differs between the two by 0.000, so no classifier on top of it can tell them apart. At stride 4 (d = 4) the best width-5 kernel scores only 0.034, so nothing fires. Width 7 recovers "right" at 0.721. Speed +3 with stride 6 gives d = 18, which looks like −2 on the 20-pixel strip, so the clip reads "moving left" (temporal aliasing).

## Worked example: attention cost per layer

Query–key pairs per attention layer for 224² frames, 16² patches (196 per frame), ignoring the class token. Computed in Python.

| Attention | Keys per query | 8 frames (1,568 tokens) | 16 frames (3,136 tokens) |
| --- | --- | --- | --- |
| Space only (per frame) | N = 196 | 307,328 | 614,656 |
| Divided: time + space | T + N | 319,872 | 664,832 |
| Joint space-time | NT | 2,458,624 | 9,834,496 |
| Joint / divided |  | 7.69× | 14.79× |

Divided attention costs barely more than attention within single frames, and it scales linearly in $T$ for fixed $N$. Joint attention grows as $T^2$: doubling the clip from 8 to 16 frames multiplies its cost by 4, while divided attention grows by about 2.1×.

```python
import numpy as np, torch
from decord import VideoReader                      # any decoder that returns frames works
from transformers import AutoImageProcessor, TimesformerForVideoClassification

ckpt = "facebook/timesformer-base-finetuned-k400"
proc = AutoImageProcessor.from_pretrained(ckpt)
model = TimesformerForVideoClassification.from_pretrained(ckpt).eval()

vr = VideoReader("video.mp4")
T, stride, n_clips = 8, 8, 5
span = (T - 1) * stride + 1                         # 57 frames per clip
starts = np.linspace(0, max(len(vr) - span, 0), n_clips).astype(int)

probs = []
for t0 in starts:
    idx = np.minimum(t0 + stride * np.arange(T), len(vr) - 1)
    frames = list(vr.get_batch(idx).asnumpy())       # T x H x W x 3, uint8
    inputs = proc(frames, return_tensors="pt")       # resize, crop, normalise -> (1, T, 3, 224, 224)
    with torch.no_grad():
        probs.append(model(**inputs).logits.softmax(-1))
p = torch.cat(probs).mean(0)                        # average the clip-level softmax
print(model.config.id2label[int(p.argmax())], float(p.max()))
```

Multi-clip inference with a Kinetics-400 TimeSformer from Hugging Face (not run here)

> **High accuracy may not mean motion understanding:** On Kinetics and UCF101 a single frame often suffices, because scenes and objects correlate with labels. A model can score well and ignore time entirely. Test on motion-defined labels such as Something-Something, or on time-reversed clips, before claiming temporal modelling (see **Action recognition**).

> **Pre-training dominates:** Much of the rise from 88.0% to 98.0% on UCF101 came from pre-training rather than from the layer type. I3D starts from inflated ImageNet weights and is pre-trained on Kinetics before fine-tuning on UCF101. Video transformers likewise depend on image (ImageNet-21K) or self-supervised video pre-training, since even Kinetics-400 is small for a ViT.

Design choices at a glance

| Approach | How time enters | Main cost | Weakness |
| --- | --- | --- | --- |
| Frame CNN + averaging | not at all (order-invariant) | T × image CNN | blind to direction and order |
| Two-stream | optical flow input | computing optical flow offline | two networks, flow is slow |
| 3-D CNN (C3D, I3D) | t × d × d kernels | about t× the 2-D FLOPs per layer | short temporal reach per layer |
| (2+1)D | separate temporal conv | ≈ same as 3-D | still local in time |
| Video transformer | attention across frames | tokens grow with T × N | needs heavy pre-training |

**Question:** A model averages per-frame CNN features over the clip and then classifies. Which pair of classes can it never distinguish?

- A. Swimming vs running
- B. Opening a door vs closing a door
- C. Playing guitar vs playing piano
- D. Day vs night scenes

**Answer:** B. Opening a door vs closing a door — Closing a door is roughly opening played backwards. The two contain the same set of frames, and a time average does not depend on frame order, so the classifier sees identical inputs. The other pairs differ in appearance.

**Question:** I3D inflates a 3 × 3 ImageNet filter to 3 × 3 × 3 and divides it by 3. Why divide?

- A. To keep the parameter count equal to the 2-D filter
- B. So that a video of one repeated frame produces the same activations as the 2-D network on that frame
- C. To make the temporal kernel causal
- D. To normalise the optical-flow input

**Answer:** B. So that a video of one repeated frame produces the same activations as the 2-D network on that frame — On a "boring" video every temporal slice sees the same frame, so the response is $\sum_{\tau=1}^{3}\tfrac13 W^{2\mathrm D}*x=W^{2\mathrm D}*x$. The pretrained network's activation statistics are preserved at initialisation. The parameter count still triples.

**Key points**

- Video classifiers read short sampled clips (T frames, stride s) and average the softmax over several clips and crops at test time.
- Late fusion of per-frame features ignores order. Early or slow fusion, flow streams, 3-D kernels and temporal attention can see motion.
- Two-stream: RGB frame plus a 20-channel stack of 10 flow frames; on UCF101, flow alone 83.7% beats RGB alone 73.0%, and fusion reaches 88.0%.
- I3D inflates 2-D filters as $W^{2\mathrm D}/k$ repeated $k$ times. (2+1)D splits each kernel into spatial and temporal parts.
- TimeSformer's divided attention gives each patch T + N keys instead of NT: 7.69× fewer pairs for 8 × 196 tokens.
- Kinetics-400 is the standard benchmark. Appearance bias means good accuracy does not prove motion modelling.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
