Definition
Action recognition is the task of recognising human actions in video. In its basic form it is video classification of a trimmed clip, such as "playing violin". Harder variants also ask when each action happens (temporal action localisation) and who does it (spatio-temporal action detection). Two widely used models are SlowFast (Feichtenhofer, Fan, Malik and He, "SlowFast Networks for Video Recognition", ICCV 2019) for RGB video and ST-GCN (Yan, Xiong and Lin, AAAI 2018) for skeleton sequences.
The backbones are the ones described in Video classification: two-stream networks, I3D, (2+1)D and video transformers. This article covers what is specific to human actions: which output is required, how much an action depends on motion rather than appearance, and how results are measured.
| Task | Output | Benchmarks | Metric |
|---|---|---|---|
| Trimmed classification | one label per clip | UCF101, HMDB51, Kinetics, Something-Something | top-1 / top-5 accuracy |
| Temporal localisation | (start, end, class, score) segments | THUMOS'14, ActivityNet | mAP at tIoU thresholds |
| Spatio-temporal detection | person box + action labels per keyframe | AVA | frame mAP at box IoU 0.5 |
| Skeleton-based | label from 2-D or 3-D joint tracks | NTU RGB+D | accuracy (cross-subject, cross-view) |
Appearance versus motion
UCF101 (13,320 clips, 101 classes) and HMDB51 (about 6.8k clips, 51 classes) were the standard benchmarks until 2017. Kinetics-400 then replaced them for training and evaluation. In all three, many classes are identifiable from one frame through the scene, objects or clothing. Something-Something (Goyal et al., ICCV 2017; 174 classes, about 220k clips in v2) was built against this bias. Its labels are templates such as "Pushing something from left to right", with everyday objects, so only the motion separates the classes. Models that pool frames without regard to order do poorly on it.
Cheap ways to add temporal reasoning to 2-D networks matter here. TSN (Wang et al., ECCV 2016) samples one snippet from each of equal segments and averages their scores, which covers the whole video. TSM (Lin, Gan and Han, ICCV 2019) shifts 1/8 of the channels one frame forward and 1/8 one frame back inside each residual block. That mixes neighbouring frames at zero extra FLOPs.
SlowFast: two frame rates
SlowFast separates appearance from motion by frame rate rather than by optical flow. The Slow pathway sees few frames with many channels, which suits the semantics of a scene that changes slowly. The Fast pathway sees times more frames with only of the channels, which suits fast motion. Lateral connections send Fast features into Slow at every stage.
- Slow temporal stride on the raw clip: for the 4×16 model, so 64 raw frames give 4 Slow frames
- frame-rate ratio: Fast reads every 2nd frame, frames
- channel ratio: Fast res2 has 32 channels to Slow's 256; per conv layer Fast costs of Slow
The Fast pathway has temporal kernels in every stage. The Slow pathway has them only in res4 and res5, because the authors found that temporal convolutions in early Slow layers hurt accuracy. Thanks to its small width, the Fast pathway adds only about 20% of the total computation. There is no optical flow anywhere.
Reported numbers (Kinetics-400 top-1, from scratch unless noted)
- SlowFast R-50 4×16
- 75.6%
- SlowFast R-50 8×8
- 77.0%
- SlowFast R-101 16×8 + non-local
- 79.8%
- AVA
- the paper also reports AVA detection, using a Faster R-CNN person detector and RoIAlign on SlowFast features
Temporal localisation and tIoU
In untrimmed video (THUMOS'14 with 20 sports classes, ActivityNet v1.3 with 200 classes from YouTube), the model outputs scored segments. A prediction is a true positive if its temporal IoU with an unmatched ground-truth segment of the same class reaches a threshold . Predictions are taken in descending order of score, and each ground-truth segment can be matched once.
- mean over classes of the area under the precision–recall curve at threshold θ
- THUMOS'14 reports mAP at 0.3, 0.4, 0.5, 0.6 and 0.7, with 0.5 as the headline number
Worked example: tIoU decides TP or FP
| Prediction (score) | Ground truth | Intersection | Union | tIoU | θ = 0.5 | θ = 0.6 |
|---|---|---|---|---|---|---|
| [11, 19] (0.95) | G1 [10, 18] | 7 | 9 | 0.778 | TP | TP |
| [60, 70] (0.85) | G3 [62, 74] | 8 | 14 | 0.571 | TP | FP |
| [64, 76] (0.50) | G3 [62, 74] | 10 | 14 | 0.714 | FP (G3 taken) | TP |
The two G3 predictions swap roles as rises. At 0.5 the higher-scored but looser segment claims G3, which leaves the tighter one as a false positive. At 0.6 the loose one fails, and G3 is recovered only at rank 7, after four false positives, so precision at that recall is lower. Boundary accuracy, not only detection, moves AP at strict thresholds.
Who is acting, and skeletons
AVA (Gu et al., CVPR 2018) labels people in movie clips with 80 atomic actions such as stand, sit, talk to and carry, on keyframes one second apart. The task is multi-label: a person can stand, talk and listen at the same time, so models use one sigmoid per class. A typical system runs a person detector on the keyframe and then pools a 3-D feature map around each box (see Faster R-CNN and Mask R-CNN for RoIAlign).
Skeleton methods take the joint coordinates from motion capture, a depth sensor or Pose estimation as input. They are cheap and ignore background, clothing and lighting. ST-GCN builds a graph whose nodes are joints at each frame. Spatial edges follow the bones and temporal edges link the same joint in consecutive frames. Each layer applies a graph convolution in space and then a temporal convolution of length 9:
- adjacency of partition : the joint itself, neighbours closer to the body's centre of gravity, and neighbours farther away
- degree matrix of for normalisation; are the weights of partition
- learnable edge-importance mask on the adjacency
On NTU RGB+D (Shahroudy et al., CVPR 2016: 56,880 clips, 60 classes, 25 Kinect joints), ST-GCN reported 81.5% cross-subject and 88.3% cross-view accuracy.
1import torch23model = torch.hub.load("facebookresearch/pytorchvideo", "slowfast_r50", pretrained=True).eval()45def pack_pathways(frames, alpha=4):6 """frames: (C, T, H, W), already resized, cropped and normalised; T = 32 here."""7 fast = frames8 idx = torch.linspace(0, frames.shape[1] - 1, frames.shape[1] // alpha).long()9 slow = frames.index_select(1, idx) # T / alpha = 8 frames10 return [slow[None], fast[None]] # add the batch dimension1112clip = torch.randn(3, 32, 256, 256) # stand-in for a decoded, normalised clip13with torch.no_grad():14 logits = model(pack_pathways(clip)) # (1, 400) Kinetics-400 scores15print(logits.softmax(-1).topk(5))Check yourself
In SlowFast, the Fast pathway has α = 8 times the frames and β = 1/8 of the channels. Ignoring kernel-size differences, how does one Fast conv layer's FLOPs compare with the matching Slow layer?
Check yourself
A predicted segment [60, 70] s overlaps a ground-truth segment [62, 74] s. At which thresholds is it a true positive (assuming no higher-scored prediction took the segment)?
Remember
- ✓Action recognition covers trimmed classification, temporal localisation (when), spatio-temporal detection (who, AVA) and skeleton-based recognition.
- ✓Kinetics and UCF101 are appearance-heavy. Something-Something's 174 template classes require motion and order.
- ✓SlowFast: Slow has T frames at stride τ with C channels, Fast has αT frames with βC (α = 8, β = 1/8), plus lateral connections. Fast is about 20% of the FLOPs.
- ✓tIoU = overlap / union of time intervals. THUMOS'14 reports mAP at 0.3–0.7, ActivityNet averages mAP over 0.5:0.05:0.95.
- ✓ST-GCN applies a partitioned graph convolution over joints and a length-9 temporal convolution: 81.5% / 88.3% on NTU RGB+D cross-subject / cross-view.