Action recognition

Action recognition labels what people do: trimmed clips (Kinetics), segments in untrimmed video (mAP at tIoU) or people in keyframes (AVA). SlowFast pairs a 4-frame Slow pathway with a 32-frame, 1/8-width Fast one.

8 min read874 wordscore0 viewsupdated 2026-10-15

Definition

Action recognition is the task of recognising human actions in video. In its basic form it is video classification of a trimmed clip, such as "playing violin". Harder variants also ask when each action happens (temporal action localisation) and who does it (spatio-temporal action detection). Two widely used models are SlowFast (Feichtenhofer, Fan, Malik and He, "SlowFast Networks for Video Recognition", ICCV 2019) for RGB video and ST-GCN (Yan, Xiong and Lin, AAAI 2018) for skeleton sequences.

The backbones are the ones described in Video classification: two-stream networks, I3D, (2+1)D and video transformers. This article covers what is specific to human actions: which output is required, how much an action depends on motion rather than appearance, and how results are measured.

TaskOutputBenchmarksMetric
Trimmed classificationone label per clipUCF101, HMDB51, Kinetics, Something-Somethingtop-1 / top-5 accuracy
Temporal localisation(start, end, class, score) segmentsTHUMOS'14, ActivityNetmAP at tIoU thresholds
Spatio-temporal detectionperson box + action labels per keyframeAVAframe mAP at box IoU 0.5
Skeleton-basedlabel from 2-D or 3-D joint tracksNTU RGB+Daccuracy (cross-subject, cross-view)
Four task forms, with their typical benchmarks and metrics

Appearance versus motion

UCF101 (13,320 clips, 101 classes) and HMDB51 (about 6.8k clips, 51 classes) were the standard benchmarks until 2017. Kinetics-400 then replaced them for training and evaluation. In all three, many classes are identifiable from one frame through the scene, objects or clothing. Something-Something (Goyal et al., ICCV 2017; 174 classes, about 220k clips in v2) was built against this bias. Its labels are templates such as "Pushing something from left to right", with everyday objects, so only the motion separates the classes. Models that pool frames without regard to order do poorly on it.

Cheap ways to add temporal reasoning to 2-D networks matter here. TSN (Wang et al., ECCV 2016) samples one snippet from each of KK equal segments and averages their scores, which covers the whole video. TSM (Lin, Gan and Han, ICCV 2019) shifts 1/8 of the channels one frame forward and 1/8 one frame back inside each residual block. That mixes neighbouring frames at zero extra FLOPs.

SlowFast: two frame rates

SlowFast separates appearance from motion by frame rate rather than by optical flow. The Slow pathway sees few frames with many channels, which suits the semantics of a scene that changes slowly. The Fast pathway sees α\alpha times more frames with only β\beta of the channels, which suits fast motion. Lateral connections send Fast features into Slow at every stage.

Slow: T frames, stride τ, C channelsFast: αT frames, stride τ/α, βC channels,α=8, β=18\text{Slow: } T\ \text{frames, stride } \tau,\ C\ \text{channels}\qquad \text{Fast: } \alpha T\ \text{frames, stride } \tau/\alpha,\ \beta C\ \text{channels},\qquad \alpha=8,\ \beta=\tfrac18
τ\tau
Slow temporal stride on the raw clip: τ=16\tau=16 for the 4×16 model, so 64 raw frames give 4 Slow frames
α\alpha
frame-rate ratio: Fast reads every 2nd frame, αT=32\alpha T=32 frames
β\beta
channel ratio: Fast res2 has 32 channels to Slow's 256; per conv layer Fast costs αβ2=1/8\alpha\beta^2=1/8 of Slow
SlowFast R-50, 4×16 (Feichtenhofer et al., ICCV 2019) · Kinetics-40064 frames224², rawstride 16T = 4stride 2αT = 32Slow pathway: low frame rate, C channelsFast pathway: α = 8× frames, β = 1/8 channelsconv1 641×7²4×56²conv1 85×7²32×56²res2 256t 14×56²res2 32t 332×56²res3 512t 14×28²res3 64t 332×28²res4 1024t 34×14²res4 128t 332×14²res5 2048t 34×7²res5 256t 332×7²lateralGAP 2048GAP 256concat 2304fc → 400t k: temporal kernel size. The Slow pathway has temporal convolutions only in res4 and res5;the Fast pathway has them in every stage, so motion is modelled at high frame rate.Lateral (violet): 5×1² conv with temporal stride α = 8, 2βC channels, fused into Slow.Per conv layer, Fast costs α·β² = 8/64 = 1/8 of Slow: 8× the frames, 1/64 of C_in·C_out.Overall the Fast pathway is about 20% of the FLOPs.Shapes are T × H × W after each stage; numbers in boxes are output channels.
fig.SlowFast R-50 4×16. A 64-frame clip is sampled at stride 16 for the Slow pathway (4 × 224²) and at stride 2 for the Fast pathway (32 × 224²). Slow widths run 64 → 2048 channels and Fast widths 8 → 256. Spatial size halves from 56² after pool1 to 7² at res5, and the temporal length never changes. Violet lateral connections (5×1² conv, temporal stride 8) feed Fast into Slow after pool1, res2, res3 and res4. Global pooling gives 2048 + 256 = 2304 features for the 400-way classifier.

The Fast pathway has temporal kernels in every stage. The Slow pathway has them only in res4 and res5, because the authors found that temporal convolutions in early Slow layers hurt accuracy. Thanks to its small width, the Fast pathway adds only about 20% of the total computation. There is no optical flow anywhere.

Reported numbers (Kinetics-400 top-1, from scratch unless noted)

SlowFast R-50 4×16
75.6%
SlowFast R-50 8×8
77.0%
SlowFast R-101 16×8 + non-local
79.8%
AVA
the paper also reports AVA detection, using a Faster R-CNN person detector and RoIAlign on SlowFast features

Temporal localisation and tIoU

In untrimmed video (THUMOS'14 with 20 sports classes, ActivityNet v1.3 with 200 classes from YouTube), the model outputs scored segments. A prediction is a true positive if its temporal IoU with an unmatched ground-truth segment of the same class reaches a threshold θ\theta. Predictions are taken in descending order of score, and each ground-truth segment can be matched once.

tIoU([s1,e1],[s2,e2])=max⁡ ⁣(0, min⁡(e1,e2)−max⁡(s1,s2))max⁡(e1,e2)−min⁡(s1,s2),ActivityNet:  110∑θ∈{0.5,0.55,…,0.95}mAPθ\mathrm{tIoU}\big([s_1,e_1],[s_2,e_2]\big)=\frac{\max\!\big(0,\ \min(e_1,e_2)-\max(s_1,s_2)\big)}{\max(e_1,e_2)-\min(s_1,s_2)},\qquad \text{ActivityNet: }\ \frac{1}{10}\sum_{\theta\in\{0.5,0.55,\dots,0.95\}}\mathrm{mAP}_\theta
mAPθ\mathrm{mAP}_\theta
mean over classes of the area under the precision–recall curve at threshold θ
θ\theta
THUMOS'14 reports mAP at 0.3, 0.4, 0.5, 0.6 and 0.7, with 0.5 as the headline number
"long jump" in a 120 s video · tIoU threshold θ = 0.50 · NMS off0 s20 s40 s60 s80 s100 s120 sscoretIoUground truthG1G2G3G40.950.778TP→ G10.900.625TP→ G20.850.571TP→ G30.800.625FPG1 already taken0.700.000FP0.600.545TP→ G40.500.714FPG3 already taken0.400.333FPG2: tIoU < θ0.300.000FPprecision–recall (score order)10recall 1AP = 0.917AP at each tIoU threshold0.10.920.20.920.30.920.40.920.50.920.60.610.70.320.80.000.90.00θ = 0.50: 4 TP, 5 FP · recall 1.00 · AP 0.917THUMOS-style mAP@[0.3:0.1:0.7] = 0.736 · ActivityNet mAP@[0.5:0.05:0.95] = 0.317
duplicates
fig.Nine scored predictions against four ground-truth "long jump" segments in a 120 s video. At θ = 0.5, four predictions are true positives, the 0.80 duplicate of G1 is a false positive, and AP = 0.917. At θ = 0.6 the 0.85-score segment (tIoU 0.571) fails. G3 is then matched by the lower-scored 0.714 segment, and G4's best match (0.545) fails, so AP drops to 0.607. At 0.8 nothing matches. The THUMOS-style mean over 0.3–0.7 is 0.736 and the ActivityNet mean over 0.5–0.95 is 0.317. Temporal NMS at 0.5 removes the duplicate and lifts AP at 0.5 to 0.950.

Worked example: tIoU decides TP or FP

Prediction (score)Ground truthIntersectionUniontIoUθ = 0.5θ = 0.6
[11, 19] (0.95)G1 [10, 18]790.778TPTP
[60, 70] (0.85)G3 [62, 74]8140.571TPFP
[64, 76] (0.50)G3 [62, 74]10140.714FP (G3 taken)TP
Three predictions from the figure (seconds). Computed in Python.

The two G3 predictions swap roles as θ\theta rises. At 0.5 the higher-scored but looser segment claims G3, which leaves the tighter one as a false positive. At 0.6 the loose one fails, and G3 is recovered only at rank 7, after four false positives, so precision at that recall is lower. Boundary accuracy, not only detection, moves AP at strict thresholds.

Who is acting, and skeletons

AVA (Gu et al., CVPR 2018) labels people in movie clips with 80 atomic actions such as stand, sit, talk to and carry, on keyframes one second apart. The task is multi-label: a person can stand, talk and listen at the same time, so models use one sigmoid per class. A typical system runs a person detector on the keyframe and then pools a 3-D feature map around each box (see Faster R-CNN and Mask R-CNN for RoIAlign).

Skeleton methods take the joint coordinates from motion capture, a depth sensor or Pose estimation as input. They are cheap and ignore background, clothing and lighting. ST-GCN builds a graph whose nodes are joints at each frame. Spatial edges follow the bones and temporal edges link the same joint in consecutive frames. Each layer applies a graph convolution in space and then a temporal convolution of length 9:

fout=∑jΛj−1/2 Aj Λj−1/2 fin Wj ⊙Mf_{\text{out}}=\sum_{j}\Lambda_j^{-1/2}\,A_j\,\Lambda_j^{-1/2}\,f_{\text{in}}\,W_j\ \odot M
AjA_j
adjacency of partition jj: the joint itself, neighbours closer to the body's centre of gravity, and neighbours farther away
Λj\Lambda_j
degree matrix of AjA_j for normalisation; WjW_j are the weights of partition jj
MM
learnable edge-importance mask on the adjacency

On NTU RGB+D (Shahroudy et al., CVPR 2016: 56,880 clips, 60 classes, 25 Kinect joints), ST-GCN reported 81.5% cross-subject and 88.3% cross-view accuracy.

python
1import torch
2
3model = torch.hub.load("facebookresearch/pytorchvideo", "slowfast_r50", pretrained=True).eval()
4
5def pack_pathways(frames, alpha=4):
6 """frames: (C, T, H, W), already resized, cropped and normalised; T = 32 here."""
7 fast = frames
8 idx = torch.linspace(0, frames.shape[1] - 1, frames.shape[1] // alpha).long()
9 slow = frames.index_select(1, idx) # T / alpha = 8 frames
10 return [slow[None], fast[None]] # add the batch dimension
11
12clip = torch.randn(3, 32, 256, 256) # stand-in for a decoded, normalised clip
13with torch.no_grad():
14 logits = model(pack_pathways(clip)) # (1, 400) Kinetics-400 scores
15print(logits.softmax(-1).topk(5))
SlowFast R-50 from PyTorchVideo's model zoo: build the two pathway inputs from one clip (not run here; this checkpoint is the 8×8 model with α = 4)

Check yourself

In SlowFast, the Fast pathway has α = 8 times the frames and β = 1/8 of the channels. Ignoring kernel-size differences, how does one Fast conv layer's FLOPs compare with the matching Slow layer?

Check yourself

A predicted segment [60, 70] s overlaps a ground-truth segment [62, 74] s. At which thresholds is it a true positive (assuming no higher-scored prediction took the segment)?

Remember

  • ✓Action recognition covers trimmed classification, temporal localisation (when), spatio-temporal detection (who, AVA) and skeleton-based recognition.
  • ✓Kinetics and UCF101 are appearance-heavy. Something-Something's 174 template classes require motion and order.
  • ✓SlowFast: Slow has T frames at stride τ with C channels, Fast has αT frames with βC (α = 8, β = 1/8), plus lateral connections. Fast is about 20% of the FLOPs.
  • ✓tIoU = overlap / union of time intervals. THUMOS'14 reports mAP at 0.3–0.7, ActivityNet averages mAP over 0.5:0.05:0.95.
  • ✓ST-GCN applies a partitioned graph convolution over joints and a length-9 temporal convolution: 81.5% / 88.3% on NTU RGB+D cross-subject / cross-view.
← back to Deep Learning roadmap