# Action recognition

> Action recognition labels what people do: trimmed clips (Kinetics), segments in untrimmed video (mAP at tIoU) or people in keyframes (AVA). SlowFast pairs a 4-frame Slow pathway with a 32-frame, 1/8-width Fast one.

Source: https://www.mlfoundry.in/learn/action-recognition  
Section: Deep Learning › Deep Learning for Computer Vision  
Level: core · 8 min read · updated 2026-10-15

**Action recognition:** is the task of recognising human actions in video. In its basic form it is **video classification** of a trimmed clip, such as "playing violin". Harder variants also ask *when* each action happens (temporal action localisation) and *who* does it (spatio-temporal action detection). Two widely used models are SlowFast (Feichtenhofer, Fan, Malik and He, "SlowFast Networks for Video Recognition", ICCV 2019) for RGB video and ST-GCN (Yan, Xiong and Lin, AAAI 2018) for skeleton sequences.

The backbones are the ones described in **Video classification**: two-stream networks, I3D, (2+1)D and video transformers. This article covers what is specific to human actions: which output is required, how much an action depends on motion rather than appearance, and how results are measured.

Four task forms, with their typical benchmarks and metrics

| Task | Output | Benchmarks | Metric |
| --- | --- | --- | --- |
| Trimmed classification | one label per clip | UCF101, HMDB51, Kinetics, Something-Something | top-1 / top-5 accuracy |
| Temporal localisation | (start, end, class, score) segments | THUMOS'14, ActivityNet | mAP at tIoU thresholds |
| Spatio-temporal detection | person box + action labels per keyframe | AVA | frame mAP at box IoU 0.5 |
| Skeleton-based | label from 2-D or 3-D joint tracks | NTU RGB+D | accuracy (cross-subject, cross-view) |

## Appearance versus motion

UCF101 (13,320 clips, 101 classes) and HMDB51 (about 6.8k clips, 51 classes) were the standard benchmarks until 2017. Kinetics-400 then replaced them for training and evaluation. In all three, many classes are identifiable from one frame through the scene, objects or clothing. **Something-Something** (Goyal et al., ICCV 2017; 174 classes, about 220k clips in v2) was built against this bias. Its labels are templates such as "Pushing something from left to right", with everyday objects, so only the motion separates the classes. Models that pool frames without regard to order do poorly on it.

Cheap ways to add temporal reasoning to 2-D networks matter here. TSN (Wang et al., ECCV 2016) samples one snippet from each of $K$ equal segments and averages their scores, which covers the whole video. TSM (Lin, Gan and Han, ICCV 2019) shifts 1/8 of the channels one frame forward and 1/8 one frame back inside each residual block. That mixes neighbouring frames at zero extra FLOPs.

## SlowFast: two frame rates

SlowFast separates appearance from motion by frame rate rather than by optical flow. The **Slow** pathway sees few frames with many channels, which suits the semantics of a scene that changes slowly. The **Fast** pathway sees $\alpha$ times more frames with only $\beta$ of the channels, which suits fast motion. Lateral connections send Fast features into Slow at every stage.

$$
\text{Slow: } T\ \text{frames, stride } \tau,\ C\ \text{channels}\qquad \text{Fast: } \alpha T\ \text{frames, stride } \tau/\alpha,\ \beta C\ \text{channels},\qquad \alpha=8,\ \beta=\tfrac18
$$

- $\tau$: Slow temporal stride on the raw clip: $\tau=16$ for the 4×16 model, so 64 raw frames give 4 Slow frames
- $\alpha$: frame-rate ratio: Fast reads every 2nd frame, $\alpha T=32$ frames
- $\beta$: channel ratio: Fast res2 has 32 channels to Slow's 256; per conv layer Fast costs $\alpha\beta^2=1/8$ of Slow

> Figure: SlowFast R-50 4×16. A 64-frame clip is sampled at stride 16 for the Slow pathway (4 × 224²) and at stride 2 for the Fast pathway (32 × 224²). Slow widths run 64 → 2048 channels and Fast widths 8 → 256. Spatial size halves from 56² after pool1 to 7² at res5, and the temporal length never changes. Violet lateral connections (5×1² conv, temporal stride 8) feed Fast into Slow after pool1, res2, res3 and res4. Global pooling gives 2048 + 256 = 2304 features for the 400-way classifier.

The Fast pathway has temporal kernels in every stage. The Slow pathway has them only in res4 and res5, because the authors found that temporal convolutions in early Slow layers hurt accuracy. Thanks to its small width, the Fast pathway adds only about 20% of the total computation. There is no optical flow anywhere.

**Reported numbers (Kinetics-400 top-1, from scratch unless noted)**

- **SlowFast R-50 4×16:** 75.6%
- **SlowFast R-50 8×8:** 77.0%
- **SlowFast R-101 16×8 + non-local:** 79.8%
- **AVA:** the paper also reports AVA detection, using a Faster R-CNN person detector and RoIAlign on SlowFast features

## Temporal localisation and tIoU

In untrimmed video (THUMOS'14 with 20 sports classes, ActivityNet v1.3 with 200 classes from YouTube), the model outputs scored segments. A prediction is a true positive if its temporal IoU with an unmatched ground-truth segment of the same class reaches a threshold $\theta$. Predictions are taken in descending order of score, and each ground-truth segment can be matched once.

$$
\mathrm{tIoU}\big([s_1,e_1],[s_2,e_2]\big)=\frac{\max\!\big(0,\ \min(e_1,e_2)-\max(s_1,s_2)\big)}{\max(e_1,e_2)-\min(s_1,s_2)},\qquad \text{ActivityNet: }\ \frac{1}{10}\sum_{\theta\in\{0.5,0.55,\dots,0.95\}}\mathrm{mAP}_\theta
$$

- $\mathrm{mAP}_\theta$: mean over classes of the area under the precision–recall curve at threshold θ
- $\theta$: THUMOS'14 reports mAP at 0.3, 0.4, 0.5, 0.6 and 0.7, with 0.5 as the headline number

> Figure: Nine scored predictions against four ground-truth "long jump" segments in a 120 s video. At θ = 0.5, four predictions are true positives, the 0.80 duplicate of G1 is a false positive, and AP = 0.917. At θ = 0.6 the 0.85-score segment (tIoU 0.571) fails. G3 is then matched by the lower-scored 0.714 segment, and G4's best match (0.545) fails, so AP drops to 0.607. At 0.8 nothing matches. The THUMOS-style mean over 0.3–0.7 is 0.736 and the ActivityNet mean over 0.5–0.95 is 0.317. Temporal NMS at 0.5 removes the duplicate and lifts AP at 0.5 to 0.950.

## Worked example: tIoU decides TP or FP

Three predictions from the figure (seconds). Computed in Python.

| Prediction (score) | Ground truth | Intersection | Union | tIoU | θ = 0.5 | θ = 0.6 |
| --- | --- | --- | --- | --- | --- | --- |
| [11, 19] (0.95) | G1 [10, 18] | 7 | 9 | 0.778 | TP | TP |
| [60, 70] (0.85) | G3 [62, 74] | 8 | 14 | 0.571 | TP | FP |
| [64, 76] (0.50) | G3 [62, 74] | 10 | 14 | 0.714 | FP (G3 taken) | TP |

The two G3 predictions swap roles as $\theta$ rises. At 0.5 the higher-scored but looser segment claims G3, which leaves the tighter one as a false positive. At 0.6 the loose one fails, and G3 is recovered only at rank 7, after four false positives, so precision at that recall is lower. Boundary accuracy, not only detection, moves AP at strict thresholds.

## Who is acting, and skeletons

**AVA** (Gu et al., CVPR 2018) labels people in movie clips with 80 atomic actions such as stand, sit, talk to and carry, on keyframes one second apart. The task is multi-label: a person can stand, talk and listen at the same time, so models use one sigmoid per class. A typical system runs a person detector on the keyframe and then pools a 3-D feature map around each box (see **Faster R-CNN** and **Mask R-CNN** for RoIAlign).

Skeleton methods take the joint coordinates from motion capture, a depth sensor or **Pose estimation** as input. They are cheap and ignore background, clothing and lighting. ST-GCN builds a graph whose nodes are joints at each frame. Spatial edges follow the bones and temporal edges link the same joint in consecutive frames. Each layer applies a graph convolution in space and then a temporal convolution of length 9:

$$
f_{\text{out}}=\sum_{j}\Lambda_j^{-1/2}\,A_j\,\Lambda_j^{-1/2}\,f_{\text{in}}\,W_j\ \odot M
$$

- $A_j$: adjacency of partition $j$: the joint itself, neighbours closer to the body's centre of gravity, and neighbours farther away
- $\Lambda_j$: degree matrix of $A_j$ for normalisation; $W_j$ are the weights of partition $j$
- $M$: learnable edge-importance mask on the adjacency

On NTU RGB+D (Shahroudy et al., CVPR 2016: 56,880 clips, 60 classes, 25 Kinect joints), ST-GCN reported 81.5% cross-subject and 88.3% cross-view accuracy.

```python
import torch

model = torch.hub.load("facebookresearch/pytorchvideo", "slowfast_r50", pretrained=True).eval()

def pack_pathways(frames, alpha=4):
    """frames: (C, T, H, W), already resized, cropped and normalised; T = 32 here."""
    fast = frames
    idx = torch.linspace(0, frames.shape[1] - 1, frames.shape[1] // alpha).long()
    slow = frames.index_select(1, idx)               # T / alpha = 8 frames
    return [slow[None], fast[None]]                 # add the batch dimension

clip = torch.randn(3, 32, 256, 256)                # stand-in for a decoded, normalised clip
with torch.no_grad():
    logits = model(pack_pathways(clip))             # (1, 400) Kinetics-400 scores
print(logits.softmax(-1).topk(5))
```

SlowFast R-50 from PyTorchVideo's model zoo: build the two pathway inputs from one clip (not run here; this checkpoint is the 8×8 model with α = 4)

> **Trimmed accuracy does not transfer to untrimmed video:** A classifier trained on trimmed clips assumes the action fills the clip. On a long video it fires on background frames that share the scene, such as the run-up area of a long-jump pit, and its boundaries are vague. Localisation needs explicit start/end modelling and is scored at several tIoU thresholds for this reason.

> **Frame rate as a design axis:** SlowFast showed that motion does not require optical flow. A thin, high-frame-rate pathway (β = 1/8, about 20% of the FLOPs) recovers most of what a flow stream adds, end to end and without precomputation.

**Question:** In SlowFast, the Fast pathway has α = 8 times the frames and β = 1/8 of the channels. Ignoring kernel-size differences, how does one Fast conv layer's FLOPs compare with the matching Slow layer?

- A. 8× more
- B. Equal
- C. 1/8
- D. 1/64

**Answer:** C. 1/8 — FLOPs scale with $T\cdot C_{in}\cdot C_{out}$. Fast has $8T$ frames and $C/8$ channels on both sides: $8\cdot\tfrac18\cdot\tfrac18=\tfrac18$. Temporal kernels in every Fast stage push the pathway's real share to about 20%.

**Question:** A predicted segment [60, 70] s overlaps a ground-truth segment [62, 74] s. At which thresholds is it a true positive (assuming no higher-scored prediction took the segment)?

- A. θ = 0.3 and 0.5, but not 0.6
- B. Only θ = 0.3
- C. All of 0.3, 0.5 and 0.7
- D. None, because the boundaries differ

**Answer:** A. θ = 0.3 and 0.5, but not 0.6 — Intersection [62, 70] = 8 s, union [60, 74] = 14 s, so tIoU = 0.571. That passes 0.3 and 0.5 and fails 0.6 and 0.7.

**Key points**

- Action recognition covers trimmed classification, temporal localisation (when), spatio-temporal detection (who, AVA) and skeleton-based recognition.
- Kinetics and UCF101 are appearance-heavy. Something-Something's 174 template classes require motion and order.
- SlowFast: Slow has T frames at stride τ with C channels, Fast has αT frames with βC (α = 8, β = 1/8), plus lateral connections. Fast is about 20% of the FLOPs.
- tIoU = overlap / union of time intervals. THUMOS'14 reports mAP at 0.3–0.7, ActivityNet averages mAP over 0.5:0.05:0.95.
- ST-GCN applies a partitioned graph convolution over joints and a length-9 temporal convolution: 81.5% / 88.3% on NTU RGB+D cross-subject / cross-view.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
