Definition
Human pose estimation is the task of locating a fixed set of body keypoints (joints and facial landmarks) for every person in an image. On COCO each person has 17 keypoints, each with an position and a visibility flag. Landmark methods are DeepPose (Toshev and Szegedy, CVPR 2014, direct coordinate regression), Convolutional Pose Machines (Wei et al., CVPR 2016), Stacked Hourglass networks (Newell, Yang and Deng, ECCV 2016), OpenPose's part affinity fields (Cao, Simon, Wei and Sheikh, CVPR 2017) and HRNet (Sun, Xiao, Liu and Wang, CVPR 2019).
Input: an RGB image with any number of people. Output: for each person, 17 points (nose, eyes, ears, shoulders, elbows, wrists, hips, knees, ankles) with confidences. It is hard because joints are small, often occluded or out of frame, left and right look alike, and limbs of nearby people cross. Before deep learning, pictorial structures modelled the body as parts on a tree with spring-like pairwise costs over hand-crafted features such as HOG.
Regressing directly from a pooled feature, as DeepPose did, throws away spatial layout. Almost every strong model since 2015 instead predicts one heatmap per keypoint, a dense map whose peak marks the joint. A fully convolutional network does this naturally and can express several candidate peaks.
Top-down and bottom-up
Top-down methods first run a person detector (see Object detection), crop each box, resize it to a fixed input such as , and predict 17 heatmaps for that one person. Bottom-up methods predict all keypoints of all people in one pass and then group them into skeletons. Top-down is more accurate on COCO, because every person is processed at a normalised scale; its cost grows with the number of people and it fails when the detector misses someone. Bottom-up cost is nearly independent of the crowd size.
Heatmap regression
The training target for keypoint is a 2D Gaussian centred on its ground-truth position , rendered on the output grid. Stacked Hourglass used px on a map; SimpleBaselines (Xiao, Wu and Wei, ECCV 2018) and HRNet use on . The loss is a plain per-pixel MSE, with invisible or unlabelled keypoints masked out:
- a pixel of the output heatmap (stride 4 relative to the network input)
- ground-truth location of keypoint on the heatmap grid, and the target width in heatmap pixels
- 1 if keypoint is labelled, 0 otherwise; on COCO
Architectures differ in how they keep resolution. The Hourglass module pools down to and upsamples back with skip connections; 8 hourglasses are stacked, each with its own loss (intermediate supervision). SimpleBaselines puts three stride-2 Transposed convolution layers on a ResNet. HRNet never drops the high-resolution stream: it adds lower-resolution branches in parallel (strides 4, 8, 16, 32 with 32, 64, 128, 256 channels in W32) and exchanges information between them at every stage.
Decoding: argmax, the quarter offset, soft-argmax
At test time the peak of each heatmap is the keypoint. A plain argmax snaps to the grid, so at stride 4 it can be up to 2 input pixels off in each axis. The SimpleBaselines/HRNet code shifts the argmax by a quarter cell toward the higher neighbour, a cheap sub-pixel refinement, and averages the heatmaps of the image and its horizontal flip. DARK (Zhang et al., CVPR 2020) replaces the quarter shift with a Taylor expansion of the log-heatmap around the peak.
Soft-argmax, used in Integral Human Pose Regression (Sun, Xiao, Liang and Wei, ECCV 2018), takes the expectation of the coordinate under a softmax of the heatmap. It is differentiable, so a coordinate loss can be trained end to end, and it is sub-pixel by construction:
- inverse temperature: large approaches argmax (and its grid snapping); small lets the whole map vote, pulling the estimate toward the map centre
Bottom-up grouping with part affinity fields
OpenPose predicts, besides part heatmaps, a part affinity field for each limb : a 2D unit vector field that points from one joint to the other on pixels lying on the limb, and is zero elsewhere. For two candidate joints , the association score is the line integral along the segment , approximated by sampling. Each limb type is then a bipartite matching problem between the candidates of its two joints, solved greedily limb by limb along a tree. The COCO model has 18 parts (COCO's 17 plus a neck) and 19 limbs, so output channels. OpenPose won the COCO 2016 keypoint challenge. Associative Embedding (Newell, Huang and Deng, NeurIPS 2017) instead predicts a scalar tag per keypoint and groups joints with similar tags.
Scoring with OKS
COCO scores a predicted person against a ground-truth person with Object Keypoint Similarity, the keypoint analogue of IoU. Each visible keypoint contributes an unnormalised Gaussian of its error, scaled by the person's size and by a per-keypoint constant measured from annotator disagreement:
- Euclidean distance between predicted and true keypoint , in image pixels
- object scale: the area of the person's segmentation mask
- 0.026 nose, 0.025 eyes, 0.035 ears, 0.079 shoulders, 0.072 elbows, 0.062 wrists, 0.107 hips, 0.087 knees, 0.089 ankles
A prediction is a true positive at threshold if OKS . COCO keypoint AP averages the AP over , exactly as box AP does over IoU thresholds. MPII instead uses PCKh@0.5: a joint is correct if it lies within half the head-segment length of the truth.
Datasets and reference configurations
- COCO keypoints
- 17 keypoints per person; train2017 has about 57K images with 150K annotated people; AP over OKS 0.50:0.05:0.95
- MPII Human Pose
- 16 joints; about 25K images with over 40K annotated people (Andriluka et al., CVPR 2014); metric PCKh@0.5
- Stacked Hourglass
- 8 stacked hourglasses, input 256×256, output 64×64 heatmaps, a loss on every stack
- HRNet-W32
- input 256×192, 64×48 heatmaps, 28.5M parameters, 7.1 GFLOPs; 74.4 AP on COCO val2017 (top-down, person boxes from a detector)
- HRNet-W48
- input 384×288: 76.3 AP on COCO val2017, 75.5 AP on test-dev
Worked example: one OKS
A person with mask area px² has five labelled keypoints; the others have and are ignored. With px, the half-score distance where a keypoint's term falls to 0.5 is 6.0 px for the nose but 24.7 px for a hip.
| Keypoint | k = 2σ | error d (px) | d² / 2s²k² | term |
|---|---|---|---|---|
| nose | 0.052 | 3 | 0.1734 | 0.8408 |
| left shoulder | 0.158 | 8 | 0.1335 | 0.8750 |
| right wrist | 0.124 | 12 | 0.4878 | 0.6140 |
| left knee | 0.174 | 10 | 0.1720 | 0.8420 |
| right ankle | 0.178 | 6 | 0.0592 | 0.9425 |
OKS , so this prediction is a true positive at the thresholds 0.50 to 0.80 (7 of the 10) and a false positive at 0.85, 0.90 and 0.95. The 3 px nose error costs about as much as the 10 px knee error: the constants make the metric strict where annotators agree.
1import numpy as np23def target(mu, shape=(64, 48), sigma=2.0):4 """Gaussian heatmap target centred at mu = (x, y) in heatmap pixels."""5 ys, xs = np.mgrid[:shape[0], :shape[1]]6 return np.exp(-((xs - mu[0]) ** 2 + (ys - mu[1]) ** 2) / (2 * sigma ** 2))78def decode_argmax(h):9 y, x = np.unravel_index(h.argmax(), h.shape)10 dx = np.sign(h[y, x + 1] - h[y, x - 1]) if 0 < x < h.shape[1] - 1 else 011 dy = np.sign(h[y + 1, x] - h[y - 1, x]) if 0 < y < h.shape[0] - 1 else 012 return np.array([x + 0.25 * dx, y + 0.25 * dy]) # SimpleBaselines/HRNet refinement1314def soft_argmax(h, beta=10.0):15 p = np.exp(beta * (h - h.max())); p /= p.sum()16 ys, xs = np.mgrid[:h.shape[0], :h.shape[1]]17 return np.array([(p * xs).sum(), (p * ys).sum()])1819SIGMAS = np.array([.026, .025, .025, .035, .035, .079, .079, .072, .072,20 .062, .062, .107, .107, .087, .087, .089, .089]) # COCO order2122def oks(pred, gt, vis, area):23 d2 = ((pred - gt) ** 2).sum(-1)24 e = np.exp(-d2 / (2 * area * (2 * SIGMAS) ** 2))25 return e[vis > 0].mean()2627h = target((20.3, 30.7))28print(decode_argmax(h), soft_argmax(h).round(2)) # [20.25 30.75] [20.45 30.74]: background votes pull soft-argmax toward the centreUse when
- +Top-down: best accuracy (HRNet, ViTPose), every person at a normalised scale, any single-person model can be plugged in.
- +Bottom-up: one forward pass for the whole crowd, no dependence on a person detector, runtime nearly constant in the number of people.
Avoid when
- −Top-down: cost grows linearly with people, missed or merged detections lose whole skeletons, crops of overlapping people contain other people's joints.
- −Bottom-up: small people get few heatmap pixels, and grouping errors swap limbs between nearby people.
Check yourself
A top-down model predicts 64×48 heatmaps for a 256×192 crop and is decoded with plain argmax. What is the worst-case quantisation error, in crop pixels, for a perfectly peaked heatmap?
Check yourself
Two predictions each place one keypoint 6 px off on the same person (s² = 9600). One error is on the nose, one on a hip. Which costs more OKS?
Remember
- ✓Pose = locate 17 COCO keypoints per person; predict one heatmap per keypoint, not raw coordinates.
- ✓Targets are Gaussians ( on 64×48 for HRNet) trained with masked MSE; decode by argmax + quarter offset or by soft-argmax.
- ✓Top-down (detector + per-crop network, HRNet-W48 76.3 AP on COCO val) is most accurate; bottom-up (OpenPose part affinity fields) scales to crowds.
- ✓OKS over visible keypoints; COCO AP averages over OKS 0.50:0.05:0.95.
- ✓3D pose is often obtained by lifting 2D keypoints with a small network, evaluated by MPJPE in mm.