Pose estimation

2D pose estimation predicts each person's 17 COCO keypoints, usually as Gaussian heatmaps at stride 4 decoded by argmax or soft-argmax, either per detected box (top-down, HRNet) or for all people at once (bottom-up, OpenPose), scored by OKS-based AP.

8 min read1108 wordscore0 viewsupdated 2026-10-15

Definition

Human pose estimation is the task of locating a fixed set of body keypoints (joints and facial landmarks) for every person in an image. On COCO each person has 17 keypoints, each with an (x,y)(x, y) position and a visibility flag. Landmark methods are DeepPose (Toshev and Szegedy, CVPR 2014, direct coordinate regression), Convolutional Pose Machines (Wei et al., CVPR 2016), Stacked Hourglass networks (Newell, Yang and Deng, ECCV 2016), OpenPose's part affinity fields (Cao, Simon, Wei and Sheikh, CVPR 2017) and HRNet (Sun, Xiao, Liu and Wang, CVPR 2019).

Input: an RGB image with any number of people. Output: for each person, 17 points (nose, eyes, ears, shoulders, elbows, wrists, hips, knees, ankles) with confidences. It is hard because joints are small, often occluded or out of frame, left and right look alike, and limbs of nearby people cross. Before deep learning, pictorial structures modelled the body as parts on a tree with spring-like pairwise costs over hand-crafted features such as HOG.

Regressing (x,y)(x, y) directly from a pooled feature, as DeepPose did, throws away spatial layout. Almost every strong model since 2015 instead predicts one heatmap per keypoint, a dense map whose peak marks the joint. A fully convolutional network does this naturally and can express several candidate peaks.

Top-down and bottom-up

Top-down methods first run a person detector (see Object detection), crop each box, resize it to a fixed input such as 256×192256\times192, and predict 17 heatmaps for that one person. Bottom-up methods predict all keypoints of all people in one pass and then group them into skeletons. Top-down is more accurate on COCO, because every person is processed at a normalised scale; its cost grows with the number of people and it fails when the detector misses someone. Bottom-up cost is nearly independent of the crowd size.

Multi-person 2D pose: 17 COCO keypoints per person, two ways to get themtop-down (detect, then one pose per box): CPM, Hourglass, SimpleBaselines, HRNetperson detector boxescrop, resize256×192 (H×W)HRNet-W32stride 4 · 32 ch8 · 6416 · 12832 · 256streams in parallel,fused repeatedly17 heatmaps64×48 eachtarget: Gaussian σ=2per map:argmax + ¼ shift× 4 → crop px→ image px17 × (x, y, conf)cost ∝ #peoplebottom-up (all keypoints, then group): OpenPose, Associative Embeddingwhole image, onceVGG-19 features(10 layers)multi-stage CNN,2 branchesS: part heatmaps19 = 18 parts + bgL: affinity fields38 = 19 limbs × 2limb scoreE = ∫ L·d̂ dualong the limbthen bipartitematch per limbskeletonscost ≈ constant in #peopleCOCO 17: nose · eyes · ears · shoulders · elbows · wrists · hips · knees · ankles (L and R)
fig.Top: the top-down pipeline. Boxes from a person detector are cropped to 256×192; HRNet-W32 keeps a stride-4 stream (32 channels) alive alongside strides 8, 16 and 32 and fuses them repeatedly; 17 heatmaps of 64×48 are decoded by argmax and mapped back to the image. Bottom: OpenPose runs once on the whole image, predicting part heatmaps S and part affinity fields L (2 channels per limb), scores every candidate limb by integrating the field along it, and assembles skeletons by bipartite matching.

Heatmap regression

The training target for keypoint jj is a 2D Gaussian centred on its ground-truth position μj\mu_j, rendered on the output grid. Stacked Hourglass used σ=1\sigma = 1 px on a 64×6464\times64 map; SimpleBaselines (Xiao, Wu and Wei, ECCV 2018) and HRNet use σ=2\sigma=2 on 64×4864\times48. The loss is a plain per-pixel MSE, with invisible or unlabelled keypoints masked out:

Hj∗(p)=exp⁡ ⁣(−∥p−μj∥22σ2),L=1J∑j=1Jvj∑p(Hj(p)−Hj∗(p))2H^*_j(p)=\exp\!\Big(-\frac{\lVert p-\mu_j\rVert^2}{2\sigma^2}\Big),\qquad \mathcal L=\frac{1}{J}\sum_{j=1}^{J} v_j\sum_{p}\big(H_j(p)-H^*_j(p)\big)^2
pp
a pixel of the output heatmap (stride 4 relative to the network input)
μj, σ\mu_j,\ \sigma
ground-truth location of keypoint jj on the heatmap grid, and the target width in heatmap pixels
vjv_j
1 if keypoint jj is labelled, 0 otherwise; J=17J = 17 on COCO

Architectures differ in how they keep resolution. The Hourglass module pools down to 4×44\times4 and upsamples back with skip connections; 8 hourglasses are stacked, each with its own loss (intermediate supervision). SimpleBaselines puts three stride-2 Transposed convolution layers on a ResNet. HRNet never drops the high-resolution stream: it adds lower-resolution branches in parallel (strides 4, 8, 16, 32 with 32, 64, 128, 256 channels in W32) and exchanges information between them at every stage.

Decoding: argmax, the quarter offset, soft-argmax

At test time the peak of each heatmap is the keypoint. A plain argmax snaps to the grid, so at stride 4 it can be up to 2 input pixels off in each axis. The SimpleBaselines/HRNet code shifts the argmax by a quarter cell toward the higher neighbour, a cheap sub-pixel refinement, and averages the heatmaps of the image and its horizontal flip. DARK (Zhang et al., CVPR 2020) replaces the quarter shift with a Taylor expansion of the log-heatmap around the peak.

Soft-argmax, used in Integral Human Pose Regression (Sun, Xiao, Liang and Wei, ECCV 2018), takes the expectation of the coordinate under a softmax of the heatmap. It is differentiable, so a coordinate loss can be trained end to end, and it is sub-pixel by construction:

μ^j=∑pp eβHj(p)∑qeβHj(q)\hat\mu_j=\sum_{p} p\,\frac{e^{\beta H_j(p)}}{\sum_{q}e^{\beta H_j(q)}}
β\beta
inverse temperature: large β\beta approaches argmax (and its grid snapping); small β\beta lets the whole map vote, pulling the estimate toward the map centre

Bottom-up grouping with part affinity fields

OpenPose predicts, besides part heatmaps, a part affinity field LcL_c for each limb cc: a 2D unit vector field that points from one joint to the other on pixels lying on the limb, and is zero elsewhere. For two candidate joints d1,d2d_1, d_2, the association score is the line integral E=∫01Lc(p(u))⋅d2−d1∥d2−d1∥ duE=\int_0^1 L_c(p(u))\cdot\frac{d_2-d_1}{\lVert d_2-d_1\rVert}\,du along the segment p(u)p(u), approximated by sampling. Each limb type is then a bipartite matching problem between the candidates of its two joints, solved greedily limb by limb along a tree. The COCO model has 18 parts (COCO's 17 plus a neck) and 19 limbs, so 19+3819 + 38 output channels. OpenPose won the COCO 2016 keypoint challenge. Associative Embedding (Newell, Huang and Deng, NeurIPS 2017) instead predicts a scalar tag per keypoint and groups joints with similar tags.

Scoring with OKS

COCO scores a predicted person against a ground-truth person with Object Keypoint Similarity, the keypoint analogue of IoU. Each visible keypoint contributes an unnormalised Gaussian of its error, scaled by the person's size and by a per-keypoint constant measured from annotator disagreement:

OKS=∑iexp⁡ ⁣(−di2/(2s2ki2)) δ(vi>0)∑iδ(vi>0),ki=2σi\mathrm{OKS}=\frac{\sum_i \exp\!\big(-d_i^2/(2s^2k_i^2)\big)\,\delta(v_i>0)}{\sum_i \delta(v_i>0)},\qquad k_i=2\sigma_i
did_i
Euclidean distance between predicted and true keypoint ii, in image pixels
s2s^2
object scale: the area of the person's segmentation mask
σi\sigma_i
0.026 nose, 0.025 eyes, 0.035 ears, 0.079 shoulders, 0.072 elbows, 0.062 wrists, 0.107 hips, 0.087 knees, 0.089 ankles

A prediction is a true positive at threshold tt if OKS ≥t\ge t. COCO keypoint AP averages the AP over t=0.50,0.55,…,0.95t = 0.50, 0.55, \dots, 0.95, exactly as box AP does over IoU thresholds. MPII instead uses PCKh@0.5: a joint is correct if it lies within half the head-segment length of the truth.

Datasets and reference configurations

COCO keypoints
17 keypoints per person; train2017 has about 57K images with 150K annotated people; AP over OKS 0.50:0.05:0.95
MPII Human Pose
16 joints; about 25K images with over 40K annotated people (Andriluka et al., CVPR 2014); metric PCKh@0.5
Stacked Hourglass
8 stacked hourglasses, input 256×256, output 64×64 heatmaps, a loss on every stack
HRNet-W32
input 256×192, 64×48 heatmaps, 28.5M parameters, 7.1 GFLOPs; 74.4 AP on COCO val2017 (top-down, person boxes from a detector)
HRNet-W48
input 384×288: 76.3 AP on COCO val2017, 75.5 AP on test-dev
predicted wrist heatmap, 16×12 cells, stride 4 (one cell = 4×4 input px)+ truth (4.30, 4.40) · target σ = 1.5 cells · noise std 0.05soft-argmax = Σ softmax(β·H)·(x, y) over all 192 cellsdecoded keypoint□ argmax(4.00, 4.00) cells · error 2.00 pxOKS 0.945◇ argmax + ¼ offset(4.25, 4.25) cells · error 0.63 pxOKS 0.994○ soft-argmax β=10(4.14, 4.18) cells · error 1.08 pxOKS 0.984OKS term exp(−d² / 2s²k²), wristk = 0.124, person scale s = 48 pxerrors in input px = cells × 4
second person's wrist in the crop
fig.A wrist heatmap (16×12 cells at stride 4, Gaussian σ = 1.5 cells, small noise) decoded three ways, each error scored by the wrist's OKS term with s = 48 px. At the default (true x = 4.3, noise 0.05, β = 10) argmax snaps to (4, 4), 2.00 px off, OKS 0.945; the quarter shift lands at (4.25, 4.25), 0.63 px, OKS 0.994; soft-argmax 1.08 px, OKS 0.984. Set noise to 0 and soft-argmax is nearly exact (0.06 px). Drop β to 3 and the near-zero background votes too: 8.14 px, OKS 0.393; at β = 30 it snaps like argmax (1.98 px). Turn on the second person's wrist: argmax is unaffected, soft-argmax drifts to 2.92 px at β = 10 and 13.47 px (OKS 0.077) at β = 3.

Worked example: one OKS

A person with mask area s2=9600s^2 = 9600 px² has five labelled keypoints; the others have vi=0v_i = 0 and are ignored. With s=98.0s = 98.0 px, the half-score distance dd where a keypoint's term falls to 0.5 is 6.0 px for the nose but 24.7 px for a hip.

Keypointk = 2σerror d (px)d² / 2s²k²term
nose0.05230.17340.8408
left shoulder0.15880.13350.8750
right wrist0.124120.48780.6140
left knee0.174100.17200.8420
right ankle0.17860.05920.9425
Per-keypoint terms exp(−d² / 2s²k²) with s² = 9600; OKS = mean = 0.823

OKS =4.1143/5=0.823= 4.1143 / 5 = 0.823, so this prediction is a true positive at the thresholds 0.50 to 0.80 (7 of the 10) and a false positive at 0.85, 0.90 and 0.95. The 3 px nose error costs about as much as the 10 px knee error: the constants make the metric strict where annotators agree.

python
1import numpy as np
2
3def target(mu, shape=(64, 48), sigma=2.0):
4 """Gaussian heatmap target centred at mu = (x, y) in heatmap pixels."""
5 ys, xs = np.mgrid[:shape[0], :shape[1]]
6 return np.exp(-((xs - mu[0]) ** 2 + (ys - mu[1]) ** 2) / (2 * sigma ** 2))
7
8def decode_argmax(h):
9 y, x = np.unravel_index(h.argmax(), h.shape)
10 dx = np.sign(h[y, x + 1] - h[y, x - 1]) if 0 < x < h.shape[1] - 1 else 0
11 dy = np.sign(h[y + 1, x] - h[y - 1, x]) if 0 < y < h.shape[0] - 1 else 0
12 return np.array([x + 0.25 * dx, y + 0.25 * dy]) # SimpleBaselines/HRNet refinement
13
14def soft_argmax(h, beta=10.0):
15 p = np.exp(beta * (h - h.max())); p /= p.sum()
16 ys, xs = np.mgrid[:h.shape[0], :h.shape[1]]
17 return np.array([(p * xs).sum(), (p * ys).sum()])
18
19SIGMAS = np.array([.026, .025, .025, .035, .035, .079, .079, .072, .072,
20 .062, .062, .107, .107, .087, .087, .089, .089]) # COCO order
21
22def oks(pred, gt, vis, area):
23 d2 = ((pred - gt) ** 2).sum(-1)
24 e = np.exp(-d2 / (2 * area * (2 * SIGMAS) ** 2))
25 return e[vis > 0].mean()
26
27h = target((20.3, 30.7))
28print(decode_argmax(h), soft_argmax(h).round(2)) # [20.25 30.75] [20.45 30.74]: background votes pull soft-argmax toward the centre
Gaussian targets, argmax with the quarter offset, soft-argmax and OKS in NumPy (runs as is)

Use when

  • +Top-down: best accuracy (HRNet, ViTPose), every person at a normalised scale, any single-person model can be plugged in.
  • +Bottom-up: one forward pass for the whole crowd, no dependence on a person detector, runtime nearly constant in the number of people.

Avoid when

  • −Top-down: cost grows linearly with people, missed or merged detections lose whole skeletons, crops of overlapping people contain other people's joints.
  • −Bottom-up: small people get few heatmap pixels, and grouping errors swap limbs between nearby people.

Check yourself

A top-down model predicts 64×48 heatmaps for a 256×192 crop and is decoded with plain argmax. What is the worst-case quantisation error, in crop pixels, for a perfectly peaked heatmap?

Check yourself

Two predictions each place one keypoint 6 px off on the same person (s² = 9600). One error is on the nose, one on a hip. Which costs more OKS?

Remember

  • ✓Pose = locate 17 COCO keypoints per person; predict one heatmap per keypoint, not raw coordinates.
  • ✓Targets are Gaussians (σ=2\sigma = 2 on 64×48 for HRNet) trained with masked MSE; decode by argmax + quarter offset or by soft-argmax.
  • ✓Top-down (detector + per-crop network, HRNet-W48 76.3 AP on COCO val) is most accurate; bottom-up (OpenPose part affinity fields) scales to crowds.
  • ✓OKS =meaniexp⁡(−di2/2s2ki2)=\mathrm{mean}_i\exp(-d_i^2/2s^2k_i^2) over visible keypoints; COCO AP averages over OKS 0.50:0.05:0.95.
  • ✓3D pose is often obtained by lifting 2D keypoints with a small network, evaluated by MPJPE in mm.
← back to Deep Learning roadmap