# Pose estimation

> 2D pose estimation predicts each person's 17 COCO keypoints, usually as Gaussian heatmaps at stride 4 decoded by argmax or soft-argmax, either per detected box (top-down, HRNet) or for all people at once (bottom-up, OpenPose), scored by OKS-based AP.

Source: https://www.mlfoundry.in/learn/pose-estimation  
Section: Deep Learning › Deep Learning for Computer Vision  
Level: core · 8 min read · updated 2026-10-15

**Human pose estimation:** is the task of locating a fixed set of body **keypoints** (joints and facial landmarks) for every person in an image. On COCO each person has 17 keypoints, each with an $(x, y)$ position and a visibility flag. Landmark methods are DeepPose (Toshev and Szegedy, CVPR 2014, direct coordinate regression), Convolutional Pose Machines (Wei et al., CVPR 2016), Stacked Hourglass networks (Newell, Yang and Deng, ECCV 2016), OpenPose's part affinity fields (Cao, Simon, Wei and Sheikh, CVPR 2017) and HRNet (Sun, Xiao, Liu and Wang, CVPR 2019).

Input: an RGB image with any number of people. Output: for each person, 17 points (nose, eyes, ears, shoulders, elbows, wrists, hips, knees, ankles) with confidences. It is hard because joints are small, often occluded or out of frame, left and right look alike, and limbs of nearby people cross. Before deep learning, pictorial structures modelled the body as parts on a tree with spring-like pairwise costs over hand-crafted features such as HOG.

Regressing $(x, y)$ directly from a pooled feature, as DeepPose did, throws away spatial layout. Almost every strong model since 2015 instead predicts one **heatmap** per keypoint, a dense map whose peak marks the joint. A fully convolutional network does this naturally and can express several candidate peaks.

## Top-down and bottom-up

**Top-down** methods first run a person detector (see **Object detection**), crop each box, resize it to a fixed input such as $256\times192$, and predict 17 heatmaps for that one person. **Bottom-up** methods predict all keypoints of all people in one pass and then group them into skeletons. Top-down is more accurate on COCO, because every person is processed at a normalised scale; its cost grows with the number of people and it fails when the detector misses someone. Bottom-up cost is nearly independent of the crowd size.

> Figure: Top: the top-down pipeline. Boxes from a person detector are cropped to 256×192; HRNet-W32 keeps a stride-4 stream (32 channels) alive alongside strides 8, 16 and 32 and fuses them repeatedly; 17 heatmaps of 64×48 are decoded by argmax and mapped back to the image. Bottom: OpenPose runs once on the whole image, predicting part heatmaps S and part affinity fields L (2 channels per limb), scores every candidate limb by integrating the field along it, and assembles skeletons by bipartite matching.

## Heatmap regression

The training target for keypoint $j$ is a 2D Gaussian centred on its ground-truth position $\mu_j$, rendered on the output grid. Stacked Hourglass used $\sigma = 1$ px on a $64\times64$ map; SimpleBaselines (Xiao, Wu and Wei, ECCV 2018) and HRNet use $\sigma=2$ on $64\times48$. The loss is a plain per-pixel MSE, with invisible or unlabelled keypoints masked out:

$$
H^*_j(p)=\exp\!\Big(-\frac{\lVert p-\mu_j\rVert^2}{2\sigma^2}\Big),\qquad \mathcal L=\frac{1}{J}\sum_{j=1}^{J} v_j\sum_{p}\big(H_j(p)-H^*_j(p)\big)^2
$$

- $p$: a pixel of the output heatmap (stride 4 relative to the network input)
- $\mu_j,\ \sigma$: ground-truth location of keypoint $j$ on the heatmap grid, and the target width in heatmap pixels
- $v_j$: 1 if keypoint $j$ is labelled, 0 otherwise; $J = 17$ on COCO

Architectures differ in how they keep resolution. The **Hourglass** module pools down to $4\times4$ and upsamples back with skip connections; 8 hourglasses are stacked, each with its own loss (intermediate supervision). **SimpleBaselines** puts three stride-2 **Transposed convolution** layers on a ResNet. **HRNet** never drops the high-resolution stream: it adds lower-resolution branches in parallel (strides 4, 8, 16, 32 with 32, 64, 128, 256 channels in W32) and exchanges information between them at every stage.

## Decoding: argmax, the quarter offset, soft-argmax

At test time the peak of each heatmap is the keypoint. A plain argmax snaps to the grid, so at stride 4 it can be up to 2 input pixels off in each axis. The SimpleBaselines/HRNet code shifts the argmax by a quarter cell toward the higher neighbour, a cheap sub-pixel refinement, and averages the heatmaps of the image and its horizontal flip. DARK (Zhang et al., CVPR 2020) replaces the quarter shift with a Taylor expansion of the log-heatmap around the peak.

**Soft-argmax**, used in Integral Human Pose Regression (Sun, Xiao, Liang and Wei, ECCV 2018), takes the expectation of the coordinate under a softmax of the heatmap. It is differentiable, so a coordinate loss can be trained end to end, and it is sub-pixel by construction:

$$
\hat\mu_j=\sum_{p} p\,\frac{e^{\beta H_j(p)}}{\sum_{q}e^{\beta H_j(q)}}
$$

- $\beta$: inverse temperature: large $\beta$ approaches argmax (and its grid snapping); small $\beta$ lets the whole map vote, pulling the estimate toward the map centre

## Bottom-up grouping with part affinity fields

OpenPose predicts, besides part heatmaps, a **part affinity field** $L_c$ for each limb $c$: a 2D unit vector field that points from one joint to the other on pixels lying on the limb, and is zero elsewhere. For two candidate joints $d_1, d_2$, the association score is the line integral $E=\int_0^1 L_c(p(u))\cdot\frac{d_2-d_1}{\lVert d_2-d_1\rVert}\,du$ along the segment $p(u)$, approximated by sampling. Each limb type is then a bipartite matching problem between the candidates of its two joints, solved greedily limb by limb along a tree. The COCO model has 18 parts (COCO's 17 plus a neck) and 19 limbs, so $19 + 38$ output channels. OpenPose won the COCO 2016 keypoint challenge. **Associative Embedding** (Newell, Huang and Deng, NeurIPS 2017) instead predicts a scalar tag per keypoint and groups joints with similar tags.

## Scoring with OKS

COCO scores a predicted person against a ground-truth person with **Object Keypoint Similarity**, the keypoint analogue of IoU. Each visible keypoint contributes an unnormalised Gaussian of its error, scaled by the person's size and by a per-keypoint constant measured from annotator disagreement:

$$
\mathrm{OKS}=\frac{\sum_i \exp\!\big(-d_i^2/(2s^2k_i^2)\big)\,\delta(v_i>0)}{\sum_i \delta(v_i>0)},\qquad k_i=2\sigma_i
$$

- $d_i$: Euclidean distance between predicted and true keypoint $i$, in image pixels
- $s^2$: object scale: the area of the person's segmentation mask
- $\sigma_i$: 0.026 nose, 0.025 eyes, 0.035 ears, 0.079 shoulders, 0.072 elbows, 0.062 wrists, 0.107 hips, 0.087 knees, 0.089 ankles

A prediction is a true positive at threshold $t$ if OKS $\ge t$. COCO keypoint **AP** averages the AP over $t = 0.50, 0.55, \dots, 0.95$, exactly as box AP does over IoU thresholds. MPII instead uses PCKh@0.5: a joint is correct if it lies within half the head-segment length of the truth.

**Datasets and reference configurations**

- **COCO keypoints:** 17 keypoints per person; train2017 has about 57K images with 150K annotated people; AP over OKS 0.50:0.05:0.95
- **MPII Human Pose:** 16 joints; about 25K images with over 40K annotated people (Andriluka et al., CVPR 2014); metric PCKh@0.5
- **Stacked Hourglass:** 8 stacked hourglasses, input 256×256, output 64×64 heatmaps, a loss on every stack
- **HRNet-W32:** input 256×192, 64×48 heatmaps, 28.5M parameters, 7.1 GFLOPs; 74.4 AP on COCO val2017 (top-down, person boxes from a detector)
- **HRNet-W48:** input 384×288: 76.3 AP on COCO val2017, 75.5 AP on test-dev

> Figure: A wrist heatmap (16×12 cells at stride 4, Gaussian σ = 1.5 cells, small noise) decoded three ways, each error scored by the wrist's OKS term with s = 48 px. At the default (true x = 4.3, noise 0.05, β = 10) argmax snaps to (4, 4), 2.00 px off, OKS 0.945; the quarter shift lands at (4.25, 4.25), 0.63 px, OKS 0.994; soft-argmax 1.08 px, OKS 0.984. Set noise to 0 and soft-argmax is nearly exact (0.06 px). Drop β to 3 and the near-zero background votes too: 8.14 px, OKS 0.393; at β = 30 it snaps like argmax (1.98 px). Turn on the second person's wrist: argmax is unaffected, soft-argmax drifts to 2.92 px at β = 10 and 13.47 px (OKS 0.077) at β = 3.

## Worked example: one OKS

A person with mask area $s^2 = 9600$ px² has five labelled keypoints; the others have $v_i = 0$ and are ignored. With $s = 98.0$ px, the half-score distance $d$ where a keypoint's term falls to 0.5 is 6.0 px for the nose but 24.7 px for a hip.

Per-keypoint terms exp(−d² / 2s²k²) with s² = 9600; OKS = mean = 0.823

| Keypoint | k = 2σ | error d (px) | d² / 2s²k² | term |
| --- | --- | --- | --- | --- |
| nose | 0.052 | 3 | 0.1734 | 0.8408 |
| left shoulder | 0.158 | 8 | 0.1335 | 0.8750 |
| right wrist | 0.124 | 12 | 0.4878 | 0.6140 |
| left knee | 0.174 | 10 | 0.1720 | 0.8420 |
| right ankle | 0.178 | 6 | 0.0592 | 0.9425 |

OKS $= 4.1143 / 5 = 0.823$, so this prediction is a true positive at the thresholds 0.50 to 0.80 (7 of the 10) and a false positive at 0.85, 0.90 and 0.95. The 3 px nose error costs about as much as the 10 px knee error: the constants make the metric strict where annotators agree.

```python
import numpy as np

def target(mu, shape=(64, 48), sigma=2.0):
    """Gaussian heatmap target centred at mu = (x, y) in heatmap pixels."""
    ys, xs = np.mgrid[:shape[0], :shape[1]]
    return np.exp(-((xs - mu[0]) ** 2 + (ys - mu[1]) ** 2) / (2 * sigma ** 2))

def decode_argmax(h):
    y, x = np.unravel_index(h.argmax(), h.shape)
    dx = np.sign(h[y, x + 1] - h[y, x - 1]) if 0 < x < h.shape[1] - 1 else 0
    dy = np.sign(h[y + 1, x] - h[y - 1, x]) if 0 < y < h.shape[0] - 1 else 0
    return np.array([x + 0.25 * dx, y + 0.25 * dy])          # SimpleBaselines/HRNet refinement

def soft_argmax(h, beta=10.0):
    p = np.exp(beta * (h - h.max())); p /= p.sum()
    ys, xs = np.mgrid[:h.shape[0], :h.shape[1]]
    return np.array([(p * xs).sum(), (p * ys).sum()])

SIGMAS = np.array([.026, .025, .025, .035, .035, .079, .079, .072, .072,
                   .062, .062, .107, .107, .087, .087, .089, .089])   # COCO order

def oks(pred, gt, vis, area):
    d2 = ((pred - gt) ** 2).sum(-1)
    e = np.exp(-d2 / (2 * area * (2 * SIGMAS) ** 2))
    return e[vis > 0].mean()

h = target((20.3, 30.7))
print(decode_argmax(h), soft_argmax(h).round(2))   # [20.25 30.75] [20.45 30.74]: background votes pull soft-argmax toward the centre
```

Gaussian targets, argmax with the quarter offset, soft-argmax and OKS in NumPy (runs as is)

> **Flip test needs a joint swap:** Averaging the heatmaps of an image and its mirror image is standard, but the flipped network output must have left and right channels swapped (left wrist ↔ right wrist) and be shifted back by one pixel in some implementations. Forgetting the swap silently averages a left wrist with a right one.

> **From 2D to 3D:** A common 3D route is **lifting**: detect 2D keypoints first, then regress 3D joints from them. Martinez et al. (ICCV 2017) showed that a small fully connected network mapping 2D joints to 3D is a strong baseline on Human3.6M; VideoPose3D (Pavllo et al., CVPR 2019) adds temporal convolutions over a sequence of 2D poses. The metric is MPJPE, the mean per-joint position error in millimetres. Depth ambiguity remains: many 3D poses project to the same 2D skeleton.

**Strengths**

- Top-down: best accuracy (HRNet, ViTPose), every person at a normalised scale, any single-person model can be plugged in.
- Bottom-up: one forward pass for the whole crowd, no dependence on a person detector, runtime nearly constant in the number of people.

**Limitations**

- Top-down: cost grows linearly with people, missed or merged detections lose whole skeletons, crops of overlapping people contain other people's joints.
- Bottom-up: small people get few heatmap pixels, and grouping errors swap limbs between nearby people.

**Question:** A top-down model predicts 64×48 heatmaps for a 256×192 crop and is decoded with plain argmax. What is the worst-case quantisation error, in crop pixels, for a perfectly peaked heatmap?

- A. 0.5 px
- B. 2 px in each axis, about 2.8 px in total
- C. 4 px in each axis
- D. 8 px

**Answer:** B. 2 px in each axis, about 2.8 px in total — The stride is $256/64 = 4$. The argmax cell centre can be half a cell, 2 px, from the true peak in each axis, so up to $2\sqrt2\approx2.83$ px. Hence the quarter offset, DARK, or soft-argmax.

**Question:** Two predictions each place one keypoint 6 px off on the same person (s² = 9600). One error is on the nose, one on a hip. Which costs more OKS?

- A. The hip error, hips are larger
- B. They cost the same, OKS uses pixel distance
- C. The nose error, because the nose constant k is about 4× smaller
- D. Neither, OKS ignores single keypoints

**Answer:** C. The nose error, because the nose constant k is about 4× smaller — The term is $\exp(-d^2/2s^2k^2)$ with $k = 0.052$ for the nose and $0.214$ for hips. At $d = 6$ px the nose term is about 0.5 (its half-score distance is 6.0 px) while the hip term is about 0.96.

**Key points**

- Pose = locate 17 COCO keypoints per person; predict one heatmap per keypoint, not raw coordinates.
- Targets are Gaussians ($\sigma = 2$ on 64×48 for HRNet) trained with masked MSE; decode by argmax + quarter offset or by soft-argmax.
- Top-down (detector + per-crop network, HRNet-W48 76.3 AP on COCO val) is most accurate; bottom-up (OpenPose part affinity fields) scales to crowds.
- OKS $=\mathrm{mean}_i\exp(-d_i^2/2s^2k_i^2)$ over visible keypoints; COCO AP averages over OKS 0.50:0.05:0.95.
- 3D pose is often obtained by lifting 2D keypoints with a small network, evaluated by MPJPE in mm.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
