# Face detection

> Face detection finds every face's box and usually 5 landmarks: from Viola–Jones Haar cascades to MTCNN's 3-CNN cascade and single-shot RetinaFace (91.4 AP on WIDER FACE hard), whose landmarks align the crop for recognition.

Source: https://www.mlfoundry.in/learn/face-detection  
Section: Deep Learning › Deep Learning for Computer Vision  
Level: core · 7 min read · updated 2026-10-15

**Face detection:** is single-class **Object detection** specialised to faces: return a box, a confidence and usually five landmarks (two eyes, nose tip, two mouth corners) for every face in an image. The milestones are the Viola–Jones boosted cascade (Viola and Jones, CVPR 2001), MTCNN (Zhang, Zhang, Li and Qiao, IEEE Signal Processing Letters 2016) and RetinaFace (Deng, Guo, Ververas, Kotsia and Zafeiriou, CVPR 2020).

It is the first stage of nearly every face pipeline: recognition, landmark tracking, expression analysis, privacy blurring, camera autofocus. Faces span a huge range of scales in one photo, from a few pixels in a crowd to the whole frame, and appear blurred, occluded, in profile, or in poor light. Small faces dominate the difficulty.

## Viola–Jones, briefly

Viola and Jones slid a $24\times24$ window over an image pyramid and classified each window with Haar-like features: differences of sums over adjacent rectangles. The **integral image** makes any rectangle sum cost four lookups:

$$
ii(x,y)=\sum_{x'\le x,\;y'\le y} I(x',y'),\qquad \sum_{\text{rect}} I = ii(D)-ii(B)-ii(C)+ii(A)
$$

- $A, B, C, D$: the top-left, top-right, bottom-left and bottom-right corners of the rectangle (each just outside it on the top and left)

AdaBoost picked a small set of discriminative features, and an **attentional cascade** of boosted stages rejected most background windows after evaluating only a few features. It ran in real time on a 2001 desktop CPU and still ships in OpenCV. It works well for frontal, well-lit faces, and degrades with pose, occlusion and scale.

## MTCNN: a cascade of CNNs

MTCNN keeps the cascade idea with three small CNNs. **P-Net**, fully convolutional, scans an image pyramid with an effective $12\times12$ window and proposes candidates. **R-Net** re-scores $24\times24$ crops of the survivors and rejects most false positives. **O-Net** looks at $48\times48$ crops and outputs the final box and five landmarks. Non-maximum suppression runs between stages. Every stage is trained on three tasks at once, face classification, box regression and landmark regression, and keeps only the hardest 70% of samples in each mini-batch (online hard sample mining).

> Figure: Top: MTCNN's three-stage cascade on an image pyramid; each net predicts a face probability, 4 box offsets and 10 landmark coordinates. Bottom: RetinaFace is single-shot. A ResNet with a feature pyramid P2–P6 (strides 4–64) and a context module per level carries three square anchors per location, and each anchor predicts a face score, a box, five landmarks and, during training only, a dense 3D mesh. Right: the five landmarks define a similarity transform onto the 112×112 template used by ArcFace-style recognisers.

## RetinaFace

RetinaFace follows **RetinaNet**'s single-stage design, with face-specific choices. Feature pyramid levels P2 to P6 have strides 4 to 64; on a $640\times640$ input there are three square anchors per location with sizes $4\cdot\text{stride}\cdot2^{k/3}$, from 16 px on P2 to 406 px on P6. That is 102,300 anchors, 75% of them on P2, because small faces need dense, small anchors. The authors manually added five-landmark annotations to the WIDER FACE training faces. The loss for anchor $i$ is

$$
\mathcal L = \mathcal L_{\text{cls}}(p_i,p_i^*) + \lambda_1 p_i^*\,\mathcal L_{\text{box}}(t_i,t_i^*) + \lambda_2 p_i^*\,\mathcal L_{\text{pts}}(l_i,l_i^*) + \lambda_3 p_i^*\,\mathcal L_{\text{pixel}}
$$

- $p_i,\ p_i^*$: predicted face probability, and 1 for a positive anchor (IoU > 0.5 with a face), 0 for a negative (IoU < 0.3)
- $t_i,\ l_i$: box offsets (4) and landmark offsets (10), both encoded relative to the anchor centre and size; smooth-L1 losses
- $\mathcal L_{\text{pixel}}$: self-supervised dense regression: a mesh decoder renders a 3D face, compared with the input pixels
- $\lambda_{1,2,3}$: 0.25, 0.1 and 0.01

Negatives vastly outnumber positives, so training uses online hard example mining at a 3:1 negative-to-positive ratio rather than **Focal loss**. With a ResNet-152 backbone RetinaFace reported 91.4% AP on the WIDER FACE hard test set. A MobileNet-0.25 variant runs in real time on a single CPU core for VGA images.

**WIDER FACE and reference results**

- **WIDER FACE:** Yang, Luo, Loy and Tang (CVPR 2016): 32,203 images, 393,703 labelled faces, 61 event categories, split 40 / 10 / 50% into train / val / test
- **Easy / Medium / Hard:** subsets defined by how often a baseline proposal method (EdgeBox) found each face; Hard is dominated by small, occluded and blurred faces and contains the other two
- **Metric:** average precision of the precision-recall curve at IoU 0.5, reported separately for the three subsets
- **RetinaFace (ResNet-152):** 91.4% AP on the Hard test set, 1.1 points above the previous best at publication
- **Downstream effect:** ArcFace verification on IJB-C reached TAR 89.59% at FAR 1e-6 with RetinaFace alignment

> Figure: RetinaFace's 15 anchor shapes on a 640×640 input, matched to one square face. At the default 14.0 px face, centred at (320, 320) between stride-4 anchor centres, only the 16 px P2 anchors match: 4 positives, best IoU 0.597. Shift the face by 2 px onto the stride-4 grid and the best IoU rises to 0.766 (5 positives). At 10.0 px even a perfectly placed 16 px anchor gives IoU 100/256 = 0.391, so the face has no positive anchor at 0.5 and only 4 at S³FD's 0.35. At 6 px the best IoU is 0.141. A 76.8 px face gets 12 positives at 0.5 and 76 at 0.35.

## Alignment: the hand-off to recognition

Recognisers such as ArcFace expect a $112\times112$ crop in which the eyes, nose and mouth sit at fixed template positions. The detector's five landmarks $p_i$ are mapped to the template points $q_i$ by the least-squares **similarity** transform (scale, rotation, translation, 4 degrees of freedom), solved in closed form by Umeyama's method:

$$
\min_{s,\,R,\,t}\ \sum_{i=1}^{5}\big\lVert sRp_i+t-q_i\big\rVert^2
$$

- $s,\ R,\ t$: isotropic scale, $2\times2$ rotation, translation; a full affine would also shear the face, which distorts identity cues
- $q_i$: template in the 112 px crop: eyes (38.29, 51.70) and (73.53, 51.50), nose (56.03, 71.74), mouth (41.55, 92.37) and (70.73, 92.20)

**Worked example: aligning a tilted face**

1. Detected landmarks in the photo: eyes (250, 180) and (305, 195), nose (272, 222), mouth corners (238, 250) and (285, 262). The eye line is tilted 15.3° and the eyes are 57.0 px apart; in the template they are 35.2 px apart.
2. Centre both point sets: the photo mean is (270.0, 221.8), the template mean (56.03, 71.90).
3. An SVD of the $2\times2$ cross-covariance gives the rotation $-14.1°$ and the scale $s = 0.590$; then $t = \bar q - sR\bar p = (-130.31,\,-16.13)$.
4. The warp matrix is $M=\begin{bmatrix}0.5720&0.1438&-130.31\\-0.1438&0.5720&-16.13\end{bmatrix}$, used as `cv2.warpAffine(img, M, (112, 112))`.
5. The mapped landmarks miss the template by 0.87, 1.34, 1.17, 0.35 and 0.63 px (RMS 0.95 px). Five points cannot be matched exactly by 4 parameters, and the residual absorbs the detector's noise and the person's face shape.

```python
import numpy as np
import cv2

ARCFACE_112 = np.array([[38.2946, 51.6963], [73.5318, 51.5014], [56.0252, 71.7366],
                        [41.5493, 92.3655], [70.7299, 92.2041]])

def similarity(src, dst):
    """Umeyama: least-squares s, R, t with s*R@src + t ~ dst. Returns a 2x3 warp matrix."""
    ms, md = src.mean(0), dst.mean(0)
    S, D = src - ms, dst - md
    U, sig, Vt = np.linalg.svd(D.T @ S / len(src))
    d = np.array([1.0, np.sign(np.linalg.det(U) * np.linalg.det(Vt))])
    R = U @ np.diag(d) @ Vt
    s = (sig * d).sum() / ((S ** 2).sum() / len(src))
    return np.hstack([s * R, (md - s * R @ ms)[:, None]])

lmk = np.array([[250, 180], [305, 195], [272, 222], [238, 250], [285, 262]], float)
M = similarity(lmk, ARCFACE_112)
print(M.round(4))   # [[0.572 0.1438 -130.3133] [-0.1438 0.572 -16.131]]

img = cv2.imread("group.jpg")
haar = cv2.CascadeClassifier(cv2.data.haarcascades + "haarcascade_frontalface_default.xml")
boxes = haar.detectMultiScale(cv2.cvtColor(img, cv2.COLOR_BGR2GRAY), scaleFactor=1.1, minNeighbors=5)
crop = cv2.warpAffine(img, M, (112, 112), borderValue=0)   # landmarks would come from RetinaFace/MTCNN
```

Classic OpenCV detection, then five-point similarity alignment to the ArcFace template. The NumPy Umeyama part runs as is and reproduces the worked example; the OpenCV lines were not run here.

> **Tiny faces fall between anchors:** Two square boxes of sides $a < b$ centred at the same point have IoU $(a/b)^2$. A face smaller than $16/\sqrt2 \approx 11.3$ px therefore cannot reach IoU 0.5 with a 16 px anchor anywhere, gets no positive anchor and is never learned. S³FD (Zhang et al., ICCV 2017) lowered the threshold to 0.35 and forced extra matches for such faces; other detectors upsample the input. Test resolution matters as much as the model on WIDER FACE Hard.

> **Landmarks are cheap and pay twice:** Five landmarks add 10 outputs per anchor. They give an extra supervision signal for the box head, and they deliver alignment for free, so the recogniser needs no separate landmark model.

- **Recognition pipelines:** detect, align to 112×112, embed; see **Face recognition**
- **Cameras:** face-priority autofocus and exposure, smile and blink triggers
- **Privacy:** automatic face blurring in street-level imagery and published video
- **AR and video calls:** landmarks seed face trackers, filters and background effects

**Question:** A 10 px face sits exactly at the centre of a 16 px anchor. The positive threshold is IoU 0.5. What happens in training?

- A. It matches, the face is fully inside the anchor
- B. It does not match: IoU = 100/256 ≈ 0.39
- C. It matches a larger 20 px anchor instead
- D. It is ignored only at test time

**Answer:** B. It does not match: IoU = 100/256 ≈ 0.39 — The intersection is the face, $10^2 = 100$, and the union the anchor, $16^2 = 256$, so IoU is 0.39. Larger anchors give even less (0.25 for 20.16 px). Without a lowered threshold or forced matching, this face contributes no positive.

**Question:** Why do face pipelines align with a similarity transform rather than a full affine or homography fitted to the 5 landmarks?

- A. It is the only transform OpenCV supports
- B. A similarity keeps the face's shape (no shear or non-uniform scaling), so identity cues are not distorted
- C. Five points are too few for any other transform
- D. It makes the crop larger

**Answer:** B. A similarity keeps the face's shape (no shear or non-uniform scaling), so identity cues are not distorted — Rotation, uniform scale and translation normalise pose and size but preserve proportions such as eye distance relative to mouth width. An affine fit (6 parameters) could shear one person's face toward the template shape, removing the very differences the recogniser needs.

**Key points**

- Face detection = one-class detection plus 5 landmarks; small faces are the hard part.
- Viola–Jones (2001): Haar features on an integral image, AdaBoost, an attentional cascade; real time on a CPU.
- MTCNN (2016): P-Net 12×12 → R-Net 24×24 → O-Net 48×48 with NMS between, each predicting score, box and landmarks.
- RetinaFace (2020): single-shot, P2–P6, 102,300 anchors at 640², loss L_cls + 0.25 L_box + 0.1 L_pts + 0.01 L_pixel; 91.4 AP on WIDER FACE Hard.
- The five landmarks fix a similarity transform to the 112×112 template that recognisers expect.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
