Face detection

Face detection finds every face's box and usually 5 landmarks: from Viola–Jones Haar cascades to MTCNN's 3-CNN cascade and single-shot RetinaFace (91.4 AP on WIDER FACE hard), whose landmarks align the crop for recognition.

7 min read806 wordscore0 viewsupdated 2026-10-15

Definition

Face detection is single-class Object detection specialised to faces: return a box, a confidence and usually five landmarks (two eyes, nose tip, two mouth corners) for every face in an image. The milestones are the Viola–Jones boosted cascade (Viola and Jones, CVPR 2001), MTCNN (Zhang, Zhang, Li and Qiao, IEEE Signal Processing Letters 2016) and RetinaFace (Deng, Guo, Ververas, Kotsia and Zafeiriou, CVPR 2020).

It is the first stage of nearly every face pipeline: recognition, landmark tracking, expression analysis, privacy blurring, camera autofocus. Faces span a huge range of scales in one photo, from a few pixels in a crowd to the whole frame, and appear blurred, occluded, in profile, or in poor light. Small faces dominate the difficulty.

Viola–Jones, briefly

Viola and Jones slid a 24×2424\times24 window over an image pyramid and classified each window with Haar-like features: differences of sums over adjacent rectangles. The integral image makes any rectangle sum cost four lookups:

ii(x,y)=∑x′≤x,  y′≤yI(x′,y′),∑rectI=ii(D)−ii(B)−ii(C)+ii(A)ii(x,y)=\sum_{x'\le x,\;y'\le y} I(x',y'),\qquad \sum_{\text{rect}} I = ii(D)-ii(B)-ii(C)+ii(A)
A,B,C,DA, B, C, D
the top-left, top-right, bottom-left and bottom-right corners of the rectangle (each just outside it on the top and left)

AdaBoost picked a small set of discriminative features, and an attentional cascade of boosted stages rejected most background windows after evaluating only a few features. It ran in real time on a 2001 desktop CPU and still ships in OpenCV. It works well for frontal, well-lit faces, and degrades with pose, occlusion and scale.

MTCNN: a cascade of CNNs

MTCNN keeps the cascade idea with three small CNNs. P-Net, fully convolutional, scans an image pyramid with an effective 12×1212\times12 window and proposes candidates. R-Net re-scores 24×2424\times24 crops of the survivors and rejects most false positives. O-Net looks at 48×4848\times48 crops and outputs the final box and five landmarks. Non-maximum suppression runs between stages. Every stage is trained on three tasks at once, face classification, box regression and landmark regression, and keeps only the hardest 70% of samples in each mini-batch (online hard sample mining).

Face detection → 5 landmarks → aligned crop for recognitionMTCNN (Zhang et al., 2016): a cascade of three small CNNspyramidP-Net12×12 windowfully conv→ proposalsNMSR-Net24×24 cropsrejectsmost false +NMSO-Net48×48 cropsfinal box+ landmarksevery stage predicts: face prob (2) · box offsets (4) · landmarks (5×2)loss weights det : box : landmark = 1 : 0.5 : 0.5 (O-Net 1 : 0.5 : 1)RetinaFace (Deng et al., 2020): single-shot, multi-level640×640ResNet + FPN + contextP2 s4 160² 16–25P3 s8 80² 32–51P4 s16 40² 64–102P5 s32 20² 128–203P6 s64 10² 256–406per anchor (×3)face score: 2box: 45 landmarks: 10mesh: train onlypos IoU≥0.5, neg<0.3NMS102,300 anchors at 640², 75% on P2 · size 4·stride·2^(k/3)loss = L_cls + 0.25 L_box + 0.1 L_pts + 0.01 L_meshalignmentdetected 5 pointssimilarity (s, θ, t)least-squares fit112×112 → recognition
fig.Top: MTCNN's three-stage cascade on an image pyramid; each net predicts a face probability, 4 box offsets and 10 landmark coordinates. Bottom: RetinaFace is single-shot. A ResNet with a feature pyramid P2–P6 (strides 4–64) and a context module per level carries three square anchors per location, and each anchor predicts a face score, a box, five landmarks and, during training only, a dense 3D mesh. Right: the five landmarks define a similarity transform onto the 112×112 template used by ArcFace-style recognisers.

RetinaFace

RetinaFace follows RetinaNet's single-stage design, with face-specific choices. Feature pyramid levels P2 to P6 have strides 4 to 64; on a 640×640640\times640 input there are three square anchors per location with sizes 4⋅stride⋅2k/34\cdot\text{stride}\cdot2^{k/3}, from 16 px on P2 to 406 px on P6. That is 102,300 anchors, 75% of them on P2, because small faces need dense, small anchors. The authors manually added five-landmark annotations to the WIDER FACE training faces. The loss for anchor ii is

L=Lcls(pi,pi∗)+λ1pi∗ Lbox(ti,ti∗)+λ2pi∗ Lpts(li,li∗)+λ3pi∗ Lpixel\mathcal L = \mathcal L_{\text{cls}}(p_i,p_i^*) + \lambda_1 p_i^*\,\mathcal L_{\text{box}}(t_i,t_i^*) + \lambda_2 p_i^*\,\mathcal L_{\text{pts}}(l_i,l_i^*) + \lambda_3 p_i^*\,\mathcal L_{\text{pixel}}
pi, pi∗p_i,\ p_i^*
predicted face probability, and 1 for a positive anchor (IoU > 0.5 with a face), 0 for a negative (IoU < 0.3)
ti, lit_i,\ l_i
box offsets (4) and landmark offsets (10), both encoded relative to the anchor centre and size; smooth-L1 losses
Lpixel\mathcal L_{\text{pixel}}
self-supervised dense regression: a mesh decoder renders a 3D face, compared with the input pixels
λ1,2,3\lambda_{1,2,3}
0.25, 0.1 and 0.01

Negatives vastly outnumber positives, so training uses online hard example mining at a 3:1 negative-to-positive ratio rather than Focal loss. With a ResNet-152 backbone RetinaFace reported 91.4% AP on the WIDER FACE hard test set. A MobileNet-0.25 variant runs in real time on a single CPU core for VGA images.

WIDER FACE and reference results

WIDER FACE
Yang, Luo, Loy and Tang (CVPR 2016): 32,203 images, 393,703 labelled faces, 61 event categories, split 40 / 10 / 50% into train / val / test
Easy / Medium / Hard
subsets defined by how often a baseline proposal method (EdgeBox) found each face; Hard is dominated by small, occluded and blurred faces and contains the other two
Metric
average precision of the precision-recall curve at IoU 0.5, reported separately for the three subsets
RetinaFace (ResNet-152)
91.4% AP on the Hard test set, 1.1 points above the previous best at publication
Downstream effect
ArcFace verification on IJB-C reached TAR 89.59% at FAR 1e-6 with RetinaFace alignment
face 14.0 px at (320, 320) on 640×640 · IoU ≥ 0.5 → positivewindow 56 px · dots: P2 centresface best anchor (16.0 px) positiveslevel anchorbest IoU with the faceIoU · # posP2 16.00.60 · 4P2 20.20.48 · 0P2 25.40.30 · 0P3 32.00.19 · 0P3 40.30.12 · 0P3 50.80.08 · 0P4 64.00.05 · 0P4 80.60.03 · 0P4 101.60.02 · 0P5 128.00.01 · 0P5 161.30.01 · 0P5 203.20.00 · 0P6 256.00.00 · 0P6 322.50.00 · 0P6 406.40.00 · 04 positive anchors of 102,300best IoU 0.597 on P2 · anchors 4·stride·2^(k/3)
positive IoU threshold
fig.RetinaFace's 15 anchor shapes on a 640×640 input, matched to one square face. At the default 14.0 px face, centred at (320, 320) between stride-4 anchor centres, only the 16 px P2 anchors match: 4 positives, best IoU 0.597. Shift the face by 2 px onto the stride-4 grid and the best IoU rises to 0.766 (5 positives). At 10.0 px even a perfectly placed 16 px anchor gives IoU 100/256 = 0.391, so the face has no positive anchor at 0.5 and only 4 at S³FD's 0.35. At 6 px the best IoU is 0.141. A 76.8 px face gets 12 positives at 0.5 and 76 at 0.35.

Alignment: the hand-off to recognition

Recognisers such as ArcFace expect a 112×112112\times112 crop in which the eyes, nose and mouth sit at fixed template positions. The detector's five landmarks pip_i are mapped to the template points qiq_i by the least-squares similarity transform (scale, rotation, translation, 4 degrees of freedom), solved in closed form by Umeyama's method:

min⁡s, R, t ∑i=15∥sRpi+t−qi∥2\min_{s,\,R,\,t}\ \sum_{i=1}^{5}\big\lVert sRp_i+t-q_i\big\rVert^2
s, R, ts,\ R,\ t
isotropic scale, 2×22\times2 rotation, translation; a full affine would also shear the face, which distorts identity cues
qiq_i
template in the 112 px crop: eyes (38.29, 51.70) and (73.53, 51.50), nose (56.03, 71.74), mouth (41.55, 92.37) and (70.73, 92.20)

Worked example: aligning a tilted face

  1. 1Detected landmarks in the photo: eyes (250, 180) and (305, 195), nose (272, 222), mouth corners (238, 250) and (285, 262). The eye line is tilted 15.3° and the eyes are 57.0 px apart; in the template they are 35.2 px apart.
  2. 2Centre both point sets: the photo mean is (270.0, 221.8), the template mean (56.03, 71.90).
  3. 3An SVD of the 2×22\times2 cross-covariance gives the rotation −14.1°-14.1° and the scale s=0.590s = 0.590; then t=qˉ−sRpˉ=(−130.31, −16.13)t = \bar q - sR\bar p = (-130.31,\,-16.13).
  4. 4The warp matrix is M=[0.57200.1438−130.31−0.14380.5720−16.13]M=\begin{bmatrix}0.5720&0.1438&-130.31\\-0.1438&0.5720&-16.13\end{bmatrix}, used as cv2.warpAffine(img, M, (112, 112)).
  5. 5The mapped landmarks miss the template by 0.87, 1.34, 1.17, 0.35 and 0.63 px (RMS 0.95 px). Five points cannot be matched exactly by 4 parameters, and the residual absorbs the detector's noise and the person's face shape.
python
1import numpy as np
2import cv2
3
4ARCFACE_112 = np.array([[38.2946, 51.6963], [73.5318, 51.5014], [56.0252, 71.7366],
5 [41.5493, 92.3655], [70.7299, 92.2041]])
6
7def similarity(src, dst):
8 """Umeyama: least-squares s, R, t with s*R@src + t ~ dst. Returns a 2x3 warp matrix."""
9 ms, md = src.mean(0), dst.mean(0)
10 S, D = src - ms, dst - md
11 U, sig, Vt = np.linalg.svd(D.T @ S / len(src))
12 d = np.array([1.0, np.sign(np.linalg.det(U) * np.linalg.det(Vt))])
13 R = U @ np.diag(d) @ Vt
14 s = (sig * d).sum() / ((S ** 2).sum() / len(src))
15 return np.hstack([s * R, (md - s * R @ ms)[:, None]])
16
17lmk = np.array([[250, 180], [305, 195], [272, 222], [238, 250], [285, 262]], float)
18M = similarity(lmk, ARCFACE_112)
19print(M.round(4)) # [[0.572 0.1438 -130.3133] [-0.1438 0.572 -16.131]]
20
21img = cv2.imread("group.jpg")
22haar = cv2.CascadeClassifier(cv2.data.haarcascades + "haarcascade_frontalface_default.xml")
23boxes = haar.detectMultiScale(cv2.cvtColor(img, cv2.COLOR_BGR2GRAY), scaleFactor=1.1, minNeighbors=5)
24crop = cv2.warpAffine(img, M, (112, 112), borderValue=0) # landmarks would come from RetinaFace/MTCNN
Classic OpenCV detection, then five-point similarity alignment to the ArcFace template. The NumPy Umeyama part runs as is and reproduces the worked example; the OpenCV lines were not run here.

Where it's used

Recognition pipelines

detect, align to 112×112, embed; see Face recognition

Cameras

face-priority autofocus and exposure, smile and blink triggers

Privacy

automatic face blurring in street-level imagery and published video

AR and video calls

landmarks seed face trackers, filters and background effects

Check yourself

A 10 px face sits exactly at the centre of a 16 px anchor. The positive threshold is IoU 0.5. What happens in training?

Check yourself

Why do face pipelines align with a similarity transform rather than a full affine or homography fitted to the 5 landmarks?

Remember

  • ✓Face detection = one-class detection plus 5 landmarks; small faces are the hard part.
  • ✓Viola–Jones (2001): Haar features on an integral image, AdaBoost, an attentional cascade; real time on a CPU.
  • ✓MTCNN (2016): P-Net 12×12 → R-Net 24×24 → O-Net 48×48 with NMS between, each predicting score, box and landmarks.
  • ✓RetinaFace (2020): single-shot, P2–P6, 102,300 anchors at 640², loss L_cls + 0.25 L_box + 0.1 L_pts + 0.01 L_pixel; 91.4 AP on WIDER FACE Hard.
  • ✓The five landmarks fix a similarity transform to the 112×112 template that recognisers expect.
← back to Deep Learning roadmap