Definition
Face detection is single-class Object detection specialised to faces: return a box, a confidence and usually five landmarks (two eyes, nose tip, two mouth corners) for every face in an image. The milestones are the Viola–Jones boosted cascade (Viola and Jones, CVPR 2001), MTCNN (Zhang, Zhang, Li and Qiao, IEEE Signal Processing Letters 2016) and RetinaFace (Deng, Guo, Ververas, Kotsia and Zafeiriou, CVPR 2020).
It is the first stage of nearly every face pipeline: recognition, landmark tracking, expression analysis, privacy blurring, camera autofocus. Faces span a huge range of scales in one photo, from a few pixels in a crowd to the whole frame, and appear blurred, occluded, in profile, or in poor light. Small faces dominate the difficulty.
Viola–Jones, briefly
Viola and Jones slid a window over an image pyramid and classified each window with Haar-like features: differences of sums over adjacent rectangles. The integral image makes any rectangle sum cost four lookups:
- the top-left, top-right, bottom-left and bottom-right corners of the rectangle (each just outside it on the top and left)
AdaBoost picked a small set of discriminative features, and an attentional cascade of boosted stages rejected most background windows after evaluating only a few features. It ran in real time on a 2001 desktop CPU and still ships in OpenCV. It works well for frontal, well-lit faces, and degrades with pose, occlusion and scale.
MTCNN: a cascade of CNNs
MTCNN keeps the cascade idea with three small CNNs. P-Net, fully convolutional, scans an image pyramid with an effective window and proposes candidates. R-Net re-scores crops of the survivors and rejects most false positives. O-Net looks at crops and outputs the final box and five landmarks. Non-maximum suppression runs between stages. Every stage is trained on three tasks at once, face classification, box regression and landmark regression, and keeps only the hardest 70% of samples in each mini-batch (online hard sample mining).
RetinaFace
RetinaFace follows RetinaNet's single-stage design, with face-specific choices. Feature pyramid levels P2 to P6 have strides 4 to 64; on a input there are three square anchors per location with sizes , from 16 px on P2 to 406 px on P6. That is 102,300 anchors, 75% of them on P2, because small faces need dense, small anchors. The authors manually added five-landmark annotations to the WIDER FACE training faces. The loss for anchor is
- predicted face probability, and 1 for a positive anchor (IoU > 0.5 with a face), 0 for a negative (IoU < 0.3)
- box offsets (4) and landmark offsets (10), both encoded relative to the anchor centre and size; smooth-L1 losses
- self-supervised dense regression: a mesh decoder renders a 3D face, compared with the input pixels
- 0.25, 0.1 and 0.01
Negatives vastly outnumber positives, so training uses online hard example mining at a 3:1 negative-to-positive ratio rather than Focal loss. With a ResNet-152 backbone RetinaFace reported 91.4% AP on the WIDER FACE hard test set. A MobileNet-0.25 variant runs in real time on a single CPU core for VGA images.
WIDER FACE and reference results
- WIDER FACE
- Yang, Luo, Loy and Tang (CVPR 2016): 32,203 images, 393,703 labelled faces, 61 event categories, split 40 / 10 / 50% into train / val / test
- Easy / Medium / Hard
- subsets defined by how often a baseline proposal method (EdgeBox) found each face; Hard is dominated by small, occluded and blurred faces and contains the other two
- Metric
- average precision of the precision-recall curve at IoU 0.5, reported separately for the three subsets
- RetinaFace (ResNet-152)
- 91.4% AP on the Hard test set, 1.1 points above the previous best at publication
- Downstream effect
- ArcFace verification on IJB-C reached TAR 89.59% at FAR 1e-6 with RetinaFace alignment
Alignment: the hand-off to recognition
Recognisers such as ArcFace expect a crop in which the eyes, nose and mouth sit at fixed template positions. The detector's five landmarks are mapped to the template points by the least-squares similarity transform (scale, rotation, translation, 4 degrees of freedom), solved in closed form by Umeyama's method:
- isotropic scale, rotation, translation; a full affine would also shear the face, which distorts identity cues
- template in the 112 px crop: eyes (38.29, 51.70) and (73.53, 51.50), nose (56.03, 71.74), mouth (41.55, 92.37) and (70.73, 92.20)
Worked example: aligning a tilted face
- 1Detected landmarks in the photo: eyes (250, 180) and (305, 195), nose (272, 222), mouth corners (238, 250) and (285, 262). The eye line is tilted 15.3° and the eyes are 57.0 px apart; in the template they are 35.2 px apart.
- 2Centre both point sets: the photo mean is (270.0, 221.8), the template mean (56.03, 71.90).
- 3An SVD of the cross-covariance gives the rotation and the scale ; then .
- 4The warp matrix is , used as
cv2.warpAffine(img, M, (112, 112)). - 5The mapped landmarks miss the template by 0.87, 1.34, 1.17, 0.35 and 0.63 px (RMS 0.95 px). Five points cannot be matched exactly by 4 parameters, and the residual absorbs the detector's noise and the person's face shape.
1import numpy as np2import cv234ARCFACE_112 = np.array([[38.2946, 51.6963], [73.5318, 51.5014], [56.0252, 71.7366],5 [41.5493, 92.3655], [70.7299, 92.2041]])67def similarity(src, dst):8 """Umeyama: least-squares s, R, t with s*R@src + t ~ dst. Returns a 2x3 warp matrix."""9 ms, md = src.mean(0), dst.mean(0)10 S, D = src - ms, dst - md11 U, sig, Vt = np.linalg.svd(D.T @ S / len(src))12 d = np.array([1.0, np.sign(np.linalg.det(U) * np.linalg.det(Vt))])13 R = U @ np.diag(d) @ Vt14 s = (sig * d).sum() / ((S ** 2).sum() / len(src))15 return np.hstack([s * R, (md - s * R @ ms)[:, None]])1617lmk = np.array([[250, 180], [305, 195], [272, 222], [238, 250], [285, 262]], float)18M = similarity(lmk, ARCFACE_112)19print(M.round(4)) # [[0.572 0.1438 -130.3133] [-0.1438 0.572 -16.131]]2021img = cv2.imread("group.jpg")22haar = cv2.CascadeClassifier(cv2.data.haarcascades + "haarcascade_frontalface_default.xml")23boxes = haar.detectMultiScale(cv2.cvtColor(img, cv2.COLOR_BGR2GRAY), scaleFactor=1.1, minNeighbors=5)24crop = cv2.warpAffine(img, M, (112, 112), borderValue=0) # landmarks would come from RetinaFace/MTCNNWhere it's used
Recognition pipelines
detect, align to 112×112, embed; see Face recognition
Cameras
face-priority autofocus and exposure, smile and blink triggers
Privacy
automatic face blurring in street-level imagery and published video
AR and video calls
landmarks seed face trackers, filters and background effects
Check yourself
A 10 px face sits exactly at the centre of a 16 px anchor. The positive threshold is IoU 0.5. What happens in training?
Check yourself
Why do face pipelines align with a similarity transform rather than a full affine or homography fitted to the 5 landmarks?
Remember
- ✓Face detection = one-class detection plus 5 landmarks; small faces are the hard part.
- ✓Viola–Jones (2001): Haar features on an integral image, AdaBoost, an attentional cascade; real time on a CPU.
- ✓MTCNN (2016): P-Net 12×12 → R-Net 24×24 → O-Net 48×48 with NMS between, each predicting score, box and landmarks.
- ✓RetinaFace (2020): single-shot, P2–P6, 102,300 anchors at 640², loss L_cls + 0.25 L_box + 0.1 L_pts + 0.01 L_pixel; 91.4 AP on WIDER FACE Hard.
- ✓The five landmarks fix a similarity transform to the 112×112 template that recognisers expect.