Face recognition

Face recognition maps an aligned face to a unit embedding and thresholds cosine similarity; FaceNet trained it with a triplet loss, ArcFace with softmax plus an additive angular margin cos(θ + 0.5) at scale 64 (99.83% on LFW).

8 min read800 wordscore0 viewsupdated 2026-10-15

Definition

Face recognition is the task of deciding who a face belongs to by comparing learned embeddings. A network maps an aligned face crop to a vector f(x)f(x) on the unit sphere so that images of the same person are close in angle and images of different people are far apart. Key papers: DeepFace (Taigman et al., CVPR 2014), FaceNet (Schroff, Kalenichenko and Philbin, CVPR 2015), SphereFace (Liu et al., CVPR 2017), CosFace (Wang et al., CVPR 2018) and ArcFace (Deng, Guo, Xue and Zafeiriou, CVPR 2019).

There are two tasks. Verification (1:1) asks whether two faces show the same person: phone unlock, border gates. Identification (1:N) searches a gallery of NN enrolled people for the best match, and in the open-set case must also reject people who are not enrolled. Both reduce to a score cos⁡(f1,f2)\cos(f_1, f_2) and a threshold. The network never sees the test identities during training, so recognition is a metric-learning problem, not a fixed classifier.

The input is the output of Face detection: a 112×112112\times112 crop warped by a five-landmark similarity transform, so the eyes and mouth sit at fixed positions. Earlier systems used hand-crafted descriptors such as LBP, or Eigenfaces (PCA of pixel intensities). DeepFace, with 3D alignment and a deep CNN, reached 97.35% on LFW in 2014, close to the human level reported for that benchmark.

Train a classifier with an angular margin, deploy only the embeddingtraining (ArcFace, MS1MV2: 5.8M images, 85K identities)112×112alignedResNet-100conv stages7×7×512 mapBN-Drop-FC-BNx ∈ ℝ^512x ← x/‖x‖ArcFace headW: 512 × C, ‖W_j‖ = 1cos θ_j = W_jᵀ xtarget: cos(θ_y + 0.5)logits × s = 64softmax CEover C = 85KidentitiesFaceNet alternative: no classifier, triplet loss directly on a 128-d embeddingthe head (512 × 85K ≈ 44M weights) is thrown away after trainingdeployment: compare embeddings by cosine similaritysame net(frozen)detect, align,embedf1f2unit vectors, 512-dverification 1:1score = f1 · f2≥ τ → same personτ fixed so thatFAR = 1e-4 onimpostor pairsreport TAR thereidentification 1:Ngallery G of Nenrolled facesĵ = argmax f · g_jopen set: rejectif max < τadd a person: enrol an embedding, no retraining
fig.Training (top): an aligned 112×112 face passes through a ResNet-100 and a BN–dropout–FC–BN head to a 512-d embedding, L2-normalised. The ArcFace head holds one unit vector per training identity (85K for MS1MV2); it adds the margin 0.5 to the angle of the true class, scales by 64 and applies softmax cross-entropy. Deployment (bottom): the head is discarded. Two embeddings are compared by their dot product against a threshold τ chosen for a target false accept rate, or one embedding is searched against an enrolled gallery.

Metric learning with triplets: FaceNet

FaceNet trained a 128-d embedding directly with the Triplet loss max⁡(0, ∥f(a)−f(p)∥2−∥f(a)−f(n)∥2+α)\max(0,\ \lVert f(a)-f(p)\rVert^2-\lVert f(a)-f(n)\rVert^2+\alpha) with α=0.2\alpha = 0.2: an anchor must be closer to a positive (same person) than to a negative by a margin. Most random triplets are already satisfied, so FaceNet mined semi-hard negatives online in large batches, training on 100 to 200 million images of about 8 million identities. It reached 99.63% on LFW. Triplet training is sensitive to mining and needs very large data, which pushed the field toward classification losses.

Margin softmax: SphereFace, CosFace, ArcFace

Train a softmax classifier over the CC training identities, but remove everything except the angle. Normalise the embedding xx and each class weight WjW_j, so the logit is cos⁡θj\cos\theta_j; multiply by a scale ss; and make the true class yy harder to win by a margin inside ψ\psi:

L=−log⁡e s ψ(θy)e s ψ(θy)+∑j≠ye scos⁡θj,ψ(θ)={cos⁡(mθ)SphereFace, m=4cos⁡θ−mCosFace, m=0.35cos⁡(θ+m)ArcFace, m=0.5\mathcal L=-\log\frac{e^{\,s\,\psi(\theta_y)}}{e^{\,s\,\psi(\theta_y)}+\sum_{j\ne y}e^{\,s\cos\theta_j}},\qquad \psi(\theta)=\begin{cases}\cos(m\theta) & \text{SphereFace},\ m=4\\ \cos\theta-m & \text{CosFace},\ m=0.35\\ \cos(\theta+m) & \text{ArcFace},\ m=0.5\end{cases}
θj\theta_j
angle between the normalised embedding xx and the normalised class centre WjW_j: cos⁡θj=Wj⊤x\cos\theta_j=W_j^\top x
ss
scale (the feature radius), 64 for CosFace and ArcFace; SphereFace kept the unnormalised feature norm here
mm
margin: multiplicative on the angle (SphereFace), additive on the cosine (CosFace), additive on the angle (ArcFace)

Plain softmax separates two classes at the bisector θ1=θ2\theta_1=\theta_2. With ArcFace, class 1 wins only when θ1+m<θ2\theta_1+m<\theta_2, so the two decision boundaries are mm radians apart everywhere on the sphere: a constant geodesic margin. CosFace's margin is constant in cosine, not in angle; SphereFace's grows with the angle and needed a softmax-blending schedule to train. ArcFace also defines a combined form cos⁡(m1θ+m2)−m3\cos(m_1\theta+m_2)-m_3 covering all three. In code, cos⁡(θ+m)\cos(\theta+m) is used only while θ+m≤π\theta+m\le\pi, because beyond that it is no longer decreasing in θ\theta; insightface falls back to cos⁡θ−msin⁡m\cos\theta-m\sin m.

After training, the classifier weights (512 × 85K ≈ 44M numbers for MS1MV2) are discarded. Only the embedding network is deployed, and adding a new person means storing one embedding.

Benchmarks and reference results

LFW
Labeled Faces in the Wild (Huang et al., 2007): 13,233 images of 5,749 people; 6,000 verification pairs in 10 folds, mean accuracy; saturated above 99.8%
IJB-B / IJB-C
NIST benchmarks of templates (sets of stills and video frames) with large pose and quality variation; reported as TAR at FAR 1e-4 or 1e-5
FaceNet (2015)
128-d, triplet loss, 99.63% on LFW
ArcFace (2019)
ResNet-100, 512-d, s = 64, m = 0.5, trained on MS1MV2 (a cleaned MS-Celeb-1M, 5.8M images, 85K identities): 99.83% on LFW
With RetinaFace alignment
ArcFace verification on IJB-C reached TAR 89.59% at FAR 1e-6 (RetinaFace paper)

Verification metrics: TAR at FAR

same  ⟺  f1⊤f2≥τ,FAR(τ)=Pr⁡[f1⊤f2≥τ∣impostor],TAR(τ)=Pr⁡[f1⊤f2≥τ∣genuine]\text{same}\iff f_1^\top f_2\ge\tau,\qquad \mathrm{FAR}(\tau)=\Pr\big[f_1^\top f_2\ge\tau\mid\text{impostor}\big],\qquad \mathrm{TAR}(\tau)=\Pr\big[f_1^\top f_2\ge\tau\mid\text{genuine}\big]
τ\tau
threshold fixed so that the false accept rate on impostor pairs equals a target such as 10−410^{-4}
TAR\mathrm{TAR}
true accept rate at that threshold; LFW accuracy hides this, because its 3,000 impostor pairs cannot measure a FAR below 1/3000
ArcFace: target logit s·cos(θ + m), others s·cos θ · W1 ⟂ W2 (90°)W1W2x (class 1)gap 28.6°class 1: θ1 < 30.7°0°45°90°135°180°10-1-2ψ(θ) targetcos θ (dashed)● cos θ2 (W2)θ1 = 30° to W1, θ2 = 60° to W2 · ψ(θ1) = 0.5203, cos θ2 = 0.5000logits: 33.30 (target) vs 32.00 (W2) · p(class 1) = 0.7857loss −log p = 0.241 · same x under plain softmax (s = 64): 6.7e-11a margin keeps pushing x toward W1 until θ1 < 30.68°; the no-man's-land between classes is 28.65°two-class toy; SphereFace's ψ uses the monotonic extension (−1)^k cos(mθ) − 2k
loss
fig.Two class centres 90° apart and an embedding x of class 1 between them. At the default (ArcFace, m = 0.5, s = 64, θ1 = 30°) the target logit is 64·cos(58.65°) = 33.30 against 32.00 for W2, so p = 0.786 and the loss is 0.241, while plain softmax already gives 6.7e-11. Class 1 wins only for θ1 < 30.68°, leaving a 28.65° gap, exactly m. CosFace m = 0.35 gives almost the same boundary here (30.67°); SphereFace m = 4 moves it to 18.0° (gap 54.0°). At θ1 = 45° ArcFace's loss is 27.24 against softmax's ln 2 = 0.693. Lower s to 16 at 30° and the loss rises to 0.544.

Worked example: what the margin does to one sample

An embedding has cos⁡θy=0.5\cos\theta_y = 0.5 (θy=60°\theta_y = 60°) to its own class and cosines 0.4 and 0.2 to two others; s=64s = 64.

Lossψ(θ_y)target logitp(y)−log p(y)
softmax (normalised)cos 60° = 0.500032.000.998340.0017
CosFace, m = 0.350.5 − 0.35 = 0.15009.601.1e-716.00
ArcFace, m = 0.5cos(60° + 28.65°) = 0.02361.513.5e-1124.09
Logits are s·ψ(θ_y), 64 × 0.4 = 25.6 and 64 × 0.2 = 12.8

Normalised softmax treats this sample as solved, because 0.5 beats 0.4. Both margin losses say it is badly classified: the sample is pushed toward its centre until cos⁡(θy+m)\cos(\theta_y+m) beats every other cosine, which is what makes embeddings compact enough for thresholds at FAR 1e-6.

python
1import numpy as np
2
3def arcface_logits(x, W, y, s=64.0, m=0.5):
4 """x: (B, 512) embeddings, W: (512, C) class centres, y: (B,) labels -> (B, C) logits."""
5 x = x / np.linalg.norm(x, axis=1, keepdims=True)
6 W = W / np.linalg.norm(W, axis=0, keepdims=True)
7 cos = np.clip(x @ W, -1.0, 1.0)
8 th = np.arccos(cos[np.arange(len(y)), y])
9 target = np.where(th + m <= np.pi, np.cos(th + m), np.cos(th) - m * np.sin(m)) # insightface fallback
10 out = cos.copy()
11 out[np.arange(len(y)), y] = target
12 return s * out # feed to ordinary softmax cross-entropy
13
14def tar_at_far(genuine, impostor, far=1e-3):
15 """Threshold = the (1 - far) quantile of impostor scores; TAR = share of genuine pairs above it."""
16 tau = np.quantile(impostor, 1 - far)
17 return tau, (genuine > tau).mean()
18
19rng = np.random.default_rng(0)
20x, W = rng.normal(size=(4, 512)), rng.normal(size=(512, 10))
21print(arcface_logits(x, W, np.array([0, 3, 5, 9])).shape) # (4, 10)
22
23genuine = np.clip(rng.normal(0.45, 0.13, 5_000), -1, 1) # cosine of same-person pairs
24impostor = np.clip(rng.normal(0.02, 0.08, 500_000), -1, 1) # different people
25for far in (1e-2, 1e-3, 1e-4):
26 tau, tar = tar_at_far(genuine, impostor, far)
27 print(f"FAR {far:.0e}: tau = {tau:.3f}, TAR = {tar:.4f}")
28# FAR 1e-02: tau = 0.207, TAR = 0.9688
29# FAR 1e-03: tau = 0.269, TAR = 0.9194
30# FAR 1e-04: tau = 0.318, TAR = 0.8542
ArcFace logits and TAR at a fixed FAR in NumPy (runs as is). The scores are synthetic Gaussians standing in for real genuine and impostor pairs.

Check yourself

ArcFace normalises both x and W_j, so every logit is a cosine in [−1, 1]. Why multiply by s = 64?

Check yourself

With ArcFace margin m, where is the decision boundary between two classes whose centres are Δ apart, for an embedding lying between them?

Remember

  • ✓Verification (1:1) and identification (1:N) both threshold the cosine between L2-normalised embeddings of aligned 112×112 faces.
  • ✓FaceNet: 128-d embedding trained with the triplet loss (α=0.2\alpha = 0.2) and semi-hard mining; 99.63% on LFW.
  • ✓Margin softmax normalises features and weights and makes the true class harder: cos⁡(mθ)\cos(m\theta) SphereFace, cos⁡θ−m\cos\theta - m CosFace, cos⁡(θ+m)\cos(\theta + m) ArcFace with s=64s = 64, m=0.5m = 0.5.
  • ✓ArcFace's boundaries are m radians apart everywhere (a geodesic margin); the classifier head is dropped at deployment.
  • ✓Report TAR at a fixed FAR (IJB-B/C), not LFW accuracy; error rates differ across demographic groups, and data collection raises privacy law issues.
← back to Deep Learning roadmap