Precision, recall & ROC

Precision = TP/(TP+FP), recall = TP/(TP+FN); the ROC curve traces TPR vs FPR across thresholds.

17 min read1731 wordscore0 viewsupdated 2026-10-08

Definition

Precision and recall describe a binary classifier at one decision threshold. Precision is the fraction of the examples it flagged that really are positive; recall is the fraction of the real positives it flagged. The ROC curve and the precision–recall curve show how these trade off as the threshold slides across every possible value.

At a glance

Applies to
any classifier that outputs a score or probability (logistic regression, trees, boosting, neural nets…)
Point metrics
precision, recall (TPR), FPR, specificity, F1 / F_β, accuracy, all from the confusion matrix at one threshold
Curve metrics
ROC AUC and average precision (area under the PR curve), which summarise all thresholds at once
ROC AUC means
P(score of a random positive>score of a random negative)P(\text{score of a random positive} > \text{score of a random negative})
Chance level
ROC AUC =0.5=0.5 whatever the class balance; PR baseline == the positive rate π\pi
Cost
one sort: O(nlog⁡n)O(n\log n) for the full sweep

The problem: accuracy lies on imbalanced data

A fraud model sees 10,000 card transactions, of which 100 (1%) are fraudulent. A "model" that answers not fraud every time is 99% accurate, and it catches no fraud at all. Accuracy averages over all examples, so when one class dominates it mostly measures how well you handle the majority. And it hides which kind of mistake you make: missing a fraud and blocking a good customer have very different costs.

The fix is to stop summarising with one number too early. First count the four kinds of outcome separately, then build metrics that look at the positive class directly.

The confusion matrix

Actual predictedPredicted positivePredicted negative
Actually positiveTP: true positive (a hit)FN: false negative (a miss, type II error)
Actually negativeFP: false positive (a false alarm, type I error)TN: true negative (a correct rejection)
Every prediction lands in exactly one cell. P = TP + FN actual positives, N = FP + TN actual negatives.

Most classifiers do not output a label; they output a score s(x)s(x), such as a probability. A label appears only once you pick a threshold tt and predict positive when s(x)≥ts(x)\ge t. Every threshold gives a different confusion matrix. Raising tt moves examples from the predicted-positive column to the predicted-negative one: fewer false alarms, more misses.

Rates built from the matrix

precision=TPTP+FPrecall=TPR=TPTP+FN=TPP\text{precision}=\frac{TP}{TP+FP}\qquad \text{recall}=\text{TPR}=\frac{TP}{TP+FN}=\frac{TP}{P}
precision\text{precision}
of everything flagged positive, the fraction that is positive (positive predictive value)
recall\text{recall}
of all actual positives, the fraction flagged; also called sensitivity or true positive rate
FPR=FPFP+TN=FPNspecificity=TNR=TNN=1−FPR\text{FPR}=\frac{FP}{FP+TN}=\frac{FP}{N}\qquad \text{specificity}=\text{TNR}=\frac{TN}{N}=1-\text{FPR}
FPR\text{FPR}
false positive rate: the fraction of actual negatives wrongly flagged (fall-out)
specificity\text{specificity}
the fraction of actual negatives correctly left alone

Look at the denominators. Recall, FPR and specificity divide by a row of the matrix: they condition on the true class, so they do not depend on how many positives there are relative to negatives. Precision divides by a column: it mixes positives and negatives, so it does depend on the class balance. That one difference explains most of the ROC-versus-PR story below.

F1 and F-beta: one number from two

Precision and recall pull in opposite directions, so we often want one number that is high only when both are high. The arithmetic mean fails that test: a classifier that flags everything has recall 1 and precision π\pi (the positive rate), so with 1% positives its arithmetic mean is about 0.5 even though it is useless. The harmonic mean is dominated by the smaller of the two:

F1=21precision+1recall=2 precision⋅recallprecision+recall=2 TP2 TP+FP+FNF_1=\frac{2}{\frac{1}{\text{precision}}+\frac{1}{\text{recall}}}=\frac{2\,\text{precision}\cdot\text{recall}}{\text{precision}+\text{recall}}=\frac{2\,TP}{2\,TP+FP+FN}
F1F_1
harmonic mean of precision and recall, in [0, 1]
2TP+FP+FN2TP+FP+FN
TN never appears: F1 ignores correct rejections entirely

For the flag-everything classifier with π=0.01\pi=0.01: F1=2(0.01)(1)/(1.01)≈0.02F_1=2(0.01)(1)/(1.01)\approx0.02, which is correctly near zero. The harmonic mean is the right average for rates over different denominators: F1F_1 is the inverse of the average of 1/precision1/\text{precision} and 1/recall1/\text{recall}, i.e. "flagged per true hit" and "positives per true hit".

Fβ=(1+β2) precision⋅recallβ2 precision+recall=(1+β2) TP(1+β2) TP+β2FN+FPF_\beta=(1+\beta^2)\,\frac{\text{precision}\cdot\text{recall}}{\beta^2\,\text{precision}+\text{recall}}=\frac{(1+\beta^2)\,TP}{(1+\beta^2)\,TP+\beta^2 FN+FP}
β\beta
how many times more you care about recall than precision; β=1\beta=1 gives F1
F2F_2
recall-heavy: screening, where a miss is expensive
F0.5F_{0.5}
precision-heavy: alerts or recommendations, where a false alarm is expensive

The second form shows what β\beta does: a false negative is weighted β2\beta^2 times as heavily as a false positive. F scores still depend on a single threshold and they ignore TN, which is fine when the negatives are an uninteresting crowd (search results, fraud) and a problem when correct rejections matter.

The ROC curve: sweeping the threshold

Instead of choosing one threshold, try them all. Sort the examples by score from high to low and lower tt past each one in turn. Each time tt passes a positive, TP goes up by one and the point moves up by 1/P1/P. Each time it passes a negative, FP goes up and the point moves right by 1/N1/N. Plotting (FPR,TPR)(\text{FPR},\text{TPR}) for every threshold traces the receiver operating characteristic curve from (0,0)(0,0) (flag nothing) to (1,1)(1,1) (flag everything).

ROC={(FPR(t), TPR(t)):t∈R},TPR(t)=P(s≥t∣y=1),  FPR(t)=P(s≥t∣y=0)\text{ROC}=\big\{\big(\text{FPR}(t),\ \text{TPR}(t)\big) : t\in\mathbb{R}\big\},\qquad \text{TPR}(t)=P\big(s\ge t\mid y=1\big),\ \ \text{FPR}(t)=P\big(s\ge t\mid y=0\big)
tt
decision threshold; only the distinct scores matter, so a finite curve has at most n + 1 points
(0,1)(0,1)
the perfect corner: every positive caught, no false alarms
diagonal\text{diagonal}
a random scorer: TPR = FPR at every threshold

If several examples share a score, the threshold passes them together, and the curve takes one diagonal step. A curve below the diagonal means the scores are anti-correlated with the labels: flip them and you are above it.

AUC: the probability of ranking a pair correctly

The area under the ROC curve summarises every threshold. It has a clean meaning: pick one positive and one negative at random; AUC is the probability that the positive gets the higher score.

AUC=P(s+>s−)+12P(s+=s−)=1PN∑i: yi=1 ∑j: yj=0(1[si>sj]+121[si=sj])\text{AUC}=P\big(s^+>s^-\big)+\tfrac12P\big(s^+=s^-\big)=\frac{1}{PN}\sum_{i:\,y_i=1}\ \sum_{j:\,y_j=0}\Big(\mathbb 1[s_i>s_j]+\tfrac12\mathbb 1[s_i=s_j]\Big)
s+, s−s^+,\ s^-
score of a random positive and of a random negative
PNPN
number of (positive, negative) pairs
12\tfrac12
a tied pair counts as half right, which matches the diagonal step in the curve

Why are the two the same? Walk the sweep again. Each step right (a negative passed) adds a column of width 1/N1/N whose height is the TPR at that moment, i.e. the fraction of positives scored above that negative. Summing those columns counts, for each negative, the positives ranked above it: exactly the pair count divided by PNPN. A tie step is a diagonal, and its trapezoid gives the ½. This sum is the Mann–Whitney U statistic divided by PNPN, so AUC is a rank statistic: any monotonic transform of the scores leaves it unchanged, and it says nothing about whether the scores are calibrated probabilities.

The precision–recall curve and average precision

The same sweep can be plotted as (recall,precision)(\text{recall},\text{precision}). The curve starts at the top left (only the highest score flagged) and ends at recall 1 with precision equal to the positive rate π=P/(P+N)\pi=P/(P+N), since flagging everything makes precision the share of positives. Unlike ROC it is not monotonic: precision drops each time a negative is passed and recovers when a positive is.

AP=∑k(Rk−Rk−1) Pk\text{AP}=\sum_{k}\big(R_k-R_{k-1}\big)\,P_k
kk
thresholds in decreasing order (each distinct score)
Rk, PkR_k,\ P_k
recall and precision at the kk-th threshold, with R0=0R_0=0
AP\text{AP}
average precision: precision averaged over the recall levels, weighted by how much recall each step adds

Recall only grows when a positive is passed, so AP is simply the precision at the rank of each positive, averaged over the positives. It is the area under the step-shaped PR curve. Do not compute it with the trapezoid rule: linear interpolation between PR points is too optimistic, because precision does not change linearly between two thresholds.

Why PR is more informative under heavy imbalance

TPR and FPR each condition on one class, so if you add ten times more negatives from the same score distribution, the ROC curve does not move. Precision depends on the balance directly:

precision=TPR⋅PTPR⋅P+FPR⋅N=TPRTPR+FPR⋅N/P\text{precision}=\frac{\text{TPR}\cdot P}{\text{TPR}\cdot P+\text{FPR}\cdot N}=\frac{\text{TPR}}{\text{TPR}+\text{FPR}\cdot N/P}
N/PN/P
negatives per positive; precision falls as it grows, even at the same ROC point

Take a ROC point with TPR 0.80 and FPR 0.05, which looks excellent. With balanced classes, precision is 0.80/(0.80+0.05)=0.940.80/(0.80+0.05)=0.94. With 100 negatives per positive it is 0.80/(0.80+5)=0.140.80/(0.80+5)=0.14: six of every seven alerts are false. The ROC curve cannot tell these two situations apart, because an FPR of 0.05 sounds small but 5% of a huge negative class is a flood. When positives are rare and what you care about is the quality of the flagged set, the PR curve (and AP) shows the problem; ROC AUC can look high while the model is unusable at any operating point.

Worked example: 12 scored examples

A model scores 12 examples; 4 are positive (P=4P=4, N=8N=8, π=1/3\pi=1/3). One positive and one negative tie at 0.35.

ExampleABCDEFGHIJKL
Score0.950.850.800.700.600.550.450.350.350.200.150.05
Label+−+−−+−+−−−−
Scores sorted from high to low. Positives highlighted.
tTPFPFNTNPrecisionRecall (TPR)FPRF1
0.9510381.0000.250.0000.400
0.8511370.5000.250.1250.333
0.8021270.6670.500.1250.571
0.7022260.5000.500.2500.500
0.6023250.4000.500.3750.444
0.5533150.5000.750.3750.600
0.4534140.4290.750.5000.545
0.3545030.4441.000.6250.615
0.2046020.4001.000.7500.571
0.1547010.3641.000.8750.533
0.0548000.3331.001.0000.500
Predict positive when score ≥ t, for every distinct score. Highlighted: the three thresholds that tie for the best Youden J = 0.375.

Accuracy is 0.75 at both t=0.95t=0.95 and t=0.80t=0.80, while predicting "negative" for everything already scores 8/12=0.6678/12=0.667. Accuracy hardly separates these thresholds, but precision and recall show what each one does.

AUC by trapezoids

  1. 1ROC points in order: (0,0)→(0,0.25)→(0.125,0.25)→(0.125,0.5)→(0.25,0.5)→(0.375,0.5)→(0.375,0.75)→(0.5,0.75)→(0.625,1)→⋯→(1,1)(0,0)\to(0,0.25)\to(0.125,0.25)\to(0.125,0.5)\to(0.25,0.5)\to(0.375,0.5)\to(0.375,0.75)\to(0.5,0.75)\to(0.625,1)\to\dots\to(1,1). Vertical steps add no area.
  2. 2Horizontal steps (width 1/8=0.1251/8=0.125 each): heights 0.250.25, 0.50.5, 0.50.5, 0.750.75 give 0.125 (0.25+0.5+0.5+0.75)=0.250.125\,(0.25+0.5+0.5+0.75)=0.25.
  3. 3The tie at 0.35 is one diagonal step from (0.5,0.75)(0.5,0.75) to (0.625,1)(0.625,1): trapezoid 0.125⋅0.75+12=0.1093750.125\cdot\tfrac{0.75+1}{2}=0.109375.
  4. 4The last three negatives are passed at TPR 1: 3×0.125×1=0.3753\times0.125\times1=0.375.
  5. 5Total: 0.25+0.109375+0.375=0.7343750.25+0.109375+0.375=\mathbf{0.734375}.

AUC by counting pairs

  1. 1There are P⋅N=4⋅8=32P\cdot N=4\cdot8=32 (positive, negative) pairs.
  2. 2A (0.95) beats all 8 negatives. C (0.80) beats 7 (all but B). F (0.55) beats 5 (G, I, J, K, L). H (0.35) beats J, K, L and ties with I: 3+12=3.53+\tfrac12=3.5.
  3. 3AUC=(8+7+5+3.5)/32=23.5/32=0.734375\text{AUC}=(8+7+5+3.5)/32=23.5/32=\mathbf{0.734375}, the same as the trapezoids.

Average precision

  1. 1Recall rises only at the thresholds where a positive is passed: t=0.95t=0.95 (precision 1/11/1), 0.800.80 (2/32/3), 0.550.55 (3/63/6), 0.350.35 (4/94/9, since the tie brings in I too).
  2. 2Each adds ΔR=0.25\Delta R=0.25: AP=0.25 (1+0.667+0.5+0.444)=0.653\text{AP}=0.25\,(1+0.667+0.5+0.444)=\mathbf{0.653} (exactly 47/7247/72).
  3. 3Compare with the PR baseline π=0.333\pi=0.333. The trapezoid rule on the same points would give 0.617, a different and not well-founded number.

scikit-learn agrees: roc_auc_score = 0.734375, average_precision_score = 0.6528. Note that the thresholds with the best F1 (0.35, F1 = 0.615) and the best Youden J (a three-way tie) need not coincide: they encode different preferences.

12 scored examples · predict positive if score ≥ t · filled = predicted positivepositives ↑negatives ↓00.20.40.60.81score →ABCDEFGHIJKLt = 0.50ROC · AUC = 0.734000.50.511FPR →↑ TPR (recall)precision–recall · AP = 0.653chance = prevalence 0.33000.50.511recall →↑ precision
pred +pred −
actual +
TP
3
FN
1
actual −
FP
3
TN
5
precision 0.500recall / TPR 0.750FPR 0.375specificity 0.625F1 0.600Youden J 0.375accuracy 0.667AUC 0.734 · AP 0.653AUC by trapezoids 0.734375 = by pairs 0.734375
datajump to
threshold t = 0.50
fig.Drag the threshold. Positives sit above the axis, negatives below; filled dots are predicted positive. The confusion matrix and the operating point on both curves follow it. Starting at t = 0.5 on the worked example: TP 3, FP 3, precision 0.50, recall 0.75, FPR 0.375. Switch to synthetic scores and change the negatives per positive: the ROC curve and AUC stay almost fixed while the PR curve, its chance line and AP sink (at d = 2, AUC is 0.92 at every ratio while AP falls from 0.91 at 1:1 to 0.55 at 10:1). Lower the separation to see both collapse toward chance.

Choosing a threshold

A curve is a menu; deployment needs one item from it. The threshold should come from what the errors cost, not from the default 0.5.

t⋆=arg⁡min⁡t [cFP FP(t)+cFN FN(t)]⟹p(y=1∣x) ≥ cFPcFP+cFNt^\star=\arg\min_t\ \big[c_{FP}\,FP(t)+c_{FN}\,FN(t)\big]\quad\Longrightarrow\quad p(y=1\mid x)\ \ge\ \frac{c_{FP}}{c_{FP}+c_{FN}}
cFP, cFNc_{FP},\ c_{FN}
cost of a false alarm and of a miss
p(y=1∣x)p(y=1\mid x)
a calibrated probability; the right-hand rule only holds if the scores are calibrated

With calibrated probabilities, flag xx when the expected cost of ignoring it, cFN pc_{FN}\,p, exceeds the expected cost of flagging it, cFP(1−p)c_{FP}(1-p). If a missed fraud costs 20 times a blocked good transaction, the threshold is 1/21≈0.051/21\approx0.05, far below 0.5. On the ROC plane, minimising expected cost means finding where a line of slope cFP NcFN P\frac{c_{FP}\,N}{c_{FN}\,P} first touches the curve.

J(t)=TPR(t)−FPR(t)=sensitivity+specificity−1J(t)=\text{TPR}(t)-\text{FPR}(t)=\text{sensitivity}+\text{specificity}-1
JJ
Youden's index: the vertical distance from the ROC point to the diagonal; maximise it when the two error rates matter equally

Youden's J is the cost rule with equal weight on the two rates (not counts). Other common rules: fix a minimum recall (a screening test must catch 95% of cases) and take the most precise threshold that achieves it; fix an alert budget (reviewers can check 200 cases a day) and take the top 200; or maximise FβF_\beta for a chosen β\beta.

In code

python
1import numpy as np
2
3def sweep(y, s):
4 """(threshold, TP, FP) at every distinct score, high to low."""
5 order = np.argsort(-s, kind="stable")
6 y, s = y[order], s[order]
7 tp, fp = np.cumsum(y), np.cumsum(1 - y)
8 last = np.r_[s[1:] != s[:-1], True] # keep the last index of each tie group
9 return s[last], tp[last], fp[last]
10
11def roc_auc_trapezoid(y, s):
12 _, tp, fp = sweep(y, s)
13 tpr = np.r_[0, tp / y.sum()]
14 fpr = np.r_[0, fp / (1 - y).sum()]
15 return np.sum(np.diff(fpr) * (tpr[1:] + tpr[:-1]) / 2)
16
17def roc_auc_pairs(y, s):
18 pos, neg = s[y == 1], s[y == 0]
19 diff = pos[:, None] - neg[None, :] # every (positive, negative) pair
20 return ((diff > 0) + 0.5 * (diff == 0)).mean()
21
22def average_precision(y, s):
23 _, tp, fp = sweep(y, s)
24 precision, recall = tp / (tp + fp), tp / y.sum()
25 return np.sum(np.diff(np.r_[0, recall]) * precision)
26
27s = np.array([.95, .85, .80, .70, .60, .55, .45, .35, .35, .20, .15, .05])
28y = np.array([1, 0, 1, 0, 0, 1, 0, 1, 0, 0, 0, 0])
29print(roc_auc_trapezoid(y, s), roc_auc_pairs(y, s)) # 0.734375 0.734375
30print(average_precision(y, s)) # 0.65277...
From scratch: the sweep, AUC two ways, and average precision.
python
1from sklearn.metrics import (roc_auc_score, average_precision_score, roc_curve,
2 precision_recall_curve, confusion_matrix, f1_score, fbeta_score)
3
4print(roc_auc_score(y, s)) # 0.734375
5print(average_precision_score(y, s)) # 0.652777...
6fpr, tpr, thr = roc_curve(y, s)
7prec, rec, thr_pr = precision_recall_curve(y, s) # prec/rec have one more entry than thr_pr
8
9tn, fp, fn, tp = confusion_matrix(y, s >= 0.55).ravel() # sklearn order: TN, FP, FN, TP
10print(tp, fp, fn, tn, f1_score(y, s >= 0.55), fbeta_score(y, s >= 0.55, beta=2)) # 3 3 1 5 0.6 0.68
11
12# Pick the threshold on VALIDATION scores, then apply it unchanged to the test set.
13# fpr, tpr, thr = roc_curve(y_val, s_val); t_star = thr[(tpr - fpr).argmax()] # Youden J
14# y_pred_test = s_test >= t_star
The same with scikit-learn, plus a threshold chosen on validation data.

Use when

  • +ROC / AUC: you want a threshold-free ranking measure that is comparable across datasets with different class balance, or both classes matter.
  • +ROC / AUC: the deployment prevalence is unknown, or you will pick the operating point later.
  • +PR / AP: positives are rare and you care about the quality of what you flag (fraud, retrieval, rare disease).
  • +F_β at a fixed threshold: you already know the operating point and the relative cost of misses vs false alarms.

Avoid when

  • −ROC / AUC can look excellent under heavy imbalance while precision at every useful threshold is poor; it also weights regions of the curve (high FPR) you would never deploy.
  • −PR / AP changes with prevalence, so AP values from datasets with different positive rates cannot be compared directly; its baseline is π, not 0.5.
  • −All curve metrics measure ranking only: they ignore calibration, and they cannot tell you which threshold to use.
  • −F1 ignores true negatives and depends on the threshold chosen.

Where it's used

Medical screening

Recall (sensitivity) first: a missed cancer is far worse than a follow-up test. Fix sensitivity at, say, 95%, then report specificity; confirm positives with a second, more specific test.

Fraud and anomaly detection

Positives are well under 1%, so ROC AUC is misleadingly high. Use PR curves, precision at a fixed alert budget, and cost-weighted thresholds.

Search and recommendation

Precision@k and average precision (averaged over queries: mean average precision, mAP) measure whether the relevant items come first.

Spam filtering

Precision-heavy (F_0.5): sending a real email to spam costs more than letting one spam through.

Check yourself

A test flags 50 people; 40 of them are sick. In total 80 people are sick. What are precision and recall?

Check yourself

A model has ROC AUC = 0.9. What does that mean?

Check yourself

You keep the same model but the test set gains 10× more negatives drawn from the same distribution. What happens?

Check yourself

Why is F1 a harmonic rather than an arithmetic mean?

Remember

  • ✓A score plus a threshold gives a confusion matrix: TP, FP, FN, TN. Accuracy hides which errors you make and is dominated by the majority class.
  • ✓Precision =TP/(TP+FP)=TP/(TP+FP) divides by a column and depends on class balance; recall/TPR =TP/P=TP/P and FPR =FP/N=FP/N divide by rows and do not.
  • ✓F1 is the harmonic mean of precision and recall; FβF_\beta weights a miss β2\beta^2 times a false alarm.
  • ✓The ROC curve sweeps the threshold over every distinct score; AUC = P(score⁺ > score⁻), with ties counting ½.
  • ✓The PR curve plots precision vs recall; AP = Σ ΔRecall × precision, the mean precision at each positive's rank. Under heavy imbalance it reveals what ROC hides.
  • ✓Choose the threshold from costs (calibrated p ≥ c_FP / (c_FP + c_FN)), Youden's J, a recall floor or an alert budget, on validation data, never on the test set.
← back to ML Algorithms roadmap