Definition
Precision and recall describe a binary classifier at one decision threshold. Precision is the fraction of the examples it flagged that really are positive; recall is the fraction of the real positives it flagged. The ROC curve and the precision–recall curve show how these trade off as the threshold slides across every possible value.
At a glance
- Applies to
- any classifier that outputs a score or probability (logistic regression, trees, boosting, neural nets…)
- Point metrics
- precision, recall (TPR), FPR, specificity, F1 / F_β, accuracy, all from the confusion matrix at one threshold
- Curve metrics
- ROC AUC and average precision (area under the PR curve), which summarise all thresholds at once
- ROC AUC means
- Chance level
- ROC AUC whatever the class balance; PR baseline the positive rate
- Cost
- one sort: for the full sweep
The problem: accuracy lies on imbalanced data
A fraud model sees 10,000 card transactions, of which 100 (1%) are fraudulent. A "model" that answers not fraud every time is 99% accurate, and it catches no fraud at all. Accuracy averages over all examples, so when one class dominates it mostly measures how well you handle the majority. And it hides which kind of mistake you make: missing a fraud and blocking a good customer have very different costs.
The fix is to stop summarising with one number too early. First count the four kinds of outcome separately, then build metrics that look at the positive class directly.
The confusion matrix
| Actual predicted | Predicted positive | Predicted negative |
|---|---|---|
| Actually positive | TP: true positive (a hit) | FN: false negative (a miss, type II error) |
| Actually negative | FP: false positive (a false alarm, type I error) | TN: true negative (a correct rejection) |
Most classifiers do not output a label; they output a score , such as a probability. A label appears only once you pick a threshold and predict positive when . Every threshold gives a different confusion matrix. Raising moves examples from the predicted-positive column to the predicted-negative one: fewer false alarms, more misses.
Rates built from the matrix
- of everything flagged positive, the fraction that is positive (positive predictive value)
- of all actual positives, the fraction flagged; also called sensitivity or true positive rate
- false positive rate: the fraction of actual negatives wrongly flagged (fall-out)
- the fraction of actual negatives correctly left alone
Look at the denominators. Recall, FPR and specificity divide by a row of the matrix: they condition on the true class, so they do not depend on how many positives there are relative to negatives. Precision divides by a column: it mixes positives and negatives, so it does depend on the class balance. That one difference explains most of the ROC-versus-PR story below.
F1 and F-beta: one number from two
Precision and recall pull in opposite directions, so we often want one number that is high only when both are high. The arithmetic mean fails that test: a classifier that flags everything has recall 1 and precision (the positive rate), so with 1% positives its arithmetic mean is about 0.5 even though it is useless. The harmonic mean is dominated by the smaller of the two:
- harmonic mean of precision and recall, in [0, 1]
- TN never appears: F1 ignores correct rejections entirely
For the flag-everything classifier with : , which is correctly near zero. The harmonic mean is the right average for rates over different denominators: is the inverse of the average of and , i.e. "flagged per true hit" and "positives per true hit".
- how many times more you care about recall than precision; gives F1
- recall-heavy: screening, where a miss is expensive
- precision-heavy: alerts or recommendations, where a false alarm is expensive
The second form shows what does: a false negative is weighted times as heavily as a false positive. F scores still depend on a single threshold and they ignore TN, which is fine when the negatives are an uninteresting crowd (search results, fraud) and a problem when correct rejections matter.
The ROC curve: sweeping the threshold
Instead of choosing one threshold, try them all. Sort the examples by score from high to low and lower past each one in turn. Each time passes a positive, TP goes up by one and the point moves up by . Each time it passes a negative, FP goes up and the point moves right by . Plotting for every threshold traces the receiver operating characteristic curve from (flag nothing) to (flag everything).
- decision threshold; only the distinct scores matter, so a finite curve has at most n + 1 points
- the perfect corner: every positive caught, no false alarms
- a random scorer: TPR = FPR at every threshold
If several examples share a score, the threshold passes them together, and the curve takes one diagonal step. A curve below the diagonal means the scores are anti-correlated with the labels: flip them and you are above it.
AUC: the probability of ranking a pair correctly
The area under the ROC curve summarises every threshold. It has a clean meaning: pick one positive and one negative at random; AUC is the probability that the positive gets the higher score.
- score of a random positive and of a random negative
- number of (positive, negative) pairs
- a tied pair counts as half right, which matches the diagonal step in the curve
Why are the two the same? Walk the sweep again. Each step right (a negative passed) adds a column of width whose height is the TPR at that moment, i.e. the fraction of positives scored above that negative. Summing those columns counts, for each negative, the positives ranked above it: exactly the pair count divided by . A tie step is a diagonal, and its trapezoid gives the ½. This sum is the Mann–Whitney U statistic divided by , so AUC is a rank statistic: any monotonic transform of the scores leaves it unchanged, and it says nothing about whether the scores are calibrated probabilities.
The precision–recall curve and average precision
The same sweep can be plotted as . The curve starts at the top left (only the highest score flagged) and ends at recall 1 with precision equal to the positive rate , since flagging everything makes precision the share of positives. Unlike ROC it is not monotonic: precision drops each time a negative is passed and recovers when a positive is.
- thresholds in decreasing order (each distinct score)
- recall and precision at the -th threshold, with
- average precision: precision averaged over the recall levels, weighted by how much recall each step adds
Recall only grows when a positive is passed, so AP is simply the precision at the rank of each positive, averaged over the positives. It is the area under the step-shaped PR curve. Do not compute it with the trapezoid rule: linear interpolation between PR points is too optimistic, because precision does not change linearly between two thresholds.
Why PR is more informative under heavy imbalance
TPR and FPR each condition on one class, so if you add ten times more negatives from the same score distribution, the ROC curve does not move. Precision depends on the balance directly:
- negatives per positive; precision falls as it grows, even at the same ROC point
Take a ROC point with TPR 0.80 and FPR 0.05, which looks excellent. With balanced classes, precision is . With 100 negatives per positive it is : six of every seven alerts are false. The ROC curve cannot tell these two situations apart, because an FPR of 0.05 sounds small but 5% of a huge negative class is a flood. When positives are rare and what you care about is the quality of the flagged set, the PR curve (and AP) shows the problem; ROC AUC can look high while the model is unusable at any operating point.
Worked example: 12 scored examples
A model scores 12 examples; 4 are positive (, , ). One positive and one negative tie at 0.35.
| Example | A | B | C | D | E | F | G | H | I | J | K | L |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Score | 0.95 | 0.85 | 0.80 | 0.70 | 0.60 | 0.55 | 0.45 | 0.35 | 0.35 | 0.20 | 0.15 | 0.05 |
| Label | + | − | + | − | − | + | − | + | − | − | − | − |
| t | TP | FP | FN | TN | Precision | Recall (TPR) | FPR | F1 |
|---|---|---|---|---|---|---|---|---|
| 0.95 | 1 | 0 | 3 | 8 | 1.000 | 0.25 | 0.000 | 0.400 |
| 0.85 | 1 | 1 | 3 | 7 | 0.500 | 0.25 | 0.125 | 0.333 |
| 0.80 | 2 | 1 | 2 | 7 | 0.667 | 0.50 | 0.125 | 0.571 |
| 0.70 | 2 | 2 | 2 | 6 | 0.500 | 0.50 | 0.250 | 0.500 |
| 0.60 | 2 | 3 | 2 | 5 | 0.400 | 0.50 | 0.375 | 0.444 |
| 0.55 | 3 | 3 | 1 | 5 | 0.500 | 0.75 | 0.375 | 0.600 |
| 0.45 | 3 | 4 | 1 | 4 | 0.429 | 0.75 | 0.500 | 0.545 |
| 0.35 | 4 | 5 | 0 | 3 | 0.444 | 1.00 | 0.625 | 0.615 |
| 0.20 | 4 | 6 | 0 | 2 | 0.400 | 1.00 | 0.750 | 0.571 |
| 0.15 | 4 | 7 | 0 | 1 | 0.364 | 1.00 | 0.875 | 0.533 |
| 0.05 | 4 | 8 | 0 | 0 | 0.333 | 1.00 | 1.000 | 0.500 |
Accuracy is 0.75 at both and , while predicting "negative" for everything already scores . Accuracy hardly separates these thresholds, but precision and recall show what each one does.
AUC by trapezoids
- 1ROC points in order: . Vertical steps add no area.
- 2Horizontal steps (width each): heights , , , give .
- 3The tie at 0.35 is one diagonal step from to : trapezoid .
- 4The last three negatives are passed at TPR 1: .
- 5Total: .
AUC by counting pairs
- 1There are (positive, negative) pairs.
- 2A (0.95) beats all 8 negatives. C (0.80) beats 7 (all but B). F (0.55) beats 5 (G, I, J, K, L). H (0.35) beats J, K, L and ties with I: .
- 3, the same as the trapezoids.
Average precision
- 1Recall rises only at the thresholds where a positive is passed: (precision ), (), (), (, since the tie brings in I too).
- 2Each adds : (exactly ).
- 3Compare with the PR baseline . The trapezoid rule on the same points would give 0.617, a different and not well-founded number.
scikit-learn agrees: roc_auc_score = 0.734375, average_precision_score = 0.6528. Note that the thresholds with the best F1 (0.35, F1 = 0.615) and the best Youden J (a three-way tie) need not coincide: they encode different preferences.
Choosing a threshold
A curve is a menu; deployment needs one item from it. The threshold should come from what the errors cost, not from the default 0.5.
- cost of a false alarm and of a miss
- a calibrated probability; the right-hand rule only holds if the scores are calibrated
With calibrated probabilities, flag when the expected cost of ignoring it, , exceeds the expected cost of flagging it, . If a missed fraud costs 20 times a blocked good transaction, the threshold is , far below 0.5. On the ROC plane, minimising expected cost means finding where a line of slope first touches the curve.
- Youden's index: the vertical distance from the ROC point to the diagonal; maximise it when the two error rates matter equally
Youden's J is the cost rule with equal weight on the two rates (not counts). Other common rules: fix a minimum recall (a screening test must catch 95% of cases) and take the most precise threshold that achieves it; fix an alert budget (reviewers can check 200 cases a day) and take the top 200; or maximise for a chosen .
In code
1import numpy as np23def sweep(y, s):4 """(threshold, TP, FP) at every distinct score, high to low."""5 order = np.argsort(-s, kind="stable")6 y, s = y[order], s[order]7 tp, fp = np.cumsum(y), np.cumsum(1 - y)8 last = np.r_[s[1:] != s[:-1], True] # keep the last index of each tie group9 return s[last], tp[last], fp[last]1011def roc_auc_trapezoid(y, s):12 _, tp, fp = sweep(y, s)13 tpr = np.r_[0, tp / y.sum()]14 fpr = np.r_[0, fp / (1 - y).sum()]15 return np.sum(np.diff(fpr) * (tpr[1:] + tpr[:-1]) / 2)1617def roc_auc_pairs(y, s):18 pos, neg = s[y == 1], s[y == 0]19 diff = pos[:, None] - neg[None, :] # every (positive, negative) pair20 return ((diff > 0) + 0.5 * (diff == 0)).mean()2122def average_precision(y, s):23 _, tp, fp = sweep(y, s)24 precision, recall = tp / (tp + fp), tp / y.sum()25 return np.sum(np.diff(np.r_[0, recall]) * precision)2627s = np.array([.95, .85, .80, .70, .60, .55, .45, .35, .35, .20, .15, .05])28y = np.array([1, 0, 1, 0, 0, 1, 0, 1, 0, 0, 0, 0])29print(roc_auc_trapezoid(y, s), roc_auc_pairs(y, s)) # 0.734375 0.73437530print(average_precision(y, s)) # 0.65277...1from sklearn.metrics import (roc_auc_score, average_precision_score, roc_curve,2 precision_recall_curve, confusion_matrix, f1_score, fbeta_score)34print(roc_auc_score(y, s)) # 0.7343755print(average_precision_score(y, s)) # 0.652777...6fpr, tpr, thr = roc_curve(y, s)7prec, rec, thr_pr = precision_recall_curve(y, s) # prec/rec have one more entry than thr_pr89tn, fp, fn, tp = confusion_matrix(y, s >= 0.55).ravel() # sklearn order: TN, FP, FN, TP10print(tp, fp, fn, tn, f1_score(y, s >= 0.55), fbeta_score(y, s >= 0.55, beta=2)) # 3 3 1 5 0.6 0.681112# Pick the threshold on VALIDATION scores, then apply it unchanged to the test set.13# fpr, tpr, thr = roc_curve(y_val, s_val); t_star = thr[(tpr - fpr).argmax()] # Youden J14# y_pred_test = s_test >= t_starUse when
- +ROC / AUC: you want a threshold-free ranking measure that is comparable across datasets with different class balance, or both classes matter.
- +ROC / AUC: the deployment prevalence is unknown, or you will pick the operating point later.
- +PR / AP: positives are rare and you care about the quality of what you flag (fraud, retrieval, rare disease).
- +F_β at a fixed threshold: you already know the operating point and the relative cost of misses vs false alarms.
Avoid when
- −ROC / AUC can look excellent under heavy imbalance while precision at every useful threshold is poor; it also weights regions of the curve (high FPR) you would never deploy.
- −PR / AP changes with prevalence, so AP values from datasets with different positive rates cannot be compared directly; its baseline is π, not 0.5.
- −All curve metrics measure ranking only: they ignore calibration, and they cannot tell you which threshold to use.
- −F1 ignores true negatives and depends on the threshold chosen.
Where it's used
Medical screening
Recall (sensitivity) first: a missed cancer is far worse than a follow-up test. Fix sensitivity at, say, 95%, then report specificity; confirm positives with a second, more specific test.
Fraud and anomaly detection
Positives are well under 1%, so ROC AUC is misleadingly high. Use PR curves, precision at a fixed alert budget, and cost-weighted thresholds.
Search and recommendation
Precision@k and average precision (averaged over queries: mean average precision, mAP) measure whether the relevant items come first.
Spam filtering
Precision-heavy (F_0.5): sending a real email to spam costs more than letting one spam through.
Check yourself
A test flags 50 people; 40 of them are sick. In total 80 people are sick. What are precision and recall?
Check yourself
A model has ROC AUC = 0.9. What does that mean?
Check yourself
You keep the same model but the test set gains 10× more negatives drawn from the same distribution. What happens?
Check yourself
Why is F1 a harmonic rather than an arithmetic mean?
Remember
- ✓A score plus a threshold gives a confusion matrix: TP, FP, FN, TN. Accuracy hides which errors you make and is dominated by the majority class.
- ✓Precision divides by a column and depends on class balance; recall/TPR and FPR divide by rows and do not.
- ✓F1 is the harmonic mean of precision and recall; weights a miss times a false alarm.
- ✓The ROC curve sweeps the threshold over every distinct score; AUC = P(score⁺ > score⁻), with ties counting ½.
- ✓The PR curve plots precision vs recall; AP = Σ ΔRecall × precision, the mean precision at each positive's rank. Under heavy imbalance it reveals what ROC hides.
- ✓Choose the threshold from costs (calibrated p ≥ c_FP / (c_FP + c_FN)), Youden's J, a recall floor or an alert budget, on validation data, never on the test set.