Reranking

A reranker rescores the first stage's top-n candidates with a model that reads query and document together (cross-encoder) or token by token (ColBERT): more precision at a cost linear in n, capped by recall@n.

8 min read724 wordscore0 viewsupdated 2026-10-21

Definition

Reranking is the second stage of a retrieve-then-rerank pipeline: a cheap retriever returns the top nn candidates from the whole corpus, and a more accurate but slower model rescores only those nn and reorders them. The standard reranker is a cross-encoder, a transformer that reads the query and one candidate as a single input. Its reference design is monoBERT (Nogueira and Cho, "Passage Re-ranking with BERT", 2019). ColBERT (Khattab and Zaharia, SIGIR 2020) is the main cheaper alternative, built on late interaction.

First-stage retrievers must be fast over millions of chunks, so they compare precomputed representations. BM25 uses an inverted index; a bi-encoder uses one vector per chunk in an ANN index (see Embedding models and Hybrid search (BM25 + dense)). Neither lets query words attend to document words, so near-misses slip in: right topic, wrong product, negated claim.

In a RAG system only the top 3–10 chunks reach the prompt. The reranker decides which ones. It sits between retrieval and prompt assembly (see RAG architecture). The first stage is tuned for recall (is the answer somewhere in the top 100?); the reranker is tuned for precision at the top.

retrieve-then-rerank (one query)corpus1,000,000 chunksfirst stageBM25 / bi-encoderANN → top-100reranker100 (q, d) pairs→ 100 scorestop-5→ LLM promptrecall: is the answer in the 100?precision: is it in the top 5?bi-encoder (first stage)querydoc (offline)encoderencoderq ∈ R^768d ∈ R^768score = cos(q, d)q and d never see each other;d is encoded once, indexedcross-encoder (monoBERT)[CLS] battery life pixel[SEP] The Pixel 8 lastsabout a day … [SEP]BERT: full attentionacross q and d tokenslinear(h_[CLS]) → softmaxp(relevant | q, d) = 0.94one forward pass per pair;nothing can be precomputedlate interaction (ColBERT)pixelbatterylastsbattery0.000.960.280.96life0.200.280.960.96pixel0.980.000.000.98maxS = Σ max = 0.96 + 0.96 + 0.98 = 2.90one 128-d vector per token;doc vectors precomputed,only the max-sim runs onlineper query: bi-encoder = 1 encoder pass + ANN lookup · cross-encoder = n passes over (q + d) tokens ·ColBERT = 1 query pass + |q| × |d| dot products per candidate
fig.Top: one query passes through a cheap first stage (1M chunks down to 100) and a reranker (100 pairs scored, 5 kept). Bottom: three ways to score a pair. A bi-encoder compares two vectors computed separately. A cross-encoder runs one forward pass over the concatenated query and document and outputs a relevance probability. ColBERT keeps one vector per token and sums each query token's best match: here 0.96 + 0.96 + 0.98 = 2.90.

Cross-encoders: monoBERT

s(q,d)=softmax⁡ ⁣(W h[CLS])rel,L=−∑j∈Jposlog⁡sj−∑j∈Jneglog⁡(1−sj)s(q,d)=\operatorname{softmax}\!\big(W\,h_{[\mathrm{CLS}]}\big)_{\text{rel}},\qquad \mathcal L=-\sum_{j\in J_{\text{pos}}}\log s_j-\sum_{j\in J_{\text{neg}}}\log\big(1-s_j\big)
h[CLS]h_{[\mathrm{CLS}]}
BERT's output at the [CLS] position for the input "[CLS] q [SEP] d [SEP]"
WW
a single linear layer to two classes (relevant, not relevant)
Jpos, JnegJ_{\text{pos}},\,J_{\text{neg}}
labelled relevant passages and sampled non-relevant ones (MS MARCO in the paper)

Every layer attends across both texts, so the model can check that "battery" in the query matches the battery figure in the passage and not the charger's. The cost is that nothing can be cached: each (query, candidate) pair needs its own forward pass. Scoring a million chunks per query is out of the question; scoring 100 is routine.

Later variants change the head or the output, not the idea. monoT5 (Nogueira et al., 2020) reads the score from the probability of the token "true" vs "false". duoBERT compares two candidates at a time. LLM rerankers prompt a chat model to sort a list of passages; RankGPT (Sun et al., 2023) does this with a sliding window over the candidates.

Late interaction: ColBERT

Sq,d=∑i=1∣Eq∣  max⁡j=1,…,∣Ed∣  Eqi⋅Edj⊤S_{q,d}=\sum_{i=1}^{|E_q|}\;\max_{j=1,\dots,|E_d|}\;E_{q_i}\cdot E_{d_j}^{\top}
EqiE_{q_i}
L2-normalised 128-d vector of query token i; the query is padded with [MASK] tokens to 32
EdjE_{d_j}
vector of document token j, computed offline and stored
max⁡j\max_j
MaxSim: each query token keeps only its best-matching document token

ColBERT sits between the two designs. Documents are encoded once, as with a bi-encoder, but the comparison is per token, so it keeps much of the cross-encoder's precision. The price is storage: one vector per token instead of one per chunk. ColBERTv2 (Santhanam et al., 2022) cut that with residual compression and trained with distillation from a cross-encoder (see Knowledge distillation).

Reference points

monoBERT (2019)
BERT-large reranking the BM25 top 1000 on MS MARCO passage ranking (8.8M passages); query cut to 64 tokens, pair to 512; MRR@10 of 36.5 on the dev set, about twice BM25 alone
ColBERT (2020)
BERT-base, 128-d token vectors; the paper reports quality competitive with BERT rerankers at two orders of magnitude lower latency and four orders fewer FLOPs per query
BEIR (Thakur et al., 2021)
zero-shot benchmark of 18 retrieval datasets; BM25 + cross-encoder reranking and late interaction had the best average quality, at the highest compute cost
RankGPT (2023)
listwise LLM reranking: the model outputs a permutation of passage ids; windows of 20 passages, moved back to front in steps of 10

What to measure, and the cost of n

nDCG@10=1IDCG@10∑j=110reljlog⁡2(j+1),T(n)≈Tfirst+n tpair\mathrm{nDCG@}10=\frac{1}{\mathrm{IDCG@}10}\sum_{j=1}^{10}\frac{\mathrm{rel}_j}{\log_2(j+1)},\qquad T(n)\approx T_{\text{first}}+n\,t_{\text{pair}}
relj\mathrm{rel}_j
relevance of the document at position j (0/1 here; graded in TREC)
IDCG@10\mathrm{IDCG@}10
the same sum for the ideal ordering, so 1.0 is perfect
tpairt_{\text{pair}}
cross-encoder time per (query, candidate) pair, amortised over a batch

The reranker can only reorder what the first stage returned, so recall@n of the first stage is a hard ceiling. A larger nn raises that ceiling, but latency grows linearly, and every extra candidate is one more distractor for an imperfect reranker to promote.

retrieve top-n, rerank, score nDCG@10 (mean of 100 queries, 5 relevant of 1000 each)0.000.250.500.751.0010301003001000n = candidates reranked (log scale)perfect rerankerlarge cross-encodersmall cross-encoderfirst stage onlyn candidates100recall@n (in the pool)0.874ceiling nDCG@100.9152nDCG@100.8457latency (0.5 ms/pair)50 msquery 1: relevant docs sit at first-stage ranks 1, 11, 14, 19, 64; 5 of 5 inside top-100cells show each doc's first-stage rank; green = relevantfirst stage#1#2#3#4#5#6#7#8#9#10reranked#1#19#14#64#11#48#32#72#98#13this query: nDCG@10 0.339 → 1.000more candidates raise the ceiling but add distractors: a weak reranker peaks, then declines
reranker
fig.100 synthetic queries, each with 5 relevant documents among 1000. Every model scores s = μ·rel + N(0, 1) with its own noise: first stage μ = 2.5, small cross-encoder μ = 3, large μ = 4. Default (small, n = 100): recall@100 = 0.874, ceiling nDCG@10 = 0.9152, reranked 0.8457 against 0.4987 for the first stage, at 50 ms (assumed 0.5 ms per pair). Push n to 1000: the ceiling reaches 1.0, but the small reranker falls to 0.7196 because it promotes more distractors. The large one (assumed 4 ms per pair) peaks at 0.9526 at n = 300, costing 1200 ms.
nrecall@nsmall: nDCG@10small: mslarge: nDCG@10large: ms
200.5980.6896100.702780
500.7940.8158250.8500200
1000.8740.8457500.9028400
3000.9720.81361500.95261200
10001.0000.71965000.93054000
Worked example 1: the n trade-off, read off the figure (latency = n × per-pair cost)

Both rerankers lift nDCG@10 far above the first stage's 0.4987. The small one is best near n=100n=100; beyond that it loses. The large one keeps gaining until n=300n=300, at 24× the latency of the small one at n=100n=100. Tune nn on your own labelled queries; do not assume more candidates help.

DocumentToken vectorsPooled cosineMaxSim per query tokenColBERT score
A: "Pixel battery lasts…"(0, .20, .98), (.96, .28, 0), (.28, .96, 0)0.98830.96, 0.96, 0.982.90
B: generic phone review(.60, .60, .53), (.53, .60, .60), (.60, .53, .60)1.00000.60, 0.60, 0.601.80
Worked example 2: why per-token matching beats one pooled vector. Query tokens battery, life, pixel as unit vectors e₁, e₂, e₃

Mean pooling blurs A's three sharp matches into one direction, and B's vague tokens average to exactly the query centroid, so a single-vector model ranks B first. MaxSim rewards A for matching every query term well and ranks it first, as a reader would.

python
1import numpy as np
2from sentence_transformers import CrossEncoder
3
4ce = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2", max_length=512)
5
6def rerank(query: str, candidates: list[dict], k: int = 5):
7 """candidates: the first stage's top-n, e.g. 50-100 dicts with a 'text' field."""
8 pairs = [(query, c["text"]) for c in candidates]
9 scores = ce.predict(pairs, batch_size=32) # one forward pass per pair
10 order = np.argsort(-scores)[:k]
11 return [(candidates[i], float(scores[i])) for i in order]
12
13query = "battery life of the pixel 8"
14top5 = rerank(query, first_stage(query, n=100)) # first_stage: your BM25 / dense retriever
Reranking first-stage candidates with an open cross-encoder (sentence-transformers; not run here)

Use when

  • +Large precision gains on top of any first stage, with no re-indexing
  • +Off-the-shelf cross-encoders transfer well across domains (BEIR)
  • +A drop-in stage: swap or fine-tune the reranker without touching the index

Avoid when

  • −Latency and GPU cost grow linearly with n, on every query
  • −Cannot recover documents the first stage missed
  • −Input length limits (512 tokens for BERT-style models) cap chunk size; ColBERT needs one stored vector per token

Check yourself

Why can a bi-encoder search 1M chunks per query while a cross-encoder cannot?

Check yourself

In the figure, the small cross-encoder scores nDCG@10 = 0.8457 at n = 100 and 0.7196 at n = 1000, although recall@n rises from 0.874 to 1.0. Why?

Remember

  • ✓Retrieve-then-rerank: a cheap first stage for recall over the corpus, an expensive reranker for precision over its top n.
  • ✓Cross-encoder (monoBERT): one forward pass over "[CLS] q [SEP] d [SEP]", score from h[CLS]h_{[\mathrm{CLS}]}; nothing precomputable.
  • ✓ColBERT: per-token vectors stored offline, score ∑imax⁡jEqi⋅Edj\sum_i \max_j E_{q_i}\cdot E_{d_j}; between bi- and cross-encoders in cost and quality.
  • ✓Recall@n of the first stage is the ceiling; latency grows as n × per-pair time.
  • ✓Tune n on labelled queries: in the figure the small reranker peaks at n = 100 (0.8457) and drops to 0.7196 at n = 1000.
← back to Generative AI roadmap