Definition
Reranking is the second stage of a retrieve-then-rerank pipeline: a cheap retriever returns the top candidates from the whole corpus, and a more accurate but slower model rescores only those and reorders them. The standard reranker is a cross-encoder, a transformer that reads the query and one candidate as a single input. Its reference design is monoBERT (Nogueira and Cho, "Passage Re-ranking with BERT", 2019). ColBERT (Khattab and Zaharia, SIGIR 2020) is the main cheaper alternative, built on late interaction.
First-stage retrievers must be fast over millions of chunks, so they compare precomputed representations. BM25 uses an inverted index; a bi-encoder uses one vector per chunk in an ANN index (see Embedding models and Hybrid search (BM25 + dense)). Neither lets query words attend to document words, so near-misses slip in: right topic, wrong product, negated claim.
In a RAG system only the top 3–10 chunks reach the prompt. The reranker decides which ones. It sits between retrieval and prompt assembly (see RAG architecture). The first stage is tuned for recall (is the answer somewhere in the top 100?); the reranker is tuned for precision at the top.
Cross-encoders: monoBERT
- BERT's output at the [CLS] position for the input "[CLS] q [SEP] d [SEP]"
- a single linear layer to two classes (relevant, not relevant)
- labelled relevant passages and sampled non-relevant ones (MS MARCO in the paper)
Every layer attends across both texts, so the model can check that "battery" in the query matches the battery figure in the passage and not the charger's. The cost is that nothing can be cached: each (query, candidate) pair needs its own forward pass. Scoring a million chunks per query is out of the question; scoring 100 is routine.
Later variants change the head or the output, not the idea. monoT5 (Nogueira et al., 2020) reads the score from the probability of the token "true" vs "false". duoBERT compares two candidates at a time. LLM rerankers prompt a chat model to sort a list of passages; RankGPT (Sun et al., 2023) does this with a sliding window over the candidates.
Late interaction: ColBERT
- L2-normalised 128-d vector of query token i; the query is padded with [MASK] tokens to 32
- vector of document token j, computed offline and stored
- MaxSim: each query token keeps only its best-matching document token
ColBERT sits between the two designs. Documents are encoded once, as with a bi-encoder, but the comparison is per token, so it keeps much of the cross-encoder's precision. The price is storage: one vector per token instead of one per chunk. ColBERTv2 (Santhanam et al., 2022) cut that with residual compression and trained with distillation from a cross-encoder (see Knowledge distillation).
Reference points
- monoBERT (2019)
- BERT-large reranking the BM25 top 1000 on MS MARCO passage ranking (8.8M passages); query cut to 64 tokens, pair to 512; MRR@10 of 36.5 on the dev set, about twice BM25 alone
- ColBERT (2020)
- BERT-base, 128-d token vectors; the paper reports quality competitive with BERT rerankers at two orders of magnitude lower latency and four orders fewer FLOPs per query
- BEIR (Thakur et al., 2021)
- zero-shot benchmark of 18 retrieval datasets; BM25 + cross-encoder reranking and late interaction had the best average quality, at the highest compute cost
- RankGPT (2023)
- listwise LLM reranking: the model outputs a permutation of passage ids; windows of 20 passages, moved back to front in steps of 10
What to measure, and the cost of n
- relevance of the document at position j (0/1 here; graded in TREC)
- the same sum for the ideal ordering, so 1.0 is perfect
- cross-encoder time per (query, candidate) pair, amortised over a batch
The reranker can only reorder what the first stage returned, so recall@n of the first stage is a hard ceiling. A larger raises that ceiling, but latency grows linearly, and every extra candidate is one more distractor for an imperfect reranker to promote.
| n | recall@n | small: nDCG@10 | small: ms | large: nDCG@10 | large: ms |
|---|---|---|---|---|---|
| 20 | 0.598 | 0.6896 | 10 | 0.7027 | 80 |
| 50 | 0.794 | 0.8158 | 25 | 0.8500 | 200 |
| 100 | 0.874 | 0.8457 | 50 | 0.9028 | 400 |
| 300 | 0.972 | 0.8136 | 150 | 0.9526 | 1200 |
| 1000 | 1.000 | 0.7196 | 500 | 0.9305 | 4000 |
Both rerankers lift nDCG@10 far above the first stage's 0.4987. The small one is best near ; beyond that it loses. The large one keeps gaining until , at 24× the latency of the small one at . Tune on your own labelled queries; do not assume more candidates help.
| Document | Token vectors | Pooled cosine | MaxSim per query token | ColBERT score |
|---|---|---|---|---|
| A: "Pixel battery lasts…" | (0, .20, .98), (.96, .28, 0), (.28, .96, 0) | 0.9883 | 0.96, 0.96, 0.98 | 2.90 |
| B: generic phone review | (.60, .60, .53), (.53, .60, .60), (.60, .53, .60) | 1.0000 | 0.60, 0.60, 0.60 | 1.80 |
Mean pooling blurs A's three sharp matches into one direction, and B's vague tokens average to exactly the query centroid, so a single-vector model ranks B first. MaxSim rewards A for matching every query term well and ranks it first, as a reader would.
1import numpy as np2from sentence_transformers import CrossEncoder34ce = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2", max_length=512)56def rerank(query: str, candidates: list[dict], k: int = 5):7 """candidates: the first stage's top-n, e.g. 50-100 dicts with a 'text' field."""8 pairs = [(query, c["text"]) for c in candidates]9 scores = ce.predict(pairs, batch_size=32) # one forward pass per pair10 order = np.argsort(-scores)[:k]11 return [(candidates[i], float(scores[i])) for i in order]1213query = "battery life of the pixel 8"14top5 = rerank(query, first_stage(query, n=100)) # first_stage: your BM25 / dense retrieverUse when
- +Large precision gains on top of any first stage, with no re-indexing
- +Off-the-shelf cross-encoders transfer well across domains (BEIR)
- +A drop-in stage: swap or fine-tune the reranker without touching the index
Avoid when
- −Latency and GPU cost grow linearly with n, on every query
- −Cannot recover documents the first stage missed
- −Input length limits (512 tokens for BERT-style models) cap chunk size; ColBERT needs one stored vector per token
Check yourself
Why can a bi-encoder search 1M chunks per query while a cross-encoder cannot?
Check yourself
In the figure, the small cross-encoder scores nDCG@10 = 0.8457 at n = 100 and 0.7196 at n = 1000, although recall@n rises from 0.874 to 1.0. Why?
Remember
- ✓Retrieve-then-rerank: a cheap first stage for recall over the corpus, an expensive reranker for precision over its top n.
- ✓Cross-encoder (monoBERT): one forward pass over "[CLS] q [SEP] d [SEP]", score from ; nothing precomputable.
- ✓ColBERT: per-token vectors stored offline, score ; between bi- and cross-encoders in cost and quality.
- ✓Recall@n of the first stage is the ceiling; latency grows as n × per-pair time.
- ✓Tune n on labelled queries: in the figure the small reranker peaks at n = 100 (0.8457) and drops to 0.7196 at n = 1000.