# Reranking

> A reranker rescores the first stage's top-n candidates with a model that reads query and document together (cross-encoder) or token by token (ColBERT): more precision at a cost linear in n, capped by recall@n.

Source: https://www.mlfoundry.in/learn/reranking  
Section: Generative AI › Retrieval-Augmented Generation (RAG)  
Level: core · 8 min read · updated 2026-10-21

**Reranking:** is the second stage of a retrieve-then-rerank pipeline: a cheap retriever returns the top $n$ candidates from the whole corpus, and a more accurate but slower model rescores only those $n$ and reorders them. The standard reranker is a **cross-encoder**, a transformer that reads the query and one candidate as a single input. Its reference design is monoBERT (Nogueira and Cho, "Passage Re-ranking with BERT", 2019). ColBERT (Khattab and Zaharia, SIGIR 2020) is the main cheaper alternative, built on late interaction.

First-stage retrievers must be fast over millions of chunks, so they compare precomputed representations. BM25 uses an inverted index; a bi-encoder uses one vector per chunk in an ANN index (see **Embedding models** and **Hybrid search (BM25 + dense)**). Neither lets query words attend to document words, so near-misses slip in: right topic, wrong product, negated claim.

In a RAG system only the top 3–10 chunks reach the prompt. The reranker decides which ones. It sits between retrieval and prompt assembly (see **RAG architecture**). The first stage is tuned for **recall** (is the answer somewhere in the top 100?); the reranker is tuned for **precision** at the top.

> Figure: Top: one query passes through a cheap first stage (1M chunks down to 100) and a reranker (100 pairs scored, 5 kept). Bottom: three ways to score a pair. A bi-encoder compares two vectors computed separately. A cross-encoder runs one forward pass over the concatenated query and document and outputs a relevance probability. ColBERT keeps one vector per token and sums each query token's best match: here 0.96 + 0.96 + 0.98 = 2.90.

## Cross-encoders: monoBERT

$$
s(q,d)=\operatorname{softmax}\!\big(W\,h_{[\mathrm{CLS}]}\big)_{\text{rel}},\qquad \mathcal L=-\sum_{j\in J_{\text{pos}}}\log s_j-\sum_{j\in J_{\text{neg}}}\log\big(1-s_j\big)
$$

- $h_{[\mathrm{CLS}]}$: BERT's output at the [CLS] position for the input "[CLS] q [SEP] d [SEP]"
- $W$: a single linear layer to two classes (relevant, not relevant)
- $J_{\text{pos}},\,J_{\text{neg}}$: labelled relevant passages and sampled non-relevant ones (MS MARCO in the paper)

Every layer attends across both texts, so the model can check that "battery" in the query matches the battery figure in the passage and not the charger's. The cost is that nothing can be cached: each (query, candidate) pair needs its own forward pass. Scoring a million chunks per query is out of the question; scoring 100 is routine.

Later variants change the head or the output, not the idea. monoT5 (Nogueira et al., 2020) reads the score from the probability of the token "true" vs "false". duoBERT compares two candidates at a time. LLM rerankers prompt a chat model to sort a list of passages; RankGPT (Sun et al., 2023) does this with a sliding window over the candidates.

## Late interaction: ColBERT

$$
S_{q,d}=\sum_{i=1}^{|E_q|}\;\max_{j=1,\dots,|E_d|}\;E_{q_i}\cdot E_{d_j}^{\top}
$$

- $E_{q_i}$: L2-normalised 128-d vector of query token i; the query is padded with [MASK] tokens to 32
- $E_{d_j}$: vector of document token j, computed offline and stored
- $\max_j$: MaxSim: each query token keeps only its best-matching document token

ColBERT sits between the two designs. Documents are encoded once, as with a bi-encoder, but the comparison is per token, so it keeps much of the cross-encoder's precision. The price is storage: one vector per token instead of one per chunk. ColBERTv2 (Santhanam et al., 2022) cut that with residual compression and trained with distillation from a cross-encoder (see **Knowledge distillation**).

**Reference points**

- **monoBERT (2019):** BERT-large reranking the BM25 top 1000 on MS MARCO passage ranking (8.8M passages); query cut to 64 tokens, pair to 512; MRR@10 of 36.5 on the dev set, about twice BM25 alone
- **ColBERT (2020):** BERT-base, 128-d token vectors; the paper reports quality competitive with BERT rerankers at two orders of magnitude lower latency and four orders fewer FLOPs per query
- **BEIR (Thakur et al., 2021):** zero-shot benchmark of 18 retrieval datasets; BM25 + cross-encoder reranking and late interaction had the best average quality, at the highest compute cost
- **RankGPT (2023):** listwise LLM reranking: the model outputs a permutation of passage ids; windows of 20 passages, moved back to front in steps of 10

## What to measure, and the cost of n

$$
\mathrm{nDCG@}10=\frac{1}{\mathrm{IDCG@}10}\sum_{j=1}^{10}\frac{\mathrm{rel}_j}{\log_2(j+1)},\qquad T(n)\approx T_{\text{first}}+n\,t_{\text{pair}}
$$

- $\mathrm{rel}_j$: relevance of the document at position j (0/1 here; graded in TREC)
- $\mathrm{IDCG@}10$: the same sum for the ideal ordering, so 1.0 is perfect
- $t_{\text{pair}}$: cross-encoder time per (query, candidate) pair, amortised over a batch

The reranker can only reorder what the first stage returned, so **recall@n** of the first stage is a hard ceiling. A larger $n$ raises that ceiling, but latency grows linearly, and every extra candidate is one more distractor for an imperfect reranker to promote.

> Figure: 100 synthetic queries, each with 5 relevant documents among 1000. Every model scores s = μ·rel + N(0, 1) with its own noise: first stage μ = 2.5, small cross-encoder μ = 3, large μ = 4. Default (small, n = 100): recall@100 = 0.874, ceiling nDCG@10 = 0.9152, reranked 0.8457 against 0.4987 for the first stage, at 50 ms (assumed 0.5 ms per pair). Push n to 1000: the ceiling reaches 1.0, but the small reranker falls to 0.7196 because it promotes more distractors. The large one (assumed 4 ms per pair) peaks at 0.9526 at n = 300, costing 1200 ms.

Worked example 1: the n trade-off, read off the figure (latency = n × per-pair cost)

| n | recall@n | small: nDCG@10 | small: ms | large: nDCG@10 | large: ms |
| --- | --- | --- | --- | --- | --- |
| 20 | 0.598 | 0.6896 | 10 | 0.7027 | 80 |
| 50 | 0.794 | 0.8158 | 25 | 0.8500 | 200 |
| 100 | 0.874 | 0.8457 | 50 | 0.9028 | 400 |
| 300 | 0.972 | 0.8136 | 150 | 0.9526 | 1200 |
| 1000 | 1.000 | 0.7196 | 500 | 0.9305 | 4000 |

Both rerankers lift nDCG@10 far above the first stage's 0.4987. The small one is best near $n=100$; beyond that it loses. The large one keeps gaining until $n=300$, at 24× the latency of the small one at $n=100$. Tune $n$ on your own labelled queries; do not assume more candidates help.

Worked example 2: why per-token matching beats one pooled vector. Query tokens battery, life, pixel as unit vectors e₁, e₂, e₃

| Document | Token vectors | Pooled cosine | MaxSim per query token | ColBERT score |
| --- | --- | --- | --- | --- |
| A: "Pixel battery lasts…" | (0, .20, .98), (.96, .28, 0), (.28, .96, 0) | 0.9883 | 0.96, 0.96, 0.98 | 2.90 |
| B: generic phone review | (.60, .60, .53), (.53, .60, .60), (.60, .53, .60) | 1.0000 | 0.60, 0.60, 0.60 | 1.80 |

Mean pooling blurs A's three sharp matches into one direction, and B's vague tokens average to exactly the query centroid, so a single-vector model ranks B first. MaxSim rewards A for matching every query term well and ranks it first, as a reader would.

```python
import numpy as np
from sentence_transformers import CrossEncoder

ce = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2", max_length=512)

def rerank(query: str, candidates: list[dict], k: int = 5):
    """candidates: the first stage's top-n, e.g. 50-100 dicts with a 'text' field."""
    pairs = [(query, c["text"]) for c in candidates]
    scores = ce.predict(pairs, batch_size=32)          # one forward pass per pair
    order = np.argsort(-scores)[:k]
    return [(candidates[i], float(scores[i])) for i in order]

query = "battery life of the pixel 8"
top5 = rerank(query, first_stage(query, n=100))   # first_stage: your BM25 / dense retriever
```

Reranking first-stage candidates with an open cross-encoder (sentence-transformers; not run here)

> **Scores are for ordering, not thresholds:** A cross-encoder's output ranks candidates for one query well, but the same score can mean different things on different queries or after a model change. A fixed cut-off such as "drop chunks below 0.5" will drop good chunks on some queries and keep junk on others. If you need a cut-off, calibrate it on labelled queries. Also mind the input limit: monoBERT-style models truncate the pair at 512 tokens, so a long chunk's tail is never read.

> **Rerank to shorten the prompt:** A good reranker lets you send fewer chunks: 5 well-ordered chunks often beat 20 unordered ones, and cost fewer tokens. Models also use passages at the start of the context better than those in the middle (see **Long-context NLP**), so put the top-scored chunk first.

**Strengths**

- Large precision gains on top of any first stage, with no re-indexing
- Off-the-shelf cross-encoders transfer well across domains (BEIR)
- A drop-in stage: swap or fine-tune the reranker without touching the index

**Limitations**

- Latency and GPU cost grow linearly with n, on every query
- Cannot recover documents the first stage missed
- Input length limits (512 tokens for BERT-style models) cap chunk size; ColBERT needs one stored vector per token

**Question:** Why can a bi-encoder search 1M chunks per query while a cross-encoder cannot?

- A. Cross-encoders use larger embeddings
- B. Bi-encoder document vectors are computed once offline; a cross-encoder needs a new forward pass for every (query, chunk) pair
- C. Cross-encoders cannot run on GPUs
- D. Bi-encoders skip tokenisation

**Answer:** B. Bi-encoder document vectors are computed once offline; a cross-encoder needs a new forward pass for every (query, chunk) pair — The cross-encoder's score depends on attention between query and document tokens, so it exists only once both are known. A bi-encoder's document side does not depend on the query, so it is precomputed and indexed for ANN search.

**Question:** In the figure, the small cross-encoder scores nDCG@10 = 0.8457 at n = 100 and 0.7196 at n = 1000, although recall@n rises from 0.874 to 1.0. Why?

- A. nDCG is computed wrongly at large n
- B. The extra 900 candidates are mostly non-relevant, and a noisy reranker promotes some of them above relevant ones
- C. Latency limits cut off the ranking
- D. The first stage gets worse at large n

**Answer:** B. The extra 900 candidates are mostly non-relevant, and a noisy reranker promotes some of them above relevant ones — Only 12.6% of relevant documents are added between n = 100 and n = 1000, alongside about 900 distractors. With noise of 1 and a margin of 3, a few of those land above relevant ones. A stronger reranker (μ = 4) gains from the extra recall instead.

**Key points**

- Retrieve-then-rerank: a cheap first stage for recall over the corpus, an expensive reranker for precision over its top n.
- Cross-encoder (monoBERT): one forward pass over "[CLS] q [SEP] d [SEP]", score from $h_{[\mathrm{CLS}]}$; nothing precomputable.
- ColBERT: per-token vectors stored offline, score $\sum_i \max_j E_{q_i}\cdot E_{d_j}$; between bi- and cross-encoders in cost and quality.
- Recall@n of the first stage is the ceiling; latency grows as n × per-pair time.
- Tune n on labelled queries: in the figure the small reranker peaks at n = 100 (0.8457) and drops to 0.7196 at n = 1000.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
