Hybrid search (BM25 + dense)

Hybrid search runs BM25 over an inverted index and a dense bi-encoder over an ANN index, then fuses the two lists, by rank (RRF, k = 60) or by a tuned convex mix of normalised scores, so exact terms and paraphrases both reach the top.

8 min read798 wordscore0 viewsupdated 2026-10-21

Definition

Hybrid search retrieves with a lexical ranker and a dense ranker in parallel and merges their result lists into one. The lexical side is almost always BM25, from the Okapi system (Robertson et al., TREC-3, 1994; reviewed in Robertson and Zaragoza, "The Probabilistic Relevance Framework: BM25 and Beyond", 2009). The dense side is a bi-encoder such as DPR (Karpukhin et al., EMNLP 2020). The two standard merges are reciprocal rank fusion (Cormack, Clarke and Büttcher, SIGIR 2009) and a convex combination of normalised scores, analysed by Bruch, Gai and Ingber ("An Analysis of Fusion Functions for Hybrid Retrieval", ACM TOIS 2023).

The two rankers fail in opposite ways. BM25 matches surface terms only. It finds "E42", part numbers, names and rare jargon, but scores "internet is really slow" against "pages load slowly" as zero, because "slow" and "slowly" are different terms without stemming. A bi-encoder (see Embedding models) matches meaning and handles that paraphrase, but compresses a chunk into one vector, where error codes E42 and E50 land almost on top of each other.

In a RAG system hybrid search is the first stage: it must get the right chunks into the top 50–100 candidates that the Reranking stage then orders (see RAG architecture). Both indexes are built from the same chunks (see Document chunking), and many Vector databases and search engines now host both.

The two scores

BM25(q,d)=∑t∈qln⁡ ⁣(1+N−nt+0.5nt+0.5) ft,d (k1+1)ft,d+k1(1−b+b ∣d∣/∣d∣‾)\text{BM25}(q,d)=\sum_{t\in q}\ln\!\Big(1+\frac{N-n_t+0.5}{n_t+0.5}\Big)\,\frac{f_{t,d}\,(k_1+1)}{f_{t,d}+k_1\big(1-b+b\,|d|/\overline{|d|}\big)}
nt,  Nn_t,\;N
chunks containing term tt, total chunks; rare terms get large idf (Lucene's always-positive form)
ft,df_{t,d}
occurrences of tt in chunk dd; the fraction saturates towards k1+1k_1+1
k1k_1
term-frequency saturation; Lucene's default is 1.2, with 0 meaning presence only
bb
length normalisation, 0 (none) to 1 (full); default 0.75; ∣d∣‾\overline{|d|} is the mean chunk length

The dense score is a cosine sdense(q,d)=eq⊤eds_{\text{dense}}(q,d)=\mathbf e_q^\top\mathbf e_d between unit-normalised embeddings, found by approximate search (see Approximate nearest neighbor search (HNSW, IVF, PQ)). BM25 is unbounded and depends on query length; cosine lies in [−1,1][-1,1] and real models crowd most scores into a narrow band. Adding the raw numbers would let whichever scale is larger dominate.

hybrid retrieval: two indexes, two ranked lists, one fused listqueryE42 errorSPARSE LANE (exact terms)DENSE LANE (meaning)tokenise[e42][error]lowercaseinverted indexe42 → d1 d10 idf 1.792error → d1 d2 d10 d13 idf 1.204BM25 top 4d102.931d12.482d131.254d21.178encoderq = unit vecerror: 1.00rest: 0ANN index (HNSW)one vector per doccosine = q · dd13 E50 doc: 0.857("error" only)cosine top 4d101.000d130.857d20.651d10.600RRF, k = 60d10 2/61.0328d13 1/63+1/62.0320d1 1/62+1/64.0318d2 1/64+1/63.0315ranks onlyreranker on top 50then top k to promptBM25: rare "e42" (in 2 of 14 docs) outweighs common "error" (4 of 14), so gold d1 ranks 2nd.Dense: E42 and E50 look alike, so d13 (E50) beats gold d1. Fusion keeps both golds in the top 3.
fig.The query "E42 error" against 14 help-centre chunks. The sparse lane looks up two posting lists; "e42" occurs in 2 chunks (idf 1.792) and "error" in 4 (idf 1.204), so BM25 ranks the two gold chunks d10 and d1 first and second. The dense lane sees only an "error" concept, so the E50 chunk d13 scores 0.857 and beats d1 (0.600). RRF with k = 60 adds 1/(60 + rank) per list: d10 gets 2/61 = 0.0328, and both golds stay in the fused top 3.

Fusion

RRF⁡(d)=∑r∈{bm25, dense}1k+rank⁡r(d),scvx(d)=α s^dense(d)+(1−α) s^bm25(d)\operatorname{RRF}(d)=\sum_{r\in\{\text{bm25},\,\text{dense}\}}\frac{1}{k+\operatorname{rank}_r(d)},\qquad s_{\text{cvx}}(d)=\alpha\,\hat s_{\text{dense}}(d)+(1-\alpha)\,\hat s_{\text{bm25}}(d)
rank⁡r(d)\operatorname{rank}_r(d)
1-based position in ranker r's list; a chunk the ranker did not return adds 0
kk
60 in Cormack et al.; larger k flattens the gap between rank 1 and rank 10
s^\hat s
a score min-max normalised over the candidate list: (s−smin⁡)/(smax⁡−smin⁡)(s-s_{\min})/(s_{\max}-s_{\min})
α\alpha
weight of the dense side, tuned on labelled queries; α=0\alpha=0 is pure BM25, α=1\alpha=1 pure dense

RRF needs no score calibration and no training data, which is why it is the default in many engines. It throws away how far apart scores are: a chunk that wins one list by a wide margin counts the same as one that wins narrowly. The convex combination keeps that information but needs normalisation and one tuned weight. Bruch et al. found that a tuned convex combination beat RRF both in and out of domain, needed only a small labelled set to tune α\alpha, and that RRF was more sensitive to its kk than usually assumed.

Reference points

RRF (2009)
Cormack et al. fused TREC and LETOR runs with k = 60; the title states the result: RRF outperformed Condorcet fusion and individual learning-to-rank methods
BEIR (2021)
Thakur et al., 18 zero-shot retrieval datasets: BM25 was a strong baseline, and several dense retrievers trained on MS MARCO fell below it out of domain; the motivation for keeping a lexical lane
DPR vs BM25
Natural Questions top-20 retrieval accuracy 78.4% for DPR vs 59.1% for BM25 (Karpukhin et al., 2020): dense wins in domain, on the data it was trained for
Learned sparse
SPLADE (Formal, Piwowarski and Clinchant, SIGIR 2021) predicts weights for expansion terms with BERT and still serves from an inverted index; it can replace BM25 in the lexical lane
Engine knobs
as of 2024: Lucene/Elasticsearch BM25 defaults k1 = 1.2, b = 0.75 and an RRF retriever with rank constant 60; Weaviate's hybrid query takes alpha (1 = pure vector, 0 = pure keyword)
query: "router stuck during firmware update" (relevant: d9, d1)14 help-centre docs; green = relevant; score 0 under BM25 = not retrievedBM25 (k1 1.2, b 0.75)1.d95.5832.d53.6663.d13.6514.d110.9305.d120.871dense (cosine)1.d50.9292.d10.8793.d90.5934.d20.3035.d140.293RRF (k = 60)1.d50.03252.d90.03233.d10.03204.d120.03055.d60.0301fused top 3 (what the prompt would receive):d5: the router firmware update adds new wifi featuresd9: the router lights blink orange during a firmware update do not unplug itd1: error E42 means the firmware update failed restart the router and retry the updateconvex fusion over all 5 queries vs αα = 0: pure BM25, α = 1: pure dense0.00.51.00.00.51.0RRF MRRrecall@3MRRmean over 5 queriesmoderecall@3MRRBM250.901.00dense0.900.80RRF k=601.000.80convex α=0.501.000.90BM25 misses paraphrase,dense blurs error codes
query
fusion
fig.Real BM25, a toy dense encoder (a word-to-concept lexicon, so paraphrases match and error codes blur) and both fusions over 14 chunks and 5 labelled queries. Defaults (k1 = 1.2, b = 0.75, RRF k = 60): recall@3 and MRR are 0.90 / 1.00 for BM25, 0.90 / 0.80 for dense, 1.00 / 0.80 for RRF and 1.00 / 0.90 for convex at α = 0.5. Any α from 0.05 to 0.45 gives 1.00 / 1.00. RRF scores stay at 1.00 / 0.80 for every k from 1 to 100. Set b = 0 and BM25's recall@3 falls to 0.80: without length normalisation the short chunk d11 ("wipe the router") drops out of the top 3 for "factory reset my router".

Worked example: where RRF and α disagree

Query "router stuck during firmware update"; gold chunks d9 (the lights blink during an update) and d1 (E42, update failed). N=14N=14, mean length 6.6436.643 terms after stop-word removal.

BM25 for d9 (8 terms, k1 = 1.2, b = 0.75)

  1. 1Length factor 1−b+b ∣d∣/∣d∣‾=0.25+0.75⋅8/6.643=1.1531-b+b\,|d|/\overline{|d|}=0.25+0.75\cdot8/6.643=1.153, so each term seen once gets f(k1+1)/(f+k1⋅1.153)=2.2/2.384=0.9229f(k_1+1)/(f+k_1\cdot1.153)=2.2/2.384=0.9229.
  2. 2idf: "during" is in 1 chunk, ln⁡(1+13.5/1.5)=2.3026\ln(1+13.5/1.5)=2.3026; "firmware" and "update" are in 3, ln⁡(1+11.5/3.5)=1.4553\ln(1+11.5/3.5)=1.4553; "router" is in 6, 0.83620.8362; "stuck" is in none and adds nothing.
  3. 3BM25=0.9229 (2.3026+1.4553+1.4553+0.8362)=5.583\text{BM25}=0.9229\,(2.3026+1.4553+1.4553+0.8362)=5.583, rank 1. The rare word "during" is worth more than "router" and "update" put together.
ChunkBM25 (rank)cosine (rank)RRF, k = 60convex α = 0.5convex α = 0.3
d53.666 (2)0.929 (1)1/62 + 1/61 = 0.0325220.82830.7597
d9 (gold)5.583 (1)0.593 (3)1/61 + 1/63 = 0.0322660.81910.8915
d1 (gold)3.651 (3)0.879 (2)1/63 + 1/62 = 0.0320020.80020.7417
d120.871 (5)0.285 (6)1/65 + 1/66 = 0.0305360.23140.2012
Fused scores (computed in numpy). d5 "firmware update adds new wifi features" is not relevant

Under RRF, d5 wins on ranks 2 and 1, although BM25 scored d9 52% above it. The convex mix sees that margin: after min-max scaling d9 has s^bm25=1\hat s_{\text{bm25}}=1 against d5's 0.6570.657, while d5 leads on the dense side 11 to 0.6380.638. The two tie at α=0.487\alpha=0.487; below that d9 is first. Recall@3 is 1.0 under every fusion here; only the order, and so MRR, changes.

In practice

Start with RRF at k=60k=60 and a candidate window of 50–100 per lane. Build a labelled set of a few hundred real queries with their gold chunks, then measure recall@50 for each lane alone and fused (see RAG evaluation). If the labels exist, try the convex mix with α\alpha swept from 0 to 1 and keep the best. Keep the tokeniser of the BM25 lane in step with the content: stemming fixes "slow"/"slowly" but can merge codes, so index identifiers as exact tokens.

python
1import numpy as np
2from rank_bm25 import BM25Okapi
3from sentence_transformers import SentenceTransformer
4
5chunks = [...] # list[str]
6bm25 = BM25Okapi([c.lower().split() for c in chunks], k1=1.2, b=0.75)
7enc = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
8E = enc.encode(chunks, normalize_embeddings=True) # offline; ANN index in production
9
10def hybrid(q, n=100, fusion="rrf", k=60, alpha=0.5):
11 s_bm = bm25.get_scores(q.lower().split())
12 s_de = E @ enc.encode(q, normalize_embeddings=True)
13 top_bm = [i for i in np.argsort(-s_bm)[:n] if s_bm[i] > 0] # what an inverted index returns
14 top_de = list(np.argsort(-s_de)[:n])
15 cand = set(top_bm) | set(top_de)
16 if fusion == "rrf":
17 score = {d: 0.0 for d in cand}
18 for lst in (top_bm, top_de):
19 for pos, d in enumerate(lst):
20 score[d] += 1 / (k + pos + 1)
21 else: # convex over min-max normalised scores
22 def mm(s):
23 v = s[list(cand)]
24 return {d: (s[d] - v.min()) / (v.max() - v.min() + 1e-9) for d in cand}
25 nb, nd = mm(s_bm), mm(s_de)
26 score = {d: alpha * nd[d] + (1 - alpha) * nb[d] for d in cand}
27 return sorted(cand, key=score.get, reverse=True)[:n] # hand to the reranker
Hybrid retrieval with both fusions (rank_bm25 + sentence-transformers; not run here). Tune alpha on a few hundred labelled queries

Use when

  • +Catches both exact-term and paraphrase matches; more robust out of domain than either ranker alone
  • +RRF needs no labels, no score calibration and one constant
  • +The two lanes run in parallel, so latency is about that of the slower one

Avoid when

  • −Two indexes to build, update and keep consistent with deletions and access rules
  • −Convex fusion needs labelled queries and re-tuning after model or corpus changes
  • −RRF ignores score margins and can promote a chunk both rankers rate as mediocre

Check yourself

Chunk A is 1st in BM25 and 40th in dense; chunk B is 5th in both. Which ranks higher under RRF with k = 60?

Check yourself

Why is min-max normalisation (or rank fusion) needed before combining BM25 and cosine scores?

Remember

  • ✓BM25 catches exact rare terms (codes, names); dense retrieval catches paraphrase. Each fails where the other succeeds.
  • ✓BM25: idf × saturating tf with length normalisation; defaults k1=1.2k_1=1.2, b=0.75b=0.75.
  • ✓RRF: ∑r1/(60+rankr)\sum_r 1/(60+\text{rank}_r); label-free and scale-free, but blind to score margins.
  • ✓Convex: αs^dense+(1−α)s^bm25\alpha\hat s_{\text{dense}}+(1-\alpha)\hat s_{\text{bm25}} after min-max scaling; needs a tuned α and often beats RRF when tuned.
  • ✓Hybrid is a first stage: optimise recall of the candidate set and leave the final order to a reranker.
← back to Generative AI roadmap