Query rewriting and HyDE

Rewrite the user's question before retrieval: resolve chat context into a standalone query, fan out paraphrases fused by RRF, step back to a broader question, or embed an LLM-written hypothetical answer (HyDE).

8 min read819 wordsadvanced0 viewsupdated 2026-10-21

Definition

Query rewriting uses a language model to turn the user's raw input into one or more search queries that a retriever handles better. Four patterns are common: conversational rewriting into a standalone question, multi-query expansion, step-back abstraction (Zheng et al., "Take a Step Back", 2023), and HyDE, Hypothetical Document Embeddings (Gao, Ma, Lin and Callan, "Precise Zero-Shot Dense Retrieval without Relevance Labels", 2022). HyDE searches with the embedding of an LLM-written answer instead of the question.

Users do not write queries the way documents are written. Questions are short, use other words than the documents, and in a chat they lean on earlier turns ("and its battery?"). Documents are long and declarative. A retriever trained mostly on one side of that gap, or on no labelled data at all, ranks poorly across it.

Rewriting sits before retrieval in a RAG pipeline (see RAG architecture). It changes only what is searched for: the answer is still generated from retrieved chunks and the user's original question. That separates it from the Reranking stage after retrieval, and from Agentic RAG, where the model decides over several steps whether and what to search.

1 · conversational rewrite (chat history → standalone query)user: Is the Nova X2 waterproof?bot:  Yes, rated IP68.user: and how long does its      battery last?LLM rewriteresolve “its”q = “How long does theNova X2 battery last?”embedding “and how long does its battery last?” alone would retrieve any battery chunk2 · transform q for the retriever (one LLM call each)multi-query1 Nova X2 battery life in hours2 Nova X2 screen-on time per  charge3 how many hours does a full  charge of the Nova X2 lastretrieve each of q, 1, 2, 3fuse: RRF Σ 1/(60 + rank)HyDE"The Nova X2's 4,500 mAhbattery lasts about a day ofmixed use, roughly 7 hoursof screen-on time…"(may be wrong; never shown)embed v = (f(q) + Σ f(d̂ₖ))/(N+1)nearest real chunks to vstep-back"What are the Nova X2'sbattery and power specs?"(a more general question whose answer contains the fact)retrieve for the original andthe step-back questiontop-k real chunks → prompt with the user'soriginal question → answer with citationscost: one LLM call on the critical path before retrieval; multi-query also runs N+1 searchesthe generated text only steers retrieval; the answer is grounded in retrieved chunks
fig.Step 1: a follow-up turn is rewritten into a standalone query, because "and how long does its battery last?" on its own does not name the phone. Step 2: three transformations of that query. Multi-query searches with several paraphrases and fuses the lists. HyDE writes a plausible answer passage and searches with its embedding; the passage may contain invented numbers and is never shown to the user. Step-back searches with a broader question too. All three end in real retrieved chunks.

Conversational rewriting

In multi-turn search the most useful rewrite is the simplest: make the latest turn self-contained by resolving pronouns and ellipsis from the history. CANARD (Elgohary et al., EMNLP 2019) collected about 40k such human rewrites of QuAC questions, and the TREC Conversational Assistance Track (CAsT, from 2019) evaluates retrieval on rewritten conversational queries. Today one LLM call with the last few turns and an instruction such as "rewrite the final question so it can be understood without the conversation" does the job.

Multi-query and step-back

Multi-query asks the LLM for NN paraphrases or sub-questions, retrieves for each, and merges the ranked lists, usually with reciprocal rank fusion ∑i1/(60+ranki(d))\sum_i 1/(60+\text{rank}_i(d)) (see Hybrid search (BM25 + dense)). Different wordings retrieve different near-misses, and a chunk that several wordings agree on rises. It costs one generation plus N+1N+1 searches.

Step-back prompting asks for a more general question first ("What are the Nova X2's battery and power specs?" for "How long does it last on one charge?"). Zheng et al. introduced it as a reasoning technique and also used it for retrieval: retrieve for both questions, so the context holds the principle or overview as well as the specific detail.

HyDE: search with a fake answer

d^k∼g(⋅∣INST, q),v^q=1N+1[f(q)+∑k=1Nf(d^k)],score⁡(d)=⟨v^q, f(d)⟩\hat d_k\sim g\big(\cdot\mid \text{INST},\,q\big),\qquad \hat v_q=\frac{1}{N+1}\Big[f(q)+\sum_{k=1}^{N} f(\hat d_k)\Big],\qquad \operatorname{score}(d)=\langle \hat v_q,\,f(d)\rangle
gg
an instruction-following LLM prompted with e.g. "write a passage to answer the question" (InstructGPT in the paper)
ff
the document encoder; in the paper the unsupervised Contriever, with no relevance labels
NN
number of sampled hypothetical documents; the query's own embedding is averaged in

The hypothetical passage is written like a document and contains answer-like content, so its embedding lands near real answer passages. Its details may be wrong. The bet is that the encoder's dense bottleneck keeps the overall content and blurs the invented specifics, and that averaging NN samples cancels errors that differ from sample to sample.

q⃗ ′=α q⃗+β∣Dr∣∑d∈Drd⃗−γ∣Dnr∣∑d∈Dnrd⃗\vec q\,'=\alpha\,\vec q+\frac{\beta}{|D_r|}\sum_{d\in D_r}\vec d-\frac{\gamma}{|D_{nr}|}\sum_{d\in D_{nr}}\vec d
Dr, DnrD_r,\,D_{nr}
documents judged (or assumed) relevant and non-relevant
α,β,γ\alpha,\beta,\gamma
weights; with γ=0\gamma=0 and DrD_r = the top retrieved hits this is pseudo-relevance feedback

This is Rocchio's relevance feedback (1971). HyDE has the same shape with γ=0\gamma=0, except DrD_r is generated before any retrieval instead of taken from the top hits. Query2doc (Wang, Yang and Wei, 2023) is the sparse cousin: append the generated passage to the query text and run BM25.

Reference points

HyDE (2022)
InstructGPT generator + Contriever encoder, zero-shot; on TREC DL19 nDCG@10 rose from 44.5 (Contriever alone) to 61.3, above BM25's 50.6
Fine-tuned encoders
the gain is largest for encoders without relevance training; a retriever already fine-tuned on in-domain labels gains less
Rewrite-Retrieve-Read (2023)
Ma et al. trained a small rewriter with reinforcement learning, using the downstream reader's answer quality as the reward
Cost
every pattern adds an LLM call on the critical path before retrieval; multi-query also multiplies search calls by N + 1
question 1 (topic 1, fact 1): HyDE, N = 4, LLM right with p = 0.50cos with the answer's fact →cos with the topic →gold doc (2)same topicother topicquestionhypothetical: ● right / ○ wrong factretrieved top-5 · large ● HyDE vector · small ○ rewritesthis question: 2 of 2 gold docs in the top 5 · 2 of 4 hypotheticals state the right fact60 docs = 3 topics × 10 facts × 2; the question names the topic but only hints at the factmean recall@5 over 30 questionsraw query0.617multi-query + RRF (N = 4)0.817HyDE (N = 4)0.917HyDE recall@5 vs p(right fact), N = 40.00.51.00.00.51.0dashed: raw query and multi-querywrong hypotheticals pull the vector toother facts of the topic
method
fig.A 48-d toy: 3 topics × 10 facts × 2 documents; each question names its topic but carries only a quarter-weight hint of its fact. Default (k = 5, N = 4, LLM right with p = 0.5): mean recall@5 over 30 questions is 0.617 for the raw query, 0.817 for multi-query + RRF and 0.917 for HyDE. Set p = 0: every hypothetical states a wrong fact from the same topic and HyDE collapses to 0.083. HyDE first beats the raw query at p = 0.25 (0.650 vs 0.617). With N = 1 at p = 0.5 it scores 0.583, below the raw query: one sample is a gamble.

Worked example: HyDE in three dimensions (axes: phone battery, "about a day", "two days")

  1. 1Question "How long does the X2 battery last?" embeds to f(q)=(0.982,0.103,0.155)f(q)=(0.982, 0.103, 0.155). Gold chunk A ("lasts about a day") is (0.6,0.8,0)(0.6, 0.8, 0) normalised; chunk B, about another phone ("two days"), is (0.6,0,0.8)(0.6, 0, 0.8). Cosines: A 0.67200.6720, B 0.71360.7136. The raw query ranks the wrong chunk first.
  2. 2The LLM writes N=3N=3 hypothetical passages: two say "about a day", (0.612,0.764,0.204)(0.612, 0.764, 0.204) and (0.621,0.777,0.104)(0.621, 0.777, 0.104); one says "two days", (0.612,0.204,0.764)(0.612, 0.204, 0.764).
  3. 3Average with the query: v^q=(0.7068,0.4620,0.3068)\hat v_q=(0.7068, 0.4620, 0.3068). Cosines: A 0.88350.8835, B 0.74520.7452. A is now first, by a margin of 0.138.
  4. 4Had the LLM produced only the wrong passage (N=1N=1): v^q=(0.797,0.1535,0.4595)\hat v_q=(0.797, 0.1535, 0.4595), cosines A 0.64440.6444, B 0.90680.9068. HyDE then makes the error worse than the raw query did.
python
1import numpy as np
2from sentence_transformers import SentenceTransformer
3
4enc = SentenceTransformer("BAAI/bge-small-en-v1.5")
5E = enc.encode(chunks, normalize_embeddings=True) # (n_chunks, d), offline
6
7def standalone(history: list[str], question: str) -> str:
8 return llm("Rewrite the final question so it can be understood without the "
9 "conversation. Return only the question.\n\n"
10 + "\n".join(history) + f"\nFinal question: {question}")
11
12def hyde_search(q: str, n: int = 4, k: int = 5):
13 fakes = [llm(f"Write a short passage that answers the question.\nQuestion: {q}\nPassage:",
14 temperature=0.7) for _ in range(n)]
15 V = enc.encode([q] + fakes, normalize_embeddings=True)
16 v = V.mean(axis=0) # (f(q) + sum f(d_k)) / (n + 1)
17 return np.argsort(-(E @ v))[:k]
18
19q = standalone(history, "and how long does its battery last?")
20context = [chunks[i] for i in hyde_search(q)] # answer from these, never from the fakes
Conversational rewrite, then HyDE with N samples (sentence-transformers; `llm` is any chat-completion call; not run here)

Where it's used

Chat assistants over documents

Conversational rewriting is close to mandatory: most follow-up turns are not searchable on their own.

Zero-shot domains

HyDE or Query2doc when you have no labelled queries to fine-tune the retriever, and the LLM knows the domain's vocabulary.

Broad or multi-part questions

Multi-query or decomposition into sub-questions, fused with RRF, when one wording cannot cover every facet.

Questions that need background

Step-back queries that pull overview or principle passages next to the specific ones.

Check yourself

Why does HyDE average several hypothetical documents (and the query) instead of using one?

Check yourself

A support bot's users ask follow-ups like "what about the EU version?". Which transformation fixes the largest share of retrieval failures?

Remember

  • ✓Query rewriting changes what is searched for, not what the answer is grounded in.
  • ✓Conversational rewriting makes follow-up turns standalone (CANARD, TREC CAsT); do it first.
  • ✓Multi-query: NN paraphrases, N+1N+1 searches, fused by RRF ∑1/(60+rank)\sum 1/(60+\text{rank}).
  • ✓HyDE: v^q=1N+1[f(q)+∑kf(d^k)]\hat v_q=\frac{1}{N+1}[f(q)+\sum_k f(\hat d_k)], the embedding of LLM-written answers; Rocchio feedback with generated documents.
  • ✓HyDE helps most with unsupervised encoders and an LLM that knows the domain; with a clueless LLM it can be far worse than the raw query.
← back to Generative AI roadmap