Definition
Query rewriting uses a language model to turn the user's raw input into one or more search queries that a retriever handles better. Four patterns are common: conversational rewriting into a standalone question, multi-query expansion, step-back abstraction (Zheng et al., "Take a Step Back", 2023), and HyDE, Hypothetical Document Embeddings (Gao, Ma, Lin and Callan, "Precise Zero-Shot Dense Retrieval without Relevance Labels", 2022). HyDE searches with the embedding of an LLM-written answer instead of the question.
Users do not write queries the way documents are written. Questions are short, use other words than the documents, and in a chat they lean on earlier turns ("and its battery?"). Documents are long and declarative. A retriever trained mostly on one side of that gap, or on no labelled data at all, ranks poorly across it.
Rewriting sits before retrieval in a RAG pipeline (see RAG architecture). It changes only what is searched for: the answer is still generated from retrieved chunks and the user's original question. That separates it from the Reranking stage after retrieval, and from Agentic RAG, where the model decides over several steps whether and what to search.
Conversational rewriting
In multi-turn search the most useful rewrite is the simplest: make the latest turn self-contained by resolving pronouns and ellipsis from the history. CANARD (Elgohary et al., EMNLP 2019) collected about 40k such human rewrites of QuAC questions, and the TREC Conversational Assistance Track (CAsT, from 2019) evaluates retrieval on rewritten conversational queries. Today one LLM call with the last few turns and an instruction such as "rewrite the final question so it can be understood without the conversation" does the job.
Multi-query and step-back
Multi-query asks the LLM for paraphrases or sub-questions, retrieves for each, and merges the ranked lists, usually with reciprocal rank fusion (see Hybrid search (BM25 + dense)). Different wordings retrieve different near-misses, and a chunk that several wordings agree on rises. It costs one generation plus searches.
Step-back prompting asks for a more general question first ("What are the Nova X2's battery and power specs?" for "How long does it last on one charge?"). Zheng et al. introduced it as a reasoning technique and also used it for retrieval: retrieve for both questions, so the context holds the principle or overview as well as the specific detail.
HyDE: search with a fake answer
- an instruction-following LLM prompted with e.g. "write a passage to answer the question" (InstructGPT in the paper)
- the document encoder; in the paper the unsupervised Contriever, with no relevance labels
- number of sampled hypothetical documents; the query's own embedding is averaged in
The hypothetical passage is written like a document and contains answer-like content, so its embedding lands near real answer passages. Its details may be wrong. The bet is that the encoder's dense bottleneck keeps the overall content and blurs the invented specifics, and that averaging samples cancels errors that differ from sample to sample.
- documents judged (or assumed) relevant and non-relevant
- weights; with and = the top retrieved hits this is pseudo-relevance feedback
This is Rocchio's relevance feedback (1971). HyDE has the same shape with , except is generated before any retrieval instead of taken from the top hits. Query2doc (Wang, Yang and Wei, 2023) is the sparse cousin: append the generated passage to the query text and run BM25.
Reference points
- HyDE (2022)
- InstructGPT generator + Contriever encoder, zero-shot; on TREC DL19 nDCG@10 rose from 44.5 (Contriever alone) to 61.3, above BM25's 50.6
- Fine-tuned encoders
- the gain is largest for encoders without relevance training; a retriever already fine-tuned on in-domain labels gains less
- Rewrite-Retrieve-Read (2023)
- Ma et al. trained a small rewriter with reinforcement learning, using the downstream reader's answer quality as the reward
- Cost
- every pattern adds an LLM call on the critical path before retrieval; multi-query also multiplies search calls by N + 1
Worked example: HyDE in three dimensions (axes: phone battery, "about a day", "two days")
- 1Question "How long does the X2 battery last?" embeds to . Gold chunk A ("lasts about a day") is normalised; chunk B, about another phone ("two days"), is . Cosines: A , B . The raw query ranks the wrong chunk first.
- 2The LLM writes hypothetical passages: two say "about a day", and ; one says "two days", .
- 3Average with the query: . Cosines: A , B . A is now first, by a margin of 0.138.
- 4Had the LLM produced only the wrong passage (): , cosines A , B . HyDE then makes the error worse than the raw query did.
1import numpy as np2from sentence_transformers import SentenceTransformer34enc = SentenceTransformer("BAAI/bge-small-en-v1.5")5E = enc.encode(chunks, normalize_embeddings=True) # (n_chunks, d), offline67def standalone(history: list[str], question: str) -> str:8 return llm("Rewrite the final question so it can be understood without the "9 "conversation. Return only the question.\n\n"10 + "\n".join(history) + f"\nFinal question: {question}")1112def hyde_search(q: str, n: int = 4, k: int = 5):13 fakes = [llm(f"Write a short passage that answers the question.\nQuestion: {q}\nPassage:",14 temperature=0.7) for _ in range(n)]15 V = enc.encode([q] + fakes, normalize_embeddings=True)16 v = V.mean(axis=0) # (f(q) + sum f(d_k)) / (n + 1)17 return np.argsort(-(E @ v))[:k]1819q = standalone(history, "and how long does its battery last?")20context = [chunks[i] for i in hyde_search(q)] # answer from these, never from the fakesWhere it's used
Chat assistants over documents
Conversational rewriting is close to mandatory: most follow-up turns are not searchable on their own.
Zero-shot domains
HyDE or Query2doc when you have no labelled queries to fine-tune the retriever, and the LLM knows the domain's vocabulary.
Broad or multi-part questions
Multi-query or decomposition into sub-questions, fused with RRF, when one wording cannot cover every facet.
Questions that need background
Step-back queries that pull overview or principle passages next to the specific ones.
Check yourself
Why does HyDE average several hypothetical documents (and the query) instead of using one?
Check yourself
A support bot's users ask follow-ups like "what about the EU version?". Which transformation fixes the largest share of retrieval failures?
Remember
- ✓Query rewriting changes what is searched for, not what the answer is grounded in.
- ✓Conversational rewriting makes follow-up turns standalone (CANARD, TREC CAsT); do it first.
- ✓Multi-query: paraphrases, searches, fused by RRF .
- ✓HyDE: , the embedding of LLM-written answers; Rocchio feedback with generated documents.
- ✓HyDE helps most with unsupervised encoders and an LLM that knows the domain; with a clueless LLM it can be far worse than the raw query.