# Query rewriting and HyDE

> Rewrite the user's question before retrieval: resolve chat context into a standalone query, fan out paraphrases fused by RRF, step back to a broader question, or embed an LLM-written hypothetical answer (HyDE).

Source: https://www.mlfoundry.in/learn/query-rewriting-and-hyde  
Section: Generative AI › Retrieval-Augmented Generation (RAG)  
Level: advanced · 8 min read · updated 2026-10-21

**Query rewriting:** uses a language model to turn the user's raw input into one or more search queries that a retriever handles better. Four patterns are common: conversational rewriting into a standalone question, multi-query expansion, step-back abstraction (Zheng et al., "Take a Step Back", 2023), and **HyDE**, Hypothetical Document Embeddings (Gao, Ma, Lin and Callan, "Precise Zero-Shot Dense Retrieval without Relevance Labels", 2022). HyDE searches with the embedding of an LLM-written answer instead of the question.

Users do not write queries the way documents are written. Questions are short, use other words than the documents, and in a chat they lean on earlier turns ("and its battery?"). Documents are long and declarative. A retriever trained mostly on one side of that gap, or on no labelled data at all, ranks poorly across it.

Rewriting sits before retrieval in a RAG pipeline (see **RAG architecture**). It changes only what is searched for: the answer is still generated from retrieved chunks and the user's original question. That separates it from the **Reranking** stage after retrieval, and from **Agentic RAG**, where the model decides over several steps whether and what to search.

> Figure: Step 1: a follow-up turn is rewritten into a standalone query, because "and how long does its battery last?" on its own does not name the phone. Step 2: three transformations of that query. Multi-query searches with several paraphrases and fuses the lists. HyDE writes a plausible answer passage and searches with its embedding; the passage may contain invented numbers and is never shown to the user. Step-back searches with a broader question too. All three end in real retrieved chunks.

## Conversational rewriting

In multi-turn search the most useful rewrite is the simplest: make the latest turn self-contained by resolving pronouns and ellipsis from the history. CANARD (Elgohary et al., EMNLP 2019) collected about 40k such human rewrites of QuAC questions, and the TREC Conversational Assistance Track (CAsT, from 2019) evaluates retrieval on rewritten conversational queries. Today one LLM call with the last few turns and an instruction such as "rewrite the final question so it can be understood without the conversation" does the job.

## Multi-query and step-back

Multi-query asks the LLM for $N$ paraphrases or sub-questions, retrieves for each, and merges the ranked lists, usually with reciprocal rank fusion $\sum_i 1/(60+\text{rank}_i(d))$ (see **Hybrid search (BM25 + dense)**). Different wordings retrieve different near-misses, and a chunk that several wordings agree on rises. It costs one generation plus $N+1$ searches.

Step-back prompting asks for a more general question first ("What are the Nova X2's battery and power specs?" for "How long does it last on one charge?"). Zheng et al. introduced it as a reasoning technique and also used it for retrieval: retrieve for both questions, so the context holds the principle or overview as well as the specific detail.

## HyDE: search with a fake answer

$$
\hat d_k\sim g\big(\cdot\mid \text{INST},\,q\big),\qquad \hat v_q=\frac{1}{N+1}\Big[f(q)+\sum_{k=1}^{N} f(\hat d_k)\Big],\qquad \operatorname{score}(d)=\langle \hat v_q,\,f(d)\rangle
$$

- $g$: an instruction-following LLM prompted with e.g. "write a passage to answer the question" (InstructGPT in the paper)
- $f$: the document encoder; in the paper the unsupervised Contriever, with no relevance labels
- $N$: number of sampled hypothetical documents; the query's own embedding is averaged in

The hypothetical passage is written like a document and contains answer-like content, so its embedding lands near real answer passages. Its details may be wrong. The bet is that the encoder's dense bottleneck keeps the overall content and blurs the invented specifics, and that averaging $N$ samples cancels errors that differ from sample to sample.

$$
\vec q\,'=\alpha\,\vec q+\frac{\beta}{|D_r|}\sum_{d\in D_r}\vec d-\frac{\gamma}{|D_{nr}|}\sum_{d\in D_{nr}}\vec d
$$

- $D_r,\,D_{nr}$: documents judged (or assumed) relevant and non-relevant
- $\alpha,\beta,\gamma$: weights; with $\gamma=0$ and $D_r$ = the top retrieved hits this is pseudo-relevance feedback

This is Rocchio's relevance feedback (1971). HyDE has the same shape with $\gamma=0$, except $D_r$ is generated before any retrieval instead of taken from the top hits. Query2doc (Wang, Yang and Wei, 2023) is the sparse cousin: append the generated passage to the query text and run BM25.

**Reference points**

- **HyDE (2022):** InstructGPT generator + Contriever encoder, zero-shot; on TREC DL19 nDCG@10 rose from 44.5 (Contriever alone) to 61.3, above BM25's 50.6
- **Fine-tuned encoders:** the gain is largest for encoders without relevance training; a retriever already fine-tuned on in-domain labels gains less
- **Rewrite-Retrieve-Read (2023):** Ma et al. trained a small rewriter with reinforcement learning, using the downstream reader's answer quality as the reward
- **Cost:** every pattern adds an LLM call on the critical path before retrieval; multi-query also multiplies search calls by N + 1

> Figure: A 48-d toy: 3 topics × 10 facts × 2 documents; each question names its topic but carries only a quarter-weight hint of its fact. Default (k = 5, N = 4, LLM right with p = 0.5): mean recall@5 over 30 questions is 0.617 for the raw query, 0.817 for multi-query + RRF and 0.917 for HyDE. Set p = 0: every hypothetical states a wrong fact from the same topic and HyDE collapses to 0.083. HyDE first beats the raw query at p = 0.25 (0.650 vs 0.617). With N = 1 at p = 0.5 it scores 0.583, below the raw query: one sample is a gamble.

**Worked example: HyDE in three dimensions (axes: phone battery, "about a day", "two days")**

1. Question "How long does the X2 battery last?" embeds to $f(q)=(0.982, 0.103, 0.155)$. Gold chunk A ("lasts about a day") is $(0.6, 0.8, 0)$ normalised; chunk B, about another phone ("two days"), is $(0.6, 0, 0.8)$. Cosines: A $0.6720$, B $0.7136$. The raw query ranks the wrong chunk first.
2. The LLM writes $N=3$ hypothetical passages: two say "about a day", $(0.612, 0.764, 0.204)$ and $(0.621, 0.777, 0.104)$; one says "two days", $(0.612, 0.204, 0.764)$.
3. Average with the query: $\hat v_q=(0.7068, 0.4620, 0.3068)$. Cosines: A $0.8835$, B $0.7452$. A is now first, by a margin of 0.138.
4. Had the LLM produced only the wrong passage ($N=1$): $\hat v_q=(0.797, 0.1535, 0.4595)$, cosines A $0.6444$, B $0.9068$. HyDE then makes the error worse than the raw query did.

```python
import numpy as np
from sentence_transformers import SentenceTransformer

enc = SentenceTransformer("BAAI/bge-small-en-v1.5")
E = enc.encode(chunks, normalize_embeddings=True)          # (n_chunks, d), offline

def standalone(history: list[str], question: str) -> str:
    return llm("Rewrite the final question so it can be understood without the "
               "conversation. Return only the question.\n\n"
               + "\n".join(history) + f"\nFinal question: {question}")

def hyde_search(q: str, n: int = 4, k: int = 5):
    fakes = [llm(f"Write a short passage that answers the question.\nQuestion: {q}\nPassage:",
                 temperature=0.7) for _ in range(n)]
    V = enc.encode([q] + fakes, normalize_embeddings=True)
    v = V.mean(axis=0)                                      # (f(q) + sum f(d_k)) / (n + 1)
    return np.argsort(-(E @ v))[:k]

q = standalone(history, "and how long does its battery last?")
context = [chunks[i] for i in hyde_search(q)]               # answer from these, never from the fakes
```

Conversational rewrite, then HyDE with N samples (sentence-transformers; `llm` is any chat-completion call; not run here)

> **HyDE needs a model that roughly knows the domain:** If the LLM knows nothing about your private corpus (internal product codes, new regulations), every hypothetical document invents consistent wrong specifics and pulls the search toward look-alike chunks. In the figure, recall@5 drops from 0.617 to 0.083 at $p=0$. Check HyDE against the raw query on labelled questions from your own corpus before shipping. Keep the query's own embedding in the average, and never show the hypothetical text to users.

> **Rewrite for the retriever you have:** A paraphrase that helps BM25 (adding synonyms and exact product names) is not the same as one that helps a dense encoder (making the query look like a passage). For hybrid search, a cheap combination works well: rewrite once for context, then send the rewritten query to BM25 and its HyDE vector to the dense index.

- **Chat assistants over documents:** Conversational rewriting is close to mandatory: most follow-up turns are not searchable on their own.
- **Zero-shot domains:** HyDE or Query2doc when you have no labelled queries to fine-tune the retriever, and the LLM knows the domain's vocabulary.
- **Broad or multi-part questions:** Multi-query or decomposition into sub-questions, fused with RRF, when one wording cannot cover every facet.
- **Questions that need background:** Step-back queries that pull overview or principle passages next to the specific ones.

**Question:** Why does HyDE average several hypothetical documents (and the query) instead of using one?

- A. To make the vector longer
- B. Errors that differ from sample to sample cancel, while the shared answer-like content adds up
- C. Because the encoder needs exactly N + 1 inputs
- D. To reduce the number of LLM calls

**Answer:** B. Errors that differ from sample to sample cancel, while the shared answer-like content adds up — Each sample invents details; if the inventions differ, their components point in different directions and shrink in the mean, while the correct content shared by most samples remains. In the figure at p = 0.5, N = 1 gives recall@5 = 0.583 and N = 4 gives 0.917.

**Question:** A support bot's users ask follow-ups like "what about the EU version?". Which transformation fixes the largest share of retrieval failures?

- A. HyDE
- B. Step-back prompting
- C. Conversational rewriting into a standalone query
- D. Increasing top-k

**Answer:** C. Conversational rewriting into a standalone query — The turn is unsearchable without its context: it names neither the product nor the topic. Resolving it against the history is the prerequisite; HyDE or multi-query can then run on the standalone query.

**Key points**

- Query rewriting changes what is searched for, not what the answer is grounded in.
- Conversational rewriting makes follow-up turns standalone (CANARD, TREC CAsT); do it first.
- Multi-query: $N$ paraphrases, $N+1$ searches, fused by RRF $\sum 1/(60+\text{rank})$.
- HyDE: $\hat v_q=\frac{1}{N+1}[f(q)+\sum_k f(\hat d_k)]$, the embedding of LLM-written answers; Rocchio feedback with generated documents.
- HyDE helps most with unsupervised encoders and an LLM that knows the domain; with a clueless LLM it can be far worse than the raw query.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
