Advanced Query Rewriting for Retrieval

In RAG and search systems, there is a counterintuitive fact: the sentence a user types into the search box is almost never the best search query. All the work you do at the retrieval stage—hybrid search, reranking, tuning the embedding model—uses a potentially flawed query as input. Query rewriting, also called query transformation, means getting this first step right. It is one of the highest-ROI optimizations in the entire retrieval pipeline, yet it is easily dismissed as merely "nice to have."

This article systematically explains five mainstream rewriting techniques, their costs and applicable scenarios, and how to implement and evaluate them in production.

I. Why the Original Query Is Never Enough

First, consider the problems an original user query can create in a retrieval pipeline. Almost every case of poor recall ultimately falls into one of these four categories:

1. Missing referents in multi-turn conversations. The user previously asked, "How is the iPhone 17 Pro?" and now asks, "What about its battery life?" The word "its" is unambiguous to a person but disastrous for a retriever. Using "What about its battery life?" for vector search will probably retrieve generic documents about batteries rather than the iPhone 17 Pro's battery. The retriever has no memory of the conversation context.

2. Too vague. "I want to learn Rust"—does the user want a beginner's tutorial, a comparison with Go, or information about async programming? A vague query maps to a diffuse cluster in vector space, so the top-k results contain a little of everything but nothing precise.

3. Multiple questions combined into one. "Compare the strengths and weaknesses of Postgres and MongoDB in transactions and horizontal scaling" is actually four atomic queries: Postgres transactions, Postgres scaling, MongoDB transactions, and MongoDB scaling. Retrieving with the entire sentence dilutes every branch. Splitting it into separate searches and then merging the results makes each branch precise.

4. Mismatch between exact terms and semantics. If the user asks, "How do I fix ErrorCode=E_4021?", that is an exact string. BM25 handles it well, but pure vector search often misses because vector encoders understand semantics better than symbols. Conversely, a semantic query such as "How do I handle an order that stays pending?" may not match a document phrased as "stuck-order troubleshooting procedure" in BM25. Rewriting cannot solve this alone—you also need hybrid search—but it can normalize "order pending" to the term "stuck order" actually used in the documents.

In short: the ceiling for retrieval is established during rewriting. Even the strongest reranker cannot rescue a query that never retrieved the relevant document in the first place.

II. Five Rewriting Techniques, Each for a Different Job

1. Coreference Resolution (Coreference / Conversational Query Rewriting)

This is the most basic and most frequently needed technique. In a multi-turn conversation, rewrite the current query containing references into a self-contained or standalone retrieval query by incorporating the conversation history.

历史: 用户: iPhone 17 Pro 的价格是多少?   助手: 999 美元起。
当前: 它的电池容量呢?
改写: iPhone 17 Pro 的电池容量是多少?

An LLM can do this with a single instruction at very low cost—one small-model call—but the benefit is enormous in multi-turn scenarios. Multi-turn RAG without coreference resolution is essentially unusable.

Implementation only requires a lightweight prompt. The key is to include the most recent N turns of conversation history, usually three to five. In production, use the fastest available model, such as a Haiku- or Flash-class model, rather than a large model. This step does not require deep reasoning; it requires low latency and low cost.

2. Sub-question Decomposition

For a compound question, first ask an LLM to split it into atomic sub-questions. Retrieve documents independently for each sub-question, then merge the results and pass them to the generation stage.

原问题: 对比 Postgres 和 MongoDB 在事务和水平扩展上的优劣
子问题:
  - Postgres 的事务支持如何?
  - Postgres 的水平扩展能力如何?
  - MongoDB 的事务支持如何?
  - MongoDB 的水平扩展能力如何?

Each sub-question is a small, precise query, producing much better recall than retrieving with the entire sentence. The tradeoff is N times as many retrieval calls, where N is the number of sub-questions, along with more complex merge logic. This works well when the question itself involves a multidimensional comparison or multi-step reasoning, such as "Which of X and Y is better suited to Z, and what are the costs of each?"

It is unsuitable for simple factual questions. Decomposing those only introduces noise while needlessly multiplying the cost by N. This step therefore usually begins with a decision: should this question be decomposed at all?

3. HyDE (Hypothetical Document Embeddings)

This is the most ingenious technique. The principle is that questions and answers can be far apart in vector space, while a hypothetical answer and the real document are close together. Both are written as answers, so their vocabulary and structure are more similar.

HyDE therefore asks an LLM to generate a hypothetical answer to the original question first, even if it contains hallucinations or is inaccurate. It then uses that hypothetical answer, rather than the original question, for vector search.

// HyDE 核心两步
const hypothetical = await llm.chat({
  messages: [
    { role: "system", content: "请针对用户问题,写一段 100 字左右的、可能包含答案的文档段落,即使你不确定也要写得像那么回事。" },
    { role: "user", content: originalQuery },
  ],
});

// 用假设答案(而非原问题)做向量检索
const docs = await vectorStore.similaritySearch(await embed(hypothetical), k);

HyDE is highly effective for open-domain semantic question answering, especially when the query is short and the documents are long—the user asks a brief question while the documents provide extensive detail. But it has two pitfalls. First, if the hypothetical answer introduces an incorrect fact, it can steer retrieval in the wrong direction by projecting misleading semantics into vector space. Second, it only helps vector search and can actually harm BM25, because the exact terms in the hypothetical answer may not match the real document. HyDE is therefore usually enabled only for the vector retrieval branch and should not be sent to BM25.

4. Multi-query (Merging Multiple Paraphrases)

Ask an LLM to rewrite the original query into N semantically equivalent queries with different phrasing, retrieve results for each one, then merge and deduplicate them.

原问题: 怎么优化 React 的渲染性能?
改写:
  - 减少 React 组件的不必要重渲染有哪些方法?
  - React 性能优化的最佳实践
  - React memo / useMemo / useCallback 什么时候用?

Different phrasings cover the different terminology that documents may use. One document may say "unnecessary renders," another may say "re-render," and another may say "performance optimization." Merging results from multiple variants significantly improves recall by reducing missed documents.

Use RRF (Reciprocal Rank Fusion) or simple score weighting to merge the results, just as you would when combining hybrid search results. The cost is again N times as many retrieval calls, making this technique suitable when recall takes priority and missing relevant documents is unacceptable.

Multi-query and sub-question decomposition look similar, but they are fundamentally different: sub-question decomposition separates different dimensions, while Multi-query expresses the same meaning in different ways. The former addresses multiple questions combined into one; the latter addresses mismatched phrasing.

5. Synonym Expansion / Terminology Normalization

This is the simplest category and does not necessarily require an LLM. Expand or normalize the user's terminology to match the vocabulary actually used in the document collection.

• Synonym expansion: "stuttering" → expand to "stuttering OR latency OR lag OR jank."
• Terminology normalization: "DB" → "database," "ES" → "Elasticsearch," and "K8s" → "Kubernetes."

You can use an operator-maintained synonym dictionary, find related terms with embeddings, or ask an LLM to normalize terminology. This is particularly valuable in corpora dense with domain terminology, such as medical, legal, or company-internal jargon. A simple synonym dictionary can often improve recall substantially at almost no cost.

III. Engineering Implementation: When to Rewrite, Cost, and Caching

Do not enable everything all the time. None of these five techniques is free. Most require an additional LLM call, while more complex techniques such as decomposition, HyDE, and Multi-query also multiply downstream retrieval calls. Blindly chaining every rewriting technique into one pipeline will drive up time to first token (TTFT) and noticeably degrade the user experience.

In production, use on-demand routing: first use inexpensive signals to determine which rewriting technique a query needs, then trigger the corresponding strategy.

async function routeAndRewrite(
  query: string,
  history: ChatMessage[],
): Promise<string[]> {
  // 1. 有历史且当前 query 短/含指代词 → 做指代消解(几乎必做)
  if (history.length > 0 && /\b(它|这个|那个|上面|刚才)\b/.test(query)) {
    return [await resolveCoreference(query, history)];
  }

  // 2. 用便宜的小模型判断:这是个复合问题吗?
  const isCompound = await cheapClassify(query, "single_or_compound");
  if (isCompound === "compound") {
    return decompose(query); // 返回多个子问题
  }

  // 3. 短而开放的语义问题 → HyDE(只用于向量路)
  // 4. 默认:轻量 Multi-query(2 个改写)兜底召回率
  return [query, ...await multiQuery(query, 2)];
}

Several practical principles:

• Coreference resolution is almost always worthwhile because it is inexpensive and essential for multi-turn conversations. Enable other techniques according to the scenario.
• Cache rewriting results. Cache retrieval results for the same rewritten query for anywhere from a few minutes to a few hours, using a query hash as the key. Repeated or similar user questions can then hit the cache directly, avoiding both LLM and retrieval costs.
• Use a fast model for rewriting and a strong model for generation. Rewriting is about making the query correct and does not require deep reasoning. Reserve the reasoning budget for final answer generation.
• Set a latency budget for rewriting. For example, cap the total rewriting time at 300ms. If it exceeds the budget, fall back to the original query instead of sacrificing response time for a perfect rewrite.

IV. Rewriting Must Also Be Evaluated, or It Is Just Guesswork

Rewriting is not automatically beneficial. Over-rewriting is a real risk: an LLM may invent assumptions and drift away from the user's intent, producing a semantically distorted query that retrieves irrelevant documents and breaks a question the system could previously answer. HyDE is especially prone to this problem. If its hypothetical answer introduces a false premise, the entire retrieval process follows it in the wrong direction.

Rewriting must therefore be tested. The most direct method is to use a fixed evaluation set and compare retrieval metrics with rewriting enabled and disabled.

• Recall@k: How many of the relevant documents that should have been retrieved are found after rewriting? This is the metric rewriting should improve most, because its core value is preventing misses. If enabling Multi-query lowers Recall instead of raising it, the rewrite has drifted and should be rolled back immediately.
• Context Precision: What proportion of retrieved documents are relevant? Decomposition and Multi-query may increase Recall while lowering Precision by retrieving more but noisier results. The downstream reranker must compensate for that.
• End-to-end answer quality (Faithfulness / Answer Relevancy): The final answer must improve for the change to count as a real improvement. If intermediate metrics rise but answer quality does not, downstream stages are absorbing the benefit of rewriting.

In practice, run an A/B comparison over an evaluation set of 50–200 问题 → 期望引用文档 examples: once with the original query and once with the rewritten query, then compare Recall@k. The net benefit of rewriting becomes immediately visible. Without this evaluation set, every rewriting optimization is based on guesswork.

V. Combining Techniques by Scenario

Mature systems do not use only one rewriting technique. They orchestrate techniques according to the scenario:

• Multi-turn conversations such as customer support or assistants: coreference resolution is mandatory, with sub-question decomposition when necessary. HyDE and Multi-query are generally unnecessary.
• Single-turn, in-depth question answering such as knowledge-base search: use HyDE for the vector branch and lightweight Multi-query to improve recall. Coreference resolution is unnecessary because there is no history.
• Comparisons or multidimensional research questions: use sub-question decomposition as the primary technique, then apply Multi-query once to each sub-question as a recall fallback.
• Terminology-dense domains such as medical, legal, or internal knowledge bases: establish a baseline with synonym expansion and terminology normalization. It is almost always worthwhile and costs very little.

More is not always better. Every additional rewriting technique adds latency and cost, along with another opportunity for semantic drift. Start with the highest-ROI technique, verify that it produces a positive improvement, and only then add the next one.

VI. Working with Hybrid Search and Reranking

Rewriting is not isolated. It belongs to the same pipeline as downstream hybrid search and reranking, so these components must be designed together:

1. Rewriting × hybrid search: route different rewrites to different retrieval branches. Send HyDE only to the vector branch, not BM25. Each decomposed sub-question can use a hybrid of vector search and BM25. OR queries from synonym expansion primarily belong in BM25 for exact-term expansion. Do not indiscriminately send the same rewritten result to every retriever.

2. Rewriting × reranking: rewriting, especially Multi-query and decomposition, tends to increase Recall at the expense of Precision. It retrieves more results, but they are noisier. That is exactly where reranking helps: first retrieve 30–50 results with rewriting and hybrid search for broad coverage, then use a reranker to select the three to five most relevant results for precision. The two are natural partners. Rewriting prevents misses, while reranking removes noise.

3. The complete pipeline: 原 query → 按需改写 → 各路混合召回(N×K) → RRF 融合 → Rerank 取 top-k → 注入 prompt → 生成. Rewriting is the first stage in this pipeline and the one with the greatest leverage.

Summary

• The original query is almost never a good search query: missing referents, vague wording, multiple questions combined into one, and vocabulary mismatches all lower the retrieval ceiling at the first step.
• Each of the five techniques has its own role: coreference resolution for multi-turn conversations, sub-question decomposition for compound questions, HyDE for closing the semantic distance between questions and answers, Multi-query for improving recall through alternate phrasings, and synonym expansion for terminology normalization. There is no silver bullet; choose by scenario.
• Route on demand instead of enabling everything: rewriting incurs LLM cost and latency. Coreference resolution is almost always worthwhile; trigger other techniques only after evaluating inexpensive signals, and set a latency budget and cache the results.
• Rewriting must be evaluated: focus on changes in Recall@k and watch for over-rewriting that shifts the query away from its original meaning. Without an evaluation set, optimization is guesswork.
• Rewriting and reranking work together: rewriting improves Recall by retrieving more, while reranking improves Precision by selecting the best results. Both are essential and operate at different stages.

Key takeaway: The ceiling for retrieval is established during rewriting—even the strongest reranker cannot rescue a query that never retrieved the relevant document. Getting the first step right matters more than perfecting everything that follows.