RAG query rewriting transforms a user’s question into one or more retrieval queries that better match the language, entities, and structure of the indexed corpus. Databricks’ current AI Cookbook and AI Search retrieval-quality guidance treat query understanding as an early stage of the RAG chain, alongside intent classification, entity extraction, filter extraction, decomposition, and multi-query retrieval. The key warning is that rewriting must be evaluated together with the retrieval component; a more polished query is not automatically a better search query.
Within Generative AI on Databricks, query rewriting sits between conversational input and Mosaic AI Vector Search—now Databricks AI Search. It can improve recall dramatically for ambiguous, conversational, or multi-turn questions when the rewrite is designed for the index rather than for human prose.
RAG chunking remains equally important: rewriting cannot retrieve context that was poorly chunked or never indexed.
Conversation history should be rewritten into a standalone retrieval query
A follow-up such as “what about the European plan?” makes sense to the chat model because previous turns contain context, but it is a poor direct vector/keyword query.
Rewrite it into a self-contained query containing the product/entity/topic established earlier.
Keep only facts already present in the conversation; a rewriter should resolve references, not invent assumptions to make the query sound complete.
Synonyms and domain vocabulary can improve recall
Users and documentation often use different terminology: “SSO” versus “single sign-on,” “incident” versus “case,” internal product acronyms versus official names.
A rewriter can expand or substitute synonyms that align the query with source language.
Maintain a domain vocabulary or examples for critical terms rather than relying entirely on general-language paraphrasing.
Spelling and normalization are cheap wins
Correct misspellings, normalize product names, expand abbreviations, and clean punctuation that interferes with full-text retrieval.
These operations can be handled deterministically before calling an LLM in many cases.
Use model-based rewriting for semantic ambiguity rather than spending model latency on transformations a text normalizer can do reliably.
Filter extraction should be separated from semantic text
A user query such as “show 2026 EU invoices over €10k” contains both semantic intent and structured filters.
Extract year, region, and amount into AI Search metadata filters while rewriting the text query to describe the relevant document/content concept.
This avoids burying exact constraints inside an embedding where they may not be enforced precisely.
Multi-query rewriting can improve ambiguous-question recall
For broad or ambiguous questions, generate several focused retrieval queries representing plausible interpretations or subtopics, retrieve for each, then merge/deduplicate results.
This can increase recall at the cost of more search calls and more candidate documents.
Use it selectively for complex queries; simple factual questions do not need three paraphrases and extra latency.
Query decomposition helps multi-hop questions
A question that requires evidence from several documents—such as policy plus current product limitation—may be better split into sequential subqueries.
Retrieve evidence for the first subquestion, use that evidence to formulate the next, then synthesize.
Keep each step traceable so evaluation can identify which hop failed rather than only seeing the final wrong answer.
Rewriting should consider the active retrieval mode
ANN, hybrid, and full-text AI Search queries respond differently to wording.
An exact error code should be preserved for keyword/full-text retrieval; a conceptual question may benefit from semantic paraphrase; hybrid can combine both.
Use retrieval evaluation to determine whether rewritten queries perform better with ANN or hybrid rather than forcing one mode universally.
Over-rewriting can destroy high-value exact terms
An LLM that replaces a precise API name, SKU, legal clause, error message, or code symbol with a natural-language synonym can make retrieval worse.
Prompt the rewriter to preserve quoted terms and identifiers, and compare original versus rewritten retrieval for exact-term queries.
The safest design often passes both the original and rewritten variants into hybrid or multi-query retrieval.
Rewrite prompts should be evaluated and versioned
Store the query-rewrite prompt in Prompt Registry with MLflow or an equivalent versioned artifact.
Evaluate rewritten query text, retrieval relevance, and final answer quality as one pipeline.
A rewrite that looks linguistically better but reduces DCG@10 or retrieval sufficiency should not be promoted.
Latency and cost should limit the number of rewrite steps
Intent classification, entity extraction, rewrite, decomposition, and multi-query expansion can easily become four or five LLM calls before retrieval starts.
Databricks guidance notes that multiple specialized steps provide control but add latency.
Use cheap deterministic parsing and compact models where sufficient, and run advanced rewriting only for queries that need it.
Query rewriting succeeds when retrieval evidence improves measurably
The mature RAG system resolves conversational references, preserves identifiers, extracts filters, expands ambiguous queries selectively, evaluates original versus rewritten retrieval, and traces every transformed query.
Rewriting is not a prose cleanup stage. It is a retrieval optimization whose success is measured by whether the right evidence appears earlier and more consistently.
Rewrite outputs should be structured rather than free-form when the pipeline needs several fields. A JSON object containing `standalone_query`, `filters`, `entities`, and `retrieval_mode` is easier to validate and trace than asking one LLM to return a paragraph that later code parses heuristically.
Filter extraction must validate values against allowed metadata. If the rewriter invents a region, tenant ID, or date range not present in the user’s request, retrieval can become incorrectly narrow or unsafe. Separate model suggestion from deterministic authorization and only apply filters that pass schema/business validation.
Multi-query retrieval needs deduplication and rank fusion. Several rewrites often return overlapping chunks; simply concatenating results wastes context and can over-weight one document. Deduplicate by stable document/chunk ID and combine ranks/scores with a defined algorithm before reranking or generation.
Rewriting can be personalized carefully using session context such as user’s product/version or locale, but authorization should never be inferred from that context. Use identity-derived filters from the application security layer and conversational context only to improve relevance.
Evaluation should include ambiguous queries where the correct behavior is to ask a clarifying question instead of rewriting aggressively. If ‘show the latest policy’ could refer to three policies, inventing one specific query may produce a confident but wrong answer. The rewriter should have an abstain/clarify path.
Prompt injection can target the query-understanding stage through conversation history or retrieved context. Keep the rewriter’s instructions isolated and restrict its output schema; it should not be able to request new tools or bypass tenant filters. Treat user text as query data, not executable policy.
Original-query retrieval should remain a useful baseline. In evaluation, compare original, rewritten, and multi-query results for each case. If rewriting harms exact-match queries, route those query classes around the rewriter rather than forcing every request through a costly transformation.
Logging should preserve both human input and derived query safely. When support teams investigate ‘the bot ignored my question,’ they need to see how the query was transformed. Redact sensitive values but retain enough structure to tell whether the error happened before search or inside search.
Rewrite quality should be segmented by query type. Follow-up conversational questions, typo-heavy support queries, natural-language filters, exact error codes, and broad research questions need different transformations. A single prompt optimized for one category can harm another; classify or use simple heuristics before rewriting.
The rewriter should preserve the user’s language when the corpus/index supports multilingual search. Translating every query to English may improve one index but lose exact terminology or locale-specific product names. Evaluate same-language versus translated retrieval and choose per corpus.
Entity extraction can produce stable metadata for analytics as well as search. Logging recognized product, account type, error code, or intent lets teams understand which query categories fail most often. Keep these derived labels privacy-safe and do not treat model-extracted entities as authoritative business IDs until validated.
Query rewriting and session memory should be decoupled. Use explicit conversation summaries or state fields as input to the rewriter rather than sending an unbounded chat transcript every time. This reduces tokens and limits the chance old irrelevant turns distort the current retrieval query.
Latency budgets should allow the application to bypass rewriting under load. If rewrite service/model latency spikes, simple queries can fall back to original or deterministic-normalized search while complex queries wait or degrade visibly. Resilience improves when query understanding is an optional optimization layer rather than a single point of failure.
Rewriting output should be bounded in length. An overlong rewritten query can dilute the key terms, increase embedding/token cost, and hurt full-text matching. Set concise output requirements and preserve only the entities/context needed to retrieve evidence.
A good production trace should record whether rewriting was bypassed, which rewrite strategy ran, model/prompt version, extracted filters, final search queries, and latency. This lets teams compare direct versus rewritten traffic and identify when the optimization is no longer paying for itself.
Keep rewrite logic measurable and easy to bypass when it does not improve retrieval.
Query rewriting should be reviewed whenever the retrieval corpus, embedding model, metadata schema, or AI Search query strategy changes because the transformation that helped the old index may no longer produce the best evidence after a retrieval-system migration.
