RAG Query Routing Policy: When to Use Keyword, Vector, Structured Queries, or Refuse to Answer

Embedding every user question and doing a single vector-similarity search is the most common RAG starting point, and the source of many production problems. A user may ask, 'Which tickets do I have pending?', which requires a structured query with permission constraints; they may look up an exact error code, for which keyword retrieval is more reliable; or they may ask about something for which the system has no evidence at all. In the last case, the best behavior is to refuse to answer or ask for more information, rather than return the three most similar text passages.

Query routing's goal is not more branches, but placing different data semantics on the correct path. The router's inputs should include a normalized question, user and tenant context, data freshness requirements, available data sources, and a risk level. A model can provide intent candidates, but the final path must be decided by deterministic policy and verifiable features.

Define Each Route's Applicability First

A practical minimum set contains four categories. Keyword or inverted-index retrieval serves identifiers, error codes, product names, and exact terms; vector retrieval serves conceptual Q&A and synonymous expressions; structured queries serve user data, metrics, inventory, and authorization-sensitive facts; refusal or clarification serves missing data, cross-tenant requests, questions beyond product scope, or high-risk decisions. Every route must declare its permitted data sources, maximum number of results, where permission filtering occurs, and fallback plan.

For example, for 'Why did order 8472 fail?', the system can first identify the structured entity ID, use a permission-protected order query, and then pass related log snippets to a language model for explanation. A direct vector search could return a similar error document from another user, which is both useless and a leakage risk.

Make Routing Decisions Evidence-Based, Not Black-Box Labels

Record the features and rules behind the router's selection: detected entities, intent confidence interval, requested freshness, candidate data sources, rejection reason, and final path. Do not record the full raw sensitive question; where necessary, save a redacted summary and hash. This makes it possible, when users challenge an answer, to distinguish 'the retrieval path was wrong,' 'the path was correct but the data was missing,' and 'the data was correct but the generated explanation was wrong.'

The key in the pseudocode is to perform deterministic boundary checks first:

if (!policy.canAccess(user, requestedSource)) return { route: "DENY", reason: "source_not_authorized" };
if (looksLikeExactIdentifier(query)) return { route: "KEYWORD_OR_RECORD_LOOKUP" };
if (needsFreshMetric(query)) return { route: "STRUCTURED_QUERY" };
if (intent.confidence < 0.65) return { route: "CLARIFY" };
return { route: "SEMANTIC_RETRIEVAL" };

Numerical thresholds are not permanent truths; calibrate them with labeled samples and online results. Different tenants, languages, and domains may need different rules; even when model classification is used, retain an auditable final path.

Fallback Paths Must Not Bypass Permission or Quality Gates

When the vector service times out, do not directly use an unfiltered full-text index as a fallback; when a structured query returns no results, do not automatically switch to another tenant's knowledge base. Fallback paths must inherit the original data scope and authorization context. For low-confidence questions, clarification is often safer than expanding search scope, for example by asking whether the user wants product documentation, personal data, or a metric.

Launch validation can prepare four kinds of samples: questions with exact IDs, questions requiring real-time values, semantic explanation questions, and questions with no evidence. Check whether routing is consistent, permission filtering takes effect on every path, and refusals clearly state what is missing. Then correlate path selection with answer satisfaction and gradually adjust the rules instead of optimizing only the vector model.

Routing Rules Must Support Replay and Version Comparison

After a router upgrade, the hardest thing to investigate is not a single error, but a class of questions suddenly taking a different data source under the new rules. Every decision should therefore include routerVersion, a policy snapshot, and an input-feature summary. Retain redacted replay samples for a limited period to compare old and new rules without reaccessing user data: which queries move from semantic retrieval to structured queries, whether zero results increase, and whether permission denials meet expectations.

Do not use replay only to improve hit rate. For high-risk questions, verify that the new version does not incorrectly expand data scope; for cost-sensitive questions, verify it does not send simple questions down an expensive reranking path. Release in shadow mode and by tenant canary, with immediate version-level rollback when anomalies occur. This turns the routing policy from an opaque classification prompt into infrastructure capable of rollout, observability, and rollback.

Further Reading

• RAG Advanced Practices
• RAG Retrieval Debugging
• RAG Latency Budgets