RAG Latency Budgets: One Performance Decision Chain for Retrieval, Reranking, and Generation

User-perceived RAG latency comes from an entire chain: authentication, query rewriting, vector retrieval, full-text retrieval, permission filtering, reranking, context assembly, time to first model token, and streamed output. Simply switching to a slightly faster model often does not solve the problem of users still waiting several seconds. The first step in performance optimization is to set an end-to-end budget for the task, then decide how much time each stage may consume.

Set SLOs by task, not endpoint

"Q&A endpoint P95 under 5 seconds" is too coarse. Looking up a ticket, comparing ten contracts, and generating cited troubleshooting steps differ in complexity and risk. Define time to first meaningful feedback, time to full answer, and allowed degradation behavior for each task type. For example, low-risk help-center queries can show retrieved sources within 800 milliseconds; complex analysis can take longer but must show progress.

Once the budget is decomposed, the team can see where the problem lies: retrieval at 150 milliseconds, reranking at 400 milliseconds, time to first model token at 1.2 seconds, generation at 3 seconds, or an occasional 2-second delay from the permission service. Every stage should have traces and P50/P95/P99 metrics; measuring only total duration is not enough. Tail latency is often caused by a small number of slow dependencies, long contexts, or retries, none of which averages reveal.

Match retrieval depth to question value

Not every request needs multi-query, cross-encoder reranking, and a dozen candidate chunks. Simple factual queries can use caching or a small amount of hybrid retrieval; high-value, low-confidence, or citation-required tasks can enable deeper recall and reranking. Strategy inputs should be task type, risk, query characteristics, and current system load, not one fixed pipeline.

Set caps for candidates, reranking, and context tokens. More candidates do not necessarily improve accuracy; common results are a slower reranker, longer model context, and answers diluted by noise. Before adding another processing stage, prove its quality benefit on an evaluation set, then decide whether it is worth the latency and cost.

Degradation must preserve correctness boundaries

When the system is busy or dependencies slow down, it can skip optional reranking, reduce candidates, return a summary of retrieval results, or ask the user to narrow the scope; but it must not skip permission filtering, citation validation, or write-operation confirmation. Write "quality enhancements that may be degraded" and "security boundaries that may not be degraded" as explicit policies, preventing data leaks or fabricated answers in the name of speed under pressure.

Caching also requires budget awareness. Caching query rewrites, embeddings, and low-risk retrieval results can reduce repeated overhead, but cache keys must include permissions and index versions. Stale or cross-permission cache hits are worse than being slightly slower. Observe cache hits, invalidations, and origin-fetch durations separately to determine which stage caching is actually improving.

Validate optimizations with real user experience

A streaming UI lets users see sources and progress earlier, but it cannot conceal a continually slowing backend. After each performance optimization, compare task completion rate, user cancellation rate, human handoff, and citation quality, rather than only benchmark load-test results. A faster pipeline that cites less evidence may reduce overall value.

The purpose of a latency budget is to let teams make tradeoffs: where it is worth spending 300 milliseconds for accuracy, where clarification should be returned immediately, and where authorization must be awaited. Turn those tradeoffs into observable policies so RAG can retain a stable experience as it scales.

Load tests must reproduce real request mixes

Load testing only short queries and a fixed document set underestimates the cost of permission filtering, long contexts, cache misses, and concurrent reranking. The test set should include frequent simple Q&A, long questions, exact strings, cross-document comparisons, and high-risk rejection scenarios, mixed at realistic proportions. Record resource usage, queue wait time, and dependency errors rather than only single-host QPS.

When a budget is breached, first determine whether it is caused by a traffic change or a system regression. Set capacity alerts separately for retrieval, reranking, and generation, and retain replayable trace samples. If a performance problem cannot be reproduced, it is difficult to decide whether to add capacity, change policy, or fix a slow query. Reproducible performance evidence is more valuable than an accidental benchmark number.

Release performance policies through low-traffic canaries and retain the old policy as a control. If a new policy shortens first-token time but increases user follow-up questions or missing-citation rates, it should not be expanded merely because it is faster. A performance budget is ultimately part of the product quality budget and must be considered alongside correctness and security in release decisions.

Regularly review whether every budget still matches real traffic, so defaults that were once reasonable do not become new bottlenecks after document scale, model context, or permission rules change.

Record budget changes in release notes. Business teams may accept an extra second for certain complex questions in exchange for more complete citations, but they may not accept unexplained slowdowns for simple queries. Stating affected task types, rollback conditions, and observation metrics in advance turns performance optimization from an infrastructure experiment into a product decision with shared expectations.

Further Reading

• RAG Streaming UI
• Advanced Query Rewriting for Retrieval
• Advanced pgvector