Implementing a Streaming UI for RAG Applications

Anyone who has built a RAG application knows that its latency structure has "two stacked stages": retrieval first (vector recall + reranking + possibly multi-path fusion), then generation (the LLM emits text token by token). Neither stage is cheap on its own, and together they are even more expensive.

A typical path is 800ms2s for retrieval + 315s for generation. If your UI is "click once, spin, then bam, show the entire answer," users stare at a loading indicator for those 8 seconds with no information in between. That is an experience disaster.

My view is straightforward:RAG is not one request and one response; it is the composition of two information streams. The retrieval stage has the high-value signal of "which materials did I find?" The generation stage has the progressive signal of "the answer is taking shape." Failing to stream either means leaving users hanging as though they were fools. This article thoroughly explains those two streams and the UI engineering around them.

1. Why RAG Must Stream: Two Latencies, Two Signals

First, build intuition. A RAG request timeline looks like this:

[0ms] 用户提问
   │  ⬇ 检索阶段(用户什么都看不到)
[1200ms] 检索完成,拿到 5 个 chunk
   │  ⬇ 生成阶段(如果全量返回,用户还是什么都看不到)
[9000ms] 大模型吐完最后一个 token,一次性渲染

With an all-at-once response, every moment in the first 9 seconds shows users only a loading spinner. During those 9 seconds, you have hidden two signals that should have been exposed:

1. At 1.2 seconds: The retrieval results are already available. What users most want to know then is, "Which materials are you using to answer me?" If irrelevant documents were retrieved, they can immediately interrupt and rephrase instead of waiting 8 seconds to see an off-topic answer and complain.
2. From 1.2 to 9 seconds: The LLM is generating token by token. The typewriter effect is not merely for "looks"; it lets users judge the direction of the answer early. If they notice it is off-topic halfway through, they can stop it directly, saving tokens and time.

So streaming is not about showing off. It is about giving users these two signals at the moment they are produced.

2. Two-Stage Streaming: Retrieval + Generation

Streaming the Retrieval Stage: First Tell Users "I Am Looking at These Materials"

As soon as retrieval finishes, push the retrieved document list (title, snippet, and source link) to the frontend. This has two benefits: confirming direction and establishing trust. Users trust the answer only after seeing its sources.

At the protocol level, I use a distinct event type for the retrieval stage, separate from generation-stage deltas:

event: sources
data: {"items":[{"id":"d1","title":"Next.js 路由文档","snippet":"...","url":"..."}]}

event: delta
data: {"text":"根据文档,"}

event: citation
data: {"marker":1,"sourceId":"d1"}

event: done
data: {}

Keeping sources and delta separate matters because they correspond to two completely different UI regions: the source list (top/sidebar) and the answer body (main area). If they are mixed together, the frontend must branch based on fields; when separated, one event drives one state slice cleanly.

Streaming the Generation Stage: Typewriter Answer Text

This is classic SSE token-by-token delivery, with frontend concatenation. But the RAG generation stage has one more concern than pure chat:citation markers. While generating, the model emits markers such as [1][2], and you need to associate them with the source list. The fourth section covers this in detail.

3. UI State Machine: Searching -> Retrieved -> Generating -> Complete

This is the skeleton of the entire streaming UI. I strongly recommend an explicit finite state machine rather than a boolean isLoading. The reason is simple: RAG has four semantically distinct phases, and isLoading cannot express the intermediate state of "materials have been found and the answer is being generated." Your UI will otherwise be full of hard-to-maintain conditions such as isLoading && hasSources && !answer.

type Source = { id: string; title: string; snippet: string; url?: string };

type RagState =
  | { phase: "idle" }
  | { phase: "searching"; query: string }
  | { phase: "retrieved"; query: string; sources: Source[] }
  | { phase: "generating"; query: string; sources: Source[]; text: string }
  | { phase: "done"; query: string; sources: Source[]; text: string }
  | { phase: "error"; query: string; message: string }
  | { phase: "aborted"; query: string };

Note several design decisions:

• I separate searching and retrieved into two states. searching means "still searching," so the UI shows a skeleton; retrieved means "materials have been found and generation is about to begin," so the sources can already render. That very short window provides the user feedback point of "I can see the materials."
• generating carries text directly instead of maintaining a separate answer state. Binding state to the data it drives reduces inconsistency.
• There are two terminal states, error and aborted. aborted is a user-initiated interruption (show "Stopped"), while error means the pipeline failed (show retry). Do not conflate them.

An SSE consumption hook drives this state machine:

function useRagStream() {
  const [state, setState] = useState<RagState>({ phase: "idle" });
  const ctrlRef = useRef<AbortController | null>(null);

  const ask = useCallback(async (query: string) => {
    ctrlRef.current?.abort();
    const ctrl = new AbortController();
    ctrlRef.current = ctrl;
    setState({ phase: "searching", query });

    try {
      const res = await fetch("/api/rag", {
        method: "POST",
        body: JSON.stringify({ query }),
        signal: ctrl.signal,
        headers: { Accept: "text/event-stream" },
      });
      // 用 SSE 的 body 读流,这里用简化的 reader
      for await (const ev of readSse(res.body!)) {
        if (ctrl.signal.aborted) break;
        setState((prev) => reduce(prev, ev));  // 纯函数 reducer
      }
    } catch (e) {
      if ((e as Error).name === "AbortError") {
        setState((prev) => ({ ...prev, phase: "aborted" }));
      } else {
        setState({ phase: "error", query, message: "检索或生成失败" });
      }
    }
  }, []);

  const stop = useCallback(() => ctrlRef.current?.abort(), []);
  return { state, ask, stop };
}

reduce(prev, ev) is a pure function that maps SSE events to state transitions: a sources event -> retrieved, the first delta -> generating, and done -> done. Keeping transition logic in a pure function makes it easy to test and replay.

The UI layer then becomes purely "render based on phase," which is very clean:

switch (state.phase) {
  case "searching":  return <SourceSkeleton />;
  case "retrieved":
  case "generating": return <><SourceList sources={state.sources} /><Typewriter text={state.text} /></>;
  case "done":       return <><SourceList sources={state.sources} /><Answer text={state.text} /></>;
}

4. Citation Rendering: Aligning Answers and Citations

This is where a RAG streaming UI is most likely to go wrong. The requirement appears simple: [1][2] appear in the answer body, and the page should have either clickable superscripts or corresponding source cards at the bottom. But streaming introduces three pitfalls:

1. Token splitting: The model may split [1] into three tokens: [, 1, and ]. You cannot assume one delta contains a complete marker.
2. Association timing: When should a source card "light up"? My recommendation is:do not light it up the instant a token appears, because it will flicker chaotically. Wait until the answer is complete (done), parse all [n] markers in the full text, and highlight matching source cards at once.
3. Index alignment: [1] must correspond to the first item in the source list. The order must not be wrong.

Use a regex to parse citations and calculate the hit set while concatenating the stream:

const CITE = /\[(\d+)\]/g;

function useCitations(text: string, sources: Source[]) {
  return useMemo(() => {
    const hits = new Set<number>();
    for (const m of text.matchAll(CITE)) hits.add(Number(m[1]));
    return sources.map((s, i) => ({ ...s, cited: hits.has(i + 1) }));
  }, [text, sources]);
}

The answer renderer replaces [1] in the body with a clickable superscript (<sup>). The key is a two-stage rendering time: during streaming, show superscripts in gray as placeholders and keep them non-clickable; after done, highlight matched source cards, make superscripts clickable, and scroll to the corresponding card on click. Visually, that becomes "the answer finishes, and citations light up in sync," which is clean.

function renderAnswer(text: string, done: boolean) {
  const parts = text.split(CITE);  // 配合全局正则需注意,实际用 split+capture
  return parts.map((p, i) =>
    i % 2 === 0 ? p : <sup key={i} aria-disabled={!done}>{`[${p}]`}</sup>
  );
}

5. React Incremental Rendering: Smooth Streaming Markdown

The most common performance issue during generation is putting the entire Markdown string into react-markdown for a full reparse and rerender with every token. After a few hundred tokens, frames begin to drop, and longer answers become increasingly sluggish.

Several practical measures:

1. Throttle rendering; do not refresh for every token. SSE may push once every 20ms, but combine updates into one render every 50~80ms (with requestAnimationFrame or a simple buffer). The human eye cannot tell a 30ms difference, but React layout work is reduced by more than half.

function useThrottledText(stream: string, ms = 60) {
  const [text, setText] = useState("");
  const buf = useRef(stream);
  buf.current = stream;
  useEffect(() => {
    const t = setInterval(() => setText(buf.current), ms);
    return () => clearInterval(t);
  }, [ms]);
  return text;
}

2. Cache Markdown parsing. During streaming, the end of the answer keeps changing, but stable earlier portions do not need to be repeatedly parsed. Use memo and render by "stable segments / active segment." A pragmatic approach is to freeze each completed paragraph (upon a newline or period) into a memoized node, so subsequent tokens render only the final segment.

3. Wrap low-priority updates in startTransition. Update streaming text with startTransition(() => setText(...)) so React can interrupt that render when busy and prioritize interactions such as clicks and scrolling. This prevents the situation where "the answer is streaming, but I cannot click Stop."

4. Throttle Markdown features. Some features, such as KaTeX formulas and syntax highlighting, are expensive. Disable them during streaming and enable them once for the full answer after done. During streaming, the output only needs to be "readable," not "perfect."

6. Engineering: SSE Protocol, Interruptions, and Errors

Interrupting is standard for RAG streaming, not optional. Generating an answer consumes real, paid tokens. If users discover halfway through that the answer is off-topic, they must be able to stop it. Frontend implementation is simple: AbortController; the stop() in the hook above does exactly that. But the backend must cooperate: when the client disconnects, it must also close the upstream LLM stream or tokens will keep burning.

// Next.js Route Handler 简化示意
export async function POST(req: Request) {
  const ctrl = new AbortController();
  req.signal.addEventListener("abort", () => ctrl.abort());

  const stream = new ReadableStream({
    async start(controller) {
      const encoder = new TextEncoder();
      const send = (ev: string, data: unknown) =>
        controller.enqueue(encoder.encode(`event: ${ev}\ndata: ${JSON.stringify(data)}\n\n`));

      try {
        const sources = await retrieve(body.query);      // 检索
        send("sources", { items: sources });
        for await (const chunk of llm.stream({ ... }, { signal: ctrl.signal })) {
          send("delta", { text: chunk.text });           // 逐 token
        }
        send("done", {});
      } catch (e) {
        send("error", { message: "上游生成中断" });
      } finally {
        controller.close();
      }
    },
  });
  return new Response(stream, { headers: { "Content-Type": "text/event-stream" } });
}

Several easy-to-miss details:

• Recover from midstream errors: If the request fails during generating, do not clear the answer already rendered. Retain the generated text, add a "Generation interrupted" notice at the end, and provide "Continue/retry" buttons. What users dislike most is "it was halfway through, the network hiccupped, and everything disappeared."
• Timeout fallback: If an SSE connection has no delta for a long time (for example, the LLM is stuck), the frontend needs a watchdog: after 30s without a new token, actively abort and notify the user.
• Protocol versioning: SSE event names (event:) are much more useful than bare data:. The frontend may use EventSource or read the stream directly, but typed events let the reducer become a switch and improve readability substantially.

7. Experience Details: Skeletons, Progress Hints, and Scroll Following

These are details that "nobody praises when done well but everyone complains about when done poorly," yet they matter especially for long RAG workflows:

• Retrieval-stage skeletons: Do not use a spinner. Use a skeleton structurally matching the source list, with several title and text-line placeholders. A structurally matching skeleton prevents layout shifts when real content appears, making the experience feel smoother.
• Progress hints: searching shows "Searching 12 documents..." and retrieved shows "Found 5 relevant sources; generating an answer..." Specific numbers are ten times more credible than "loading..."
• Scroll following: As the answer grows during generation, the container should auto-scroll to the bottom. But there is a classic trap:when users manually scroll upward, do not seize scroll control. Only follow automatically when scrollHeight - scrollTop - clientHeight < threshold; otherwise, respect the user's current reading position.

useEffect(() => {
  const el = ref.current;
  if (!el) return;
  const atBottom = el.scrollHeight - el.scrollTop - el.clientHeight < 80;
  if (atBottom) el.scrollTop = el.scrollHeight;  // 只在贴近底部时跟随
}, [text]);

Summary

• RAG has two streams, not one request: the retrieval and generation stages each have signals that should be exposed. Do not hide them behind a single loading state.
• Use a state machine, not a boolean: clearly distinguish searching / retrieved / generating / done, and the UI becomes a pure function of phase.
• Light citations in two stages: show placeholder superscripts during streaming; after done, parse [n] together and highlight source cards to avoid flicker.
• Throttle incremental rendering: batch token refreshes, freeze stable segments, and lower priority with startTransition, so even long answers remain smooth.
• Interruptions and errors are standard: support stopping (AbortController + backend cascading abort) and do not clear content that has already been generated after an interruption.

The RAG experience is won when "users can see progress every second," and lost when "everything lands with a bang after 8 seconds." Streaming is not decoration; it is the foundational interaction model for RAG applications.