Advanced RAG: Hands-On Chunking, Vector Retrieval, and Reranking

Load company documents into a vector database, then have the model retrieve answers with top-k: this is how 90% of people understand RAG, and why 90% of RAG demos deliver mediocre results. What truly determines RAG quality is never "which embedding model you used," but how content is chunked before retrieval and reranked afterward.

1. First, Admit It: Why Naive RAG Falls Short

The naive RAG pipeline is: 文档 → 固定长度切块 → embedding → 余弦 top-k → 拼进 prompt. It has three fundamental flaws:

1. Chunking breaks semantics. A hard split every 512 tokens can cut a complete argument in half, leaving retrieval with only a fragment.
2. Vector retrieval favors semantics but is weak at exact keyword matching. Ask for "Q3 sales," and vectors can retrieve a document containing "third-quarter revenue"; ask for an exact string such as "ErrorCode=E_4021," and vectors often lose to BM25.
3. The top-k results contain noise. Only two of the top five results may be truly relevant, while the other three reduce answer quality.

The following four levers address these problems one by one.

2. Chunking: Get It Right, and Retrieval Is Half Solved

Chunking offers the highest ROI of any RAG optimization, yet it is the one most often overlooked. Common strategies include:

• Fixed length: The simplest approach, using 512 tokens with an overlap of 50. It works for introductory use cases with uniformly structured documents, but it can fragment semantics.
• Recursive splitting: Split recursively according to the \n\n → \n → 。 → 空格 hierarchy, staying as close as possible to natural paragraph boundaries. LangChain's RecursiveCharacterTextSplitter belongs to this category.
• Structure-aware splitting: Split on Markdown headings, HTML tags, or code block boundaries, and attach metadata identifying the section each chunk belongs to ({section: "计费/退款规则"}). This is the preferred choice for document-centric RAG.
• Semantic chunking: Use embeddings to measure similarity between adjacent sentences and split where similarity drops sharply. This works well for free-form text, but it is slower.

Two easily overlooked details:

• Chunks should be neither too short nor too long. Short chunks lack context ("It supports this feature"; what does "it" refer to?), while long chunks dilute the signal and consume more tokens. Paragraph-sized chunks of 200-500 tokens are usually the sweet spot for documents.
• Preserve the context window. When storing each chunk, also store its "parent paragraph" or "section path." At retrieval time, expand to the parent when needed. This is small-to-big / parent-child document retrieval: use small chunks for retrieval precision and larger chunks for answer context.

3. Retrieval: When Vectors Are Not Enough, Use Hybrid Search

Production-grade RAG almost always uses hybrid retrieval: vector retrieval for semantics plus BM25 keyword retrieval, followed by fusion.

// 伪代码：向量召回 + BM25 召回，RRF 融合
const dense = await vector.search(embed(query), k);   // 语义
const sparse = await bm25.search(query, k);           // 精确词
const fused = rrf(dense, sparse);                     // Reciprocal Rank Fusion

RRF (Reciprocal Rank Fusion) is the most commonly used fusion method. For each document, it calculates a score from its positions in the two lists using Σ 1/(rank + 60), then merges the results. It is simple, requires no tuning, and delivers stable results. PostgreSQL's pgvector plus full-text search, Elasticsearch, and Qdrant all provide built-in support.

Why is hybrid retrieval essential? Because user queries come in two modes: semantic requests such as "find the section that explains the refund process," and exact strings such as "troubleshoot ERR_CONNECTION_REFUSED." Pure vector retrieval cannot handle the latter reliably.

4. Reranking: Retrieve More, but Answer with Only the Best Results

Retrieval (recall) and reranking (precision) should be separate. First, use fast vector retrieval to fetch 20-50 results so relevant content is not missed. Then use a slower but more accurate reranker to select the 3-5 most relevant results, minimizing noise.

A reranker is a cross-encoder: it processes [query, doc] together and outputs a relevance score. It is far more accurate than a bi-encoder, which encodes the query and document separately before calculating cosine similarity, but it is also much slower. That is why it should be used only on the small candidate set produced by retrieval.

Common choices include bge-reranker, Cohere Rerank, and jina-reranker. A one-line call looks like this:

const reranked = await cohere.rerank(query, candidates, topN: 4);

In practice, reranking is one of the highest-impact individual RAG optimizations. Adding it often produces an immediate improvement in answer accuracy. It is not cheap, but the cost remains manageable because it runs on only a few dozen candidates.

5. Query Rewriting: User Wording Is Rarely Ideal for Retrieval

A user's original question is often a poor retrieval query. Common rewriting techniques include:

• Coreference resolution: In a multi-turn conversation, "What about its price?" becomes "What is the price of the iPhone 17 Pro?"
• Subquestion decomposition: Break a complex question into multiple atomic queries and retrieve them separately ("Compare the strengths and weaknesses of A and B" becomes separate searches for A, B, and their comparison).
• HyDE: First have the model generate a "hypothetical answer," then use it instead of the original question for vector retrieval. The hypothetical answer is closer to the real document in semantic space.
• Synonym expansion / Multi-query: Have the model rewrite the query in three different ways, retrieve results for each version, and merge them.

These rewrites require an additional LLM call and therefore incur a cost, but they are highly effective for ambiguous questions and multi-turn scenarios.

6. Without Evaluation, There Is No Optimization

The worst way to tune RAG is "by feel." Build a fixed evaluation set containing 50-200 问题 → 期望答案/期望引用文档 examples and run it after every change. Only then can you tell whether the system improved or regressed.

Common metrics from the RAGAS framework include:

• Context Precision/Recall: How much of the retrieved context is actually relevant, and how much of the context that should have been retrieved was retrieved.
• Faithfulness: Whether every generated answer is supported by the context, without "hallucinations."
• Answer Relevancy: Whether the answer addresses the question.

Practical advice: First raise Context Recall above 0.9 so everything that should be retrieved is retrieved, then improve Faithfulness so the model does not invent facts. The former depends on chunking and hybrid retrieval; the latter depends on reranking and prompt constraints.

Summary

Advanced RAG is fundamentally about turning "one retrieval call" into a pipeline:

query → 改写 → 混合召回(50) → Rerank(5) → 注入 prompt → 生成 → (评估)

The four levers are chunking, hybrid retrieval, reranking, and query rewriting. In ROI order, start with chunking and reranking because they require small changes and deliver large gains, then add hybrid retrieval and query rewriting. Finally, build an evaluation set. Without one, every "optimization" is just guesswork.