Tuning Relevance in AI Search

Search for "refund process" and the top result—does it actually cover refunds, or is it some tangential scrap about "return policy"? Whether search works hinges on exactly this one judgment. Recall answers "is it there at all?", relevance answers "is it the right one?". A search that returns 20 entirely irrelevant results is worse than just saying "not found"—it wastes the user's time and burns through trust.

1. Relevance Is the Soul of Search

Many people equate "search" with "retrieval": vectorize the query, do top-k recall, done. But recall only pulls "possibly relevant" candidates into the pool; what users see and click is the top few after ranking. Whether these top few are actually relevant is the real determinant of the search experience.

First establish two key understandings:

• Recall ≠ relevance. Recall chases completeness (did everything that should come back come back?); relevance chases precision (are the things that came back the right ones?). Pure vector top-k recall can be decent while relevance collapses—"semantically close" and "actually answers this question" are not the same thing. Searching "iPhone battery life" and recalling a paragraph about "Android batteries" is semantically close but off-topic.
• Returning a pile of irrelevant results is worse than returning nothing. Return "not found" and the user will retry with different words; return ten irrelevant results and the user concludes "this system is useless" and leaves. There's a classic phenomenon in search engines: click-through rate falls off a cliff after the first page of results. Poor relevance is the same as no relevance.

2. The Six Core Relevance Signals

Judging "relevant or not" is not a single dimension; it's a multi-signal synthesis. Production-grade search stacks at least these:

Going through them one by one. Semantic similarity (vector) excels at "rephrased queries"—searching "refund" can recall a paragraph about "return and refund process"—but it's weak on exact strings and proper nouns; "ErrorCode=E4021" often fails to vector-match. Keyword matching (BM25) is complementary to vectors; it wins hands down on proper nouns, code, model numbers, and error codes—these exact-match queries. Production search is almost always a hybrid of the two.

Authority decides who gets promoted: two articles on the same disease—one a tertiary hospital guideline and the other a content-mill rewrite—the authority signal is what separates them. Academic search looks at citation counts; general search looks at domain authority / backlinks; internal knowledge bases look at author seniority and document upvotes. On freshness: searching "Next.js deployment" returns a 2020 tutorial and a 2026 doc that are both "relevant," but their content may be entirely different (App Router, Server Actions). For news-type queries the freshness weight must be maxed; surfacing a five-year-old "latest update" is an incident. Personalization handles disambiguation: the same "apple" query—orchardists want produce news, developers want company news. Clicks (implicit feedback) are free, massive-volume relevance labels, on the order of thousands of times more data than manual labels—covered separately below.

3. Weighted Fusion: Tune Weights by Scenario

These signals are not simply added; they're weighted scoring. A typical fusion:

// Multi-signal weighted fusion (pseudocode)
score =
    0.35 * normalize(semantic_sim) +   // semantic
    0.25 * normalize(bm25)         +   // keyword
    0.15 * normalize(authority)    +   // authority
    0.10 * normalize(freshness)    +   // freshness
    0.15 * normalize(personalized);    // personalization

The key is not the formula but the weights must be tuned per scenario:

• Academic search: crank authority up (citation counts and impact factors decide everything); freshness weight can be low (a ten-year-old foundational paper is still relevant).
• News / information: max out freshness; yesterday's headline is already old news today.
• Internal knowledge base: authority comes from author and department; personalization is weak (everyone queries the same document set).
• E-commerce search: personalization + sales + review rating carry high weight; freshness is secondary.
• Code / docs search: keyword weight is highest (exact-match symbol names); semantics is auxiliary.

Rule of thumb: there is no "universal weight," only "a sensible weight for the current scenario." For cold start you can copy a comparable product's mix, but you must tune it against your own evaluation set. On fusion order, a common approach is "two-stage": first use cheap signals (vector + BM25) to coarsely recall 50–100 results, then use expensive signals (personalization, authority) + rerank to refine down to 10. Balancing performance and quality.

4. Rerank: Promote the Most Relevant

Weighted fusion is linear scoring with a limited ceiling—it cannot capture fine-grained interaction between query and document. Enter rerank.

A reranker is a cross-encoder: it pairs [query, doc], runs them through the model together, and outputs a refined relevance score. Compared to a bi-encoder (encoding query and doc separately and then computing cosine), it can see the interaction between the two and is much more accurate—but about an order of magnitude slower, so it can only run on the small set that survives recall.

// Two-stage: coarse recall + refined rerank
const candidates = await hybridSearch(query, 50);             // vector + BM25 coarse recall
const ranked = await reranker.rerank(query, candidates, 8);   // cross-encoder refined rerank

Why is the effect immediate? Because the recall stage (vector + BM25) has large ranking error; the most relevant item in the top-50 may sit at position 30, and the cross-encoder can push it into the top 3. For the user, position 3 vs. position 30 is the difference between "usable" and "garbage." On model selection, bge-reranker, Cohere Rerank, and jina-reranker are all mature, and cost is controllable—you only run a few dozen candidates. If your search doesn't yet have rerank, this is the single highest-ROI optimization; do it first.

5. Offline Evaluation: Turn "Feels Relevant" into "Scores Relevant"

The biggest taboo in tuning relevance is "going by feel." You need a labeled evaluation set to quantify "relevant or not."

Building the set: pick 100–300 real queries, and for each one manually label "which documents are relevant, and to what degree" (usually three tiers, 0/1/2: not relevant / relevant / highly relevant). This is grunt work, but without it every tuning decision is mysticism. Run these core metrics:

• nDCG@k (Normalized Discounted Cumulative Gain): the most common metric for relevance ranking; it accounts for both relevance degree and position—the earlier the position and the higher the relevance, the higher the score. nDCG@10 = 0.82 means the top-10 ordering is decent quality.
• MRR (Mean Reciprocal Rank): the average position of the first relevant result. MRR = 0.5 means the first relevant result is, on average, at position 2.
• Recall@k: whether the relevant document(s) made the top k; this leans toward recall and should be read alongside nDCG.

DCG@k  = Σ rel_i / log2(i + 1)      (the later the position, the larger the discount)
nDCG@k = DCG@k / iDCG@k            (actual ranking / ideal ranking)

Usage: every time you tune weights / swap rerankers / change the fusion strategy, run the evaluation set and check whether nDCG went up. Ship if it goes up, roll back if it goes down—the same logic as RAG evaluation. An advanced practice is to slice by query type (navigational / informational / transactional) to see which type drags things down, and tune weights accordingly. Bad-case analysis is always more useful than staring at the total score.

6. Click Feedback: Free Massive Labeling—But Watch Out for Position Bias

Manual labeling is expensive and slow, while every user click is an implicit label: clicked = probably relevant, skipped = probably not relevant, clicked and immediately bounced = unsatisfied. This is implicit feedback.

Several uses:

• CTR fed back into ranking as a relevance signal: if a result's CTR is high for a query, boost its weight.
• Dwell time / bounce rate to correct click quality: a click that bounces in 3 seconds is not real relevance.
• Click skip pairs: "user jumps from result A to B" implies A failed and B is more likely relevant.

But click data has a giant pit: position bias. Results at the top naturally have higher CTR—not because they're more relevant, but because users are too lazy to scroll down. Using raw CTR as relevance will keep pushing "what's already on top" further up; the Matthew effect locks in a suboptimal solution. The fix is debiasing: use Expected Reciprocal Rank (ERR), position randomization, or counterfactual estimation to strip out the position factor. Production-grade systems all do this layer. Click feedback without debiasing is more dangerous than no feedback at all—it systematically locks you into a suboptimal ranking.

7. Avoid "Looks Relevant but Is Useless"

Finally, a commonly overlooked but fatal issue: relevance ≠ usefulness. A result can match every keyword and be semantically close, yet have garbage content—rewrites, padding, outdated, or SEO-farm articles. Promoting these "looks relevant" results gives a worse experience than irrelevance.

A few lines of defense:

• Quality filtering: run a quality score before indexing (content length, information density, source reputation); low-quality content never enters the index, or gets demoted.
• Deduplication / near-dedup: the same content gets rewritten dozens of times—keep only the authoritative original source. SimHash / MinHash for near-dedup.
• SEO spam detection: keyword stuffing, clickbait, content-farm signatures—train a dedicated classifier to filter them out.
• Content freshness check: for time-sensitive queries, content older than a threshold is filtered out, no matter how high the "relevance score" is.
• User feedback channels: thumbs-down, "result not relevant" button. Negative signals are a goldmine and can be fed straight back into the ranking model.

Rule of thumb: the people building search obsess over "is the recall accurate," but the real reason users leave is usually "the top results are garbage." Quality gating matters as much as relevance ranking—don't optimize only the latter.

Summary

• Relevance is the soul of search: recall answers "is it there," relevance answers "is it the right one." Returning a pile of irrelevant results is worse than returning nothing.
• Multi-signal fusion: semantic, keyword, authority, freshness, personalization, clicks—tune weights per scenario; there is no universal mix.
• Rerank is the highest-ROI single optimization: the cross-encoder surfaces relevant results buried in the recall stage.
• Offline evaluation turns feel into numbers: labeled set + nDCG/MRR; run it after every change, ship only if it goes up.
• Click feedback is free labeling, but you must debias: without removing position bias, the feedback gets more skewed the more you use it.

Search-tuning maturity comes down to one thing: can you say "this revision took nDCG@10 from 0.71 to 0.83" instead of "it feels more accurate." Run this loop—signals, fusion, rerank, evaluation, feedback—and search graduates from "a working demo" to "a good product."