Code Search Stack Migration: Lexical, AST, and Embedding Shadow Evaluation and Switch Gates

"Find the definition of Invoice" suits a symbol index, "where is refund timeout handled" may need semantic retrieval, while "all scripts that call this environment variable" is better served by exact text search. Treating a set of polished natural-language demos as selection evidence easily yields a system that explains code but misses renames, uncommitted files, or restricted directories. Code Intelligence: AST + LLM introduces the combined capabilities; this article only governs search-stack comparison and migration acceptance points.

Freeze queryPortfolio.v1

The search owner publishes codeSearch.queryPortfolio.v1. Each queryId binds a repository tree, query text and language, user intent category, authorized principal, required and forbidden blobs/symbols, allowed equivalent results, review owner, and difficulty. The portfolio covers at least exact identifier, definitions/references, call paths, configuration keys, error semantics, cross-language interfaces, renames, deletions, and dirty working trees, plus "should return nothing" negative queries.

schema: codeSearch.queryPortfolio.v1
queryId: billing-refund-timeout
tree: sha256:18c...
kind: behavior-location
principal: role:payments-dev
relevant: [blob:91a...#RefundWorker, blob:77c...#retryPolicy]
forbidden: [path:legal-private/**]

Relevance cannot be generated solely by the candidate engine itself. Two reviewers familiar with the repository annotate without knowing the engine identity; disagreements are arbitrated by a third person. Reference graphs or LSP can provide candidates but cannot directly become ground truth. The portfolio records the incompletely annotated pool and judgment depth to avoid treating all unreviewed results as irrelevant. A new generation is published after repository upgrades; high scores on an old tree cannot prove the current code works.

Lexical, Tree-sitter/AST, LSP, and Embedding each pin their parser, tokenizer, model, dimensions, normalization, chunk, and rerank configuration. The index manifest binds tree, submodule, generator, ACL policy, build time, watermark, and shard digest. Programming Agent Repository Map decides which structural hints to provide; search selection must prove these hints come from the same repository snapshot, not a mix of stale indexes.

Dual Indexes Answer the Same Authoritative Query

The shadow gateway sends the normalized query, principal, target tree, and dirty overlay digest to both the old and new stacks simultaneously. Responses retain ordered results, scores, explanation categories, index generation, elapsed time, and errors, but only old results are handed to the Agent. Exact queries compare recall@k, MRR, and the first correct result; exploratory queries report judged recall, nDCG, and the no-result rate; all types also report p50/p95 latency, index cost, and permission violations.

Vector similarity scores cannot be compared directly with text BM25 scores; the cross-engine winner is decided by task metrics. Duplicate queries are deduplicated by event ID; timeouts and partial shard failures are counted as INCOMPLETE_RESULT, and the returned top-k must not masquerade as complete. Queries containing secrets or source code do not enter ordinary analysis logs; only version, category, and a controlled summary are retained. Vector backends such as pgvector provide nearest-neighbor capability; relevance, ACLs, and freshness remain the responsibility of the upper-layer contract.

Incremental updates are accepted separately: after indexing tree N, consecutively perform additions, modifications, renames, deletions, permission tightening, and force-push, recording the latency from commit event to searchable generation. Monorepo Index Freshness defines the incremental context watermark; this migration brings the same watermark into the old vs. new stack comparison. A dirty working tree must be applied as a request-level overlay and never written into the shared canonical index.

Deliberately Test Common Defects and Permission Leakage

The fault suite injects new syntax the parser doesn't support, LSP server crashes, wrong-generation embedding models, a missing vector shard, renames that only handle the new path, lost deletion tombstones, ACL caches holding an old epoch, case and Unicode identifiers, symlink boundary crossing, and a reranker that promotes a forbidden path to the top. It also attacks vector retrieval with semantically similar but behaviorally opposite functions, and attacks lexical ranking with old API names from comments.

The gate requires zero forbidden results returned, zero stale results after the declared deletion and de-authorization watermark, 100% exact-symbol recall@10, a lower bound for critical behavioral query recall@20 meeting the pre-registered value, no increase in negative-query false-positive rate, INCOMPLETE not counted as success, and p95 latency and per-query cost within budget. Any ACL leakage directly disqualifies the candidate and cannot be offset by average relevance. RAG Embedding Migration handles knowledge-retrieval vector replacement; the code stack must additionally preserve blob, symbol, tree, and permission identity.

The switch first builds both indexes and shadows for at least one full release cycle, then routes 5% of read-only searches into the new stack via stable per-principal bucketing. During the rollback window, the old stack must keep consuming commit, ACL, and tombstone events; it periodically issues a readiness receipt binding the current tree, ACL epoch, deletion watermark, and complete shard set. The alias CAS can only switch to a generation consistent with the authoritative watermark. If critical-layer recall drops, freshness p99 exceeds the limit, UNKNOWN shards exceed the threshold, or a single permission leak occurs, switch back only if the old receipt is still fresh; otherwise, use a controlled lexical fallback against the current tree, and reject the query when the permission or deletion watermark cannot be proven. After the failing stack exits serving, it can be replayed in isolation, but you cannot stop consuming events and then treat a stale index as a rollback point. The old index is cleaned up only after deletion, rename, and rollback drills all pass.