Index Freshness in Large Monorepos: Incremental Context for Coding Agents

Symbol, reference, and summary indexes for a large monorepo can take tens of minutes to build. When a coding agent queries function definitions and callers, seeing results does not mean they belong to the current HEAD: files may have been renamed, exports deleted, and the worktree may contain uncommitted changes. A stale index is more dangerous than no index because it provides incorrect evidence that appears precise.

Repository Maps for Coding Agents describes the repository fact layer. This article focuses on how an index proves which commit it covers, which changes have not yet been incorporated, and how query results should degrade.

All commands in this article are intended to run in a Linux/GNU CI container with Bash, GNU coreutils, and Git. macOS and Windows runners must use the same container.

Freshness Is a Version Relationship, Not an Update Time

The index manifest records the source commit, indexer and language service versions, configuration digest, inclusion and exclusion rules, shards, build state, and content digest. updatedAt=刚刚 is meaningless: the index may have just finished building an old branch. Only a version whose publication state is complete and whose manifest digest passes validation may serve queries.

{
  "sourceCommit": "7a50f4d",
  "indexer": "code-indexer@sha256:31c9...",
  "configDigest": "sha256:90ab...",
  "shards": 128,
  "completedShards": 128,
  "state": "complete"
}

The query service compares sourceCommit, the current HEAD, and the dirty worktree. If the indexed commit equals HEAD and the worktree is clean, the index can be used directly. If the indexed commit is an ancestor of HEAD and the diff is within budget, the service loads an incremental overlay. If the branches have diverged, the configuration has changed, or events have been lost, freshness cannot be proven, so the index must be rebuilt or the source read directly.

HEAD_SHA=$(git rev-parse HEAD)
INDEX_SHA=$(jq -r .sourceCommit .agent-index/manifest.json)
test "$(git merge-base "$INDEX_SHA" "$HEAD_SHA")" = "$INDEX_SHA"
git diff --name-status --find-renames "$INDEX_SHA..$HEAD_SHA"

TMP_ROOT=$(mktemp -d)
TMP_INDEX="$TMP_ROOT/index"
cleanup() {
  test ! -e "$TMP_INDEX" || unlink "$TMP_INDEX"
  test ! -e "$TMP_INDEX.lock" || unlink "$TMP_INDEX.lock"
  rmdir -- "$TMP_ROOT" 2>/dev/null || true
}
trap cleanup EXIT
GIT_INDEX_FILE="$TMP_INDEX" git read-tree HEAD
GIT_INDEX_FILE="$TMP_INDEX" git -c core.filemode=true add -A -- .
WORKTREE_TREE=$(GIT_INDEX_FILE="$TMP_INDEX" git write-tree)
printf 'head=%s\nworktreeTree=%s\n' "$HEAD_SHA" "$WORKTREE_TREE"
cleanup
trap - EXIT

Starting from HEAD, the temporary index incorporates tracked modifications, deletions, and unignored untracked files. The tree OID covers the actual blobs, complete paths, and Git modes without requiring filename parsing. Queries return indexCommit, headCommit, worktreeTree, and source ranges. A dirty submodule state is not represented in the parent tree, so each submodule's HEAD and tree must be recorded separately.

Incremental Logs Must Represent Deletions and Renames

File watchers are useful for low-latency signals, but they cannot be the sole source of truth. Process downtime, queue overflows, and Watchman recrawls can all cause events to be lost. The incremental indexer uses Git commit diffs as its reconstructible mainline, worktree events as temporary overlays, and assigns each batch a monotonic sequence number and checkpoint.

After an added file is parsed, its symbols and dependency edges are written. For a modified file, edges derived from the old content are removed before new edges are written. A deleted file produces a tombstone that prevents old definitions from continuing to appear in results. A rename preserves both the identity migration and the path change. Processing only files that currently exist leaves ghost symbols behind, which is especially misleading when searching for callers.

A batch is completed in a staging index. After validating document and symbol counts, reverse edges, and tombstones, the system atomically switches the current pointer. A read request must see either the old complete version or the new complete version, never a mixed state in which only 90 of 128 shards have finished. This consistency model resembles Consistency in Incremental RAG Indexing, but a code index must also handle symbol binding, compiler configuration, and generated-file boundaries.

Context Packages Combine a Base Index with Overlays

A task starts from a target symbol or file and expands definitions, callers, tests, and configuration over the base graph, then applies the commit diff and dirty-worktree overlay. Modified files are read directly from disk first, deleted nodes are removed from the candidates, and new files are parsed immediately. A summary may be reused only when the source content hash is unchanged.

Context selection is governed by an explicit budget: retain task targets, public contracts, and high-risk callers first; add implementations and tests by dependency distance next; and include historical notes only at the end. Context Budgets for Coding Agents defines the compression strategy. The index layer must ensure that the candidates it supplies are not stale.

If the incremental change set exceeds 500 files, the workspace configuration or lockfile changes, or the language service cannot restore the same project graph after restarting, trigger a shard rebuild immediately. Thresholds come from load testing and should not be copied from this example. While over budget, the system can run LSP queries and read source directly for a small number of targets, explicitly labeling the result partial. It must not return an empty caller list and claim there is no impact.

Calibrate the Freshness Gate with Known Changes

The acceptance set should include cross-package renames, deletion of public exports, file moves, path-alias changes, generated-file updates, uncommitted edits, forced branch switches, and lost watcher events. For each scenario, query definitions and references and require the results to match a golden index built from the complete current source tree. Test Impact Analysis for Coding Agents offers a model for assessing affected tests, but index acceptance must also verify that returned content and line numbers actually belong to the current version.

Production metrics track index lag in commits, incremental queue age, recrawls, tombstones, query fallback rate, dirty-overlay parsing time, and freshness rejections. Monitoring only whether the index job succeeded misses cases where the current pointer was never switched, shards are missing, or the query service is still reading an old version.

Roll Back the Index, Not the Source Code

When a new index is defective, atomically switch the current pointer back to the previous complete version that matches its sourceCommit, clear incompatible overlays, and rebuild from the current HEAD. If an old index trails the current code, it may be used only for diagnostics. Queries must continue reading the source directly or automated rewrites must be paused. Never roll back developer code merely to make the index "match" again.

A local version directory can be restored with a single atomic pointer replacement. LAST_GOOD must come from a signed publication record:

test "$(jq -r .state ".agent-index/versions/$LAST_GOOD/manifest.json")" \
  = "complete"
ln -sfn "versions/$LAST_GOOD" .agent-index/current.next
mv -Tf .agent-index/current.next .agent-index/current

For object storage, use compare-and-swap with a condition on the old version so two recovery jobs cannot overwrite each other. After switching the pointer, compare that version's watermark with the current HEAD. If it is behind, force direct source reads and enqueue a rebuild.

Large repositories can use Monorepo Tooling to provide project boundaries and task graphs, but whether an agent can trust a query ultimately depends on whether the commit watermark, event completeness, overlays, and failure fallback are all verifiable. Freshness is not a timestamp. It is a version proof that can be recomputed from the current code to every fragment of context.