Agent Memory Compression and Consolidation: Summary Lineage, Reversible Merges, and Loss Gates

Long-term collaboration accumulates thousands of events; stuffing all the raw text into the prompt crowds out the current task, while letting the model "summarize and overwrite the old summary" every few days loses negations, numbers, and unfinished constraints round by round. Context engineering discusses window tradeoffs; this post only deals with persistent memory consolidation: a summary is an invalidatable derived index, not a new raw fact.

Bind an immutable input snapshot to every consolidation

A worker reads the high-water mark H, the current inputEpoch, and the alias expected tuple (parentArtifactId, sourceEpoch, coversThrough), then selects valid entries with sequence <= H. A normal tail append only advances H; only undo, replacement, a new conflict, or a trust change advances inputEpoch. The manifest records both fences, input IDs and hashes, the selection rule, and the prompt and model versions; generating completion does not equal readable.

{
  "artifactId": "summary:user-7:000184",
  "newCoversThrough": 184,
  "inputEpoch": 27,
  "expectedAlias": {
    "parentArtifactId": "summary:user-7:000090",
    "sourceEpoch": 26,
    "coversThrough": 90
  },
  "inputIds": ["m-91", "m-103", "m-180"],
  "inputRoot": "sha256:9f5c8a7d2e3b14c6a0d981f7e4b52c368a1d0f9e7c6b4a32518d7e0c9b6a4f21",
  "selectorVersion": "v3",
  "promptVersion": "consolidate-v5",
  "modelSnapshot": "approved-model-2026-06"
}

Hash inputs using the RFC 8785 JSON Canonicalization Scheme or an equivalent fixed-byte protocol; do not rely directly on object field order. The generated result outputs structured facts, open items, conflict references, and a natural-language summary at the same time; every claim must list derivedFrom, forming a W3C PROV-style derivation graph. The claim-to-evidence approach of RAG citation provenance can be reused, but the evidence is an internal memory version.

The fidelity gate checks loss-intolerant fields first

A fluently written model does not mean correct compression. Before consolidation, extract active constraints, negation facts, exact numbers, dates, entity IDs, permission scopes, and unresolved conflicts into a must-preserve set. The output is deterministically parsed and checked item by item; missing entries, polarity flips, unit changes, or writing a conflict as a definite fact are all rejected from publication.

for (const fact of mustPreserve) {
  const hit = output.claims.find(c => c.key === fact.key);
  assert(hit, `missing:${fact.id}`);
  assert(canonicalValue(hit.value) === canonicalValue(fact.value));
  assert(hit.polarity === fact.polarity);
  assert(hit.derivedFrom.includes(fact.id));
}
assert(output.unresolvedConflicts.sort().join() === expectedConflicts.sort().join());

Open-ended events can be evaluated for coverage by a human or model judge, but it cannot replace the precise invariants above. When the summary exceeds the token budget, prefer keeping low-value events in the cold-retrieval layer over truncating JSON or deleting open items. The bucketing principle from Coding Agent context budgets applies here too.

Separate the raw layer, the derived layer, and the read alias

The summary is first written to a new artifact ID. The switch transaction locks the source clock and alias, re-reads each input version, confirms it is still valid, the hash is unchanged, no new conflict has appeared, and the clock still equals $inputEpoch, then performs a three-way CAS on the expected alias:

UPDATE memory_summary_alias
SET artifact_id = $artifact, source_epoch = $inputEpoch,
    covers_through = $newCoversThrough
WHERE subject_id = $subject
  AND artifact_id = $parentArtifactId
  AND source_epoch = $expectedSourceEpoch
  AND covers_through = $expectedCoversThrough
  AND $newCoversThrough >= covers_through;

The source clock condition can be guaranteed by a SELECT ... FOR UPDATE in the same transaction or an UPDATE subquery. If the affected row count is not 1, rebuild; parentArtifactId + sourceEpoch + coversThrough must all match, and coverage must be monotonically non-decreasing. Retrieval requires the alias epoch to equal the clock; when the summary hasn't been rebuilt after a change, fall back to raw retrieval. Artifacts must carry lineage and must not be written back as new facts with no source.

When a source is revoked, a temporal version is replaced, or a taint policy is upgraded, mark all descendants STALE along derivedFrom, remove them immediately from the recallable set, then rebuild from still-valid inputs. Old summaries can be kept for audit but must not participate in answers. Temporal conflict resolution decides which versions are valid; the consolidator has no authority to adjudicate on its own.

Use mutation tests to prove compression doesn't silently change meaning

Fixed samples cover double negation, close numbers, unit conversion, same-name entities, cross-project preferences, revoked sources, and mutually conflicting decisions. Randomly insert unrelated events, regroup input batches, and retry the worker twice; the must-preserve results and the conflict set should be unchanged. Race tests let the H=220 new task complete the switch first, then let the same-parent H=184 old task complete, asserting the old task's CAS fails and coverage does not regress; another test revokes an input after the candidate passes the gate but before the switch, asserting that the epoch change or validity recheck blocks the switch. Then inject generation timeout, missing output field, and a crash after writing the artifact but before switching the alias; no half-finished result must be retrievable.

Every day, rebuild sample summaries from the raw layer and compare key-fact recall rate, erroneous-new-fact rate, conflict retention rate, average token reduction, and number of stale artifacts visible. An erroneous-new-fact rate > 0 or a non-zero stale-visible count is a blocking failure; do not mask it with average similarity.

Release by shadow-generating first, having humans review, then switching aliases per tenant. On failure, bypass the summary and query the raw layer; do not roll coverage backward; then build a new artifact from currently valid inputs using the previous selector/model with coverage no lower than the current value, and forward-recover using the same CAS. Raw events, manifests, and failure artifacts are not deleted, and the failed summary must not be summarized again.

Further reading

• Context Engineering
• RAG Citation Provenance
• Coding Agent Context Budget
• Agent Replay Evaluation