Agent Memory System Design: Short-term, Long-term, and Forgetting

You chat with an Agent halfway, switch tasks, come back, and it has forgotten everything—this is "no memory." An Agent that can collaborate continuously must remember: who you are, what was decided before, your preferences, and which paths you've taken. The memory system is what turns an Agent from a "one-off tool" into a "long-term partner."

But the most common mistake in memory engineering is "remembering everything"—you end up storing noise that makes the Agent more confused. This article covers how to do memory right.

1. The Three-Tier Memory Model

Borrowing the cognitive science framework, we divide Agent memory into three tiers, each with different responsibilities:

Many people mix these three tiers into the "context window," resulting in an overstuffed window with key information flushed out. The right approach is to store by tier and retrieve on demand—working memory is "right now," short-term is "today," long-term is "the past." Each turn brings only the relevant bits into working memory.

2. Long-Term Memory: What to Store, How to Store It

Long-term memory is where differentiation lies (the first two tiers are similar for everyone). The core comes down to two questions: what to store, and how to store it.

What to store (types):

• Fact type: "User is a backend engineer" "Project uses Next.js"—structured, stable.
• Preference type: "Likes short answers" "Use TypeScript for code"—semi-stable.
• Event type: "Refactored the payment module together last week"—time-stamped, easily expires.
• Relationship type: "A is B's owner"—graph structure.

How to store (storage):

• Structured tables (Postgres): Fact type, preference type. Precise queries ("this user's role"), fast and accurate.
• Vector database (pgvector/Qdrant): Event type, free text. Semantic retrieval ("history related to this task").
• Graph database (optional): Complex relationship networks. Most projects don't need this.

Lesson: Don't go all-vector from the start. Facts/preferences go in structured tables (precise, cheap), and only "semantically related" retrieval uses vectors. A hybrid of the two is the mainstream approach.

3. The Most Overlooked: Write Strategy

What to record and what not to record is more important than "how to retrieve." Blindly storing every turn into the memory store leaves it full of "okay," "hmm," "please continue" noise after a week, which drowns out real information during retrieval.

A write strategy:

1. Explicit memory > implicit inference. "Remember: I like to use pnpm" (explicit) is worth storing; "infer from the conversation that the user might like pnpm" (implicit) must be done carefully—incorrectly inferred memory is pollution.
2. Extract then store, don't store the raw text. Extract key facts from the conversation into structured entries ({type: "preference", key: "package_manager", value: "pnpm"}) rather than storing the entire conversation. Storing raw text = the memory store becomes a chat log, unusable.
3. Importance filter. Only store high-value facts (explicitly stated by the user, recurring, decision-related). One-off pleasantries and transient questions are not stored.
4. Deduplication and merging. The same fact appearing multiple times is merged into one entry (with "last confirmed time"), rather than stored as N entries.

Writing can be handled by a dedicated lightweight model ("from this turn, are there any facts worth remembering long-term? Extract into structured entries"), while the main Agent only handles the conversation. Writing is independent, asynchronous, and can be human-reviewed.

4. Retrieval: Bringing the Right Memories into Working Memory

At the start of each turn, recall a few relevant entries from long-term memory and inject them into working memory. This is isomorphic to RAG:

• Isolate by user/session: Only search this user's memory (don't mix in others' preferences).
• By relevance + recency: Prioritize high relevance, down-weight outdated entries.
• Retrieve a small, precise set: Take the 3–5 most relevant, not dozens that pollute the context.
• Attach metadata: Each memory carries "source/time/confidence," so the model knows how trustworthy it is.

5. Forgetting: The Other Half of the Memory System

A memory store that only records and never forgets gets worse over time: stale information ("user uses Vue"—actually switched to React long ago), erroneous inferences, conflicting records pile up, and retrieval quality eventually collapses. The forgetting mechanism is as important as the writing mechanism.

Several kinds of forgetting:

• Update overwrite: New facts overwrite old facts ("uses React now" replaces "uses Vue"). For time-stamped fields, the latest wins.
• Expiry eviction: Event-type memories (time-stamped) are archived/deleted when expired.
• Confidence decay: Low-confidence, long-unreferenced memories are gradually down-weighted.
• Conflict arbitration: When multiple records of the same fact conflict, use rules to arbitrate (latest, highest confidence, or human confirmation).

In a healthy memory store, the entry count should be stable (with both additions and removals), not growing infinitely. If your memory store only grows, it means you're missing a forgetting mechanism.

6. Privacy and Security

The most common pitfalls in memory:

• Cross-user leakage: Memory is strictly isolated by user_id, and retrieval must enforce a user_id filter—this is a hard constraint at the code layer, not a line in the prompt.
• Sensitive data: ID numbers, passwords, and keys should not enter the memory store (even if the user "tells" them). Run sensitive-information detection before writing.
• Deletable: When the user says "forget everything about me," you must be able to delete it completely (compliance requirement, GDPR, etc.).

7. Practical Architecture Summary

A minimal architecture for a production-grade Agent memory system:

Conversation turn → Main Agent (working memory)
                     │
                     ├─ Write pipeline (async): extract facts → filter → dedupe → store (table/vector)
                     └─ Retrieval (each turn start): by user_id + relevance, take top-3~5 and inject into working memory

Background: forgetting tasks (expiry/overwrite/decay) + privacy cleanup

Key points: Separate writing from retrieval, layer memory by type, have a forgetting mechanism, and isolate by user.

Summary

For Agent memory, quality matters far more than quantity. The three-tier model (working/short-term/long-term) manages lifetime, the write strategy manages "what to record" (explicit > implicit, extraction not raw text, importance filtering), retrieval manages "what to use" (small precise retrieval, user isolation), and forgetting manages "don't become a garbage dump" (update overwrite, expiry, decay).

Remember this: A memory system without forgetting is not a good memory system. A memory system that judiciously records only what's right and actively forgets what's wrong keeps the Agent far more stable than one that "remembers everything." Memory engineering is the foundation of an Agent's long-term value, and deserves to be designed as a first-class citizen.