Long-term memory for LLM agents

For NAO + anyone building agents that have to *remember*. An LLM is stateless: between two API calls it forgets everything that isn't in the prompt. "Agent memory" is the engineering that makes a stateless next-token predictor behave as if it had a durable, growing mind. This article is the honest mechanics — MemGPT/Letta's OS-style paging, the Generative-Agents memory stream, and the plain vector/summary baselines everyone actually ships — plus how OpenAlice's file-first memory maps onto all of it.

The one-sentence idea

An LLM has no memory; the *system around it* has memory. Long-term memory = a storage layer outside the context window + a policy that decides what to write, what to forget/compress, and what to read back into the prompt at the right moment. Every named system below is just a different answer to those three verbs: write, compress, retrieve.

Why it matters

The context window is RAM, not disk. It is finite (even a 1M-token window is finite), expensive (you pay per token, every turn), and — crucially — lossy in the middle: models reliably attend to the start and end of a long prompt and miss facts buried in the middle ("lost in the middle"). So "just use a bigger window" does not solve memory. You need a hierarchy: keep a small, curated slice in-context and a large, searchable slice outside it.

How much this matters is measurable. On long conversational-memory benchmarks (35 sessions, 300+ turns), even RAG-augmented LLMs lag far behind humans, especially on temporal/causal reasoning ("what changed since last week, and why") — the survey literature is blunt that this is unsolved (Memory for Autonomous LLM Agents, 2026). MemGPT's own headline: on its Deep Memory Retrieval consistency task, GPT-4 with a fixed window scored 32.1%; the same model wrapped in MemGPT scored 92.5% — the lift is entirely memory engineering, same weights.

How it actually works (real mechanics)

There are two foundational architectures worth knowing cold, plus the two baselines they're reacting against.

Baseline 0 — buffer + summary (what most apps ship)

The simplest durable memory is a rolling buffer: keep the last N turns verbatim; when you'd overflow, summarize the oldest turns into one paragraph and prepend that summary. LangChain's ConversationSummaryBufferMemory is exactly this. Cheap, no infra. The failure mode: summaries are lossy and compounding — summarizing a summary, turn after turn, erodes specifics (names, numbers, who-said-what) until the agent confidently misremembers.

Baseline 1 — vector memory (RAG over conversation)

Embed every message (or fact) into a vector and store it. At each turn, embed the current query and pull the top-k nearest neighbours back into the prompt. This is retrieval-augmented memory: unbounded storage, sub-linear lookup, nothing is "forgotten." Failure modes: (a) relevance ≠ usefulness — cosine similarity surfaces topically-similar text, not the causally needed fact; (b) no notion of recency or "this got overwritten last week"; (c) it retrieves fragments, so the agent sees ten disconnected snippets with no narrative. These baselines are where the two real architectures improve.

Architecture A — MemGPT / Letta: the LLM as its own memory manager

MemGPT (Packer et al., 2023; now the Letta framework) borrows the OS virtual-memory metaphor directly. There are two tiers:

Main context (= physical RAM, the actual prompt the model sees), in three parts:

System instructions — read-only; the control-flow rules and the memory- function schemas.
Working context — a small fixed-size read/write scratchpad for stable facts about the user and the agent's persona. Always in-prompt.
FIFO message queue — rolling recent messages, with a recursive summary of everything already evicted sitting at its head.

External context (= disk, never directly in-prompt, only via function call):

Recall storage — the full database of all past messages (searchable).
Archival storage — a read/write store for arbitrary long text the agent chooses to keep.

The defining move: the LLM edits its own memory by calling functions, and the results of those calls are fed back to it. The model emits structured calls like working_context.append(...), archival_memory.insert(...), archival_memory.search(query), conversation_search(query). A parser executes them and returns the output as a new message. So memory management is in-band — it's just more LLM turns, not a separate hand-written controller.

Two control-flow mechanisms make this work:

Memory-pressure warnings. When the prompt hits ~70% of the window, the system injects a warning so the model can proactively flush facts to external storage before data is lost. At 100%, the queue manager evicts ~50% of messages and folds them into the recursive summary (new_summary = summarize(old_summary + evicted_messages)) — a multi-generational summary that degrades gracefully rather than dropping data on the floor.
Heartbeats / yields. Each function call carries request_heartbeat. If true, MemGPT immediately runs another inference step (so the model can chain: search → read page 2 → search again → finally answer) — this is how it pages through a document larger than its window. If absent, the agent yields control back and waits for the next external event. This is literally an OS interrupt/yield loop with the LLM in the driver's seat.

Letta productionizes this as memory blocks: a block has a label (persona, human, …), a description (the prompt the agent uses to decide how to read/write it), a value, and a character limit. Blocks are always visible — compiled into the context window in XML-ish form, no retrieval needed — and edited via tools (core_memory_append, core_memory_replace, memory_insert, memory_rethink). Blocks can be shared across agents: attach the same block to two agents and an edit by one is instantly visible to the other (synchronized shared memory). Letta also adds sleep-time agents — a background agent that reorganizes/consolidates memory between user turns, so the consolidation cost is off the critical path.

Architecture B — Generative Agents: the memory stream

Park et al. (2023) — the "Smallville" simulation of 25 believable NPCs — uses a different and very influential design: the memory stream, an append-only list of natural-language observations, each timestamped with its creation and last- access time. There is no working-context block; everything is retrieved. The intelligence is in the retrieval score, computed per memory at query time as:

score = α_recency · recency + α_importance · importance + α_relevance · relevance

with all three weights α = 1 in the paper, and the three components each min-max normalized to [0,1]:

Recency = exponential decay over time since the memory was last accessed, decay factor 0.995 per sandbox-hour. (Note: decay on last access, not creation — frequently-used memories stay "warm." A subtle, good idea.)
Importance = the LLM rates the memory 1–10 at write time ("1 = brushing teeth, 10 = a breakup / college acceptance"). This is a learned salience signal the vector baselines lack entirely.
Relevance = cosine similarity between the query embedding and the memory embedding (the only "RAG" component).

The top-scoring memories are pulled into the prompt. On top of this sit two more LLM-driven processes:

Reflection. Periodically the agent synthesizes higher-level memories. Trigger: when the sum of importance scores of recent observations exceeds a threshold (150 in the paper). The agent then asks the LLM "what are the 3 most salient high-level questions about these statements?", retrieves memories for each question, and asks the LLM to extract ~5 insights — which are written back into the stream as new memories, with citations to the source observations. Reflections can themselves be reflected on, forming a tree of increasingly abstract beliefs ("Klaus is passionate about research"). This is the part vector memory cannot do: it manufactures new knowledge by compression, not just retrieval.
Planning uses the same retrieval to turn beliefs into a daily schedule, then recursively decomposes it, reacting and re-planning as new observations arrive.

The key insight to carry away: MemGPT lets the model manage memory imperatively (it calls functions); Generative Agents manages memory declaratively (a scoring function decides) plus an LLM consolidation pass (reflection). Most production systems in 2025–26 are hybrids of these two.

Key ideas & tradeoffs

In-context vs. retrieved. Always-in-prompt memory (working context / Letta blocks) is reliable but costs tokens every turn and is size-capped. Retrieved memory (recall/archival, vector store, memory stream) is unbounded but only as good as the retrieval. Real systems keep a small always-on block + a large retrieved tail. Picking the split is the whole game.
Compression is lossy and compounds. Recursive summaries and reflections buy capacity by destroying detail. MemGPT's recursive-summary-of-summary and Park's reflection-of-reflection both degrade; the engineering trick is to keep the raw observations underneath and only treat the summary as an index.
Salience beats similarity. The single biggest idea the vector baseline misses is importance — an explicit "how much does this matter" score (Park's 1–10, or MemGPT's let-the-agent-decide-what-to-archive). Recency + importance + relevance together retrieve far better than relevance alone.
Who manages memory? Three options: a hand-written controller (cheap, brittle), the LLM itself in-band (MemGPT — flexible, but each memory op costs an inference and can hallucinate the wrong edit), or a separate background agent (Letta sleep-time, OpenAlice's circadian consolidation — moves cost off the hot path). 2026 surveys note a shift toward learned memory control (MoE-gated retrieval weights, learnable stopping) over these hand-set α's.
Long context ≠ long-term memory. They're complementary. A bigger window reduces write-time compression pressure (you can keep more raw text), but middle-of-context loss and per-token cost mean you still want an external tier.

Honest caveats & limitations

It's still unsolved at the hard part. Temporal and causal reasoning over long histories ("what did the user believe last month vs. now, and what changed it") remains far below human level even with RAG (2026 surveys). Don't promise a memory system "remembers everything correctly over months" — it doesn't.
Self-editing memory can corrupt itself. When the LLM writes its own memory (MemGPT/Letta), a hallucinated core_memory_replace can overwrite a true fact with a false one, and that error becomes the new ground truth, compounding. This is an active security topic ("memory poisoning"): an adversarial input that gets stored persists across sessions. Treat agent-written memory as untrusted and version/audit it.
The numbers are domain-specific. Park's 0.995 decay and 150-importance threshold are tuned to a sandbox where one "hour" is a game tick. Lift them into a real product unchanged and they'll be wrong; they are defaults to tune, not laws.
MemGPT's lift depends on a strong base model. The function-calling / self-management loop needs a model that reliably emits correct structured calls and reasons about when to search. Weaker models flail in the loop, burning tokens without converging — the 92.5% figure is GPT-4-class.
Retrieval quality dominates everything. All of these architectures sit on top of an embedding model + ranker. Bad embeddings or naive top-k make the most elegant memory architecture useless. Memory is, underneath, a [[graphrag]] / ranking problem as much as a storage one.
Evaluation is immature. "Did it remember well?" is hard to score; most benchmarks test fact recall, not the harder belief-update / contradiction- handling that real assistants need.

How it connects to OpenAlice

OpenAlice already runs a memory architecture that maps cleanly onto the two tiers above — and deliberately chose the file-first route over a database, which matters here.

Recall storage = `sessions.jsonl`, per chat. Alice's full conversation history is appended turn-by-turn to per-chat JSONL files (crates/.../memory/<id>/chat/...), file-first by policy (feedback-storage-file-first-simplified). That is MemGPT's recall storage: the durable, replayable, auditable log outside the prompt. The AGI goal openalice.agi.q7-cot-trace extends each assistant turn with a structured chain-of-thought trace in that same sessions.jsonl — i.e. storing reasoning, not just text, as memory.
Working context = charter / scope memory keys. charter_memory_key and the scoped memory CRUD in crates/alice-cli/src/handlers/memory.rs + api-handlers/.../routes/memory.rs (get/delete/delete_all_memory) are OpenAlice's version of Letta memory blocks: small, labelled, always-in-prompt key/value facts the agent keeps about a user/charter. They're file-backed and editable, which is exactly the Letta-block read/write contract — but managed by explicit tooling rather than fully self-edited, dodging the memory-poisoning risk above.
Reflection / consolidation = the circadian memory goal. openalice.agi.q9-circadian-memory ("Memory consolidation cycles — Critic-gated adaptive consolidation with circadian baseline scheduler") is exactly Park's reflection and Letta's sleep-time agent, with one safety upgrade: the consolidation pass is Critic-gated, so a hallucinated insight gets vetoed before it's written back. That directly addresses the "self-editing memory can corrupt itself" caveat — OpenAlice's answer to memory poisoning is a critic in the write path.
Procedural memory = the Voyager goal. openalice.agi.q6-voyager-curriculum (Critic-gated skill versioning) is the procedural sibling of all the episodic/semantic memory above: remembering how to do things (versioned skills), not just facts. Worth keeping distinct in your head — most "agent memory" discussion is episodic, but durable agents need both.
Retention/archival policy is already real. lounge-archive-policy in openalice-persona (per-room retention + archive cron) is the eviction half of the loop: long-running rooms slide past the in-memory window without losing audit history — the file-first equivalent of MemGPT's FIFO eviction into recall storage.

The honest OpenAlice gap, same as everyone's: the retrieval layer over sessions.jsonl. Recall storage that you can only replay linearly isn't yet a memory stream — it needs scored retrieval (recency + importance + relevance) and a real ranker, which is where this connects to [[graphify]] and [[graphrag]] for turning the flat log into a searchable, structured belief store.

Long-term memory for LLM agents — recall, archival, summary & memory streams