Continual learning & model editing

What it is (intuition first)

A pretrained model is frozen in time. Its weights encode whatever the training data said as of the cutoff, and nothing else. The moment the world moves — a new CEO, a renamed product, a fact you got wrong, a person Alice just met — the model is stale in exactly that spot. The naive fix is to retrain from scratch on updated data, but a frontier pretraining run costs millions of dollars and weeks of compute, so nobody does that to fix one fact.

Continual learning and model editing are the two families of answers to the same question: how do you keep a model current without paying for a full retrain?

The two families pull in opposite directions, and that tension is the whole subject:

• Continual learning wants the model to keep learning new things over time — task after task, fact after fact — the way a person does. The enemy here is catastrophic forgetting: when you train a network on task B, gradient descent happily overwrites the weights that encoded task A, and A is gone. The brain doesn't do this; backprop does.
• Model editing is the surgical opposite. You don't want to teach the model a new skill — you want to reach in and change one specific fact ("the Eiffel Tower is in Rome") with a scalpel, leaving everything else byte-identical. The enemy here is collateral damage: edits that bleed into unrelated facts, or that fail to propagate to the obvious implications of the edit.

The mental model that makes it click: a model's knowledge lives in its weights as a giant, entangled associative memory. Continual learning is adding to that memory without erasing it; model editing is find-and-replace on a single entry without corrupting its neighbors. Both are hard for the same root reason — the memory is distributed and superposed, so there is no clean "row" to overwrite.

Why it matters

• It is the maintenance problem for every deployed model. Knowledge decays. A model shipped in January is wrong about everything that happened in February. The question isn't whether to update it but which mechanism — retrain, fine-tune, edit, or route around the staleness with retrieval ([[graphrag]], [[embeddings]]).
• Full retraining is economically off the table for fixes. You retrain to add capability, not to correct a typo in the model's beliefs. Everything in this article exists because the gap between "I need one fact changed" and "the cost of a pretraining run" is six or seven orders of magnitude.
• Forgetting is silent. Fine-tune a model on your support docs and it may quietly lose math, coding, or instruction-following. There's no error message — just a regression you discover in production. Knowing why it happens (and how EWC, replay, and PEFT mitigate it) is the difference between a safe update and a bricked model.
• For OpenAlice specifically: Alice needs to stay current about NAO's world — new people, projects, preferences, corrections. The honest engineering answer is almost always memory + retrieval, not weight surgery ([[agent-memory-systems]]), and the literature backs that. This article is partly a map of when weight-level updating is the wrong tool.

How it works (real mechanics)

1. Catastrophic forgetting — the core obstacle

Train a neural net sequentially on task A then task B, and performance on A collapses. The weights that mattered for A get repurposed by B's gradients because plain SGD has no notion of "this weight is load-bearing for something I learned earlier." This is catastrophic forgetting, and it's the reason you can't just keep fine-tuning a model on a stream of new data forever.

Elastic Weight Consolidation (EWC) (Kirkpatrick et al., 2017, PNAS) is the canonical mitigation and a clean intuition pump. The idea, borrowed from synaptic consolidation in neuroscience: after learning task A, estimate how important each weight was to A (via the diagonal of the Fisher information matrix), then when training on task B add a quadratic penalty that pulls important weights back toward their A-values and lets unimportant ones move freely. You "freeze the load-bearing walls and renovate the rest." The paper showed a single network learning multiple Atari games sequentially without catastrophic forgetting — something plain SGD cannot do.

The broader continual-learning toolbox falls into three buckets, all still active research, none a solved problem:

• Regularization-based — penalize moving important weights (EWC, Synaptic Intelligence).
• Replay / rehearsal — interleave a buffer of old examples (or generated pseudo-examples) with new ones so the old distribution stays "in view."
• Architecture-based — give new tasks new parameters (adapters, expanding networks) so old weights are never overwritten. This is where [[lora-and-peft]] connects: a frozen base plus per-task low-rank adapters is forgetting-proof by construction because the base never moves.

2. Model editing — the scalpel

Model editing asks a narrower question: change one fact with minimal side effects. The seminal line of work is ROME and MEMIT from David Bau's lab.

ROME — Rank-One Model Editing (Meng et al., 2022, arXiv:2202.05262, NeurIPS'22). Two contributions, and the first is what makes the second believable:

1. Causal tracing (locating). Run the model on a factual prompt, corrupt the subject tokens with noise, then restore clean activations one layer at a time and watch which restorations recover the correct answer. This causal-intervention method localizes factual recall to a specific band of mid-layer feed-forward (MLP) modules acting on the last subject token. The MLP behaves like a key–value associative store: the key is the subject, the value is the attribute.
2. Rank-one editing. To change "the Eiffel Tower is in Paris" to "Rome," compute a single rank-one update to one MLP's down-projection weight that maps the subject-key to a new value-vector encoding "Rome." It is closed-form (no gradient descent on the whole model) and surgical. On counterfactual edits, ROME held both specificity (don't change unrelated facts) and generalization (the edit survives paraphrases) where prior methods sacrificed one for the other.

MEMIT — Mass-Editing Memory in a Transformer (Meng et al., 2022, arXiv:2210.07229). ROME edits one fact in one layer. MEMIT generalizes it to thousands of facts at once by spreading the update across a range of mediating MLP layers and solving a least-squares update that inserts many key–value pairs together. The authors reported scaling to thousands of associations on GPT-J (6B) and GPT-NeoX (20B) — orders of magnitude beyond ROME's single-edit regime. MEMIT is still the standard baseline for batch knowledge editing.

A neighboring family worth knowing: hypernetwork / meta-learned editors (e.g. MEND, KnowledgeEditor) that train a small network to predict the weight edit from a gradient, and memory-based editors (e.g. SERAC) that don't touch weights at all — they store edits in an external cache and route relevant queries to a patch model. That last one is essentially [[agent-memory-systems]] wearing an editing hat, and it sidesteps every weight-corruption failure below.

3. Where editing breaks (flag the contested claims)

The clean "scalpel" story has been steadily punctured by follow-up evaluation work. Two findings every practitioner should hold:

• Edits don't propagate to their implications — the *ripple effect*. Cohen et al. (2024, TACL, arXiv:2307.12976) point out that injecting "Jack Depp is the son of Johnny Depp" logically entails a cascade of related facts (Johnny now has an additional child; Jack's sibling has a new brother; …). Their RippleEdits benchmark shows current editors update the target fact but largely fail to update its logical consequences. Later work (ChainEdit, 2025) reports baseline logical-generalization accuracy on RippleEdits around ~20% — i.e. editors get the headline fact right and the entailed facts mostly wrong. Treat any "edit success rate" number that ignores ripple effects as overstated.
• Editing at scale quietly degrades the model. Gupta et al. (2024, arXiv:2401.07453, "Model Editing at Scale leads to Gradual and Catastrophic Forgetting") show that as you apply sequential edits, even methods like ROME/MEMIT cause progressive degradation — first gradual forgetting, then a sharper collapse — and can disable the model's broader abilities. The scalpel dulls and starts cutting things it shouldn't.

Net: model editing is a genuine research capability and a beautiful interpretability result (it validates the localize-then-edit story of [[mechanistic-interpretability]]), but it is not yet a production knowledge-maintenance tool for a system that needs hundreds of evolving facts to stay mutually consistent.

4. The practical decision: fine-tune vs RAG (vs edit vs memory)

For injecting new factual knowledge, the most-cited empirical comparison is Ovadia et al. (2024, EMNLP, arXiv:2312.05934, "Fine-Tuning or Retrieval?"). Their headline finding, across multiple models and knowledge-intensive tasks: RAG consistently outperformed unsupervised fine-tuning for knowledge injection, for both existing and entirely new knowledge. On a current-events ("new knowledge") task, RAG roughly doubled fine-tuning's score in their setup (vendor-neutral academic result, but it's a single study on specific models — read it as a strong prior, not a law). A key mechanistic takeaway: LLMs struggle to absorb new facts via unsupervised fine-tuning at all, and exposing them to many paraphrased variations of the same fact helps — the model needs the fact in many surface forms before it sticks.

The working decision table:

The load-bearing rule, and the one most teams learn the expensive way: fine-tuning is for teaching *how to behave*, not *what is true*. Facts belong in retrieval or memory; behavior belongs in weights.

Where it shows up in OpenAlice

• Alice's "stay current" need is a memory problem, not an editing problem. New people, corrections, and preferences flow into [[agent-memory-systems]] + retrieval, not into Alice's weights. This article is the citation for why we don't reach for ROME when NAO corrects a fact — ripple effects and scale-degradation make weight surgery the wrong tool for evolving, mutually-dependent knowledge.
• PEFT as the forgetting-safe specialization path. When we do want behavioral specialization, [[lora-and-peft]] adapters over a frozen base give us continual specialization without bricking the base model — the architecture-based answer to catastrophic forgetting.
• Editing as an interpretability probe. ROME's causal tracing is a working instance of the localize-a-fact program in [[mechanistic-interpretability]]; even if we never edit weights in prod, the locating half is useful for understanding where Alice's knowledge lives.
• It bounds the retrain question. When someone proposes "let's just fine-tune Alice on the new data," this article is the pushback: measure forgetting, prefer retrieval for facts, and reserve weight updates for behavior.

Open questions & frontier

• Lifelong editing without collapse. Can sequential editing be made stable over thousands of edits (the Gupta et al. failure mode)? Null-space-constrained and locality-preserving editors are the current attack.
• Editing that respects entailment. Closing the ripple-effect gap — making one edit propagate to its logical consequences — is open and benchmarked (RippleEdits, ChainEdit).
• Unified theory of memory. Where should a fact live: weights, an external store, or a retrieved corpus? The field is converging on "facts → retrieval/memory, behavior → weights," but the boundary is fuzzy and product-specific.
• Forgetting as a feature. Machine unlearning — deliberately removing a fact (privacy, copyright, safety) — is editing's mirror image and a fast-growing area; it shares all the same locality and collateral-damage problems in reverse. Connects to [[ai-safety-and-jailbreaks]] and [[rlhf-and-alignment]].

See also

• [[lora-and-peft]] — the forgetting-safe specialization path; adapters over a frozen base
• [[agent-memory-systems]] — where evolving knowledge actually lives for an agent like Alice
• [[graphrag]] — retrieval as the preferred fact-injection mechanism (the RAG side of the decision table)
• [[mechanistic-interpretability]] — ROME's causal tracing is this program in action
• [[model-merging]] — a different "update without retrain" lever (combine fine-tunes via weight arithmetic)
• [[rlhf-and-alignment]] — shapes behavior in weights; the canonical "fine-tune for behavior, not facts" case