Context engineering

"Context engineering is the natural progression of prompt engineering." — Anthropic

>

"the delicate art and science of filling the context window with just the right information for the next step." — Andrej Karpathy

What it is (intuition)

A language model is a function from a sequence of tokens to a distribution over the next token. Everything it "knows" in the moment — your instructions, the conversation so far, retrieved documents, tool outputs, examples — lives in one place: the context window that you feed in on each call. The model has no other working memory. Between calls it remembers nothing.

Prompt engineering is the craft of writing the static instruction block well: clear wording, good role, useful examples. That craft still matters, but it answers only one question — "what should I say to the model?"

Context engineering answers a bigger one: "of all the information that could go into the window right now, which exact tokens, in which order, freshly assembled for this specific step, give the best next action?" It treats the window not as a prompt you write once but as a state you curate on every single turn of a running agent. LangChain's working definition: "the set of strategies for curating and maintaining the optimal set of tokens (information) during LLM inference."

The shift happened because agents happened. A one-shot chatbot turn has a small, mostly-static context. An agent that runs for hundreds of steps — reading files, calling tools, accumulating observations — generates a torrent of tokens, and someone has to decide what stays in the window and what gets thrown out or stored elsewhere. That someone is the context engineer. Cognition (the Devin team) put it bluntly: context engineering is "effectively the #1 job of engineers building AI agents."

The mental model that makes this click (Karpathy's): the LLM is a CPU and the context window is its RAM — small, fast, precious working memory. Context engineering is the operating-system job of paging the right data in and out of that RAM at the right time. See [[agentic-loops]] for the loop that this RAM-management wraps around.

Why it matters

Two hard facts force the discipline.

1. The window is finite — and "finite" arrives sooner than the advertised limit. A 200K-token model does not actually use 200K tokens well. Performance degrades long before the limit, a phenomenon now called context rot. Chroma tested 18 frontier models (GPT-4.1, Claude 4, Gemini 2.5, Qwen3) and found every single one gets worse as input grows — "model performance degrades as input length increases, often in surprising and non-uniform ways." The classic "lost in the middle" result ([[long-context]], Liu et al. 2023) shows a U-shaped accuracy curve: a fact is found reliably if it sits at the start or end of the context and is missed if it sits in the middle, with accuracy dropping by >30% in the worst position. The NoLiMa benchmark, which strips literal keyword overlap so the model must reason associatively, shows GPT-4o falling from 99.3% accuracy at 1K tokens to 69.7% at 32K — and 11 of 13 models dropping to half their short-context score by 32K. So more context is not free; past a point it actively hurts. Anthropic frames this as an attention budget: every token spent dilutes the model's ability to attend to the tokens that matter. The architectural root cause is real — a transformer builds n² pairwise attention relationships for n tokens ([[attention-and-transformers]]), and models have "less experience with, and fewer specialized parameters for, context-wide dependencies."

2. Tokens cost money and latency, and most agent tokens are input tokens. Manus reports a 100:1 input-to-output token ratio in production — the agent reads vastly more than it writes. At that ratio, how you manage input is your cost structure. (More on the KV-cache lever below.)

Put together: the goal of context engineering is, in Anthropic's phrase, to "find the smallest set of high-signal tokens that maximize the likelihood of your desired outcome." Not the most context — the right context.

The failure modes when you get it wrong have names (Drew Breunig's taxonomy):

• Context poisoning — a hallucination enters the context and then gets cited as fact on later turns.
• Context distraction — so much context piles up that it drowns out the model's own trained priors.
• Context confusion — irrelevant-but-present information bends the answer.
• Context clash — two parts of the context contradict each other and the model picks wrong.

How it works (the four real moves)

LangChain's taxonomy is the one practitioners actually use. Every concrete technique is one of four moves on the context window: write, select, compress, isolate.

1. Write — put information outside the window so you can get it back later

The window is RAM; you need disk. "Writing" means persisting state where it survives compaction and across sessions.

• Scratchpads — the agent writes intermediate notes/plans to a tool-backed file or a state field during a task. Anthropic calls this structured note-taking: "notes persisted to memory outside of the context window [that] get pulled back into the context window at later times," giving "persistent memory with minimal overhead."
• Memories — cross-session persistence. The agent reflects on a session and synthesizes durable facts (user prefers X, project uses Y) for next time. ChatGPT, Cursor, Claude Code's CLAUDE.md, and Windsurf all do versions of this. This is the deep topic of [[agent-memory-systems]] — episodic, semantic, and procedural memory, MemGPT-style paging.

The Manus team's sharpest version: use the file system as externalized, unbounded context. Files are "unlimited in size, persistent by nature, and directly operable." When the agent shrinks its window, it drops the content of a web page but keeps the URL — compression that is restorable, never lossy in a way it can't undo.

2. Select — pull the right external information back into the window, just in time

This is retrieval, broadly. The frontier insight: don't pre-load everything; load lazily.

• Retrieval (RAG) — embed a query, fetch the top-k relevant chunks, paste them in. For code agents this means AST parsing, embedding search, grep/file-search, knowledge graphs ([[graphrag]]), and a re-ranking step. (Full mechanics live in [[embeddings]] and [[graphrag]].)
• Tool selection — when an agent has dozens of tools, putting all their schemas in context every turn causes context confusion. Apply RAG to the tool descriptions themselves: retrieve only the relevant tools. The survey notes this gives roughly a 3× improvement in tool-selection accuracy. See [[tool-use-function-calling]] for how tool schemas are wired.
• Just-in-time retrieval (the Anthropic shift) — instead of pre-computing and stuffing all possibly-relevant data up front, keep "lightweight identifiers (file paths, stored queries, web links)" in context and let the agent "dynamically load data into context at runtime using tools." This is progressive disclosure: the agent "incrementally discover[s] relevant context through exploration." The trade-off is explicit — "runtime exploration is slower than retrieving pre-computed data" — so the recommendation is a hybrid: retrieve a little up front for speed, explore for the rest.

Pseudocode — just-in-time vs. pre-loaded selection:

# Pre-loaded (old RAG default): one big retrieval, hope it's right.
chunks = vector_db.search(query, k=20)          # may pull 20 chunks, 15 noise
context = system_prompt + "\n".join(chunks) + query
answer  = llm(context)                            # noise causes context confusion

# Just-in-time (agentic): keep pointers, fetch on demand inside the loop.
context = system_prompt + file_index + query      # file_index = paths/summaries only
while not done:
    action = llm(context)                         # model decides what to open
    if action.kind == "read_file":
        content = fs.read(action.path)            # paged in only when needed
        context = append(context, content)        # ...and droppable later (see compress)
    ...

3. Compress — keep only the tokens still doing work

When the window fills, reduce. The two levers:

• Summarization / compaction — Claude Code triggers auto-compact past ~95% of window capacity: it summarizes the full interaction history and reinitializes a fresh window with the compressed version. Anthropic's caution is the whole art: "the selection of what to keep versus what to discard, as overly aggressive compaction can result in the loss of subtle but critical context." The safest, lowest-risk compaction is tool-result clearing — old tool outputs are usually safe to drop because their conclusions are already baked into later reasoning.
• Trimming / pruning — heuristic (drop oldest messages) or learned (a trained pruner like Provence that scores and removes low-value spans).

Compaction sketch:

def maybe_compact(messages, model_window, keep_recent=10):
    if token_count(messages) < 0.95 * model_window:
        return messages                           # plenty of room, do nothing
    recent  = messages[-keep_recent:]             # never summarize the live tail
    history = messages[:-keep_recent]
    summary = llm(SUMMARIZE_PROMPT + serialize(history))  # lossy but bounded
    return [summary] + recent                     # fresh window, compressed past

4. Isolate — split context across separate windows/agents/sandboxes

If one window can't hold everything cleanly, use more than one.

• Sub-agents / multi-agent — a coordinator holds the high-level plan; specialized sub-agents each run in a clean window on a focused task and return only a "condensed, distilled summary of [their] work (often 1,000–2,000 tokens)." The cost is real: Anthropic's multi-agent researcher used ~15× more tokens than a single chat thread — you trade tokens for cleaner separation of concerns and parallelism. (Orchestration patterns: [[agentic-loops]], [[mixture-of-agents]].)
• Sandboxes — run code in an isolated environment (e.g. E2B) so heavy objects (images, audio, large JSON) live outside the LLM context until a specific value is actually needed.
• State-schema isolation — design the agent's state object with fields that are not exposed to the model. Store a giant tool result in a hidden field; show the LLM only a handle or a one-line summary.

The fifth move nobody lists but everyone needs: ordering & cache

Two production realities cut across all four moves:

Ordering matters because of lost-in-the-middle. Put the most important instructions at the start or end, never buried in the middle. Manus operationalizes this with recitation: the agent maintains a todo.md and rewrites it every few steps, "push[ing] the global plan into the model's recent attention span" — deliberately moving the goal into the high-attention recency zone in tasks that average ~50 tool calls.

The KV-cache is the cost lever. During inference the model caches the key/value tensors for every prefix token; if the next call shares a prefix, that work is reused. Manus calls KV-cache hit rate "the single most important metric for a production-stage AI agent" — cached input tokens cost $0.30/MTok vs $3/MTok uncached on Claude Sonnet, a 10× difference. The rules that follow:

• Stable prefix. "Even a single-token difference can invalidate the cache from that token onward." A timestamp in your system prompt silently destroys cache reuse on every call.
• Append-only context. Never edit earlier turns; only append. Reordering or rewriting history busts the cache from the edit point forward.
• Mask, don't remove, tools. When a tool shouldn't be available this turn, don't delete its schema (that changes the prefix → cache miss, and dangles references in history). Instead use logit masking / a constrained decoder so the model can't select it while its definition stays put. (Mechanics: [[tool-use-function-calling]].)

Key ideas & tradeoffs

The throughline: context engineering is almost entirely about tradeoffs between completeness and signal-to-noise, paid in tokens, latency, dollars, and the ever-present risk of lossy compression. There is no "just give it everything" — that's the option that demonstrably loses.

A few non-obvious findings worth internalizing:

• Keep your errors in the context. Manus deliberately leaves failed actions and stack traces in the window: it helps the model "implicitly update its internal beliefs" and stop repeating the mistake. Hiding failures removes the evidence the model needs to recover.
• Break few-shot ruts. When every recent turn looks the same, the model mimics the pattern even when it's wrong. Inject "small amounts of structured variation" to keep it from over-imitating its own history.
• Coherent haystacks can be *worse*. Chroma found models sometimes do worse when the surrounding text reads as a logical, coherent narrative than when it's shuffled — a deeply counterintuitive result that warns against assuming "well-organized = easier to retrieve from."
• Semantic distance hurts more at length. As needle-question similarity drops (the answer doesn't share keywords with the question), "model performance degrades more significantly with increasing input length," and even a single distractor measurably lowers accuracy.

Honest caveats & open questions

This is a freshly-named, fast-moving, partly-folk discipline. The term went mainstream in mid-2025. The first unified academic survey (Mei et al., 1,411 papers reviewed) is from July 2025 — months old as of this writing. Much of the strongest practical knowledge lives in engineering blog posts (Anthropic, LangChain, Manus, Chroma, Cognition), not peer-reviewed papers. That's not a knock — it's where the real production experience is — but it means best practices are empirical, version-specific, and shift with each model release. A KV-cache trick or compaction threshold tuned for one model-vendor may not transfer.

The benchmarks are imperfect. "Needle in a haystack" is easy and over-optimized; NoLiMa and LongMemEval are harder and more honest, but the field openly admits it lacks good evaluations for generation under long context. The survey names the central open gap a comprehension–generation asymmetry: "while current models … demonstrate remarkable proficiency in understanding complex contexts, they exhibit pronounced limitations in generating equally sophisticated, long-form outputs." In plain terms — we've gotten much better at packing the right tokens *in*; getting good, long, coherent output *out* of that context is still weak. That asymmetry is, per the authors, "a defining priority for future research."

Unsettled questions, honestly stated:

• Why exactly does context rot happen, mechanistically, beyond "attention spreads thin"? It's measured far better than it's explained. ([[mechanistic-interpretability]] has barely touched it.)
• Is there a principled, model-agnostic way to decide what to compact, or will it always be a hand-tuned heuristic per task?
• Does ever-longer native context (1M+ windows) make compaction obsolete, or does rot scale with it so the discipline never goes away? Current evidence points firmly at the latter — bigger windows have *not* repealed context rot.
• Multi-agent isolation buys clean windows but the ~15× token cost and the lossy summary-handoff boundary are unsolved at scale; when isolation beats one well-managed window is still case-by-case.

Treat anyone selling a single "correct" context architecture with suspicion. The honest state of the art is: measure your KV-cache hit rate and your task accuracy, instrument what's actually in your window, and tune the four moves empirically for your model and your task.

How it connects to OpenAlice

Context engineering is not a side topic for OpenAlice — it is the substrate Alice runs on. Every concern above shows up in the codebase:

• The agentic loop. Alice's [[agentic-loops|agentic loop]] is exactly the read-decide-act cycle where the window is re-assembled every turn. The historical "turn-skip / tail-drop" bugs were context-management bugs: the wrong tokens (or none) reaching the model at the entry point. The standing rule that integration tests must originate at the event/handler entry point exists because context assembly is where things silently break.
• Memory & storage. Alice's files-first storage and per-chat memory directories are the write move — externalized context that survives compaction, persisted per chat (memory/.../chat/.../settings/...). The lane-state / boot-recovery gotcha (clear lane_state.db before restart or it mass-replays old messages) is a context-management failure mode in production. Deep dive: [[agent-memory-systems]].
• Prompt composition is byte-sensitive. The directive that the prompt-composer stay byte-identical for all personality content, and that Alice's Soul/Identity blocks be preserved verbatim, is a stable-prefix discipline: changing the prefix isn't just a personality risk, it's a KV-cache and behavioral-envelope risk. The ordering of personality, mode, and memory blocks is literal context engineering.
• The three mode axes (InteractionMode / PersonalityMode / ChatMode) are a select-and-isolate mechanism: each chat composes a different context from the same substrate. Worker-mode (for tasks like Katya's diploma) selects different tools and memory than Companion-mode.
• Atlas itself is a select-layer. The org-wide rule "query Atlas before grepping" is just-in-time retrieval applied to the codebase: Atlas (code search, semantic search, call-graph, [[graphrag]]) is the lightweight-identifier index that lets an agent page exactly the right symbols into context instead of pre-loading whole repos. The known caveat — call-graph resolution is candidate-grade (~15–19% resolved) — is the retrieval-quality tradeoff from the select section, made concrete.
• Sub-agent delegation. The lab's delegation pattern (Opus orchestrates; Sonnet sub-agents read; Codex does mechanical edits) is the isolate move — each sub-agent gets a clean window and returns a distilled summary, paying the token cost for separation of concerns, exactly as Anthropic describes.

In short: the same four moves — write, select, compress, isolate — plus stable-prefix / cache discipline are the daily engineering reality of keeping Alice coherent across hundreds of turns. This article is the theory; the OpenAlice agentic loop, memory dirs, prompt-composer, and Atlas are the implementation.

See also

[[agentic-loops]] · [[agent-memory-systems]] · [[long-context]] · [[tool-use-function-calling]] · [[graphrag]] · [[embeddings]] · [[attention-and-transformers]] · [[mixture-of-agents]] · [[mechanistic-interpretability]]