Agentic loops & orchestration

What it is (intuition first)

A plain LLM call is one-shot: text in, text out, done. An agent is what you get when you wrap that call in a loop and give the model a way to act on the world between turns — call a tool, run code, search the web, read a file — and then see the result before deciding what to do next. The loop is the whole trick. There is no magic "agent" object; an agent is a while loop around an LLM where the model's output decides whether the loop continues.

The mental model that makes everything click:

perceive → reason → act → observe → repeat, until the model says "I'm done."

The model reasons in natural language ("I need to find the population of France, let me search"), emits a structured action (search("population of France")), your harness executes it, feeds the observation ("67.7 million") back into the conversation, and the model reasons again. That cycle — and the dozen design decisions around when to stop, how to recover from a bad step, who decides the plan — is the entire field of "agentic loops & orchestration."

Two words you'll see constantly, and they are not the same thing (Anthropic draws this line sharply):

• Workflow — LLM calls wired together by your code on predefined paths. You decide the steps; the model fills in the blanks. Predictable, cheap, debuggable.
• Agent — the model decides the steps dynamically, including how many. Flexible, powerful, harder to bound and to debug.

Most production "agents" are actually workflows, and that's usually the right call. The honest rule (straight from Anthropic): use the simplest thing that works; add the loop only when it demonstrably helps.

Why it matters

• It's the substrate of every real LLM product that does more than chat. Coding agents, deep-research tools, customer-support resolvers, computer-use agents — all are loops over an [[test-time-compute-reasoning]]-augmented model plus tools.
• It converts a frozen model into something that can act and self-correct. A single forward pass can't recover from a wrong assumption. A loop can: act, observe the failure, re-reason, retry. This is test-time capability gain with zero weight updates — closely related to [[test-time-compute-reasoning]] (paying for thinking at inference) and to [[rlhf-and-alignment]] (which shapes whether the model behaves well inside the loop).
• It's where the cost, latency, and failure live. A 7-step agentic loop is 7 LLM calls. The same task through an orchestrator that plans once and dispatches can be 2 calls — ~70% cheaper. Loop design is cost design. See [[model-routing]] for the routing half of that equation.
• It's the boundary between "demo" and "reliable." Loops fail in characteristic ways — getting stuck repeating an action, hallucinating tool results, never terminating. The maturity of an agent is mostly the maturity of how its loop handles those.

How it works (real mechanics)

1. The base building block: the augmented LLM

Before any loop, you augment the model with three capabilities (Anthropic's "augmented LLM"):

• Tools — function schemas the model can call (search, read_file, run_python, …).
• Retrieval — the model issues queries to fetch context ([[graphrag]], [[embeddings]]).
• Memory — what to carry across turns / sessions ([[agent-memory-systems]]).

Everything below loops this augmented unit.

2. ReAct — the canonical loop (Reason + Act)

ReAct (Yao et al., 2022, arXiv:2210.03629) is the foundational pattern and still the default. The insight: don't make the model only reason (chain-of-thought, which hallucinates because it never checks the world) and don't make it only act (which flails because it never plans). Interleave them. Reasoning traces let the model "induce, track, and update action plans and handle exceptions"; actions let it "interface with external sources" to ground that reasoning.

The trajectory is a sequence of three repeating elements:

Thought:      free-form reasoning about what to do next
Action:       a structured call from a fixed action space
Observation:  the result the environment returns

For HotpotQA, ReAct's action space was literally three verbs:

search[entity]    → returns the Wikipedia page (or top-5 similar titles if missing)
lookup[string]    → returns the next sentence on the current page containing string
finish[answer]    → emits the final answer and terminates the loop

A real trajectory looks like:

Question: Which magazine was started first, Arthur's Magazine or First for Women?
Thought 1: I need to find when Arthur's Magazine and First for Women started.
Action 1:  search[Arthur's Magazine]
Obs 1:     Arthur's Magazine was an American literary periodical ... founded 1844 ...
Thought 2: Arthur's Magazine started 1844. Now First for Women.
Action 2:  search[First for Women]
Obs 2:     First for Women is a woman's magazine ... started in 1989 ...
Thought 3: 1844 < 1989, so Arthur's Magazine was first.
Action 3:  finish[Arthur's Magazine]

Pseudocode for the bare ReAct loop:

messages = [system_prompt, few_shot_examples, question]
for step in range(MAX_STEPS):
    out = llm(messages, stop=["\nObservation:"])   # model writes Thought + Action
    thought, action = parse(out)
    if action.name == "finish":
        return action.arg
    obs = env.execute(action)                       # the only line that touches the world
    messages += [out, f"\nObservation: {obs}\n"]
return "no answer (hit step cap)"

Results: ReAct gave +34% absolute success on ALFWorld and +10% on WebShop over imitation/RL baselines, with only one or two in-context examples. On HotpotQA/FEVER it beat act-only and cut the hallucination that pure chain-of-thought suffers. Best results came from ReAct + CoT-SC (self-consistency): sample several trajectories, and back off between ReAct and chain-of-thought based on internal-knowledge vs. retrieved-evidence confidence.

Failure modes the authors documented (these are universal, memorize them):

1. Reasoning errors — wrong logic despite having the right facts.
2. Search errors — never retrieving the needed entity.
3. Hallucination — fabricating "observations" the tool never returned.
4. Repetitive actions / loops — "getting stuck repeatedly taking the same action," never terminating. Every production loop needs a step cap and a repetition detector because of this one.

3. Plan-and-Execute (plan first, then act)

ReAct decides one step at a time, which is reactive and can wander. Plan-and-Solve (Wang et al., ACL 2023, arXiv:2305.04091) splits the work in two: (a) devise a plan that divides the task into ordered subtasks, then (b) carry out the subtasks in order. It's still zero-shot — the trigger prompt is roughly:

"Let's first understand the problem and devise a plan to solve it. Then, let's carry out the plan and solve the problem step by step."

The PS+ variant adds explicit instructions ("extract relevant variables and their numerals," "calculate intermediate results") to attack the three error classes the paper targets: calculation errors, missing-step errors, and semantic misunderstanding. PS+ matched 8-shot manual CoT on math while staying zero-shot.

In agent terms this becomes plan-and-execute orchestration: one "planner" call produces a task list, then an executor (often a ReAct loop per subtask) works the list, optionally re-planning when reality diverges. Tradeoff vs. ReAct: cheaper and more directed when the task is decomposable, but brittle when early steps reveal the plan was wrong (mitigation: a re-plan step after each subtask).

4. Reflection / self-critique (Reflexion)

Reflexion (Shinn et al., 2023, arXiv:2303.11366) adds a second outer loop: when a trajectory fails, the agent writes a verbal self-reflection ("I assumed the file was JSON but it was CSV; next time check the format first") and stores it in an episodic memory buffer. On the next attempt that reflection is prepended to the context, steering behavior — no gradient updates, learning purely through language.

Four roles:

• Actor — runs the task (often itself a ReAct loop), produces a trajectory.
• Evaluator — scores the trajectory (a scalar, a unit-test pass/fail, or an LLM judge).
• Self-Reflection model — turns "you failed + here's why" into a concise lesson.
• Memory — short-term (the current trajectory) + long-term (accumulated reflections).

memory = []
for trial in range(MAX_TRIALS):
    trajectory = actor.run(task, hints=memory)      # inner ReAct loop
    score = evaluator(trajectory)                   # tests / judge / reward
    if score.success:
        return trajectory
    reflection = reflect(task, trajectory, score)   # "what went wrong + fix"
    memory.append(reflection)                        # carried into next trial

Result: 91% pass@1 on HumanEval, beating the then-SOTA GPT-4 at 80%, plus large gains on ALFWorld and HotpotQA — driven entirely by the reflection feedback, not retraining. This is the "evaluator-optimizer" pattern in Anthropic's vocabulary: generate → critique → refine, looped while a clear evaluation signal exists.

5. Orchestration patterns (who decides, who does)

Once a single loop isn't enough, you compose them. Anthropic's six production patterns, ordered by how much control you cede to the model:

The orchestrator-workers pattern is the multi-agent heart: a lead model decomposes the task, spawns worker sub-agents (each its own ReAct loop, possibly its own context window), and synthesizes their outputs. The 2025 survey (arXiv:2508.17692) and the 2026 multi-agent-orchestration survey organize this space as three topologies + one adaptivity axis:

• Centralized / hierarchical — one orchestrator routes and aggregates (manager → workers). Clear authority, single point of failure, easiest to bound.
• Decentralized — peers talk directly (debate, blackboard). More robust, harder to control, can deadlock — hence "trigger-based verification agents" to break ties.
• + Dynamic/adaptive control — the topology itself changes at runtime: route to a different specialist, spawn an extra critic, allocate diverse solution paths based on the live state.

Dynamic routing is the cheap, high-leverage version: a fast classifier (or the model itself) inspects the request and dispatches to the right model/tool/sub-agent — short questions to a small model, hard ones to a frontier model, code to a code agent. Routing is workflow-level orchestration and is often the single biggest cost/latency win; the mechanics live in [[model-routing]], and the "many-models-beat-one" extreme in [[mixture-of-agents]] and [[fusion-and-llm-councils]].

6. A production-grade loop has more than three lines

The toy ReAct loop above is missing everything that makes a real one survive contact with users. A hardened tool-loop adds: a max-iteration cap (or it never terminates), bounded-parallel tool execution (the model can request 5 tools at once), an all-tools-failed break (don't burn iterations on a dead environment), length-continuation (if the model truncates mid-answer, prompt "continue" — bounded so it can't loop forever), and a final tool-free call to force a clean answer. OpenAlice's run_agentic_loop (see below) is exactly this shape.

Key ideas & tradeoffs

• Reactive (ReAct) vs. deliberative (plan-execute). Step-at-a-time adapts to surprises but wanders and costs a call per step; plan-first is directed and cheaper but brittle if the plan is wrong. Real systems blend them (plan, then ReAct each subtask, re-plan on divergence).
• Single agent vs. multi-agent. Multi-agent buys parallelism, specialization, and bigger effective context (each sub-agent has its own window). It costs token multiplication, coordination/communication overhead, and a much harder failure surface. Many "multi-agent wins" reproduce with one good ReAct loop plus better tools — try the simple loop first.
• More autonomy ↔ less control. Ceding the path to the model buys flexibility and costs predictability, debuggability, and bounded cost. Match the pattern to the task's actual unpredictability, not to ambition.
• Tools matter more than prompts. Anthropic reports spending more time on the agent-tool interface (ACI) than on the agent prompt for their SWE-bench work. Clear schemas, examples, edge-cases, and poka-yoke designs (e.g. require absolute paths) remove whole classes of loop failure. (OpenAlice's global rule "require absolute file paths in sub-agents" is this exact principle.)
• Reflection is real but bounded. Verbal self-critique genuinely lifts pass rates when there's a trustworthy evaluator (unit tests, a checker). With a noisy or sycophantic LLM-judge it can entrench errors instead of fixing them.
• Frameworks are optional and can hurt. LangGraph/CrewAI/AutoGen/OpenAI-Agents-SDK speed the boring parts but add abstraction that "obscures the underlying prompts and responses." Anthropic's guidance: start by calling the API directly — most patterns are a few lines — and only adopt a framework once you understand what it generates underneath.

Honest caveats & open questions

• Termination is unsolved in general. No principled stopping rule exists; everyone uses heuristics — step caps, repetition detectors, confidence thresholds, an explicit finish action. Loops still over-run and under-run.
• Compounding error. Each step has a success probability < 1; over a long horizon they multiply. A 95%-reliable step is ~60% reliable over 10 steps. Long agentic tasks are fragile for this arithmetic reason, and benchmarks on short tasks oversell real-world reliability.
• Evaluation is genuinely hard. Trajectory-level quality, partial credit, "right answer for wrong reasons," and reproducibility (sampling makes runs non-deterministic) are all unresolved. Self-reported success from an agent is unreliable — verify with an independent gate, never trust the agent's own "done."
• Multi-agent claims are noisy. Plenty of reported gains are confounded by more total compute or better tools rather than the topology itself. Ablate against a single strong loop before believing a multi-agent architecture earned its cost.
• Hallucinated observations. A model can fabricate a tool result inside its own output if your harness doesn't strictly separate model text from real tool returns (enforce the stop-sequence / role boundary).
• Security surface. A looping, tool-using model is an execution engine. Prompt injection in a fetched page, a file, or another agent's message can hijack the loop. Sandbox tools, gate dangerous actions behind human-in-the-loop approval, and never let a channel message trigger privileged actions.

How it connects to OpenAlice

OpenAlice's agent core is a hardened ReAct/tool-loop. Atlas indexes two implementations:

• `crates/core/src/loop_runner.rs` → `run_agentic_loop` — the reusable engine. Its shape is textbook-plus-hardening: - The classic cycle: call the LLM (syscall) → if it requests tools, execute them → push results back as Role::Tool messages → repeat. - DEFAULT_MAX_TOOL_ITERATIONS = 10 — the step cap that prevents the ReAct "stuck-in-a-loop" failure mode. - Bounded-parallel tool execution — up to MAX_PARALLEL_TOOLS = 8 tools run concurrently behind a tokio::Semaphore (the model can fan out; the harness fans back in). This is the parallelization/sectioning pattern living inside a single agent step. - `all_failed` break — if every tool in a round errors, stop rather than burn iterations on a dead environment. - Length-continuation — on FinishReason::Length, push "Continue from where you left off," bounded by MAX_AUTO_CONTINUATIONS = 3 so it can't itself become an infinite loop. - A final tool-free `syscall` forces a clean natural-language answer after the loop.
• `crates/api-handlers/src/http/routes/chat_tools/agentic_loop.rs` → `run_agentic_loop` — the chat-facing wrapper that threads conversation context, effort, and model selection (the [[model-routing]] hook) into the core loop, and is exercised by integration/regression tests (phase_8kk_three_fixes.rs, steer_r2_mid_stream.rs, worker_run_tool_loop.rs) that pin down silent-restart, mid-stream steer, and clean-completion behavior.

Around this core the ecosystem realizes the broader patterns from this article:

• `openalice-agent-basic` is the embed-agnostic perceive → reason → act core being extracted from openalice-agent-lite — explicitly a "reusable ReAct loop" shared by Persona/World/Live/Social, with the net-new bridge that turns perception (visual/spatial) into an observation line the way a voice utterance becomes a user line. That's ReAct's "observation" channel generalized beyond text tools.
• Effort/model selection per call is OpenAlice's [[test-time-compute-reasoning]] dial inside the loop; routing decisions connect to [[model-routing]].
• The AGI / proactive-Alice track (Critic-gate + CoT-traced autonomous outreach, and the Voyager-style skills "cycle") is the evaluator-optimizer / Reflexion pattern applied to a long-lived agent — generate an action, gate it through a critic, learn across cycles — with explicit human-in-the-loop approval gates, matching the "autonomous agents pause for human judgment at checkpoints" guidance and OpenAlice's safety bar (HITL approvals, kill switch, standing approvals).

The OpenAlice house rules read like the failure-mode list above turned into policy: require absolute paths in sub-agent tools (poka-yoke ACI), never trust a sub-agent's self-reported "done" — re-gate with the real build+test (independent evaluator), and never let a channel message trigger privileged actions (loop-hijack / injection defense).

See also

[[test-time-compute-reasoning]] · [[agent-memory-systems]] · [[model-routing]] · [[mixture-of-agents]] · [[fusion-and-llm-councils]] · [[graphrag]] · [[autoresearch]] · [[rlhf-and-alignment]]