OpenAlice → Game-Engine Re-Architecture: Full Analysis + Roadmap

Date: 2026-06-17 · Author: Norbert Wiener (alice-core) · Method: 10-agent Opus analysis (9 dimensions + synthesis) at HEAD c627af8 · Status: analysis + proposal — NOT yet approved to implement (open questions for NAO below).

NAO's directive: "we over-SCRIPTED bespoke agent features (e.g. a TaskList tool) that Alice could self-serve by composing BASE PRIMITIVES (bash/python/fs + reasoning); reorganize abstractly like a GAME ENGINE (factories/spawners/global modules/dynamic composition); strip excess unused functions." This is the minimal-primitives + LLM-composition paradigm (Voyager / CodeAct).

Verdict: the thesis is correct AND already half-validated in-tree

OpenAlice's over-scripting is industrialized and quantitatively confirmed — but the composition substrate NAO wants ~80% already exists, half-built or flag-gated-off. The work is to COMMIT to it + delete the redundant bespoke layer, not invent new architecture.

Diagnosis — 3 compounding pathologies (quantified)

1. Over-scripted tool surface

• 207 `register_tool!` invocations (~24K LOC across 32 tools_*.rs in api-handlers), shadowed by a parallel ~140-entry static `BuiltinTool` registry in agents-tools (~16.5K LOC), bridged by an 874-LOC re-forwarding shim (tools_builtin_bridge.rs) + 59 stub handlers that literally return "executed at transport layer".
• Only ~24 are genuine irreducible PRIMITIVES. The other ~150–180 are bespoke get_store() + store.put("prefix/{id}", json!{...}) CRUD/formatter/config wrappers Alice could self-serve via bash → alice-cli (whose verb tree already mirrors them: task/calendar/sop/worker/goal/mood/rank/router/appstore — 34 alice-cli modules).
• NAO's thesis is already validated in-tree: Wave M Phase 3 (2026-05-03) deleted 23 read-side tools and migrated them to bash 'alice-cli <verb>' — but the write-side twins were left behind, producing triple-implementation per domain (dedicated tool + alice-cli verb + dead static stub).

2. Dead / theatrical higher-order engines

agents-soul hosts 9 "capability engines" (~5,538 LOC); only 3 are earned. Verified dead/theatrical (~3,240 LOC, ~58%):

• deep_agent (866 LOC) — registered to DRIVER_AGENT_TREE, never `get_driver`'d.
• cron_scheduler (825 LOC) — zero non-test callers; ticks but on_trigger never set.
• flows (791 LOC) — FlowEngine registered with a no-op executor, never fetched.
• goal_runner (508 LOC) — "execution" = tokio::sleep(10ms) + auto-mark-Done.
• goal_engine::decompose_naive (~250 LOC) — fixed 3-item "Plan/Execute/Verify" template for any input.

Earned & kept: TaskRunner (494, real Semaphore+dedup), TickEngine (227, 2 real subscribers), mission_engine+executor (1,015, real OODA via llm_router).

3. Rigid wiring

• AppState = 16-field god-struct hand-threaded as State<Arc<AppState>> into ~228 routes.
• register_drivers = 250-line imperative new()-and-insert with documented load-bearing ordering; main.rs boot adds ~57 manual register/install/spawn calls.
• Kernel driver registry (54 DRIVER_*, Arc<dyn Any>+downcast) silently no-ops on downcast-miss (doc-comment cites "three incidents").
• 71 non-test `#[allow(dead_code)]` hide stranded clusters: a second duplicate HNSW (1,292 LOC), storage-v2 dual_write_* (comment-only ref), UnusedTts.

Cleanly reclaimable: ~3.2K dead engines + ~2K stub layer + ~3.3K reg_*.rs boilerplate + ~1.3K duplicate HNSW + ~6.4K CRUD bodies ≈ 12–16K LOC removable, collapsing tool surface from ~207 to <30 (and the prompt tool-schema token cost with it). 78% of 666 FEATURES are beta/experimental.

Target architecture — commit to the substrate that already exists

• ENGINE = the extracted agentic loop behind AgenticLoopServices (5-field service struct, shipped today) + ToolExecFn. This is THE seam. Generalize it: every subsystem that reaches into AppState gets a narrow typed *Services struct from kernel drivers (DeliveryServices, MemoryServices, SpawnServices). AppState shrinks to a thin facade.
• MINIMAL PRIMITIVE SET (~24, the "instruction set"): bash (sandboxed: cwd jail + 40-pattern denylist + env scrub + unshare -n + timeout), view/edit/write/glob/grep, web_search/web_fetch/browser_navigate (SSRF/WAF gate), 6×lsp_*, repo_map, memory_read/write/list/search, config_get/set, send_file, image_gen/audio_generate/play_sound, agent_spawn, dossier_link/anchor, map_screenshot. Rule (codified as a registry-build assertion): a bespoke tool earns existence ONLY if it gates a primitive (approval gate, atomic multi-file write, SSRF/connector egress, provider quota) — never if it merely wraps store.put. Gap: add a python/code-exec mode to bash (currently NoOpWasmHost).
• ONE REGISTRY / FACTORY: promote plugin-host::PluginRegistry (RwLock<HashMap<PluginKind, Vec<FactoryFn>>>, 5 adapter kinds) out of Experimental/default-OFF to THE factory. Route all action sources (native tools, MCP, WASM packs, Voyager skills) through it. Keep the linkme TOOL_DESCRIPTORS distributed_slice as the self-registration mechanism for native primitives; collapse the parallel static map + bridge + legacy dispatch into it. Wire FeatureStatus into the factory so Beta/Experimental modules are config-toggled at boot.
• SPAWNERS: unify AgentRegistry + (DELETE deep_agent) + microagent templates + modules-hands TOML workers into ONE prefab-backed Spawner (wire RoleRegistry, currently None). Higher-order work (missions, goals, scheduled) all become "enqueue a task whose prompt is the goal, whose ACT is the shared agentic loop" — TaskRunner+TickEngine is the ONE earned higher-order substrate. Route mission_executor through run_agentic_loop (currently a tool-less CompletionRequest).
• SKILL PERSISTENCE (Voyager): two-tier — (1) SKILL.md = LLM-authored markdown + sibling .sh, bash-backed, Critic-gated (agi-skills crate, 15 pre-seeded, skill_manage tool); (2) WASM pack via pack-sdk for heavyweight/untrusted. Policy: a new "feature" is a SKILL first; compiled Rust tool/WASM pack ONLY if it provably needs sandboxing, auth, or a hot-path latency budget. This structurally stops the 207→Nth bespoke-tool accretion.

Current → target mapping

Roadmap (risk-aware; gate every step; lab-canary structural+deep before mvp)

Quick-wins (low risk, NOT Cat-A) — start now, independently re-gate each deletion:

1. Finish Wave M — delete the write-side CRUD tool twins (task/calendar/worker/sop/goal/mood/router/tts/appstore/microagent write verbs); each has an alice-cli verb on identical store keys. ~120 tools / ~6.4K LOC. (policy-gated — see open Q1)
2. Execute the stalled bash-first deletions (docs/bash-first-refactor-blueprint-2026-05-02.md): delete 7 git_* + workspace_list/delete/patch; merge workspace_read/stat→view, workspace_write→write. Pre-approved 2026-05-02, never executed.
3. Delete the 5 dead/theatrical engines (~3,240 LOC) + their tests + the DRIVER_AGENT_TREE/DRIVER_FLOW_ENGINE constants; verify each get_driver chain dead first.
4. Dead-code sweep — dedupe the 2 HNSW impls→one core primitive, delete UnusedTts/dead RAG structs/test-only stub tools; ground-truth pass removing the 71 #[allow(dead_code)] crate-by-crate and letting cargo enumerate the true dead set (trust the compiler over the candidate-grade inspector call-graph: 1753/1766 inspector hits were tests).

Structural (medium):

1. Collapse the dual tool registry into ONE (TOOL_DESCRIPTORS as single source; delete static map + builtin_tools_map() + 874-LOC bridge + legacy dispatch arm + 59 transport-layer stubs).
2. Retire closed migrations behind cutover-confirm (storage-v2 dual_write_* — golden-gated; reg_skills_legacy; 10 deprecated-but-registered tools).
3. Self-registering ModuleFactory + topo-sorted boot orchestrator (replaces fragile load-bearing ordering); add typed kernel.get_driver_as::<T>() to kill the silent-downcast-miss bug class.
4. Extend the AgenticLoopServices seam (medium, Cat-A-adjacent): DeliveryServices/MemoryServices/SpawnServices from kernel drivers; AppState → thin facade. Loop body stays byte-identical.

Deep (high, Cat-A — lab-canary mandatory):

1. Promote plugin-host to THE factory registry; route native tools + MCP + WASM packs + skills through it; wire ToolExecFn to resolve through it.
2. Unify spawners into ONE prefab-backed Spawner; route missions/goals/scheduled work through TaskRunner→run_agentic_loop; missions become the single OODA engine.
3. Flip Voyager off→shadow→on (after lab acceptance); skill_manage = the unit of feature growth; codify the skill-first policy.
4. Crate-map reorg (medium, mostly labeling): 6-tier Cargo member banners + docs/CRATE_MAP.md; merge agi-{core,safety,skills}→openalice-agi; fold deploy→xtask; deprecate modules+transport facade shims.

Cat-A immovables (byte-identical-protected — gate every touching step)

• Agentic loop hot-path invariants: NEVER emit a second Role::System message (providers read the LAST system msg as the SOLE instruction set — a second silently deletes Alice's whole soul/safety prompt); ToolCall.id byte-round-trip; LoopGuard; 128k ContextBudget.
• Soul/personality/prompt-composition path byte-identical (drift is a feature; the mechanism must not change content).
• connector_bridge 32-stage ORDER — observable timing side-effects, guarded by stage_order_is_stable. A registry may only be introduced BEHIND these tests with byte-identical output.
• AppState 16 fields — a contract for agentic_loop_services_from_appstate (blast radius ~228 routes); freeze + grow via kernel drivers.

Process: independently re-gate EVERY deletion (full build + TEST target + drift + symbol grep — cargo is truth, not sub-agent self-reports or LSP). Cap -j4 --test-threads=4. Architectural refactors on branch + lab BEFORE mvp. docker builder prune -af between rebuilds; keep ≥20G. No push without NAO.

Open questions for NAO (answers unblock implementation)

1. CRUD deletion policy: blanket "delete any pure store.put/get tool that has an alice-cli verb", or per-family ack (task/calendar/worker/sop/mood/router/tts/appstore/microagent) before each batch?
2. Voyager gate: comfortable flipping off→shadow→on (Alice authors + Critic-gates her own skills) after lab acceptance, or hold at shadow until you personally review the first promoted skills?
3. Map tools: keep map_screenshot (real CDP) as primitive, delete map_create/search/list/delete? Or retire the map subsystem?
4. Knowledge stubs (kg_/rag_ ship canned "no results"): delete the 5 now, or are real KG/RAG drivers near-term (keep names reserved)?
5. Crate reorg appetite: labeling-only pass + safe merges now, or hold crate moves until tool/engine deletions land?
6. plugin-host promotion timing: part of the registry-collapse, or land single-registry on TOOL_DESCRIPTORS first + promote plugin-host later (smaller blast radius)?
7. /goal: delete outright (fold into a normal agentic turn) or keep as a thin alias enqueuing a mission?

Dimension evidence

Full per-dimension findings (tool audit, capability engines, dead-code, game-engine arch, primitive sufficiency, SOTA paradigm, bespoke sweep, wiring/coupling, crate topology): workflow run wf_0f47447a-546, 9 dimensions + synthesis, 10 Opus agents, ~1.1M tokens.