Storage-v2 consolidation — one canonical per-chat root (#518), 2026-06-15

Status: design (RCA done). Implementation is a careful, backup-first, phased migration of live per-chat data — NAO-gated before any step that moves data.

Problem

Per-chat data is currently spread across five filesystem roots, with several orphan/duplicate directories left by earlier migrations. Goal: ONE canonical per-chat root, orphans folded in safely (no data loss).

Current map (verified)

Orphans / duplicates (live, on b7775dc9): telegram---{id} (triple-dash bug), chat_telegram--{id} (pre-Wave-S prefix), tg--{id} vs telegram--{id} (prefix-rename era), mode.json (pre-interaction_mode_store), connector_history_embeddings/, refs/people/{slug} mirroring identities/{slug}, and a parallel data/alice/ tree (agent-alias early-boot artifacts). Plus lane_state.db lives flat at ~/.openalice/ (ephemeral, intentionally outside the per-chat root — not migrated).

Proposed canonical layout

One root per chat: ~/.openalice/data/{agent}/chat/{scope}/ with sessions/ · inbox/ · outbox/ · voice/ · browser/ · traces/ · generated_images/ · image_index.jsonl · dossiers/by-identity/ · settings/{interaction_mode,charter,…} · onboarding/ · persona · people/ · topics/ · facts/ · events/ · lessons/ · policy/ · meta/ · observations/ · skills/ · _audit.jsonl · schema.json.

scope = {connector}-{chat_id} — byte-identical to today's ctx.chat_scope. Global identity data (profile.yaml, aliases.yaml, global dossier.md, avatar) stays agent-scoped in identities/{slug}/.

Migration plan (6 phases, each independently rollback-safe)

1. Snapshot + dry-run — cp -al backup; a MigrationStep (storage_v2_consolidation_per_chat) with detect() enumerating all per-chat dirs across the 5 roots → a no-op MigrationPlan (gated …_DRY_RUN=1).
2. Unify scope-name variants within chat/ (triple-dash, chat_ prefix, tg--) — merge by content hash (never blind mv; cp --backup=numbered).
3. Sessions → move sessions/{scope}/ into chat/{scope}/sessions/; one-line path change in paths.rs + a dual-read shim (new path first, old fallback) deployed BEFORE the writer path flips.
4. memory/chat → fold data/memory/{agent}/chat/{scope}/ into the canonical root. Safest first step: a symlink shim (data/memory/{agent}/chat → data/{agent}/chat) so existing FsStore keys resolve transparently (zero-code); the explicit key migration follows.
5. Per-chat dossier fragments → identities/{slug}/by-chat/{scope}.md → chat/{scope}/dossiers/by-identity/{slug}.md + dual-read shim.
6. Drain legacy connector_history blob — only after v2 sessions are authoritative per chat; back-fill pre-v2 chats (blob→TurnRecords) FIRST to avoid amnesia; archive (don't delete) the blob.
7. Remove orphans — after 2 deploy cycles of dual-read confirmation.

Risks (the ones that bite)

• Scope-name collisions → dedup by content hash, never blind move.
• FsStore key-prefix cliff — changing memory/{agent}/chat/... key prefix makes ALL per-chat settings invisible at once (mode/charter/onboarding drop to defaults). The symlink shim (Phase 3) avoids this entirely.
• Pre-v2 blob amnesia — chats older than STORAGE_LAYOUT=v2 have NO v2 sessions; back-fill before draining the blob.
• `tg--` vs `telegram--` chronology — merge sessions by chronological sort, not naive concat.
• `resolve_data_root()` triplicated in 3 crates — unify to one call BEFORE moving any files, or reads/writes can split across roots.
• `data/alice/` alias tree — fold into data/b7775dc9/, don't delete.

Recommended first increment (lowest-risk, high-value)

Phase 0 (dry-run detector) + Phase 2 (sessions dual-read shim — no data move yet) + unify resolve_data_root(). These are reversible, data-safe, and surface the full per-chat inventory before anything is moved. Everything after is NAO-gated.

DONE so far (2026-06-15, lab/branch — no data moved)

• `resolve_data_root()` unified (commit 1124471d8 + 5dfa8a72e): was 4 copies (core/dossier, agents-memory, alice, telegram-host inline) → ONE canonical in openalice_core::paths. The telegram-host copy had diverged (empty OPENALICE_DATA_DIR skipped the secondary). This is the "all readers/writers agree on the root BEFORE moving files" prerequisite, now satisfied.

Phase 0 — LIVE inventory (read-only, prod ~/.openalice/data, 2026-06-15)

The dry-run, run read-only against the live root. Confirms the fragmentation:

• 7 real Telegram chats, each split across 2–3 dash-variant folders under b7775dc9/chat/: | chat id | variant folders present | |---|---| | 1003770697028 | chat_telegram--, telegram--- (triple), telegram-- | | 5144606208 | chat_telegram--, telegram---, telegram-- | | 5181834298 | chat_telegram--, telegram---, telegram-- | | 5220582659 | 3 variants | | 5042302347 | chat_telegram--, telegram--- | | 1003987391011 | chat_telegram--, telegram--- | | 5060344127 | telegram-- (canonical only — clean) |
• Orphan classes counted: 6× chat_telegram-- (pre-Wave-S prefix), 6× telegram--- (triple-dash bug), 5× canonical telegram--, 4× single-dash test.
• 12 test/probe artifacts polluting the PROD data root: phase8aa3, phase8aaa-fix2-*, phase8ddd1-empty, probe-memory-003/005, probe-skill-002/003, probe-speaker-002, probe-subagent-002, telegram-11111, telegram-22222. These are safe to archive (test pollution, not real chats).
• Other roots disagree on chat count (sessions=9, connector_history=14, memory/chat=12) → confirms data is split across roots, not just within chat/.
• data/alice/ parallel tree present (agent-alias artifact → fold into b7775dc9/); b7775dc9/refs/people present.

Consolidation target made concrete: 7 canonical chat roots, fold the 2–3 variants per chat by content hash (Phase 1), archive the 12 test artifacts, fold data/alice/. Data move stays NAO-gated + runs on the 212M backup copy first.

⚠️ CRITICAL CORRECTION (2026-06-15 22:22 — live-data check before prod move)

A live mtime check of ~/.openalice/data before applying the move to prod reversed the canonical assumption above — and would have caused catastrophic data loss if applied blindly. The actual live state:

Findings that invalidate the original plan:

1. The active binary writes each chat to TWO folders near-simultaneously: chat_telegram--{id} (main, more files) and telegram---{id} (triple-dash sibling). They are written within the same minute → both LIVE, complementary (different file counts → not pure duplicates).
2. telegram--{id} (double-dash) — the folder the plan called "canonical" — is DEAD (last written May/early June). Choosing it as the merge target would archive the LIVE `chat_telegram--` + `telegram---` folders → Alice loses all active chat history.
3. The copy-validation passed (lossless) but keyed on folder names, not live activity, so it merged into the wrong (dead) canonical. Lossless ≠ correct.

Corrected prerequisites before ANY prod data move:

• Map the dual-write: why does the binary write both chat_telegram-- and telegram--- per chat? (different subsystems / redundant / triple-dash bug?) This decides what merges into what.
• Canonical = `chat_telegram--{id}` (the live main), NOT telegram--{id}.
• The move cannot run while Alice writes these folders (race) → it MUST happen inside an Alice-quiesce window (e.g. the prod-deploy restart), not live.
• Re-run the copy-validation with the corrected canonical + writer-map first.

Until this is resolved the live move is HALTED. Fresh backup taken (backups/data-pre-livemove-20260615-222214.tar.gz, 213M). Prod data untouched.

ROOT CAUSE of the fragmentation (RCA 2026-06-15 23:xx — verified)

The three chat/ folder shapes are NOT duplicate copies of one scope — they come from a scope-derivation inconsistency in the connector-bridge stages:

1. Canonical (main data) = `chat_telegram--{id}`. prompt_assembly sets ctx.chat_scope = derive_ctx_chat_scope_v2_gate(...) → "chat/{connector}-{source}" (prompt_assembly.rs:185 / :460). The Wave-S helpers pass that scope to chat_folders::chat_root, whose sanitize() (core/chat_folders.rs) maps / → _. So "chat/telegram--{id}" becomes the folder `chat_telegram--{id}` under chat/. This holds inbox/outbox/traces/ dossiers — the bulk of each chat. (The chat_ prefix is a redundant artifact of the chat/-prefixed scope being sanitized while already living under chat/.)
2. Fragmenting fallback = `telegram---{id}` / `telegram--{id}`. When a write path runs with ctx.chat_scope EMPTY, resolve_scope in chat_folders_inbox.rs / chat_folders_outbox.rs / traces.rs falls back to format!("{}--{}", ctx.connector_name, ctx.source) (double-dash literal, NO chat/ prefix). For a signed group source="-1003..." that yields telegram---{id} (triple); historically unsigned sources yielded telegram--{id} (double). These diverge from the canonical → the same chat's data splits.
3. `telegram---{id}` is ALSO the intentional LANE scope (coalesce/src/ lane_key_canon.rs:59 documents the triple dash as canonical + round-tripping). That lane state lives in its own root — do NOT "fix" the triple dash in the lane subsystem; only the chat-folder fallback divergence is the bug.

Corrected fix design (CORE ROUTING — needs lab validation + data migration, NOT a rushed change; alice-core territory):

• Make the resolve_scope fallback in the 3 Wave-S stages produce the SAME scope as prompt_assembly (i.e. chat/{connector}-{source} → sanitizes to chat_telegram--{id}), OR ensure ctx.chat_scope is always populated before those stages run. Either eliminates future fragmentation.
• Optional cleanup: drop the redundant chat/ prefix so the folder is just telegram--{id} under chat/ (cosmetic; bigger migration).
• Blast radius: touches per-chat write routing → must lab-validate that group + private + ambient paths all land in ONE folder, and that lane keys / history scoping (the just-shipped #1/#2) are unaffected.

Corrected canonical for the data consolidation: `chat_telegram--{id}` (where the main data lives), NOT telegram--{id}. The one-time consolidation folds the fallback-divergence folders (telegram---{id} chat-folder copies, telegram--{id}) INTO chat_telegram--{id} — and must run in an Alice-quiesce window.

RESOLUTION (2026-06-16) — ideal fix shipped + migration validated

The "corrected canonical = chat_telegram--{id}" conclusion above is superseded. That choice merely bent to where the stale-fallback data happened to sit. The ideal fix removes the divergence at its source instead:

Code (commit `b858d82b5`, on mvp). A single shared stages::canonical_chat_scope(ctx) = "{connector}-{source}" that all three chat-folder stages (inbox/outbox/traces) delegate to. This is byte-identical to the v2-sessions canonical (chat_id_from_conv_id → {connector}-{source}), so chat-folders and sessions converge on `telegram--{src}` (double dash for signed groups; single for unsigned DMs). The lane keeps its intentional triple-dash key (separate store). Regression test pins both the signed and unsigned shapes.

Lab-validated with real Codex: a group message lands ALL data in ONE telegram--{id} folder matching its sessions/ folder, zero split folders, and history recall reads back correctly (no read/write mismatch).

Migration (commit `5536370bc`, `scripts/migrate_chat_scope_canonical.sh`). Folds the three legacy variants (chat_telegram--, telegram---, telegram--) into the canonical telegram--{src} per chat. Canonical is derived by stripping the leading chat_ from the live chat_<conn>-… variant (sign-agnostic; provably equals the sessions/ name, which is asserted post-merge). Newest-mtime-first union with rsync --ignore-existing; differing collisions preserved as <file>.conflict-<variant>; backup-first; originals archived to chat/_archive-scope-migration/ (reversible); --dry-run supported.

Validated on a fresh copy of prod data (data-pre-livemove-20260615-222214.tar.gz): 6/6 chats folded, 0 conflicts, 0 alignment-warnings, 1358→1358 files (lossless), all jsonl+json valid. All six prod chats are signed, so each canonical is the double-dash form, 1:1 with the sessions/ names. (sessions/ also holds 3 stale tg-- orphans from an older connector name — out of scope for chat-folders; note for a later sessions-side sweep.)

Deploy coupling. Code + migration ship TOGETHER in an Alice-quiesce window: the new binary writes telegram--{id}, so deploying it without migrating would leave history in the old variant folders until the migration runs. Sequence: quiesce Alice → migrate_chat_scope_canonical.sh (backup-first) → swap binary → restart. Gated on NAO's explicit deploy go (prod still on 50a084c12).