Atlas
kb://incidents/2026-05-21-target-disk-95-percentstable2026-05-21

2026-05-20 — target/ disk runaway to 95%

incidentpostmortemcargodisksub-agents

A sub-agent dispatched on 2026-05-20 ran cargo test --workspace and inflated its worktree's target/ directory to 60 GB. Combined with 4 other active worktrees, disk usage hit 95% / 23 GB free and the monitor fired.

Timeline

  • 2026-05-20 evening — Phase 8JJJJ wave dispatched the ToolDispatch sub-agent against wt-8jjjj-tooldispatch worktree. Prompt did not cap test scope.
  • ~30 min later — target/ in that worktree at ~25 GB.
  • ~60 min later — 60 GB. Combined with other worktrees averaging 5-10 GB each, disk approaching alert threshold.
  • Alert — Monitor at df --output=pcent /home/blal ≥ 90% fired.
  • Recovery — cargo clean -p against the runaway crate freed 55 GB. Other worktrees pruned with git worktree remove --force for the ones that had already committed.
  • Resume — disk back to ~68%. Subsequent sub-agent dispatches re-scoped per the new memory rule.

Root cause

cargo test --workspace compiles every crate in the workspace. For the openalice main repo with ~430k LOC across many crates, this generates a 60 GB build artefact set. A sub-agent that needs to test a single crate has no reason to compile the workspace, but the default test command happily does so.

Why the existing -j 4 rule didn't catch this

The previous memory (2026-05-16) capped parallelism (-j 4, --test-threads=4) but not scope. A cargo test --workspace -j 4 still compiles the whole workspace, just with 4 rustc instances instead of 12. CPU pressure goes down; disk pressure stays the same.

Fix — scope rule, not just parallelism rule

The new rule (kb://infrastructure/cargo-j4-discipline) narrows the build surface, not just the thread count:

NEVER cargo test --workspace.
Use cargo test -p <SINGLE-CRATE> --lib -j 4 -- --test-threads=4.
If target/ grows past 5 GB, cargo clean -p <crate> and continue narrower.

Every dispatched sub-agent prompt now includes this rule explicitly. The discipline lives at prompt-template level, not at runtime — by the time the sub-agent is running, it's too late.

Pre-cleanup checklist (added to dispatch ritual)

  1. du -sh ~/projects/org-openalicelabs/openalice-wt-*/target → identify > 5 GB targets
  2. Remove obsolete worktrees (git worktree remove --force)
  3. Confirm disk < 75% before dispatching
  4. Max concurrent cargo-heavy agents: - disk < 75% → 3 - 75-85% → 2 - > 85% → 1

Why this incident matters

blal.de is a 464 GB single host shared across many projects. One runaway target/ is enough to threaten the whole lab. The Soul/Identity layer of Alice can't be refactored if disk is full and tests can't run. The scope rule is non-negotiable.

Open follow-up

A periodic du-based cleanup cron would catch this proactively. Not yet scheduled. Until then, every dispatch is gated by the manual checklist above.