Rust build-speedup toolchain

TL;DR

3 binaries + 1 config file = ~5-15x faster Rust cycle across the openalice org. Bootstrap with:

bash openalice-atlas/scripts/setup-rust-toolchain.sh

Idempotent, safe to re-run, never overwrites existing config without warning.

What's in the stack

sccache — the big win

The org has ~30 Rust crates (axum-common, persona-server, auth, chat-bridge, voice, rt, voice-agent, voice-client, broadcaster, scheduler, radio, tenants, stream-host, …). Every cargo invocation in any of them recompiles the same shared deps (sqlx, axum, tokio, serde, etc.) from scratch unless something caches.

sccache sits between cargo and rustc. It hashes the inputs (source + flags + compiler version) and writes the resulting .rmeta/.rlib to ~/.cache/sccache/. Next compile with the same inputs — anywhere in any worktree — hits the cache. Cargo's incremental cache stops at the workspace boundary; sccache crosses it.

Verified speedup on openalice-radio (full rebuild after rm -rf target/debug):

• Cold: 40.2s (cache miss on 308 Rust + 32 C/C++ + 28 Assembler artifacts)
• Warm: 14.8s (368 cache hits, 49.5% hit rate)
• 2.7x on a single cold→warm cycle

This grows: every distinct crate that compiles sqlx-0.8.6 writes one set of cache entries. Subsequent crates pulling sqlx-0.8.6 hit those entries. After 5-10 sessions across the org, cross-worktree hit rate climbs and the speedup compounds.

mold — the link-time win

Default rustc on Linux uses GNU ld for the final link step. On a workspace like openalice-persona/server with hundreds of object files, ld can take 5-30 seconds linking after rustc finishes compiling everything. mold replaces it.

Wiring (in ~/.cargo/config.toml):

[target.x86_64-unknown-linux-gnu]
rustflags = ["-C", "link-arg=-fuse-ld=mold"]

This passes -fuse-ld=mold to gcc (the C-frontend rustc uses to link). gcc resolves mold from PATH (lives at /home/blal/.local/bin/mold).

cargo-nextest — the test-time win

cargo test runs tests in a single process, one at a time per crate, serially across crates. cargo nextest run runs each test in its own process with smart parallelization. Same observable behaviour, ~3x faster on test-heavy crates.

Adoption note: still call cargo test in agent prompts (it's the reference behaviour). Switch to cargo nextest run in CI / local sanity loops where speed matters.

What's intentionally NOT wired

CARGO_TARGET_DIR (shared target dir across worktrees)

Tempting — would dedupe ~100-200GB of repeated target/debug across the 27+ worktrees. But our docker-compose.dev.yml files mount the local target/release/<binary> paths:

volumes:
  - ./server/target/release/persona-server:/usr/local/bin/persona-server:ro

A global CARGO_TARGET_DIR=/home/blal/.cargo-shared-target would break every one of those mounts. The cost (rewriting 10+ docker-compose files + verifying all rebuilds) exceeds the disk-space saving on blal.de's 464GB NVMe.

Workaround: clean target/debug dirs periodically. Use ~/projects/_shared/scripts/cleanup.sh --aggressive monthly per CLAUDE.md.

profile.dev opt-level=1 (workspace Cargo.toml)

Per-crate Cargo.toml change. Faster builds + faster tests at the cost of touching N files. Apply ad-hoc on crates whose dev compile time hurts (e.g. persona-server's 30+ binary build). Not worth the global churn.

Direct mold invocation via MOLD_PATH

Some setups bypass gcc and invoke mold directly. Our setup uses gcc-via-rustflags because it composes with rustc's existing link flags without surprises.

Server safety

Per kb://infrastructure/cargo-j4-discipline: cap cargo at -j 4 and max 3 cargo-heavy agents concurrent. blal.de (12-thread Ryzen 5 3600, 62 GB RAM) once froze at load 672 from 15+ parallel rustc. sccache + mold reduce the per-compile load — they don't multiply parallelism. The -j 4 cap stays.

Monitoring

sccache --show-stats           # cache hit rate + size
sccache --zero-stats           # reset counters before measuring
du -sh ~/.cache/sccache        # cache disk usage

The cache auto-evicts via LRU when it hits the default 10GB limit. Tune via SCCACHE_CACHE_SIZE env var if needed (set in ~/.bashrc or a wrapper script — don't bake into ~/.cargo/config.toml since that's compiler-side, not server-side).

2026-06-16 update — gotchas + release-build + Docker (the missing half)

The stack above optimizes the dev/incremental cycle. Three things it did NOT cover surfaced during the alice 1/2/5 prod deploy (where a cargo build --release of the alice binary ran the disk to 100% and ENOSPC'd on the LTO link). These apply org-wide — hand them to whoever owns each repo.

⚠️ GOTCHA #1 — sccache and incremental are MUTUALLY EXCLUSIVE

This is the big one and it was silently costing us the entire sccache win.

sccache cannot cache a crate built with `-C incremental`. It detects the incremental flag and falls through to plain rustc — no cache read, no cache write. If you have BOTH RUSTC_WRAPPER=sccache AND incremental = true, sccache is dead weight: it wraps every compile and caches nothing.

How to spot it: sccache --show-stats shows Cache hits: 0 / Cache misses: 0 / Compile requests executed: 0 even after a full build. (Confirmed on openalice 2026-06-16 — global [build] incremental = true in .cargo/config.toml → 0% cache.)

Fix — pick the right tool per profile:

# .cargo/config.toml  — do NOT set a global [build] incremental
# (a global one forces incremental onto release too, killing sccache there).

[profile.dev]
incremental = true      # local iteration: incremental beats sccache

[profile.release]
incremental = false     # deploy builds: let sccache cache the deps instead

Rule of thumb:

• dev / local edit-loop → incremental = true, sccache irrelevant (it's bypassed; that's fine).
• release / CI / deploy → incremental = false so sccache caches the (rarely-changing) dependency .rlibs. Then a repeat deploy build only recompiles your changed crates, not 500 deps.

Release/deploy builds: thin-LTO instead of fat-LTO

The single biggest lever for the deploy build (and the ENOSPC fix). Our current release profile is the slowest, most disk/RAM-hungry config that exists:

[profile.release]
lto = true            # ← FAT LTO: serial, whole-program, huge peak temp/RAM
codegen-units = 1     # ← maximal runtime perf, zero codegen parallelism

lto = true (fat) + codegen-units = 1 makes the LTO link a single serial pass holding the whole program's bitcode — that's the 6-10 min link and the multi-GB temp that ENOSPC'd on a 91%-full disk. For a server binary (not a hot micro-benchmark) switch to:

[profile.release]
lto = "thin"          # parallel, ~95-99% of fat-LTO runtime perf
codegen-units = 16    # parallel codegen
strip = true
panic = "abort"

Expected: release link time drops sharply, peak disk/RAM falls (no more ENOSPC), for a ~1-3% runtime cost that is invisible on an I/O- and LLM-bound service like Alice. Keep fat-LTO only for a genuinely CPU-bound hot binary if one ever exists.

Docker-built services: BuildKit cache mounts

sccache/mold help host-native builds (alice is built on the host). Services that build inside Docker (tenants, agent-lite, auth, atlas, gateway, …) get nothing from them across image rebuilds, because a source change busts the Docker layer that runs cargo build, re-compiling every dependency.

Use BuildKit cache mounts. They persist the cargo registry + target/ across image rebuilds, so a source-only change recompiles just the changed crate. They are transparent to our repo layout and need no extra tooling on the host.

# syntax=docker/dockerfile:1.7
FROM rust:1-bookworm AS builder
WORKDIR /app
COPY . .
RUN --mount=type=cache,target=/usr/local/cargo/registry \
    --mount=type=cache,target=/app/target \
    cargo build --release && \
    cp target/release/<bin> /app/<bin>      # cp OUT of the mount before runtime stage
# runtime stage copies /app/<bin>, NOT /app/target (the mount is build-only)

Measured (`openalice-tenants`, 2026-06-16): cold cargo step 177.5s → warm 14.1s (~12.6×); rebuilt + deployed + all SAML/SCIM endpoints green. Enable BuildKit with DOCKER_BUILDKIT=1 (or compose >= v2, which is on by default).

Do NOT use `cargo-chef` for our services. It panics reconstructing skeletons for crates that depend on out-of-workspace siblings (../openalice-axum-common, …): cargo chef cook dies at recipe.rs:224 — No such file or directory. Every deployed service here uses sibling path-deps, so chef is a dead end for us — cache mounts give the same warm-rebuild win on our single host without the layout constraint.

Pre-build disk-guard

The 2026-06-16 ENOSPC was preventable: blal.de sat at 91% (29G free) and the fat-LTO link of alice pushed it to 0. Rust target/ dirs are 5-44GB each (openalice/target alone = 44G). Guard every release build:

FREE=$(df --output=avail / | tail -1)      # KiB
[ "$FREE" -lt 26214400 ] && { echo "ABORT: <25G free, prune first"; exit 1; }

Reclaim sources, safest-first: docker builder prune -af (build cache), finished lab-data dirs, cargo clean --release on idle sibling repos. Avoid docker volume prune on a multi-project host (a dangling volume can be a stopped project's DB). See kb://infrastructure/cargo-j4-discipline for the parallelism cap that also bounds peak disk during concurrent builds.

Priority for a repo owner

1. Fix the sccache↔incremental conflict (free; un-deads an already-installed tool).
2. thin-LTO for release (free; biggest deploy-build + ENOSPC win).
3. BuildKit cache mounts for any Docker-built Rust service (NOT cargo-chef — see above).
4. disk-guard in deploy scripts.

Adoption history

• 2026-05-22 — Minsky (alice-core agent) installed sccache + mold + cargo-nextest binaries on blal.de. Set RUSTC_WRAPPER=sccache + SCCACHE_DIR in ~/.bashrc for interactive shells.
• 2026-05-23 — Tycho (alice-persona agent) noticed non-interactive shells (docker exec, sub-agent Bash tool) don't source .bashrc. Created ~/.cargo/config.toml that wires sccache + mold globally without depending on shell env vars. Wrote setup-rust-toolchain.sh for future bootstrap.
• 2026-06-16 — Org-wide rollout. Norbert (alice-core) fixed the sccache↔incremental conflict (per-profile incremental) + applied thin-LTO + disk-guard to openalice. Tycho applied thin-LTO to 10 streaming/persona/social/live repos + converted openalice-tenants to BuildKit cache mounts (177.5s → 14.1s warm, ~12.6×). Key correction: cargo-chef was found to PANIC on our out-of-workspace sibling path-deps (recipe.rs:224), so the Docker-build recommendation is now BuildKit cache mounts, not chef (section above amended).

Origin

NAO referenced Minsky's optimization list in TG message 3641 (2026-05-23 05:14 UTC). Confirmed binaries installed; gap was wiring. Decision: persist setup script in openalice-atlas/scripts/ (alongside git-hooks/install.sh) so future dev boxes can re-bootstrap with one command. Atlas is the canonical place for org-wide developer tooling per its cockpit role.

Cross-links

• cargo-j4-discipline — the safety rule that caps parallelism
• streaming-product-modularization-wave — the 13-wave refactor where this toolchain would have saved hours