openalice-lab

Registers arbitrary agents via agent.yaml + Dockerfile. Loads YAML benchmark suites (tool-calling, memory-recall, safety, latency, multi-turn). Scores each agent with the same 5-metric engine used by the Incubator, producing a LeaderboardEntry per agent. Head-to-head comparison and historical tracking across agent versions. ═══════════════════════════════════════════════════════

BudgetGuard enforces a daily USD limit so accidental full-suite runs don't drain Codex quota. CacheBackend (SQLite-backed or in-memory) deduplicates identical (model, prompt, tool-hash) calls — cache-hit = zero tokens. CostModel provides calibrated per-provider rates. TinyRunner orchestrates adapters: swe_bench, humaneval_mini, terminal_bench, codeclash. Tier-1 target cost ≈ $0.30/commit. @feature lab.bench.integrity-certificate bench-integrity OIC M1: ed25519-signed IntegrityCertificate wraps each benchmark run. AttestationSurface captures git SHA, model ID, provider, workspace allowlist SHA256, harness provenance SHA256, sample counts, cost, and a terminator_test_passed flag. Bundle written as certificate.json + bundle.tar.zstd for offline third-party verification without re-running Alice. WHY: Codex subscription quota is finite. Running SWE-bench Verified unmetered would burn a month's budget in one commit. Every design choice here is subordinate to "first runs must be cheap". Layout: error.rs — `BenchError` enum. FULL IMPL. budget.rs — `BudgetGuard` daily-limit enforcement. FULL IMPL. cost.rs — `CostModel` pure cost estimator. FULL IMPL. cache.rs — `CacheBackend` trait + rusqlite + in-mem impls. FULL IMPL. adapter.rs — `BenchAdapter` trait + `Tier` + `TinyRunner` (stub). report.rs — `RunReport` + `render_markdown()` (stub). runners/ — one adapter impl per bench (stubs; slices 3–5). Related docs: openalice-lab/docs/bench-harness-spec-2026-04-22.md — full design. openalice-lab/HARNESS.md — generic lab-harness (this crate layers on). ═══════════════════════════════════════════════════════════════════════════

SQLite control plane (WAL mode, 6 tables, 9 indexes) storing every run, test result, score, xray snapshot, session recording, and patch proposal. Provides heatmap(), score_history(), list_runs() aggregates consumed by lab-dashboard and lab-cli. Migrations are idempotent (IF NOT EXISTS) — safe to re-run on every startup. ═══════════════════════════════════════════════════════

Axum WebSocket server broadcasting real-time test progress to a vanilla-JS frontend. REST endpoints: /api/runs (pass/fail per run), /api/scores (trend data), /api/stats (table counts), /api/heatmap (feature × tier grid), /api/capsules (live Docker container states), /metrics (Prometheus format). Optional LAB_DASHBOARD_KEY API-key auth via X-Lab-Key header or ?key= query param. Served at atlas.blal.pro. ═══════════════════════════════════════════════════════

Root-cause heuristic classifies failures as prompt/code_bug/config/ missing_feature/flaky_test/external. Patch proposals auto-trigger when the same cause recurs 3+ times. A/B testing computes score delta with statistical significance (p<0.05). Canary gate: tier-0=100%, tier-1>90%, regression<10% — produces a CanaryDecision with reason. ═══════════════════════════════════════════════════════

`render_html(html, out_dir, opts) -> RenderOutput`. Locates a Chromium binary across the common install locations, writes the HTML to a temp file under `out_dir`, and shells out to `--headless=new --screenshot`. No new browser-driver dependency: we reuse the same headless-Chrome substrate the streaming / Tier-3 stack already relies on, invoked through `std::process::Command` so the helper stays dependency-light. Determinism (design doc §5): fixed viewport, a fixed virtual-time budget (`settle_ms`) that advances the page's clock deterministically before the snapshot, and `--force-color-profile=srgb` so the PNG bytes are stable across hosts. This module is intentionally LLM-free. The render path is fully validatable without any model call — the only LLM cost (the vision judge) lives behind the `VisionJudge` trait in `vision_judge`. ═══════════════════════════════════════════════════════════════════════════

Task/Solver/Scorer triad modelled on UK AISI Inspect. Solver and Scorer receive physically split views of each Sample — the solver never sees the reference answer (oracle_mode is the single documented exception for ReferenceSolver pipeline validation). BudgetTracker enforces per-sample caps on tokens, tool calls, wall-time, and USD cost. @feature lab.harness.alice-solver AliceSolver: HTTP client to Alice /v1/chat with budget recording. AliceMultiTurnSolver: multi-turn loop for stateful probes — copies fixtures into the bind-mount before turn 1, captures Prometheus pre/post snapshots, honours delay_ms_after between turns. @feature lab.harness.side-effect-scorer SideEffectScorer: offline scorer for master-plan probes. Asserts file_exists, dir_nonempty, audit_row (JSONL scan), prometheus_inc, prometheus_lte_delta, prometheus_eq_delta. Never touches the network at score time — reads only the bind-mounted data dir and the Prometheus snapshots captured by AliceMultiTurnSolver. @feature lab.harness.trace-regression TraceStore persists per-(task, solver) JSONL traces. compare() diffs two runs and returns a RegressionReport. probe_alice exits 2 on regression so CI catches it without human review. @feature lab.harness.master-plan-probes Five JSONL datasets (memory, speaker, skill, subagent, coalesce) covering Wave Q1–Q8 Alice capabilities. MasterPlanProbe loader with typed multi-turn input schema. SAMPLE_FILTER env for single-sample iteration without burning full budget. Layout: task.rs — Task + Sample types. A Task is "a dataset + what a solver must produce + how to score it". `Dataset` is a sync or async stream of samples. solver.rs — Solver trait. The "agent under test". Implementors include Alice (HTTP), Aider (CLI), Claude Code (CLI), raw model APIs. Each gets a `SolverContext` with sandbox access + budget + approved tools. scorer.rs — Scorer trait. Given (Sample, SolverOutput) → Score. Built-ins: ExactMatch, TestsPass, ModelGraded. budget.rs — Token + time caps enforced per sample. Any solver going over is terminated; its sample counts as failure. report.rs — Aggregation: per-task summary + per-sample trace. Feeds lab-dashboard. Stored as JSONL so diffs across versions are grep-able. runner.rs — The orchestrator: iterate samples, call the solver within budget, score the output, roll up to a Report. Handles parallelism (configurable) + cost tracking. error.rs — Harness-specific error enum. The crate is intentionally dependency-light: every concrete agent / benchmark lives in a downstream crate that pulls `lab-harness`. This keeps the core stable while we iterate on individual adapters. ═══════════════════════════════════════════════════════════════════════════

`check_self_contained(html)` — the "Completeness pre-filter" from the pipeline. Pure logic, no network, no LLM: returns the list of violations so reports can show exactly why an artifact was rejected before any (paid) vision judge runs. @feature lab.harness.ui-bench-dataset `UiBench::from_jsonl` loads the 12 version-pinned UI tasks. Each row is {slug, title, prompt, rubric[], interaction_script?}. The solver sees the prompt; the scorer-side reference carries the rubric so the vision judge can score against concrete checkable items. The render itself lives in `cdp_render`; the scoring lives in `vision_judge` (the only LLM cost, behind a trait). This module never calls a model. ═══════════════════════════════════════════════════════════════════════════

`VisionJudge` trait: given a rendered PNG + task title + rubric, return a `VisionVerdict` (completeness / visual_quality / interactivity, each 0–100, plus notes). `VisionJudgeScorer` turns a verdict into a single 0–100 score via configurable weights (default 0.4 / 0.4 / 0.2). `build_judge_prompt` assembles the rubric-anchored prompt text: it lists each rubric item, asks the judge to score every dimension 0–100 with brief justification, and demands strict JSON back. Pinning the prompt here (versioned) is what lets the attestation surface record "which judge + which prompt" produced a score. v1 scope: trait + a `MockVisionJudge` for tests. NO real network judge is invoked by any test. A real Codex/OpenRouter-backed judge is a later, explicitly-gated slice (the trait makes that a drop-in). v1 INTERACTIVITY NOTE: scripted interaction (arrow-key / click events) is deferred (the dataset's `interaction_script` is null for every task). Until then the judge returns a fixed `interactivity` value (a coarse "does it look like it would respond" prior) and `VisionJudgeScorer` weights it lightly (0.2). When the scripted-interaction slice lands, this dimension becomes execution-verified rather than judged. ═══════════════════════════════════════════════════════════════════════════

Isolated Docker sandbox for Alice under test. Four profiles (minimal/standard/full/stress) controlling LLM backend, worker set, browser, A2A, and token budget. Security: lab-internal network (no internet), cap_drop ALL, no-new-privileges, read-only rootfs. Bind-mounted data dir (Phase 8L) lets the host scorer read Alice's filesystem side-effects. UID mapping (Phase 8S-perm) ensures host user can write fixtures pre-run. ═══════════════════════════════════════════════════════

Parses YAML TestFixture files (inject / expect / verify / wait steps) from tests/tier-{0..4}-* directories. Executes steps sequentially: inject via mock /_lab/inject, expect via mock /_lab/messages poll, verify via Alice REST calls. Computes 5-metric scores per fixture, stores TestResult + Score to lab-control SQLite, returns pass/fail summary. Filters by --tier and --feature flags for targeted runs. ═══════════════════════════════════════════════════════ FLOW: 1. Load YAML fixtures from tests/tier-{N}-*/ directories 2. For each fixture: a. Reset mock services (POST /_lab/reset) b. Execute steps sequentially: - inject: POST to mock-telegram /_lab/inject - expect: wait + GET mock-telegram /_lab/messages + compare - verify: GET Alice API endpoints + check conditions c. Compute metrics (correctness, latency, tokens, safety, memory) d. Record TestResult + Score to database 3. Return summary with pass/fail counts and scores

In-process axum servers that impersonate Telegram (sendMessage / getUpdates / /_lab/inject), Discord, and LLM providers (OpenAI /v1/chat/completions + Codex /v1/responses in echo|canned|record|replay modes). Zero real API calls; all interactions are recorded for assertion. Service discovery via shared MockPorts JSON file. ═══════════════════════════════════════════════════════

GET /v1/prompts/ab-tests — list all A/B tests for a block. POST /v1/prompts/ab-tests — create a new A/B test comparing two block versions with a configurable traffic split. Results and status (running / concluded) stored in ab_tests table in prompts.db.

POST /v1/prompts/assemble — render the stored prompt blocks for a given (model, mode, domain, speaker_ctx) triple and persist the resulting assembly in prompt_assemblies for diffing and replay. Delegates rendering to assembler::assemble(); records the actor from the JWT AuthUser extension.

Read-only block endpoints: GET /v1/prompts/blocks — paginated list with category/mode_gate/tag filters. GET /v1/prompts/blocks/{slug} — full block detail including current body. GET /v1/prompts/blocks/{slug}/versions — version timeline (all statuses). GET /v1/prompts/blocks/{slug}/diff — unified diff between two semver refs.

Axum Router factory for the Prompt Evolution Studio microservice. Wires PromptStudioStore (SQLite), JWT middleware, all 15 /v1/prompts/* routes, the /healthz endpoint, and the embedded studio.html SPA. Entry point for both the library binary (port 3700) and the standalone prompts-studio-svc (port 8090). Auth: HS256 JWT verified via SHARED_JWT_SECRET; STUDIO_AUTH_BYPASS=1 disables check for dev. Boot-time inspector seed via INSPECTOR_PATH + OPENALICE_SOURCE_DIR if prompt_blocks table is empty.

Draft lifecycle: POST /v1/prompts/blocks/{slug}/draft — create a new draft version (status=draft, semver auto-bumped or explicit hint). POST /v1/prompts/blocks/{slug}/rollback — roll the block back to a prior published version. POST /v1/prompts/drafts/{id}/approve — publish the draft (flips prompt_versions.status draft→published, archives prior head, bumps block's current_version). POST /v1/prompts/drafts/{id}/reject — mark draft as rejected (status=archived), preserving audit trail.

POST /v1/prompts/playground — LLM round-trip test bench for assembled prompts. Phase 8MMMM ships a deterministic stub (PLAYGROUND_STUB response) so the SPA can be built end-to-end without live LLM cost. Phase 8NNNN (goal: lab.prompts-studio.live-llm-playground) wires real Codex / OpenRouter calls behind explicit NAO sign-off.

DB-backed reflexion proposal review (legacy path): GET /v1/prompts/reflexion-proposals — list proposals filterable by status (pending / approved / rejected). POST …/{id}/approve — mark proposal approved + record approver + decision_note; persisted to reflexion_proposals table. POST …/{id}/reject — mark rejected. Distinct from reflexion_b (file-backed YAML proposals from the alice runtime scheduler).

File-backed reflexion B-hybrid proposal review (Phase 8MMMM-PPPP): GET /v1/reflexion/proposals — lists YAML proposals written by the alice runtime scheduler under OPENALICE_PROMPTS_ROOT/prompts/proposals/*.yaml. POST …/{id}/accept — materialises an overlay YAML + removes source file. POST …/{id}/reject — renames to *.yaml.rejected (archive sweep picks up after 30d). NEVER deletes (feedback-memory-archive-not-delete). Distinct from routes::reflexion (DB-backed, at /v1/prompts/reflexion-proposals).

Standalone binary wrapping the lab-prompts-studio library (NAO task #454, 2026-05-22). Binds on PROMPTS_STUDIO_BIND (default 0.0.0.0:8090) — a distinct port from the library's own binary (:3700) so both can run side-by-side in dev. Boot sequence: open/create SQLite at PROMPTS_DB_PATH → optional inspector seed → JWT setup → serve. Deployment surface (Dockerfile.prompts-studio-svc, compose stanza, Traefik labels) evolves independently of the library. «без деплоев» — NOT live until NAO approval (goal: lab.prompts-studio.svc-deploy).

Rust rewrite of openalice-ranker (Python). Consumes Alice's features.json and competitor YAML fixtures; produces CoverageDiff, GapRanker priority list, and a ranked leaderboard. Pipeline: FeaturesCatalog + CompetitorRegistry → CoverageDiff → FuzzyGapRanker → JsonRenderer / MarkdownRenderer. Token-free — pure data analysis. ═══════════════════════════════════════════════════════

Pure-Rust scoring engine: correctness (40%), safety (20%), latency (15%), tokens (15%), memory quality (10%). Aggregates by feature and tier (T0-T4 weighted). Regression detection: >5% drop = warn, >10% = block. Cost estimator in USD per 1K tokens. Zero I/O — every function is synchronous and deterministically testable. ═══════════════════════════════════════════════════════

Canonical domain types shared across all lab crates: Run, TestResult, Score, XraySnapshot, Recording, Patch, TestFixture, LabConfig. Every struct maps to a SQLite table or API wire contract. Zero I/O — pure data definitions with serde + full unit-test coverage. ═══════════════════════════════════════════════════════

Stage-1 AST (zero tokens): subprocess call to openalice-inspector for module/function/import inventory. Stage-2 LLM: priority-ranked file list (prompt=10, safety=9, agent-loop=8, tools/memory=7, config=4) — top 30 files, ~5-15 K tokens. Stage-3 Diff: snapshot comparison for competitor change detection. WatchedRepo git-pull + diff pipeline runs on zero LLM tokens until a significant change is detected. ═══════════════════════════════════════════════════════