Ultra-SOTA Benchmark System — Design Doc

Date: 2026-06-09 Author: Tycho Brahe (agent) per NAO directive — «улучши всю систему чтобы бенчмаркать Алису … делай ultra state of the art deluxe 2088 … но пока подходим экономно … что ещё забыли для топов бенчей» Status: DESIGN ONLY. No expensive benchmark RUNS without NAO's explicit OK (economy gate). Related: #193 (ecosystem unification), #194 (minimalist coder + Live-SWE), #198 (benchmark-integrity cert), [[project-alice-agi-coding-capability]], [[reference-dashboard-architecture]].

0. TL;DR

We already own ~half of a world-class benchmark rig: a Docker incubator, an Inspect-style lab-harness (Task/Solver/Scorer/trace/golden), a BenchAdapter portfolio runner with cache+budget+cost, an ed25519 cert-integrity pipeline (#198), live token + cost_usd instrumentation, and a rank-evaluator that already scores honesty/hallucination. What's missing is (a) completing the 4 runner stubs to real Docker execution, (b) a visual/UI axis (CDP render → screenshot/MP4 → vision-judge + ELO), and (c) folding the remaining BridgeBench-v2 dimensions (Algorithms, Debugging, Refactoring, Generation, Security, Multi-File, BS-pushback, Reasoning, Speed, Cost) into one unified multi-axis runner.

This doc maps BridgeBench v2 (the viral "vibe-coding" benchmark) onto our infra, adds the heavy academic benches BridgeBench omits (SWE-bench Verified, Terminal-Bench, GAIA), and specifies our differentiator: every run is cryptographically attested (anti-cheat) — stronger than BridgeBench's append-only journal.

Build philosophy: cheap code first, micro-validate on 1–3 samples, gate all full/expensive LLM runs on NAO's word.

1. What we already have (ground truth, 2026-06-09)

All in the openalice-lab repo unless noted.

1.1 bench-harness (portfolio runner)

• BenchAdapter trait: slug() / display_name() / load(tier) -> Vec<BenchInstance> / async run_instance(inst, ctx) -> InstanceResult / cost_estimate_usd().
• TinyRunner::run(tier, ctx) — top-level orchestrator over a portfolio of adapters. STUB (slice 2).
• Tier = Tiny | Small | Full (e.g. SWE-bench Tiny=3, Small=20, Full=500).
• RunContext { cache, budget, cost_model, integrity }.
• Real today: cache (SQLite + in-mem), budget (daily-limit guard), cost (pure estimator), error. Stub: report render, TinyRunner::run.
• Runner stubs (all `todo!()`): swe_bench (slice 4), humaneval_mini (slice 3), terminal_bench (slice 4), codeclash (slice 5, contingent on public dataset).

1.2 lab-harness (Inspect-style eval framework)

• Task / Sample; Solver trait (impls: Alice via HTTP, Aider CLI, Claude Code, raw models) with SolverContext + budget + approved tools.
• Scorer trait; built-ins ExactMatch, TestsPass, SideEffectScorer (reads Prometheus counter deltas + file_exists/audit_row/dossier_contains — strict & flexible variants).
• run_task(task, solver, scorer, cfg) -> Report; JSONL per-sample trace.
• trace::TraceStore (regression detection); golden::{GoldenAssertions, ReplaySolver} (offline replay, no live LLM).
• Bundled datasets: swebench_lite, aider_mini, gaia_mini, master_plan.

1.3 lab-incubator (sandbox)

• Docker lifecycle via docker compose CLI; 4 profiles minimal|standard|full|stress.
• bind-mount of Alice's data dir so the host scorer reads filesystem side-effects; UID mapping for pre-run fixtures.
• Security: lab-internal network (no internet), cap_drop ALL, no-new-privileges, read-only rootfs; health-gate on /health.

1.4 bench-integrity (#198 — anti-exfiltration cert)

• AttestationSurface (25+ fields: git SHAs, model hash, dataset hash, workspace_allowlist_sha256, harness_provenance_sha256, path_integrity_sha256, container_digests, Prometheus scope hash, timestamps, solver/scorer names).
• IntegrityCertificate (schema-version pinned) → ed25519 sign (key ~/.openalice/lab/integrity.ed25519, 0600) → bundle.tar.zstd → offline verify (no network, no re-run).
• Existing artifact: certs/swebench-lite-2026-04-23.* (pipeline-validated with mock solver/scorer — i.e. the cert plumbing is real, the solve was mock).

1.5 Alice-side instrumentation that already maps to benchmark axes

• Speed/Cost: Wave-2 per-feature LLM token + call counters (c8efd3b89) + `cost_usd` rate table (11 models, cost_usd_from_usage(), bundle 7432ebd7d). → directly produces throughput, TTFT-adjacent, and cost-per-correct.
• Hallucination/Pushback: rank_evaluation.rs — hallucination penalty (−20 at honesty<0.3), honest-uncertainty bonus, combined() weights 0.30/0.30/0.25/0.15. → maps to BridgeBench BS-pushback + Hallucination.
• Visual render substrate: CDP/VNC Tier-3 isolation + the browse tool + the streaming stack already drive headless Chrome. → reusable for UI-axis screenshot/MP4.
• Personality non-regression: openalice-golden-snapshots (byte-identical Cat-A) + master-plan probes (15 samples).

1.6 CLI surface

lab (db/xray/incubator/arena/evolution/mocks/dashboard/report/watch/replay/ranker), bench-integrity, probe_alice, probe_master_plan, swebench_leaderboard, harness_history.

1.7 What does not exist yet

• No headless-browser screenshot capture wired into a scorer.
• No vision-LLM judge, no ELO/pairwise scoring, no UI/visual task set.
• TinyRunner::run + all 4 runners are stubs.
• No Algorithms/Debugging/Refactoring/Generation/Security/Multi-File task sets.
• No GAIA/WebArena/Terminal-Bench real execution.

2. Reference: BridgeBench v2 methodology (NAO-sourced, 2026-03-26)

The "vibe-coding" benchmark by BridgeMind (open-source; reposted ~100M X views). Ground truth from NAO's paste.

7 core categories (each scored 0–100 = weighted(Correctness, Completeness, Quality); overall = weighted avg):

Standalone benchmarks (separate pages/scores):

UI v2 — the 12 tasks (familiar scenes "everyone can judge at a glance"): Lava Lamp · Thunderstorm Over City · Vinyl Record Player · Underwater Coral Reef · Neon Sign Flickering · Campfire Under Stars · Space Invaders · Breakout · Flappy Bird · Snake Game · Analog Clock · Music Visualizer. Has a "Visual Comparison" + "Export Video Comparison" (branded MP4 of one scene across models).

Reproducibility principles (BridgeBench's own):

• Calls model APIs directly from source (bypass aggregators) → real model, not middleman.
• Append-only run journal, version-pinned scoring, resumable runs, provider abstraction = "adding a model is two files".
• DGX Spark integration → cloud API vs locally-hosted open-weight on the same chart, with GPU/power/energy-per-token.

Roadmap (v2 → v3): BridgeBench CLI (local runs), community submissions w/ reproducible configs, category deep-dives, SpeedBench integration; v3 adds 10 tracks (code review, spec gen, clarification, repo orientation, test repair, writing, integration, UX polish, launch readiness, multi-step tool use).

3. Unified taxonomy — our axes

We do not clone BridgeBench; we superset it. Five axes, each a portfolio of BenchAdapters, all run through the same incubator + cert pipeline.

AXIS 1 — Code correctness (repo + function level)
  ├─ SWE-bench Verified / Lite        (real Docker solve — completes the stub)
  ├─ HumanEval+ / MBPP+               (function-level; HumanEval-10 already REAL: 11/11)
  ├─ Aider Polyglot                   (edit-format fidelity; aider_mini exists)
  ├─ LiveCodeBench                    (contamination-resistant, time-windowed)
  └─ BridgeBench-core: Algorithms · Debugging · Refactoring · Generation · Multi-File · Security

AXIS 2 — Terminal / computer-use
  ├─ Terminal-Bench 2.0              (containerised terminal tasks — completes the stub)
  └─ OSWorld (later; GUI computer-use)

AXIS 3 — Agentic / web / tool-use
  ├─ GAIA (gaia_mini exists)         (multi-step, tool-use, real-world QA)
  └─ WebArena / Mind2Web (later)

AXIS 4 — Visual / creative (NEW — the bridgebench "lava lamp" axis)
  └─ UI-bench: 12 tasks → CDP render → screenshot + MP4 → vision-judge + rubric + pairwise ELO

AXIS 5 — Honesty / personality (our differentiator, already partly built)
  ├─ BS-pushback (100 nonsense-premise tasks) → uses rank-evaluator honesty
  ├─ Hallucination (code-execution-verified)
  ├─ Reasoning (grounded multi-step, conflicting artifacts)
  └─ Personality non-regression (golden Cat-A + master-plan probes)

CROSS-CUTTING (every axis emits these)
  ├─ Speed  — tok/s + TTFT (from token counters)
  ├─ Cost   — cost-per-correct (from cost_usd)
  └─ Integrity — ed25519 attestation bundle per run (#198)

Why this beats just-BridgeBench: BridgeBench is strong on breadth-of-dimensions but shallow on repo-level agentic difficulty (no SWE-bench-Verified, no Terminal-Bench, no GAIA). We get both — the viral dimensions and the academic-credibility benches — under one attested runner.

4. Per-axis design

Each adapter implements BenchAdapter. Common shape: load(tier) reads a version-pinned JSONL (with a dataset SHA the cert captures); run_instance drives a Solver (Alice-HTTP in-incubator, or a raw-model solver for comparison) and a Scorer; cost/budget enforced by RunContext.

Axis 1 — Code correctness

• SWE-bench (Verified + Lite). Completes `swe_bench.rs` slice 4. load: read JSONL, verify dataset hash, slice per tier. run_instance: spin a per-instance Docker image (repo @ base commit), hand Alice the issue text + repo, capture her patch, apply, run the hidden test_patch, score TestsPassScorer (FAIL→PASS transition on the gold tests + no regressions). Isolation: incubator full profile, no-internet. This is the flagship for the #194 autonomous-coding capability ([[project-alice-agi-coding-capability]]).
• HumanEval+ / MBPP+. Completes `humaneval_mini.rs` slice 3. Already have a REAL HumanEval-10 run (11/11). Extend to HumanEval+ (EvalPlus extra tests) + MBPP+. Function-level, cheap → good for micro-validation of the whole pipeline.
• Aider Polyglot. Reuse aider_mini; scores whether the model emits a correctly-applicable edit (diff/whole-file) that passes the exercise tests across languages.
• LiveCodeBench. Time-windowed problems (post-training-cutoff) → contamination-resistant; report by window.
• BridgeBench-core six. Author/port task sets for Algorithms, Debugging, Refactoring, Generation, Multi-File, Security. Scorers: - Algorithms/Generation/Multi-File → TestsPass (hidden cases) + a Completeness sub-score (fraction of spec acceptance-criteria met) + a Quality sub-score (lint/AST heuristics; over-engineering penalty). - Debugging → FAIL→PASS on the repro test + hidden-bug tests + a regression guard + (optional) root-cause LLM-judge. - Refactoring → behavior-preservation (pre/post test parity) + structural-intent AST checks (e.g. "extracted N functions", "removed the singleton") + hidden-regression tests. - Security → secure-gen test cases (500+ incl. adversarial) + a vuln-detection sub-task scored against ground-truth CWE labels.

Axis 2 — Terminal / computer-use

• Terminal-Bench 2.0. Completes `terminal_bench.rs` slice 4. Parse the TB manifest, spawn the TB Docker runner per task, Alice acts via the bash tool surface inside the incubator; score TB's own pass criteria.

Axis 3 — Agentic / web

• GAIA (gaia_mini → full). Multi-step tool-use QA; exact-match / normalized-answer scorer. WebArena/Mind2Web deferred (needs a hosted web sandbox — heavier).

Axis 4 — Visual / creative (the new build)

See §5 — this is the largest net-new component.

Axis 5 — Honesty / personality

• BS-pushback. 100 tasks across 5 domains seeded with fabricated jargon / reversed relations. Solver = Alice; scorer = LLM-judge → {pushback, partial, accepted}, cross-checked against the rank-evaluator honesty score (we already compute it) to reduce judge variance.
• Hallucination. Code-analysis questions verified by code execution + ground truth (deterministic, no judge where possible).
• Reasoning. Grounded multi-step over conflicting artifacts; score evidence-citation and final conclusion.
• Personality non-regression. Golden Cat-A byte-identical + 15-sample master-plan probes. This axis must stay green on every Alice change (it already gates deploys).

5. Axis 4 in detail — Visual / UI bench (CDP → screenshot/MP4 → judge + ELO)

This is the "lava lamp / neon sign / snake" axis. Pipeline per task:

1. PROMPT      task spec (e.g. "Build a Lava Lamp in a single self-contained HTML file")
2. SOLVE       Solver returns one HTML string (no external assets allowed)
3. STATIC GATE  parse HTML; must be self-contained, no network <script src> to CDN beyond allowlist;
                size cap; basic validity (the "Completeness" pre-filter)
4. RENDER      headless Chrome via CDP (reuse Tier-3/browse substrate):
                 - load the HTML in a fixed viewport (e.g. 1280×720)
                 - wait for paint + a fixed settle time
                 - capture: (a) still PNG at t=0; (b) N frames over T seconds → MP4
                 - capture console errors + unhandled exceptions (interactivity/robustness signal)
5. INTERACT    for games (Snake/Breakout/Flappy/Invaders): scripted input events via CDP
                (arrow keys / clicks), assert state changes (score increments, element moves)
6. SCORE       three sub-scores → 0–100:
                 - Completeness  : did it produce the requested scene/widget? (static gate + element checks)
                 - Visual quality: vision-LLM judge on the PNG/MP4 against a per-task rubric
                 - Interactivity : console-clean + scripted-interaction assertions pass
7. ARENA (opt) pairwise ELO: show judge two models' MP4s for the same task → winner;
                accumulate Elo across the 12 tasks (BridgeBench-style "visual comparison")
8. ATTEST      screenshots + MP4 hashes go into the AttestationSurface → cert bundle

Design decisions:

• Reuse, don't rebuild, the browser. The streaming/Tier-3 stack already runs headless Chrome via CDP. Add a thin cdp_render(html, opts) -> {png, mp4, console_log} helper in lab-harness; do not add a new chromedriver dependency.
• Vision-judge cost control. The judge is the expensive part. Default to a rubric-anchored single-judge call per artifact (cheap), reserve pairwise-ELO panels for explicit "arena" runs (gated). Cache judge verdicts by (task, html-hash) so re-runs are free.
• Determinism. Fixed viewport, fixed settle time, fixed RNG seed injected into the page where possible, fixed frame count → reproducible MP4 hashes for the cert.
• Anti-judge-gaming. Keep the rubric per-task and concrete ("is there a visible lamp blob that rises and falls?") rather than "is it beautiful"; record the judge model + prompt version in the cert.

The 12 tasks ship as a version-pinned ui-bench-v1.jsonl; each row = {slug, prompt, rubric, interaction_script?}.

6. Scoring & reproducibility

We adopt BridgeBench's reproducibility discipline and exceed it with cryptographic attestation.

Scoring model (per BridgeBench, kept compatible): each task 0–100 = weighted(Correctness, Completeness, Quality). Per-axis = weighted mean over tasks. We also emit the cross-cutting Speed (tok/s, TTFT) and Cost (USD-per-correct) for every axis, from the existing counters — so a single run yields a quality and an efficiency number, like BridgeBench's Speed/Cost pages.

Surface — one unified `bench` site (NAO decision 2026-06-09). Not per-bench subdomains: a single internal bench surface (e.g. bench.blal.pro, internal-auth) hosting all axes — the exact BridgeBench model (one site → category pages). Structure:

• Landing = overview table of every axis: Alice's score + cost-per-correct + speed (like the BridgeBench home).
• Per-bench pages: SWE-bench · Terminal · GAIA · UI-bench (gallery) · Honesty/BS · Reasoning · …
• UI-bench page = the live gallery: the actual generated HTML (openable + animating) + still screenshots + (later) MP4 model-vs-model comparisons.
• Every result row links its ed25519 cert fingerprint — the anti-cheat differentiator BridgeBench lacks.
• Reads from the bench-harness RunReport JSON + rendered artifacts; backed by the multi-axis bench_leaderboard (extends swebench_leaderboard, md + JSON). Internal-first; public later.

7. Architecture wiring

                ┌──────────────────────────────────────────────┐
                │            bench-harness::TinyRunner           │  (complete slice 2)
                │   portfolio of BenchAdapter, tier-sliced       │
                └───────────────┬───────────────┬───────────────┘
        ┌───────────────────────┘               └────────────────────────┐
        ▼                                                                 ▼
  BenchAdapter (per bench)                                       RunContext
  load(tier) → Vec<BenchInstance>                          cache · budget · cost · integrity
  run_instance(inst, ctx):
     ├─ Solver  (lab-harness): Alice-HTTP in incubator | raw-model | aider | claude-code
     ├─ Scorer  (lab-harness): TestsPass | SideEffect | ExactMatch | VisionJudge(new) | EloArena(new)
     └─ emit InstanceResult {score, sub-scores, tokens, cost_usd, artifacts[]}
        ▼
  lab-incubator (Docker, no-internet, cap_drop ALL, bind-mount)  — for axes 1/2/3 real execution
        ▼
  bench-integrity: AttestationSurface(dataset SHA, harness SHA, container digests, artifact hashes)
        → ed25519 sign → bundle.tar.zstd → offline verify
        ▼
  bench_leaderboard (md + json) → unified internal `bench` site (landing + per-bench pages + UI gallery)

New code is small and additive:

• bench-harness/runners/{swe_bench,humaneval_mini,terminal_bench}.rs — fill the stubs.
• bench-harness/runners/{gaia,ui_bench,bridgecore_*}.rs — new adapters.
• lab-harness/scorer: add VisionJudgeScorer + EloArenaScorer + cdp_render helper.
• bench-harness: finish TinyRunner::run + report::render_markdown.
• bench_leaderboard binary (extend swebench_leaderboard).

8. Economical build plan (cheap-first, gated)

Ordered so that each step is cheap to build and validated on 1–3 samples before any full run. Full/expensive runs (SWE-bench Full, 12-task ELO panels, BS-100) are gated on NAO's explicit OK.

Phase A — Pipeline spine (no/low LLM cost).

1. Finish TinyRunner::run + report::render_markdown (pure code; unit-tested).
2. Complete humaneval_mini (slice 3) — cheapest real bench; re-confirm the existing 11/11 to prove the spine end-to-end.
3. Wire bench-integrity as the integrity hook in RunContext so even the Phase-A run produces a signed cert. (Sub-agent-buildable; verify vs cargo.)

Phase B — Code-correctness real execution.

1. Complete swe_bench (slice 4) against SWE-bench-Lite Tiny (3) in the incubator; micro-validate. Then gate Lite-Small/Full on NAO.
2. Complete terminal_bench (slice 4) Tiny (3); micro-validate.
3. Author BridgeBench-core six task sets (start with Debugging + Generation — highest signal); 2–3 samples each.

Phase C — Visual axis.

1. cdp_render helper + VisionJudgeScorer; validate on 1 task (Snake) end-to-end (render→MP4→judge→score) before authoring all 12.
2. Author ui-bench-v1.jsonl (12 tasks); single-judge scoring. Gate the pairwise-ELO arena (12× pairwise judge calls) on NAO.

Phase D — Honesty/agentic.

1. GAIA (gaia_mini) + BS-pushback (validate on 3 nonsense tasks, cross-check rank-evaluator). Gate BS-100 + Hallucination-30 on NAO.

Phase E — Surface (the unified `bench` site).

1. bench_leaderboard multi-axis (md + JSON) → one internal bench site (landing table + per-bench pages + UI-bench live gallery + cert-fingerprint links). Internal-first; standing up the service = a deploy (NAO's explicit OK).

Each phase: build via Sonnet/Codex sub-agents per the delegate pattern, re-gate against cargo (build + target tests + golden 2/2), commit atomically. Disk-guard every build (#499).

Build status (2026-06-09)

• Phase A — DONE (the runnable + attested spine): TinyRunner::run + report::render_markdown (commit 9c223e9); CertIntegrityHook bridging the runner → ed25519 cert (b6cbc43). Steps 2 (humaneval real solve) deferred — needs a gated run.
• Phase C visual — slices 1+2 DONE: cdp_render + check_self_contained + ui-bench-v1.jsonl (12 tasks) + VisionJudgeScorer (single-judge, judge behind a trait) — 2b62ecb; ui_bench BenchAdapter (dataset→solver→render→judge→scored, mock-tested, real-chromium e2e proven) — 85343cd. Render path validated FREE (real PNGs). MP4/ELO = the later "occasional" mode.
• Pending a gated run: the first real solve+judge (Alice generates the 12 UIs → render → real vision-judge), SWE-bench/Terminal/GAIA real execution, the bench site deploy.

9. Cost model & gates

• Free: this doc; all Phase-A code; humaneval_mini re-confirm (tiny, already-paid baseline); golden/unit tests.
• Cheap (micro-validation, 1–3 samples): each new adapter's first run — single-digit cents to low dollars depending on model.
• Gated (NAO OK required): SWE-bench Lite-Small/Full, Terminal-Bench Small/Full, the 12-task ELO arena, BS-100, Hallucination-30, any full leaderboard sweep across multiple models.
• Cap discipline: BudgetGuard daily limit enforced by RunContext; cargo capped -j 4; per-feedback memory.
• Provider: Codex-OAuth (shared quota — check ~/.codex/sessions/ before big runs) and/or OpenRouter for cost-truth (mirrors BridgeBench's single-billing-source approach).

10. "What did we forget for topping the benches?" (NAO's question)

Gaps to be genuinely top-tier, beyond the obvious benches:

1. Contamination resistance — without it, high scores are suspect. Adopt LiveCodeBench's time-windowing + record training-cutoff in the cert; prefer held-out/freshly-authored tasks for our own sets.
2. Pass@k + variance, not pass@1 — report mean ± stdev over N seeds (stochasticity is real; we saw it in speaker probes). One number hides flakiness.
3. Hidden vs visible test separation — BridgeBench leans on this (Debugging/Algorithms). Our scorers must keep gold/hidden tests out of the solver's view (the incubator no-internet + allowlist already helps; make it explicit per task).
4. Cost-per-correct as a first-class metric, not an afterthought — already feasible via cost_usd. A model that's 2% better at 10× cost should rank lower on the efficiency frontier.
5. Energy/locality (BridgeBench's DGX angle) — we have no GPU; we can't do energy-per-token for local models, but we can report wall-clock + cost and clearly scope that we benchmark Alice-the-agent, not raw model throughput.
6. Agentic difficulty ladder — single-turn (HumanEval) → multi-file (Multi-File) → repo-level (SWE-bench) → long-horizon tool-use (GAIA/Terminal). Report where Alice falls off; that's the roadmap signal for #194.
7. Judge robustness — vision/text LLM-judges drift. Pin judge model+prompt version in the cert; periodically spot-check judge vs human; prefer execution-verified scoring wherever possible (Hallucination does this).
8. Refactoring's "structural intent" — needs AST tooling, not just test parity. We have LSP/tree-sitter surfaces to lean on.
9. Reproducible "two-file provider" — keep the provider abstraction so a new model is trivial to add; mirrors BridgeBench's submission story and enables community/B2B (#196).
10. Negative controls — include a few intentionally-unsolvable or trick tasks to detect models that bluff (ties into BS-pushback). A benchmark that never catches overconfidence isn't measuring honesty.

11. Decisions (NAO, 2026-06-09)

1. Surface — RESOLVED: one unified internal `bench` site hosting all benches (not per-bench subdomains). See §6/§8 Phase E.
2. Public vs internal — RESOLVED: internal now, public later (let the numbers get solid first).
3. Comparison set — RESOLVED: Alice only (self-measurement) first; gpt-5.x + open-weights added once the rig is settled (each extra model multiplies run cost).
4. ELO arena / MP4 — RESOLVED: single-judge first (rubric→screenshot, cheap) is the primary path; build the MP4/ELO "occasional" mode too but invoke it sparingly (paid video-judging only on explicit arena runs).
5. Task authoring vs porting — port public sets (SWE-bench/HumanEval+/Terminal-Bench/GAIA) first, author BridgeBench-core + UI sets ourselves. (Default; adjust on request.)
6. WebArena/OSWorld — out of the first cut (heaviest, lowest marginal credibility per dollar). (Default; adjust on request.)

No expensive runs executed for this doc or the Phase-A / visual-slice builds — all validated with mocks + free chromium renders. Next gated step on NAO's OK: the first real solve+judge (Alice → 12 UIs → render → 1 real vision-judge) and/or standing up the internal `bench` site.