openalice-canary
**Behavioral E2E canary for the OpenAlice product surface.** Tests *real behavior* against the live stack and asserts **outcomes, not status codes** — because `compiles` + `returns 200` ≠ `works` (see `openalice-agent-lite/docs/BUG_CATEGORIES.md`).
Architecture rating
README present with clear purpose statement referencing BUG_CATEGORIES.md; manifest valid and descriptive (100/100); but 8 missing docstrings across only 3 files, no inline architecture notes, no @feature annotations (features=20/100), no env var documentation despite 9 undeclared vars
Manifest present and valid (100/100 in Atlas); follows shared eval contract for result JSON; but 17 missing type hints, 8 missing docstrings, 1 god module, and 9 undeclared env vars show poor adoption of project conventions; Atlas drift score 10/100 confirms significant divergence from substrate norms
5 bare/broad excepts suppress real failures; no evidence of graceful shutdown, timeouts, retries, or idempotency; max cyclomatic complexity 13 in a god module suggests monolithic error-prone flow; FLEET_LOOP_THRESHOLD/FLEET_RESTART_THRESHOLD env vars hint at restart logic but undeclared and unvalidated
9 undeclared env vars (CANARY_ORIGIN, CANARY_PROMPT, etc.) with no .env schema or secret-handling layer; no authn/authz on the canary itself; 5 bare/broad excepts risk swallowing injection or SSRF errors silently; no evidence of input sanitization on CANARY_PROMPT before use
5 console.log statements are the only output mechanism; no structured logging, no metrics export, no tracing, no health probe (Atlas confirms 'no https endpoint in manifest exposes', up=false); canary result JSON is the sole observable artifact
No database, no migrations, no schema; writes a flat JSON file to /home/blal/shared/eval/canary.json with no FK constraints, no tenant isolation, no GDPR controls; eval_history.py likely appends without rotation beyond EVAL_GZIP_AGE_DAYS env var
Zero test files detected in repo structure; the canary IS a test harness for other services but has no tests for its own logic (fleet_scan, eval_history, prompt handling); complexity-13 god module and 3 source files with no self-test coverage
source · auto-grader-2026-06-27
Ecosystem manifest
.atlas-deps.ymlBehavioral E2E canary — tests real product behavior against the live stack (outcomes, not 200s) and writes a result-JSON per the shared eval contract for Atlas to ingest. Shared eval-infra; Embed pack first.
- no purpose stated
- no purpose stated
- no purpose stated
- file: /home/blal/shared/eval/canary.json
Atlas grepped the repo's source for env reads (rust env::var · ts process.env · py os.environ · shell $VAR) and diffed against the manifest's consumes_env. Two lists below: stale declarations and undeclared reads.
- CANARY_EVAL_DIR
- CANARY_NO_LLM
- CANARY_ORIGIN
- CANARY_PROMPT
- CANARY_VERSION
- CANARY_WIDGET
- EVAL_GZIP_AGE_DAYS
- FLEET_LOOP_THRESHOLD
- FLEET_RESTART_THRESHOLD
Code composition
tokei · loc- Python100.0%
- Python100.0%
- Python100.0%· 541
| Language | % code | code | comments | blanks | files |
|---|---|---|---|---|---|
| Python | 100.0% | 541 | 37 | 86 | 3 |
| Markdown | 0.0% | 0 | 69 | 21 | 1 |
Telemetry ribbon · last 90 days
5 eventsNo @feature annotations recorded in this repo yet.