Agent evaluation & observability

For NAO + anyone who has built an LLM agent and then asked "…but is it actually any good, and how would I even know?" This is the honest version. Evaluating agents is the hardest, least-solved problem in the whole stack — harder than building them. The build is a weekend; knowing whether the build regressed last Tuesday is an open research area. This article maps the four tools the field actually uses today — trajectory evaluation, OpenTelemetry traces, tool-call correctness scoring, and live-site benchmarks — and is blunt about where each one breaks.

What it is (intuition first)

A normal program is deterministic: same input → same output, and a unit test either passes or fails. An LLM agent is not that. It is a loop — perceive, think, call a tool, read the result, think again, call another tool, eventually answer — where every step is a non-deterministic sample from a model, the environment talks back, and there is rarely one "correct" path. Run the same task twice and you get two different trajectories (the ordered sequence of thoughts, tool calls, tool results, and messages).

So "is the agent good?" splits into questions that have nothing to do with each other:

1. Did it get the right final answer? (outcome / task success)
2. Did it get there sanely — right tools, right arguments, right order, no pointless detours, no dangerous actions? (trajectory quality)
3. Does it do this *reliably* — or did it succeed once and fail the next four times? (consistency)
4. When it failed, *where* in the loop did it go wrong — bad plan, wrong tool, malformed arguments, tool errored, model misread the result, infinite loop? (observability / diagnosis)
5. What did it cost — tokens, latency, dollars, number of steps?

A single accuracy number answers only (1). Everything that makes agents useful in production — that they don't randomly delete a row, don't loop forever, don't burn $3 per query — lives in (2)–(5). Building tooling for (2)–(5) is the subject of this article.

Two complementary disciplines:

• Evaluation = offline, batch. Run the agent over a fixed task suite, score it, compare versions. "Did v2 regress vs v1?" Think benchmark / test suite.
• Observability = online, per-run. Instrument the live agent so that every real trajectory emits a structured trace (a tree of timed spans) you can inspect, search, alert on, and later replay as eval data. Think logs-but-structured-for-loops.

Mature teams need both, and increasingly they share a substrate: the same trace format you use to debug a production incident is the input to your trajectory evaluator. (For the broader memory/retrieval substrate an agent runs on, see [[agent-memory-systems]]; for what "an agent doing research autonomously" looks like end-to-end, [[autoresearch]].)

Why it matters

• Agents fail silently and confidently. A wrong arithmetic answer is obvious; an agent that looked like it booked the right flight but used the wrong passenger ID is not. The failure is buried mid-trajectory. Without trace-level visibility you find out from the customer.
• The reliability gap is enormous and under-reported. Headline accuracy hides it. On Sierra's τ-bench, GPT-4o-class function-calling agents solve under 50% of retail tasks on a single try — and `pass^8` falls below 25%, meaning fewer than 1-in-4 tasks succeed across all eight independent attempts. A demo that "works" is routinely a coin-flip in production.
• Outcome ≠ process. An agent can reach the right answer through a wrong, unsafe, or absurdly expensive path (10 tool calls where 2 suffice; reading a customer's full PII to answer a yes/no). Outcome-only eval rewards luck and hides cost, latency, and safety regressions.
• Regression detection is the daily job. Once an agent is live, the real question is never "is it smart" but "did this prompt tweak / model swap / new tool quietly break 6% of tasks?" That requires repeatable eval + diffable traces, not vibes.
• It's the bottleneck on improvement. You cannot RL-tune, prompt-optimize, or even prioritize fixes for what you cannot measure. Good eval/observability is the gradient signal for the whole agent program.

How it works (real mechanics)

1. Trajectory evaluation

A trajectory is the full ordered transcript of a run. Trajectory evaluation scores the path, not just the destination. Concretely, given a reference (golden) trajectory or a set of acceptance predicates, you decompose the score into per-step checks. The clearest recent formalization is TRAJECT-Bench (2025), which synthesizes executable tool-use tasks varying in two orthogonal dimensions and reports trajectory-level diagnostics alongside final accuracy:

• Breadth — parallel tool calls a single step can/should fan out.
• Depth — interdependent chains, where tool N's output feeds tool N+1's arguments. (Their finding: the hard wall is the short→mid-length transition; performance degrades non-linearly with depth.)

Its three diagnostic axes — adopt these names, they're becoming standard:

TRAJECT-Bench named two failure modes worth memorizing because every agent exhibits them:

• Similar-tool confusion — two tools with overlapping descriptions (get_user vs get_user_profile) and the model picks the wrong one.
• Parameter-blind selection — the model picks the right tool but fills arguments wrongly (hallucinated IDs, wrong units, defaults left blank).

A simple, framework-agnostic trajectory scorer:

# Score one trajectory against a reference. Outcome AND process.
def score_trajectory(traj, ref):
    called = [s.tool for s in traj.tool_calls]
    expected = [s.tool for s in ref.tool_calls]

    tool_sel = jaccard(set(called), set(expected))          # selection
    arg_ok = mean(arg_match(c.args, e.args)                  # arguments
                  for c, e in align(traj.tool_calls, ref.tool_calls))
    order_ok = dependency_order_satisfied(traj, ref.dag)     # dependencies
    outcome = check_goal_state(traj.final_env, ref.goal)     # did it work

    # weighted blend — weights are a product decision, not a law
    return dict(tool_selection=tool_sel, args=arg_ok,
                order=order_ok, outcome=outcome)

Three ways to source the reference predicate, in increasing realism / cost:

1. Golden trajectory — a human-authored canonical path. Brittle: punishes valid alternative paths, so use it for set/DAG checks, not exact match.
2. State-based goal check — ignore the path; assert the final environment state. This is τ-bench's trick (below) and is far more robust than path-matching because it admits any correct route.
3. LLM-as-judge over the trajectory — a model reads the transcript and rates it. Flexible, scales to fuzzy criteria, and the least trustworthy (see §4).

2. State-based outcome eval + the reliability metric (τ-bench / τ²-bench)

τ-bench (Sierra Research, 2024) is the cleanest design for interactive agents. Setup: a simulated user (an LLM following a hidden persona/intent) talks to your agent, which has domain API tools (retail / airline) and a policy document of rules it must obey. The genius is the scoring: it does not grade the conversation. It runs the conversation, then compares the database's final state to an annotated goal state. State match → pass. This sidesteps path-matching and judge-noise entirely for anything that mutates a backend.

Its second contribution is the metric that exposed the reliability crisis, `pass^k` (read "pass-hat-k"): the probability that an agent solves a task on all k independent attempts. For a task solved on c of n trials, the unbiased estimator is the same combinatorial form as Codeforces/HumanEval's pass@k but with the logic inverted (all-must-pass, not any):

pass^k(task) = C(c, k) / C(n, k)        # prob. all k sampled trials succeed
pass^k       = mean over tasks

pass@k (≥1 of k succeeds) measures best-case potential; pass^k (k of k succeed) measures worst-case dependability — exactly what production cares about. The gap between a model's pass^1 and pass^8 is its reliability debt. τ²-bench extends this to dual-control (both user and agent can call tools — e.g. telecom troubleshooting where the user must toggle settings on their phone) and adds a telecom domain, because single-actor worlds were too easy.

3. Tool-call correctness (BFCL)

When you only care about function calling — does the model emit the right call with the right args — the reference benchmark is the Berkeley Function-Calling Leaderboard (BFCL) (ICML 2025). 2,000+ question-function-answer items. Two scoring engines:

• AST checking — parse the model's emitted call into an Abstract Syntax Tree and structurally match function name + argument names/types/values against the reference. Scales to thousands of functions without executing anything (cheap, deterministic, no live side-effects). It covers simple, parallel (one turn → many calls), and multiple (pick the right one of several candidate functions) cases across Python/Java/JS/REST.
• Executable checking — actually run the produced call against real/mock APIs and assert the return. Catches things AST can't (a structurally-valid call that errors at runtime).

BFCL's evolution mirrors the field: V1 single-turn → V2 "live" / real user-contributed functions (less overfit, more API noise) → V3 multi-turn & agentic (state carried across turns). The lesson baked into BFCL: AST match is a cheap proxy; execution is ground truth; you want both, because each catches what the other misses.

4. LLM-as-a-judge (the glue — and the weak link)

For anything you can't pin to a database state or an AST — "was this answer helpful / grounded / on-policy?" — the field reaches for a model as the grader. Two paradigms (per the Survey on LLM-as-a-Judge, 2024):

• Pointwise scoring — judge rates one response on a scale (1–5, 0–1).
• Pairwise comparison — judge picks the better of A vs B. More reliable: the survey reports pairwise has higher positional consistency and better human alignment than absolute scoring.

It is indispensable and systematically biased. The documented, reproducible biases:

• Position bias — prefers whichever candidate is shown first (or last). A dedicated study (Shi et al., 2024) shows it's large and model-family-wide. Fix: run both orders (A,B) and (B,A); keep only verdicts that agree on swap.
• Verbosity / length bias — longer answers score higher regardless of quality. Fix: length-control, or instruct + penalize.
• Self-preference / self-enhancement — a judge rates outputs in its own style/model higher. Fix: judge with a different model family; or a jury / panel of diverse judges and aggregate (reduces any single model's bias).
• Other mitigations: reference-guided grading (give the judge the gold answer), chain-of-thought rationales before the verdict, calibration, fine-tuned specialist judge models.

Treat the judge as a noisy instrument you must calibrate against human labels, not an oracle. Always report judge-human agreement on a held-out set before you trust a judge-driven number. (This is the same council/panel idea as [[fusion-and-llm-councils]] and [[mixture-of-agents]] — diversity beats a single opinionated grader.)

5. Observability: OpenTelemetry GenAI semantic conventions

Eval needs data; observability produces it. The field standardized — fast — on OpenTelemetry (OTel) GenAI semantic conventions, so every framework emits the same shape of trace and any backend (Grafana/Tempo, Datadog, Phoenix, MLflow, Langfuse) can read it. A trace is a tree of spans (timed, attributed operations). The GenAI conventions define the span operations and attributes:

Operation names (gen_ai.operation.name):

• `chat` / text_completion / generate_content — a single model call.
• `invoke_agent` — one agent invocation. Span name "invoke_agent {agent.name}". Span kind = INTERNAL for in-process, CLIENT for a remote agent (split formalized in spec v1.41).
• `create_agent` — agent construction ("create_agent {agent.name}", CLIENT).
• `execute_tool` — one tool/function invocation.
• `invoke_workflow`, `plan` — predefined-workflow and planning steps.

The resulting span tree for a multi-step agent run (this is the whole point — the loop becomes a readable, timed tree):

invoke_agent support-router            (INTERNAL)
├── chat gpt-4o                        (CLIENT)   ← turn 1: model decides to call a tool
├── execute_tool query-orders          (CLIENT)   ← tool call
│   └── tools/call query-orders        (SERVER)   ← MCP server side, linked via W3C Trace Context
└── chat gpt-4o                        (CLIENT)   ← turn 2: model reads result, answers

Key attributes you get for free on every span: gen_ai.request.model, gen_ai.usage.input_tokens / gen_ai.usage.output_tokens (cost!), gen_ai.response.finish_reasons (stop vs tool_calls), gen_ai.agent.name / gen_ai.agent.id, gen_ai.tool.name / gen_ai.tool.call.id, error.type. MCP tool calls add mcp.method.name, mcp.session.id and propagate trace context across the agent↔server boundary — so a tool failure inside an MCP server links back to the exact agent step that called it.

Content-capture policy — the part people get wrong. By default, no prompt content or tool arguments are recorded (security/PII). Three modes:

1. Not recorded (default).
2. On the span — full messages as gen_ai.input.messages / gen_ai.output.messages, gated behind an opt-in env flag (OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true).
3. External reference — content in S3/DB, span holds only a URL (keeps traces cheap; common at scale).

The payoff loop: captured trajectories become eval fixtures. You sample real production traces, (auto-)label them, and feed them back into the offline suite — closing the loop between observability and evaluation. This is the substrate behind tools like [[graphify]]/[[graphrag]]-style trace stores.

6. Live-site benchmarks (the realism frontier)

The hardest setting: agents acting on real, changing websites. The tension is realism vs reproducibility, and you cannot fully have both.

• WebArena (2023): a self-hosted, reproducible clone of real site categories (shopping, GitLab, Reddit, CMS) — 812 long-horizon tasks from 241 templates, scored by functional state checks. Reproducible, but frozen — agents can overfit, and it ages.
• WebVoyager and successors run on the live public web — maximal realism, but ground truth rots: "how many recent papers mention X" changes daily, so evaluation leans on LLM-as-a-judge over screenshots (inheriting all of §4's biases) and results are hard to reproduce across days.
• BrowserGym (2024) is the unifier: one Gym-style observation/action interface (DOM + accessibility tree + screenshot; click/type/scroll actions) wrapping WebArena, WebVoyager, MiniWoB++, WorkArena, VisualWebArena, etc., so one agent harness runs against all of them comparably. It standardizes the plumbing; it cannot standardize away the underlying realism/reproducibility tradeoff.

Live-site success rates remain low and noisy — concrete proof that outcome accuracy alone is a weak, unstable signal and that you need trajectory + trace + reliability metrics underneath it.

Key ideas & tradeoffs

• Outcome vs process. Outcome (state-match) is robust and cheap but blind to how; process (trajectory) catches unsafe/expensive paths but is brittle to valid alternatives. Use state-match for the verdict, trajectory diagnostics for the autopsy.
• Path-match vs state-match vs judge. Exact path-match over-penalizes good alternative routes. State-match (τ-bench) is the gold standard when the task mutates checkable state. LLM-judge fills the rest but adds bias + cost. Pick by how checkable your goal is.
• `pass@k` vs `pass^k`. pass@k flatters (best-of-k); pass^k is honest (all-of-k). Report pass^k for anything customer-facing. The spread between them is your reliability story.
• AST (proxy) vs execution (ground truth). AST is fast/deterministic/safe but misses runtime errors; execution is true but slow, stateful, and has side-effects. BFCL ships both for a reason.
• Reproducibility vs realism. Sandbox (WebArena) = repeatable + overfittable; live web (WebVoyager) = real + unreproducible. There is no free lunch; pick per decision and disclose which you ran.
• Single judge vs jury. One judge is cheap and biased; a diverse panel is costlier and fairer. Mitigate position bias with order-swap always — it's nearly free.
• Standards as a moat-remover. OTel GenAI conventions mean you're not locked to one vendor's trace format — instrument once, swap backends, and reuse traces as eval data. Adopt them rather than rolling your own span schema.

Honest caveats & open questions

• There is no agreed metric. Unlike accuracy for classifiers, agent eval has no canonical number. Every benchmark invents its own scoring; cross-paper comparisons are mostly apples-to-oranges. Be suspicious of any single headline figure.
• LLM-judges are a load-bearing crutch. A huge fraction of agent "scores" — especially on live web — ultimately rest on an LLM grading another LLM, with documented, unfixed position/verbosity/self-preference bias. Some 2025 work (e.g. "Neither Valid nor Reliable?") questions whether judge scores are valid measurements at all. Always anchor to human labels on a sample.
• Benchmark rot & contamination. Public suites leak into training data; live suites' answers change under you. A frozen leaderboard number can be simultaneously stale, gamed, and contaminated.
• Reliability collapses under composition. Per-step accuracy compounds: a 90%-per-step agent over a 10-step task is ~0.9¹⁰ ≈ 35% end-to-end if errors are independent. pass^k exposes this; most demos hide it. Long-horizon dependability is unsolved.
• Cost of eval is real. State-based and executable evals need a runnable environment (DB, mock APIs, browser); judge-based evals burn tokens at scale. Good eval can cost more compute than the agent it grades.
• Trace ≠ understanding. OTel gives you a beautiful span tree; it does not tell you why the model chose the wrong tool. Root-causing semantic failures from traces is still manual, expert work.
• Open problems: automatic, cheap, unbiased trajectory scoring; reliable live-web ground truth; eval that's robust to contamination; standardized cross-benchmark reliability reporting; and judging multi-agent trajectories (whose error was it?).

How it connects to OpenAlice

Alice is a long-running, tool-using, multi-connector agent (Telegram, browser, pty, code→PR) — which means everything above is operational, not academic, for us:

• Alice already emits a trajectory per turn. The agentic loop produces exactly the perceive→think→tool→observe sequence this article scores. Aligning Alice's internal step records to OTel GenAI span conventions (invoke_agent / chat / execute_tool, with gen_ai.tool.call.id, token usage, finish reasons) would let any standard backend read her traces and turn live turns into eval fixtures — the §5 payoff loop. This is the natural successor to the fixture-runner architecture already in the lab.
• The AGI-coding capability needs `pass^k`, not `pass@1`. NAO's directive that autonomous repo→PR is a base Alice capability measured via SWE-bench makes the reliability metric central: report pass^k so we see the dependability gap, not a flattering best-of-N. The bench platform (bench.blal.pro) is where this lives.
• Tool-call correctness is a daily Alice failure surface. Alice has many similar connectors/tools — exactly the similar-tool confusion and parameter-blind selection modes from §1. A BFCL-style AST check over Alice's emitted tool calls is a cheap, deterministic regression gate to run before any prompt/model change reaches mvp.
• State-match eval fits Alice's connectors. For connector actions that mutate state (sending a message, opening a PR, editing memory), τ-bench-style final-state assertions are more robust than judging her prose — and dodge the judge-bias trap.
• Atlas is the trace store. Atlas already indexes the ecosystem; a trajectory zone (per-turn span trees, searchable, diffable across versions) is the concrete observability backend this article argues every agent program needs. It would feed [[graphify]]/[[graphrag]]-style queries over what Alice actually did.
• Judge discipline applies to our critics. Wherever Alice uses an LLM critic (the critic-400 fail-open path, prompt-evolution scoring), §4's rules hold: order-swap for pairwise, prefer a different model family as judge, and validate against human labels — same council logic as [[fusion-and-llm-councils]] / [[mixture-of-agents]].

Bottom line: for OpenAlice the move is outcome via state-match, process via OTel trajectory diagnostics, reliability via `pass^k`, tool-calls via AST gate, and the judge treated as a calibrated-but-suspect instrument — and to make Alice's live traces first-class eval data, not throwaway logs.