Agent observability & tracing

What it is (intuition first)

A single LLM call is a black box you can almost reason about: one prompt in, one completion out, a token count on the receipt. An agent is a black box wrapped in a loop wrapped in a tree. It calls a model, reads the model's tool requests, runs tools, feeds results back, calls the model again, sometimes spawns a sub-agent, and only after N of these rounds produces something a user sees. When that goes wrong — it loops twice, calls the wrong tool, hallucinates a billing policy, burns $4 of tokens on a question that needed one call — a normal application log shows you a 200 OK and a final string. It tells you nothing about why.

Agent observability is the discipline of making that hidden execution graph visible: recording every model call, tool call, retrieval, and sub-agent handoff as a structured event, linked by parent/child relationships, so you can reconstruct exactly what the agent did and why. The unit of recording is the span — a timed, attributed record of one operation — and a tree of spans sharing a request ID is a trace. This is the same span/trace model that distributed tracing has used for microservices for a decade; the new part is a schema for the AI-specific bits: which model, how many tokens, which tool with which arguments, what the model decided.

The cleanest framing of why this needs its own name comes from the practitioner literature (Braintrust, 2026): LLM observability tracks a single model call; agent observability tracks the full workflow — multiple LLM calls, tool use, memory access, sub-agent handoffs, and decision branches. Traditional application-performance monitoring sees latency and errors but is blind to the semantic failures that define agent bugs — a clean 200 that did the wrong thing.

The mental model:

A span = one operation (a model call, a tool call, a retrieval). A trace = the whole tree of spans for one user request. Observability = traces + the AI-specific attributes (model, tokens, tool args, cost) + the ability to query, evaluate, and replay them.

Why it matters

It is the only record of agent reasoning. An agent's "thought process" is ephemeral — it exists in a sequence of prompts and completions that vanish unless you capture them. A trace is the only artifact that says which tool the agent chose, what it passed, and what came back. Without it, debugging is guesswork over a final string.
It is where cost actually accumulates. A single agent task can fan out into dozens of model calls across a growing prefix ([[agentic-loops]]). Token and cost accounting per span — rolled up to the trace — is how you find the one step that re-sends a 50k-token document every loop, or the retry storm that triples the bill. Cost is invisible at the request level and obvious at the span level.
It closes the loop between production and quality. The strongest pattern in the field is eval-in-loop: run evaluators (LLM-as-judge, rule assertions) over live traces, and when one fails, convert that trace into a test case. The trace is not just a debugging artifact — it is the raw material for the eval set that prevents the next regression. (See [[agent-evaluation]] for the evaluation methodology; this article is about the infrastructure that feeds it.)
It makes failures reproducible. "It worked on my machine" is fatal for agents because their behavior depends on the exact prompt, model version, tool outputs, and sampling. A trace captures all of that, so a failure can be replayed — re-run against a fixed input — instead of described.

How it works (real mechanics)

1. The span/trace model, specialized for GenAI

The de-facto standard is the OpenTelemetry GenAI semantic conventions (canonical repo: open-telemetry/semantic-conventions-genai), developed by the GenAI SIG since April 2024. It does for AI spans what HTTP semantic conventions did for web requests: fixes the names so a trace emitted by LangChain, CrewAI, or your own code carries the same attribute keys and any backend can read it.

The core idea is a small set of operation types carried in gen_ai.operation.name:

operation	what it spans
`chat`	one model invocation (the LLM call itself)
`invoke_agent`	one agent's full turn — parent of the calls it makes
`execute_tool`	one tool/function execution
`embeddings`	an embedding request (e.g. for retrieval)

These compose into a tree. Per the OTel agent-span spec, a typical agent trace is a top-level `invoke_agent` span with child `chat` spans for each model call and `execute_tool` spans for each tool the model requested:

invoke_agent "research-assistant"          (the agent turn)
├── chat        gpt-5.5                     (model decides: call search)
│     finish_reasons = ["tool_calls"]
├── execute_tool web_search                 (the tool runs)
│     gen_ai.tool.name = "web_search"
├── chat        gpt-5.5                     (model reads results, answers)
│     finish_reasons = ["stop"]
└── ...

2. The attributes that carry the signal

A span name tells you what; the attributes tell you the rest. The GenAI conventions standardize keys including:

Model identity — gen_ai.request.model (what you asked for), gen_ai.response.model (what served it). Version drift hides here.
Token accounting — gen_ai.usage.input_tokens and gen_ai.usage.output_tokens. These are the raw inputs to cost: multiply by the per-token price for that model and you have per-span spend, which rolls up to a trace-level total.
Why it stopped — gen_ai.response.finish_reasons ("stop", "tool_calls", "length"). A "length" truncation or an unexpected "tool_calls" is often the root of a failure.
Agent / tool identity — gen_ai.agent.name, gen_ai.tool.name — so multi-agent handoffs and per-tool latency are queryable.

Content capture is off by default. Per the OTel 2026 guidance, no prompt content or tool arguments are captured unless an operator explicitly enables it (e.g. an otel.captureContent / instrumentation flag), at which point attributes like gen_ai.system_instructions, gen_ai.input.messages, and gen_ai.output.messages carry the full conversation. This is a deliberate privacy-by-default stance — prompts and tool arguments routinely contain PII and secrets — and it is a trap for the newcomer: you turn on tracing, see a beautiful span tree, and the one thing you wanted (the actual prompt) is blank until you opt in. Flag this both ways: off-by-default protects users, but capturing full prompt/response content (when you do enable it) turns your trace store into a secrets store that needs the same redaction and access controls as the data itself.

3. Token & cost accounting

Cost is not a provider feature you read off an invoice after the fact — it is something you reconstruct from spans. Each chat span carries token counts; you attach a price table (per-model, per-token, distinguishing input / output / cached input — see [[semantic-caching-for-llms]]) and compute per-span estimated cost, then trace-level rollups (Braintrust frames it exactly this way). This is what surfaces the expensive step: the loop that re-sends the whole transcript, the sub-agent that re-runs the same retrieval, the retry that doubles a call. Caveat worth flagging: cost is an estimate — cached-token discounts, prompt caching, and provider-side billing nuances mean span-derived cost can diverge from the actual invoice; treat it as a high-signal relative metric, not an exact ledger.

4. Eval-in-loop (online evaluation)

Tracing tells you what happened; evaluation tells you whether it was good. Online evaluation runs evaluators over a sampled subset of live production traces — LLM-as-judge checks, custom scorers, rule-based assertions — to catch quality regressions and drift as they happen rather than in a weekly review (LangSmith, Braintrust both center this). The loop, as the LangSmith "agent reliability loop" puts it: trace every run → turn real failures into datasets → run repeatable experiments with evaluators → promote only the changes that win. The trace store and the eval set become the same substrate. (The mechanics of how to score a trajectory — exact-match, judge rubrics, tool-call correctness — belong to [[agent-evaluation]] and [[llm-evaluation]]; here the point is that the trace is the thing you evaluate.)

5. Replay & failure triage

Because a trace captures the exact inputs (prompt, model, tool outputs, params), a failed run is reproducible: you can re-feed that input to the agent and watch it fail again deterministically enough to debug — instead of staring at a final string. The triage workflow that recurs across platforms:

Surface — an online evaluator (or a user report, or an error span) flags a bad trace.
Inspect — open the span tree; find the step where it went wrong (wrong tool chosen? bad arguments? a "length" truncation? a tool that returned garbage?).
Capture — convert that trace into an eval case (the "production failure → dataset in one click" pattern).
Fix & replay — change the prompt/model/tool, re-run the captured case, and confirm the fix without shipping to production.

This is the difference between debugging an agent and describing an agent problem in a bug report.

Key ideas & tradeoffs

Span model is borrowed; the schema is new. The trace/span machinery is classic distributed tracing — the OTel contribution is the GenAI semantic conventions (standard attribute names) so traces are portable across frameworks and backends. Adopting the standard buys you vendor-neutrality; rolling your own buys you nothing but lock-in.
Granularity vs. overhead. More spans (every reasoning step, every memory read) = richer triage but more storage and instrumentation cost. Most teams span at the operation level (model call, tool call, retrieval) and selectively add reasoning/state spans for the agents that need them.
Content capture is a privacy fork in the road. Off-by-default protects users but blinds your debugging; on means your trace store inherits the sensitivity of every prompt. There is no setting that gives you full visibility and zero exposure — you pick redaction + access control as the price of seeing prompts.
Cost is an estimate, not a ledger. Span-derived cost is excellent for relative triage ("this step is 10× the others") and unreliable as an exact bill once caching and provider discounts enter.
Observability ≠ evaluation, but they share a substrate. Tracing records; evaluation judges. They are different jobs ([[agent-evaluation]]) that operate on the same traces — which is why the two words keep appearing together and why conflating them leads to teams that trace everything and measure nothing.

Honest caveats & open questions

The standard is still experimental. As of early 2026, most GenAI semantic conventions are marked experimental — attribute names and shapes can still change, and the spec recently moved out of the main OTel semconv repo into a dedicated semantic-conventions-genai repository. Adoption is real (Datadog, Honeycomb, New Relic support them; LangChain, CrewAI, AutoGen emit them) but you are building on a moving target; pin versions and expect churn.
Vendor framings are vendor framings. The crisp "LLM vs agent observability" distinction and the "failure → dataset in one click" / "agent reliability loop" narratives come from Braintrust and LangSmith — they describe real patterns but are also product marketing. The concepts generalize; the one-click magic is platform-specific. Flagged as such.
No fabricated numbers here. Unlike caching or inference, this space has few clean, citable benchmark numbers — "richer traces find more bugs" is true and unquantified. Be suspicious of any "X% faster debugging" claim; the value is real but the literature is qualitative.
Reasoning spans are aspirational for reasoning models. Capturing "intermediate chain-of-thought" assumes the reasoning is in the trace — but provider reasoning models increasingly hide or summarize their internal tokens, so the most interesting span (the actual deliberation) may be exactly the one you can't see. Trace what crosses the API boundary; the rest is inferred.
Sampling hides tail failures. Online eval runs on a sampled subset for cost reasons; rare-but-severe failures can slip through the sample. Full-trace retention + targeted (error-triggered) eval is the partial fix, at higher cost.

How it connects to OpenAlice

OpenAlice's agent core is a hardened tool-loop ([[agentic-loops]], [[tool-use-function-calling]]) that, on every turn, calls a model, may invoke tools, and re-sends a growing conversation prefix — i.e. exactly the invoke_agent → chat → execute_tool span tree the GenAI conventions describe. Several house disciplines are agent-observability practice under other names:

"Always verify" + read `usage`. OpenAlice's house rule to confirm behavior from usage (cache reads, token counts) rather than assume is, mechanically, span-level token/cost accounting — the same gen_ai.usage.* signal, read to catch a silent prefix invalidator ([[semantic-caching-for-llms]]) or a runaway step.
The bench + lab loop is eval-in-loop. OpenAlice runs an internal UI-bench platform and a fixture-runner harness; pointing evaluators at captured agent runs and promoting only green changes is the "trace → dataset → experiment → promote" reliability loop in OpenAlice form ([[agent-evaluation]]).
MCP tool calls are `execute_tool` spans. OpenAlice speaks MCP ([[mcp]]) and A2A ([[a2a-protocols]]); each tool invocation and each cross-agent handoff is a natural span boundary, and parent/child links across handoffs are what make a multi-agent run reconstructable.
Privacy-by-default matches the house bar. Alice's content is personality- and PII-sensitive; OTel's don't capture prompt content unless opted in default aligns with OpenAlice's gov-level / GDPR posture — a trace store that captured every prompt verbatim would inherit the soul-preservation and data-handling constraints wholesale.

The throughline: observability is not a dashboard you bolt on afterward — it is the substrate that makes an agent debuggable, costable, and evaluable, and every one of those is a precondition for trusting an autonomous loop in production.

Agent observability & tracing — span/trace models for tool calls + reasoning, token/cost accounting, eval-in-loop, replay & failure triage