Tool use & function calling — the mechanics under every agent

What it is (intuition first)

A language model on its own can only do one thing: predict the next token. It cannot look up today's price of a stock, run a SQL query, send an email, or even reliably multiply two ten-digit numbers. Tool use (a.k.a. function calling) is the plumbing that lets the model reach outside itself — but with a crucial twist: the model never runs anything. It only emits a structured request saying "please call get_weather with {location: "Berlin"}," and your code runs the actual function and hands the result back.

The cleanest mental model, straight from Anthropic's docs, is a contract: you declare what functions exist and what shape their inputs take; the model decides when and how to call them; your application executes and returns a result that flows back into the conversation. As the docs put it, this "makes the model behave less like a text generator and more like a function you call" — except the caller choosing which function to invoke is itself a language model reading the conversation.

If you've ever written a regex to scrape a decision out of model output ("did it say yes or no?"), that's the tell you wanted a tool call. The structure you were trying to recover belonged in the schema all along.

Beginner ramp: think of it as the LLM equivalent of a REST client. You publish an API (the tool schema), the model writes the request body (validated JSON), you run the endpoint, you return JSON. The novelty is only who writes the request.

Why it matters

Tool use is the single highest-leverage primitive in agent design. Almost everything people mean by "AI agent" — coding agents, research agents, customer-support bots, computer-use — is a loop around tool calls. Three reasons it's load-bearing:

1. Capability. Anthropic notes that on benchmarks like SWE-bench (real software engineering) and LAB-Bench FigQA (scientific figure reading), adding even basic tools produces outsized gains, sometimes surpassing human-expert baselines. The model's reasoning is multiplied by access to ground truth.
2. Grounding & freshness. Anything outside training data — current data, your private DB, a file on disk — requires a tool. This is the execution arm of retrieval; see [[graphrag]] and [[embeddings]] for the retrieval half.
3. Structure. A tool schema is also the most reliable way to force a model to emit a specific JSON shape instead of prose. "Structured outputs" and "function calling" are the same machinery viewed from two angles.

The Tool Learning survey (arXiv:2405.17935) frames the workflow as four stages that every agent implicitly walks through: task planning → tool selection → tool calling → response generation. Function calling is the third stage made concrete, but a good schema and description quietly drive the first two as well.

How it works (real mechanics)

1. The tool definition (a JSON Schema with a name and a description)

A tool is three things: a name, a natural-language description, and an input_schema / parameters expressed in JSON Schema. Anthropic calls the field input_schema; OpenAI nests it under function.parameters. Same idea.

{
  "name": "get_weather",
  "description": "Get the current weather for a location. Use when the user asks about
                  current conditions, temperature, or whether to bring an umbrella.
                  Returns temperature and a short text summary.",
  "input_schema": {
    "type": "object",
    "properties": {
      "location": { "type": "string", "description": "City and region, e.g. 'Berlin, DE'" },
      "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "default": "celsius" }
    },
    "required": ["location"],
    "additionalProperties": false
  }
}

Two non-obvious truths here:

• The description is the most important field you will write. The model decides whether to call the tool, and how to fill its arguments, almost entirely from the name + description + per-property descriptions. Anthropic's own guidance: when the model misuses or under-uses a tool, the first fix is more detail in the description — what it does, when to use it, what each parameter means, edge cases, and what not to do. The schema constrains shape; the description conveys intent.
• The schema is tokens, and tokens cost money and context. The tools array is injected into the prompt on every request, plus a hidden tool-use system prompt (Anthropic publishes the exact counts — e.g. ~290 tokens for Opus 4.6 under tool_choice: auto). Twenty fat tool schemas can quietly eat thousands of tokens before the user even speaks. (This is the pressure behind tool search / lazy tool loading.)

2. The model emits a tool_use block

When the model decides to call a tool, the response comes back with stop_reason: "tool_use" and one or more structured blocks:

{
  "stop_reason": "tool_use",
  "content": [
    { "type": "text", "text": "I'll check the weather in San Francisco." },
    {
      "type": "tool_use",
      "id": "toolu_01A09q90qw90lq917835lq9",
      "name": "get_weather",
      "input": { "location": "San Francisco, CA", "unit": "celsius" }
    }
  ]
}

The id is the join key — you'll need it to return the matching result. (OpenAI's wire format differs cosmetically: it returns a tool_calls array where function.arguments is a JSON-encoded *string* you must json.loads yourself, rather than Anthropic's already-parsed input object. Same semantics, one extra parse step, one extra failure mode if the string is malformed.)

3. You execute and return a tool_result

Your code runs the real function and replies with a user message containing a tool_result block keyed by tool_use_id:

{
  "role": "user",
  "content": [
    {
      "type": "tool_result",
      "tool_use_id": "toolu_01A09q90qw90lq917835lq9",
      "content": "15°C, partly cloudy"
    }
  ]
}

Formatting rules that bite people (Anthropic returns HTTP 400 if you break them): the tool_result must immediately follow its tool_use turn with nothing in between, and inside the user message tool_result blocks come FIRST, any accompanying text AFTER.

4. The agentic loop

Client-executed tools require your application to drive the loop. The canonical shape is a while keyed on stop_reason:

messages = [{"role": "user", "content": user_prompt}]
while True:
    resp = client.messages.create(model=..., tools=TOOLS, messages=messages)
    messages.append({"role": "assistant", "content": resp.content})

    if resp.stop_reason != "tool_use":
        break  # "end_turn" | "max_tokens" | "stop_sequence" | "refusal" → done

    results = []
    for block in resp.content:
        if block.type == "tool_use":
            try:
                out = dispatch(block.name, block.input)   # YOUR code runs here
                results.append({"type": "tool_result",
                                "tool_use_id": block.id, "content": str(out)})
            except Exception as e:
                results.append({"type": "tool_result", "tool_use_id": block.id,
                                "content": f"{type(e).__name__}: {e}", "is_error": True})
    messages.append({"role": "user", "content": results})  # all results in one turn

That loop is an agent. Wrap it with memory ([[agent-memory-systems]]), a system prompt, and a stopping policy, and you have the substrate every coding/research agent is built on.

Client vs. server tools. Anthropic splits tools by where the code runs. Client tools (your functions, plus trained-in schemas like bash/text_editor/ computer/memory) you execute in the loop above. Server tools (web_search, code_execution, web_fetch, tool_search) run inside Anthropic's infra — you enable them and read the answer; the provider runs the loop for you and may return stop_reason: "pause_turn" while it iterates.

5. Parallel tool calls

A single assistant turn can contain multiple `tool_use` blocks — e.g. fetch weather for three cities at once, or read a file and query a DB simultaneously. You execute them (ideally concurrently), then return all the tool_result blocks in one user message. This is a real latency win: N independent calls cost one round trip, not N.

You can disable it for determinism: Anthropic uses tool_choice with disable_parallel_tool_use: true; OpenAI uses parallel_tool_calls: false. Forcing serial calls is sometimes necessary when calls have ordering dependencies or side effects, or when you want strictly one action per turn for auditability. BFCL treats parallel and parallel-multiple as their own hard categories precisely because generating several coherent, independent calls at once is a distinct, harder skill than emitting one.

6. Steering whether a tool fires: tool_choice

auto is a nudge, not a guarantee — and it's steerable via the system prompt. Anthropic notes "Use the tools to investigate before responding" measurably raises tool use; "Use your judgment…" keeps it conservative. For a hard guarantee, use tool_choice. Note also the cost asymmetry: forcing tool use (any/tool) injects a larger hidden system prompt than auto/none.

7. Structured outputs / strict mode (schema conformance for free)

The same machinery doubles as a JSON generator. Define a tool whose schema is the shape you want, force it with tool_choice, and read the validated input. But plain function calling only makes the schema likely, not guaranteed — the model can still hallucinate a field or pass a string where you wanted a number.

Strict mode closes that gap. Set strict: true (both Anthropic and OpenAI) and the provider uses constrained decoding: at each step the token sampler is masked to only those tokens that keep the output a valid prefix of your JSON Schema. The model cannot emit a key you didn't declare or a wrong-typed value. OpenAI's requirements make the mechanism visible: additionalProperties: false, every field listed in required, optional fields modeled as ["string", "null"]. The tradeoff: strict schemas are more rigid (no free-form keys; first request with a new schema pays a compilation cost), but they eliminate an entire class of parse-time bugs. Anthropic notes strict mode "eliminates invalid tool calls entirely."

8. Error handling — the part that separates toys from production

Tools fail: networks drop, APIs 500, the model passes a bad argument. You report failures back in-band with is_error: true:

{
  "type": "tool_result",
  "tool_use_id": "toolu_01A09q90qw90lq917835lq9",
  "content": "ConnectionError: weather service unavailable (HTTP 500)",
  "is_error": true
}

The model reads the error and adapts — apologizes, retries, or tries a different approach. The highest-value, least-obvious lesson from the docs: write error messages *for the model*, not for a log file. "failed" is useless; "Rate limit exceeded. Retry after 60 seconds." or "Error: missing required 'location' parameter" gives the model the context to self-correct. For a missing-parameter error, Claude typically retries 2–3 times with corrections before giving up — but only if your message says what was missing.

This is exactly where Toolformer (arXiv:2302.04761) and the survey converge: self-correction from execution feedback is a learned skill, and the structured reflection literature (e.g. "Failure Makes the Agent Stronger," 2025) shows that turning errors into clean, structured feedback measurably improves multi-turn tool reliability.

Security footnote, non-negotiable. tool_result content is untrusted input — web pages, emails, third-party APIs. An attacker who controls it can embed instructions (indirect prompt injection). Keep tool output inside tool_result blocks (never promote it into the system prompt), and treat the agent's tool surface as an attack surface.

Key ideas & tradeoffs

• Schema vs. description division of labor. Schema enforces shape (types, required fields, enums). Description conveys intent (when to call, what arguments mean). Both matter; the description is where most real-world failures get fixed.
• More tools ≠ better. Every tool costs prompt tokens and raises the chance the model picks the wrong one. BFCL's hardest categories are multiple-function selection and relevance detection (knowing when no tool fits). Beyond ~10–20 tools, models degrade and you want retrieval-over-tools (tool search) rather than dumping all schemas in-context.
• Parallel vs. serial. Parallel = lower latency, but the model must produce several coherent independent calls (a distinct skill) and you lose strict ordering. Serialize when calls have side effects or dependencies.
• Strict mode vs. flexibility. Constrained decoding guarantees valid JSON but forbids open-ended keys and adds a small schema-compile cost. Worth it for anything downstream code parses.
• Native function calling vs. prompt-and-parse. Fine-tuned native tool-calling (trained-in schemas, special tokens à la Toolformer) is far more reliable than asking a base model to "respond in JSON" and regex-ing the result. The whole point of formats like Anthropic's bash/text_editor is that the model was trained on those exact signatures and calls them more reliably.
• AST vs. executable evaluation (BFCL). You can grade a tool call two ways: parse the proposed call and check names/types/values against ground truth (AST), or actually run it and check the result. AST is cheap and deterministic; executable catches bugs AST misses but is flaky because real APIs change. Most eval suites use AST as the workhorse.

Honest caveats & open questions

• `auto` is a probability, not a promise. Even with a perfect schema the model may skip a tool it should use, or call one it shouldn't. BFCL's relevance-detection category exists because models hallucinate calls to irrelevant tools — a real, measured failure mode.
• Implicit argument coercion is weak. BFCL found models stumble on conversions like "5%" → 0.05 or string→float, and chat-tuned (non-tool-tuned) models frequently emit malformed, non-executable syntax. Strict mode helps with types but not with semantic coercion.
• Provider lock-in at the wire level. Anthropic returns parsed input; OpenAI returns an arguments string; roles differ (tool_result inside a user message vs. a dedicated tool/function role). Portable agent code needs an adapter layer. MCP (Model Context Protocol) is the emerging attempt to standardize the tool side of this.
• Long-horizon, multi-tool planning is still fragile. ToolLLM (arXiv:2307.16789) introduced DFSDT — a depth-first decision-tree search over reasoning traces — because linear ReAct/CoT prompting gets stuck and can't backtrack when a tool returns garbage. Robust planning over many tools is an open research area, not a solved one.
• Benchmark ≠ production. A 16,464-API benchmark (ToolBench) or BFCL leaderboard score doesn't tell you how the model handles your idiosyncratic internal API, your error formats, or adversarial tool output. Self-reported tool-call success is unreliable; execution is the only source of truth.
• Token budget is a silent ceiling. Tool schemas + tool-use system prompt + accumulated tool_use/tool_result history grow every loop iteration. Long agentic runs blow the context window; this is why memory/compaction ([[agent-memory-systems]]) is part of the same problem, not a separate one.

How it connects to OpenAlice

Tool use is the entire substrate of Alice's agentic capability — the thing NAO's memory calls "a BASE AGI capability of Alice." Every concrete behavior is a tool call:

• The agentic loop above is literally Alice's run loop. Alice's agentic_loop drives exactly the while stop_reason == "tool_use" cycle described here — emit calls, your code executes (browse, pty/bash, file edits, connector sends), return results, repeat. The HITL approval gates, kill switch, and standing approvals in NAO's quality bar are all interceptors on the tool-dispatch step of this loop (gate the call before dispatch(), resume on approval) — see the recent #489 approval-protocol work (B2 suspend/resume around tool execution) on this mvp branch.
• Connectors are tools. Telegram, YouTube, web fetch — each is a function the model calls; the "never auto-mutate NAO's YouTube" rule is a forced approval on a specific tool, the production face of tool_choice + an HITL gate.
• Atlas is a tool surface. The whole "query Atlas before grepping" directive works because atlas.code.search / atlas.code.callers / atlas.context are exposed as MCP tools and REST endpoints — a textbook example of giving an agent structured access to ground truth instead of letting it free-text-grep. Good tool descriptions are why an agent reaches for them.
• Structured outputs everywhere. Anywhere OpenAlice parses model output into a typed struct (scene directives, ranker decisions, the critic's verdicts), the right move is a strict-mode tool schema, not regex-on-prose — the exact "if you're writing a regex to extract a decision, it should have been a tool call" rule.
• Provider portability matters here. OpenAlice defaults to Codex/gpt-5.5 but keeps the option of Claude; the wire-format differences (parsed input vs. arguments string, role conventions) are precisely why Alice's tool-dispatch layer abstracts over providers.

Related reading in this library: [[agent-memory-systems]] (what the loop remembers across tool calls), [[test-time-compute-reasoning]] (reasoning models that interleave thinking with tool calls), [[graphrag]] and [[embeddings]] (the retrieval tools agents call most), and [[model-routing]] (choosing which model drives the tool loop cost-effectively).