Tool Calling and Execution Loops

Train a model to emit schema-valid tool calls, decide when not to call, and recover from malformed results.

Tool calling is a posttraining outcome, not a prompt-engineering feature bolted onto a finished model. The model must choose among available tools, emit arguments that satisfy a typed schema, decide when answering directly is better than calling, read results back, and continue within a step budget. Each of those is a separate learned behavior with a separate failure mode, and the evaluation protocol keeps them separate.

Tool traces and execution

Extend the Chapter 08 chat template with tool roles: an assistant message carries a tool name and JSON arguments, and a tool message carries the execution result. The tool inventory reaches the prompt as typed schemas. An episode interleaves assistant turns, tool calls, and tool results:

{
  "role": "assistant",
  "tool_calls": [
    {"name": "get_weather", "arguments": {"city": "Oslo", "unit": "celsius"}}
  ]
}

The loop is: generate, parse, validate against the schema, execute against a registry of deterministic fake tools (weather, calculator, file lookup, calendar), append the result, repeat until a final answer or the step budget is exhausted. Malformed calls are training data, not exceptions: return a structured error message and record whether the next turn recovers. Training loss applies to tool-call spans only, using the masked cross-entropy from Chapter 08; compare free decoding with post-hoc validation against JSON-schema-constrained decoding for the argument span.

Tool-call validity

A tool call is correct only if three independent conditions hold, and each has its own metric:

Condition	Failure mode	Metric
The tool exists and fits the request	hallucinated or wrong tool	exact tool-choice accuracy
Arguments parse and satisfy the schema	schema violation	schema-validity rate
Calling was the right decision	unnecessary call	when-to-call accuracy, false-call rate

A model can score perfectly on one row and fail the others, which is why a single “tool accuracy” number is uninterpretable.

Evaluation protocol

On held-out requests, report exact tool choice and argument correctness, schema-validity rate, when-to-call accuracy on prompts where calling helps and prompts where it does not, hallucinated-tool and hallucinated-parameter rates, recovery success after a deliberately corrupted tool result, multi-turn completion within the step budget, and the calibration curve of call decisions. Hold out tools and their schemas entirely from training, not just their names: generalization to an unseen schema is a different claim from generalization to an unseen phrasing of a seen tool.

Tool reliability

Each failure mode in this chapter maps to one from the course vocabulary: a hallucinated API is proxy optimization (the model learned the shape of calls from traces), a schema violation is a specification error, an unnecessary call is a calibration failure, and ignoring a corrected tool result is a correction failure. The protocol above measures each separately, and it carries into the Chapter 14 capstone unchanged.

Summary

Tool training is SFT over scripted trajectories with call-span masks, optionally sharpened by DPO pairs on borderline prompts (correct call versus hallucinated tool, well-scoped call versus unnecessary call). The evaluation protocol, not the training loss, defines what was actually learned.

Shared typed tool registry

The chapter’s protocol is now backed by a deterministic proof-tool registry. Schema validation happens before execution, unknown tools and malformed arguments return structured error codes, and a valid proof result can be checked independently. Those outcomes are the training traces: errors are data for recovery, not exceptions that disappear from the dataset.

from proof_lm.logic import generate_examples
from proof_lm.tools import ProofToolRegistry, ToolCall

registry = ProofToolRegistry()
positive_proof = generate_examples(count=1, seed=11)[0].proof_text
parsed = registry.execute(ToolCall("parse_formula", {"formula": "P -> Q"}))
checked = registry.execute(ToolCall("check_proof", {"proof": positive_proof}))
malformed = registry.execute(ToolCall("check_proof", {"proof": "<not a proof>"}))
print("registry id:", registry.registry_id)
print("tools:", [item["name"] for item in registry.schema()])
print("parse formula:", parsed.value)
print("valid proof:", checked.value)
print("malformed status:", malformed.error_code)
assert parsed.ok and checked.ok and checked.value["valid"]
assert not malformed.ok and malformed.error_code == "invalid_arguments"

registry id: tools-0d40b519fe44f7c4
tools: ['parse_formula', 'check_proof', 'find_countermodel']
parse formula: {'formula': 'P → Q', 'kind': 'implies'}
valid proof: {'valid': True, 'errors': []}
malformed status: invalid_arguments

Exercises

Use the exercises to test the chapter’s invariants and connect the derivations to the reusable implementation. Solutions are hidden in the notebook source and are available through the course tooling when needed.

[P11.1] Correct tool-call conditions

List the three independent conditions for a correct tool call and one metric for each.

[P11.2] Separating tool-call failures

Design three held-out episodes that separate wrong-tool selection from correct-tool malformed arguments from unnecessary calls. For each, state which metric from the protocol fails and which still passes.

[P11.3]

A model emits a known tool name but omits a required argument. Which part of the tool-call protocol fails, and what structured result should the training loop record?