Model versus Harness

agents
engineering
From a stateless completion to a measured think-act-observe loop.

A language model can predict the next message in a conversation, but prediction alone does not read a file, call a function, or learn what happened after an action. This week draws the boundary between that raw model and the harness around it. The build is intentionally small: a raw completion, one hand-parsed tool call, and a roughly fifty-line multi-turn loop.

The examples use a deterministic offline model-shaped function. The protocol is the object of study, so no API key, network connection, or particular provider is required.

Design rule: every new capability must come with an observable contract and a failure test before it is called agency.

By the end of the chapter, the episode, the think-act-observe pattern, and its loop invariant will be executable rather than just named. The final ledger records what this small harness can establish, what it cannot, and why Week 2 starts with transport rather than a larger agent framework.

Model and harness

A completion endpoint consumes messages and returns more messages. Unless the caller adds state, each request has no memory of a previous request. The model has no intrinsic persistent state: a variable in a Python process, a file in a repository, or a previous observation exists only if the harness serializes it back into the next context.

The model also has no intrinsic effect on the world. A sentence such as “I wrote the file” is still a sentence until a local program performs the write and reports its result. Finally, a model has no guaranteed ground truth about the current filesystem, clock, network, or tool state. Its useful knowledge can be stale, and a plausible answer is not an observation.

These are separate gaps:

Gap What the model emits What the harness must add
State a continuation conditioned on supplied messages history, session state, or durable storage
Effect a proposed action or tool call a dispatcher that executes a permitted function
Ground truth a claim about the world an observation returned by the world or a tool

A harness does not make the model omniscient. It creates a controlled channel through which the model can request an action and receive evidence about that action.

from dataclasses import dataclass
import json
from pprint import pprint
from typing import Any, Callable


@dataclass(frozen=True)
class StatelessModel:
    """A model-shaped function with no state or side effects."""

    name: str = "offline-raw-model"

    def complete(self, messages: list[dict[str, Any]]) -> dict[str, Any]:
        return {
            "role": "assistant",
            "content": (
                "I can suggest a procedure, but I have no filesystem observation "
                "and cannot change the repository."
            ),
        }


raw_model = StatelessModel()
request = [{"role": "user", "content": "Read README.md and summarize its license."}]
first = raw_model.complete(request)
second = raw_model.complete(request)
assert first == second
pprint(first)
print("same request, same supplied context -> identical model state:", first == second)
{'content': 'I can suggest a procedure, but I have no filesystem observation '
            'and cannot change the repository.',
 'role': 'assistant'}
same request, same supplied context -> identical model state: True

The baseline is not a claim that every model is deterministic. It isolates the boundary: this function receives no repository bytes, has no file-writing capability, and returns no observation from the outside world. Repeating the call with the same supplied messages cannot create persistence or an effect. Randomness or a stronger model would change the text, not those missing channels.

A harness therefore has to preserve a conversation, translate a model decision into an executable call, and append the result without pretending that the model observed it directly.

Episodes and think-act-observe

An episode is a message stream

m_{0:t}=(m_0,m_1,\ldots,m_t), \qquad m_i \in \{\text{system},\text{user},\text{assistant},\text{tool}\}.

An assistant message may contain zero or more tool calls. Each call has an identifier, a tool name, and serialized arguments. Before the next model turn, that call must receive exactly one tool message with the same identifier. A final assistant message has no tool calls and ends the episode.

A tool call is correct only when three conditions hold: the selected tool exists and is appropriate for the request; its arguments parse and satisfy the tool schema; and calling it is the right decision rather than answering directly. The model may propose all three, but the harness is where their runtime consequences become observable.

The smallest controller alternates three conceptual steps:

  1. Think: ask the model for its next assistant message. “Think” names a decision turn; it does not require exposing private chain-of-thought.
  2. Act: if the message contains calls, execute the selected local functions under whatever policy exists.
  3. Observe: append each function’s returned string as a tool message, then give the expanded history to the model again.

The loop invariant is the integrity anchor:

Every assistant turn ends in either a final answer or well-formed tool calls; every tool call produces exactly one observation; no observation is fabricated by the harness.

A trajectory \tau is the episode plus the harness record around it: events, token counts, costs, permission decisions, hook verdicts, and context operations. Week 1 records only messages and simple events. Later weeks add the rest without changing the episode contract.

def validate_episode(messages: list[dict[str, Any]]) -> bool:
    """Check the structural part of the Week 1 loop invariant."""
    pending: dict[str, dict[str, Any]] = {}
    finished = False

    for index, message in enumerate(messages):
        role = message.get("role")
        if finished:
            raise AssertionError(f"message {index} appears after a final answer")

        if role == "assistant":
            calls = message.get("tool_calls") or []
            if calls:
                for call in calls:
                    call_id = call["id"]
                    if call_id in pending:
                        raise AssertionError(f"duplicate pending call id: {call_id}")
                    function = call["function"]
                    if not isinstance(function["name"], str):
                        raise AssertionError("tool name must be a string")
                    if not isinstance(function["arguments"], str):
                        raise AssertionError("serialized arguments must be a string")
                    pending[call_id] = call
            else:
                if not isinstance(message.get("content"), str):
                    raise AssertionError("a final answer must contain text")
                if pending:
                    raise AssertionError("a final answer has unanswered tool calls")
                finished = True
        elif role == "tool":
            call_id = message["tool_call_id"]
            if call_id not in pending:
                raise AssertionError(f"observation without one pending call: {call_id}")
            if not isinstance(message.get("content"), str):
                raise AssertionError("tool observations must be strings")
            pending.pop(call_id)
        elif role in {"system", "user"}:
            if not isinstance(message.get("content"), str):
                raise AssertionError(f"{role} message must contain text")
        else:
            raise AssertionError(f"unknown message role: {role}")

    if pending:
        raise AssertionError(f"unanswered calls: {sorted(pending)}")
    return True


good_episode = [
    {"role": "user", "content": "What is 2 + 3?"},
    {
        "role": "assistant",
        "content": None,
        "tool_calls": [
            {
                "id": "call-1",
                "type": "function",
                "function": {"name": "add", "arguments": '{"a": 2, "b": 3}'},
            }
        ],
    },
    {"role": "tool", "tool_call_id": "call-1", "name": "add", "content": "5"},
    {"role": "assistant", "content": "5"},
]
assert validate_episode(good_episode)
print("structural invariant holds for a two-turn episode")
structural invariant holds for a two-turn episode

The validator checks pairing and ordering, not whether the tool told the truth. That distinction matters. The harness can establish that a result came back through the expected channel, but it needs a tool contract, a second check, or an evaluator to establish semantic correctness.

The controller is also budget constrained. In the complete course, execution stops when a turn, cost, or context budget binds:

t \le B_{\text{turns}}, \qquad c(\tau) \le B_{\text{cost}}, \qquad n(\tau) \le B_{\text{context}}.

This week implements only the hard turn bound. A hard stop is an honest trajectory outcome, not a final answer. Cost and context become real control surfaces in Weeks 2 and 6.

Artifact lineage

The loop is not a new name for chain-of-thought. Four influential lines of work supply different pieces of the story:

Work Contribution Boundary that remains
Chain-of-Thought prompting (Wei et al., 2022) elicits intermediate reasoning steps so a model can solve some multi-step problems reasoning remains inside the completion; it does not itself execute or observe an action
ReAct (Yao et al., 2022) interleaves reasoning, actions, and observations in a trajectory the environment and parser still need explicit contracts
Toolformer (Schick et al., 2023) trains a model to decide when and how to insert API calls into text learned call propensity is not a permission layer or a durable session
Generative Agents (Park et al., 2023) makes memory streams, retrieval, reflection, and planning part of an agent simulation persistent memory adds retrieval and consistency failure modes

ReAct is the closest conceptual predecessor for this week’s loop: an assistant turn proposes an action, the environment returns an observation, and the next turn conditions on the resulting history. Toolformer focuses on learning the call behavior. Generative Agents shows what happens when the trace becomes a persistent memory stream. CoT is useful as a prompting technique, but an agent should not be defined by whether it prints a reasoning trace.

The thesis under examination is the course shorthand that Claude Code is a thin loop plus twenty good decisions. Week 1 isolates the loop before those decisions are hidden in a framework; later weeks add and measure the decisions one mechanism at a time.

Where reasoning prompts fail

A reasoning trace can improve a result and still fail in ways a harness must expose:

  • Error propagation: an early wrong premise can be repeated and elaborated until the final answer sounds more certain than the evidence supports.
  • Sycophancy: the model can accept a user’s leading claim or a previous assistant statement instead of checking it against an observation.
  • Distraction: extra retrieved facts or irrelevant instructions can consume context and pull the reasoning away from the task.

The think step in this chapter is therefore an abstraction over a model turn, not a request to reveal hidden reasoning. The observable artifact is the message and the tool call; the evaluation target is whether the trajectory reaches a correct, constrained result. Later context and instruction mechanisms create deliberate probes for distraction and correction survival.

Naive implementations

The three baselines make the boundary visible:

  1. the model-only completion above, which cannot act;
  2. one raw API-shaped tool call, parsed and executed by hand, which can act once but cannot complete a tool-mediated answer; and
  3. a roughly fifty-line multi-turn loop, which sends history, executes Python functions, appends results, and stops at a final answer or a turn budget.

None of these implementations has retries, schema validation, permissions, context management, or a durable event journal. That omission is deliberate. The failures are easier to attribute before the later mechanisms arrive.

Implementation 2: one raw tool call

A provider API normally returns a JSON envelope containing a choice, an assistant message, and possibly tool_calls. The next cell uses that envelope without an SDK. offline_raw_api is a deterministic stand-in for the HTTP response, so the parser and the control boundary can be tested offline.

This baseline does one request, parses one call, executes one local function, and returns the observation to its caller. It does not send the observation back to the model. That is the exact missing turn the loop will add.

def add_ints(a: int, b: int) -> str:
    if type(a) is not int or type(b) is not int:
        raise TypeError("a and b must be integers")
    return str(a + b)


def count_letters(text: str) -> str:
    if not isinstance(text, str):
        raise TypeError("text must be a string")
    return str(len(text.replace(" ", "")))


def lookup_fact(topic: str) -> str:
    if not isinstance(topic, str):
        raise TypeError("topic must be a string")
    facts = {"sky": "blue", "grass": "green"}
    return facts.get(topic, "unknown")


def echo(value: str) -> str:
    if not isinstance(value, str):
        raise TypeError("value must be a string")
    return value


TOOLS: dict[str, Callable[..., str]] = {
    "add": add_ints,
    "count_letters": count_letters,
    "lookup_fact": lookup_fact,
    "echo": echo,
}

TOOL_DESCRIPTIONS = [
    {
        "type": "function",
        "function": {
            "name": "add",
            "description": "Add two integer values.",
            "parameters": {
                "type": "object",
                "properties": {
                    "a": {"type": "integer"},
                    "b": {"type": "integer"},
                },
                "required": ["a", "b"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "count_letters",
            "description": "Count non-space characters in text.",
            "parameters": {
                "type": "object",
                "properties": {"text": {"type": "string"}},
                "required": ["text"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "lookup_fact",
            "description": "Look up a small offline fact by topic.",
            "parameters": {
                "type": "object",
                "properties": {"topic": {"type": "string"}},
                "required": ["topic"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "echo",
            "description": "Return the supplied string unchanged.",
            "parameters": {
                "type": "object",
                "properties": {"value": {"type": "string"}},
                "required": ["value"],
            },
        },
    },
]


def raw_response(message: dict[str, Any]) -> str:
    finish_reason = "tool_calls" if message.get("tool_calls") else "stop"
    return json.dumps(
        {
            "id": "offline-response",
            "choices": [
                {"index": 0, "message": message, "finish_reason": finish_reason}
            ],
        }
    )


def offline_raw_api(request: dict[str, Any]) -> str:
    messages = request["messages"]
    goal = next(message["content"] for message in messages if message["role"] == "user")
    observations = [message for message in messages if message["role"] == "tool"]

    if observations:
        return raw_response({"role": "assistant", "content": observations[-1]["content"]})
    if goal == "What is 2 + 3?":
        return raw_response({"role": "assistant", "content": "5"})
    if goal == "What is 19 + 23?":
        call = {
            "id": "call-1",
            "type": "function",
            "function": {"name": "add", "arguments": '{"a": 19, "b": 23}'},
        }
        return raw_response({"role": "assistant", "content": None, "tool_calls": [call]})
    if goal == "How many letters are in harness?":
        call = {
            "id": "call-1",
            "type": "function",
            "function": {"name": "count_letters", "arguments": '{"text": "harness"}'},
        }
        return raw_response({"role": "assistant", "content": None, "tool_calls": [call]})
    if goal == "What color is the sky?":
        call = {
            "id": "call-1",
            "type": "function",
            "function": {"name": "lookup_fact", "arguments": '{"topic": "sky"}'},
        }
        return raw_response({"role": "assistant", "content": None, "tool_calls": [call]})
    return raw_response({"role": "assistant", "content": "unknown task"})


def naive_one_shot(
    goal: str,
    api: Callable[[dict[str, Any]], str] = offline_raw_api,
    tools: dict[str, Callable[..., str]] = TOOLS,
) -> dict[str, Any]:
    request = {
        "model": "offline-script-v1",
        "messages": [{"role": "user", "content": goal}],
        "tools": TOOL_DESCRIPTIONS,
    }
    payload = json.loads(api(request))
    assistant = payload["choices"][0]["message"]
    calls = assistant.get("tool_calls") or []
    if not calls:
        return {"status": "final", "answer": assistant["content"], "messages": [assistant]}

    call = calls[0]
    name = call["function"]["name"]
    arguments = json.loads(call["function"]["arguments"])
    observation = tools[name](**arguments)
    return {
        "status": "needs_follow_up",
        "answer": None,
        "assistant": assistant,
        "observation": observation,
    }


demo_one_shot = naive_one_shot("What is 19 + 23?")
pprint({key: demo_one_shot[key] for key in ("status", "observation")})
{'observation': '42', 'status': 'needs_follow_up'}

The one-shot output is an observation, not a completed answer. A caller could display 42, but the model did not receive that evidence and could not explain or use it in a subsequent decision. The parser also assumes every envelope, choice, call, function name, and argument string has exactly the expected shape. Those assumptions are useful for a teaching baseline and unsafe as a production contract.

Optional live API adapter

The protocol can be pointed at an OpenAI-compatible endpoint, but this chapter does not need a key. The helper below is definition-only: it is never invoked during the offline run. Set OPENAI_API_KEY and call it explicitly in a separate experiment if you want to compare a provider’s response envelope with the fixture.

import os
from urllib.request import Request, urlopen


def optional_openai_compatible_call(
    messages: list[dict[str, Any]],
    tools: list[dict[str, Any]],
    base_url: str = "https://api.openai.com/v1",
) -> dict[str, Any]:
    api_key = os.environ.get("OPENAI_API_KEY")
    if not api_key:
        raise RuntimeError("Set OPENAI_API_KEY before opting into a live request")
    payload = {"model": "replace-me", "messages": messages, "tools": tools}
    request = Request(
        base_url.rstrip("/") + "/chat/completions",
        data=json.dumps(payload).encode("utf-8"),
        headers={
            "Authorization": f"Bearer {api_key}",
            "Content-Type": "application/json",
        },
        method="POST",
    )
    with urlopen(request, timeout=30) as response:
        return json.loads(response.read())


print("live adapter defined; no network request made")
live adapter defined; no network request made

Implementation 3: the small multi-turn loop

The loop adds only the missing turn. It keeps a list of messages, asks the model for a raw JSON response, executes every call in that response, appends each result unchanged, and asks again. It returns a final answer or an explicit max_turns status.

The code is intentionally naive. It catches a tool exception and turns it into an error string, but it does not yet validate schemas, retry transport failures, ask for permission, or limit cost. Those are mechanisms for later chapters, not hidden behavior in this baseline.

def naive_agent_loop(
    goal: str,
    model: Callable[[dict[str, Any]], str] = offline_raw_api,
    tools: dict[str, Callable[..., str]] = TOOLS,
    max_turns: int = 4,
) -> dict[str, Any]:
    messages = [{"role": "user", "content": goal}]
    events: list[dict[str, Any]] = []

    for turn in range(max_turns):
        request = {
            "model": "offline-script-v1",
            "messages": messages,
            "tools": TOOL_DESCRIPTIONS,
        }
        payload = json.loads(model(request))
        assistant = payload["choices"][0]["message"]
        messages.append(assistant)
        events.append({"kind": "assistant", "turn": turn, "message": assistant})
        calls = assistant.get("tool_calls") or []
        if not calls:
            return {
                "status": "final",
                "answer": assistant["content"],
                "messages": messages,
                "events": events,
            }

        for call in calls:
            name = call["function"]["name"]
            try:
                arguments = json.loads(call["function"]["arguments"])
                result = tools[name](**arguments)
            except Exception as exc:
                result = f"tool_error: {type(exc).__name__}: {exc}"
            observation = {
                "role": "tool",
                "tool_call_id": call["id"],
                "name": name,
                "content": result,
            }
            messages.append(observation)
            events.append(
                {
                    "kind": "tool",
                    "turn": turn,
                    "name": name,
                    "call_id": call["id"],
                    "content": result,
                }
            )

    return {
        "status": "max_turns",
        "answer": None,
        "messages": messages,
        "events": events,
    }


loop_demo = naive_agent_loop("What is 19 + 23?")
print(loop_demo["status"], "->", loop_demo["answer"])
assert validate_episode(loop_demo["messages"])
final -> 42

The four marked lines are the entire state transition. <1> creates the user history; <2> preserves the assistant message, including its serialized calls; <3> records the local function’s result or an explicit error; and <4> makes that exact string visible to the next model turn. The event list is not yet a journal, but it makes the trajectory inspectable without changing the message protocol.

Notice what the loop does not do: it does not infer that a tool succeeded, rewrite an error into friendly prose, or manufacture a result when a call is malformed. Its error string is a correction delivered through the same observation channel. Whether the model uses that correction is an empirical question.

Deterministic offline tests

The tests below are ordinary assertions rather than a call to a provider or a stochastic benchmark. Each fake model is a pure function of the supplied request, and each tool has fixed output. They encode the three minimum Week 1 contracts:

  1. a final answer terminates the loop;
  2. repeated calls terminate at max_turns; and
  3. a tool result, including punctuation and newlines, reaches the next turn verbatim.

The last test also runs the structural episode validator. A hard stop may end after all calls in the final turn have been answered, so it is a valid incomplete trajectory but never a fabricated final answer.

def final_only_model(request: dict[str, Any]) -> str:
    return raw_response({"role": "assistant", "content": "done"})


def thrashing_model(request: dict[str, Any]) -> str:
    tool_count = sum(message["role"] == "tool" for message in request["messages"])
    call = {
        "id": f"thrash-{tool_count + 1}",
        "type": "function",
        "function": {"name": "add", "arguments": '{"a": 1, "b": 1}'},
    }
    return raw_response({"role": "assistant", "content": None, "tool_calls": [call]})


def verbatim_model(request: dict[str, Any]) -> str:
    if not any(message["role"] == "tool" for message in request["messages"]):
        raw_value = "line 1\nline 2 | no summary"
        call = {
            "id": "verbatim-1",
            "type": "function",
            "function": {"name": "echo", "arguments": json.dumps({"value": raw_value})},
        }
        return raw_response({"role": "assistant", "content": None, "tool_calls": [call]})
    return raw_response({"role": "assistant", "content": "observed"})


final_run = naive_agent_loop("finish", model=final_only_model, max_turns=2)
assert final_run["status"] == "final"
assert final_run["answer"] == "done"
assert validate_episode(final_run["messages"])

thrash_run = naive_agent_loop("keep going", model=thrashing_model, max_turns=3)
assert thrash_run["status"] == "max_turns"
assert thrash_run["answer"] is None
assert validate_episode(thrash_run["messages"])

verbatim_run = naive_agent_loop("preserve this", model=verbatim_model, max_turns=2)
assert verbatim_run["status"] == "final"
assert validate_episode(verbatim_run["messages"])
tool_messages = [m for m in verbatim_run["messages"] if m["role"] == "tool"]
assert len(tool_messages) == 1
assert tool_messages[0]["content"] == "line 1\nline 2 | no summary"

print("PASS: final answer terminates")
print("PASS: thrashing stops at max_turns")
print("PASS: tool result is appended verbatim")
PASS: final answer terminates
PASS: thrashing stops at max_turns
PASS: tool result is appended verbatim

These tests establish termination and message integrity, not task intelligence. The thrashing case is especially important: a loop that keeps asking for work without a hard boundary is not more agentic, it is an unbounded process. The verbatim assertion leaves semantic interpretation to the next model turn and makes any harness paraphrase observable.

One-shot versus loop

Use one fixed task set and the same deterministic response policy in both conditions. A task is successful only when the returned answer exactly equals its known answer. The one-shot baseline receives one model response and can execute one tool, but a tool-mediated task is marked incomplete because no final model turn occurs. The loop receives the same first response and then replays the tool observation.

Protocol. Model offline-script-v1; greedy scripted decoding; seed 0; four fixed tasks; max_turns=4; no network; no retries; no hidden judge. Report completion, tool calls, and exact task success together. This is a protocol experiment, not a population estimate for a frontier model.

TASKS = [
    {"goal": "What is 2 + 3?", "answer": "5"},
    {"goal": "What is 19 + 23?", "answer": "42"},
    {"goal": "How many letters are in harness?", "answer": "7"},
    {"goal": "What color is the sky?", "answer": "blue"},
]


def exact_success(result: dict[str, Any], expected: str) -> bool:
    return result.get("status") == "final" and result.get("answer") == expected


comparison: list[dict[str, Any]] = []
for task in TASKS:
    one_shot = naive_one_shot(task["goal"])
    loop = naive_agent_loop(task["goal"])
    comparison.append(
        {
            "goal": task["goal"],
            "one_shot_success": exact_success(one_shot, task["answer"]),
            "loop_success": exact_success(loop, task["answer"]),
            "one_shot_status": one_shot["status"],
            "loop_status": loop["status"],
            "loop_tool_calls": sum(event["kind"] == "tool" for event in loop["events"]),
        }
    )

for row in comparison:
    print(
        f"{row['goal']:<38} one-shot={row['one_shot_success']} "
        f"({row['one_shot_status']:<16}) loop={row['loop_success']} "
        f"({row['loop_status']:<5}) tools={row['loop_tool_calls']}"
    )

one_shot_successes = sum(row["one_shot_success"] for row in comparison)
loop_successes = sum(row["loop_success"] for row in comparison)
completed_one_shots = sum(row["one_shot_status"] == "final" for row in comparison)
completed_loops = sum(row["loop_status"] == "final" for row in comparison)
print(f"one-shot success: {one_shot_successes}/{len(TASKS)}")
print(f"loop success:     {loop_successes}/{len(TASKS)}")
print(
    "success-rate delta: "
    f"{(loop_successes - one_shot_successes) / len(TASKS):+.0%}"
)
assert one_shot_successes == 1
assert loop_successes == 4
assert completed_one_shots == 1
assert completed_loops == 4
What is 2 + 3?                         one-shot=True (final           ) loop=True (final) tools=0
What is 19 + 23?                       one-shot=False (needs_follow_up ) loop=True (final) tools=1
How many letters are in harness?       one-shot=False (needs_follow_up ) loop=True (final) tools=1
What color is the sky?                 one-shot=False (needs_follow_up ) loop=True (final) tools=1
one-shot success: 1/4
loop success:     4/4
success-rate delta: +75%

The smallest harness change raises exact success from 1/4 to 4/4 on this fixed set, a +75 percentage-point delta, while adding one follow-up model turn to each tool-mediated task. The result attributes the gap to completion of the protocol, not to a more capable tool: the one-shot condition already executed the same local functions.

This number is intentionally narrow. It says that a response parser without a return path is incomplete on tasks requiring evidence. It says nothing about cost, latency, unsafe actions, or whether the model would notice a wrong observation. The next experiments probe those failure surfaces.

Lying tool

Now break one tool without raising an exception. lying_add accepts valid integers but returns 43 for 19+23. The harness delivers that string exactly as it delivered the honest result. Two deterministic response policies expose the model-side distinction:

  • a trusting policy repeats the observation as its answer;
  • a checking policy independently recomputes the expression and reports a conflict.

The fixture alternates those policies across eight cases. The frequency is a count of prescribed response styles, not a claim about how often an unobserved real model would check its tools.

def lying_add(a: int, b: int) -> str:
    add_ints(a, b)  # retain argument validation while corrupting the result
    return "43"


def make_lie_probe_model(policy: str) -> Callable[[dict[str, Any]], str]:
    def model(request: dict[str, Any]) -> str:
        observations = [m for m in request["messages"] if m["role"] == "tool"]
        if not observations:
            call = {
                "id": "lie-1",
                "type": "function",
                "function": {"name": "add", "arguments": '{"a": 19, "b": 23}'},
            }
            return raw_response({"role": "assistant", "content": None, "tool_calls": [call]})

        reported = observations[-1]["content"]
        if policy == "checking" and reported != "42":
            answer = f"CONFLICT: tool reported {reported}; independent check gives 42"
        else:
            answer = reported
        return raw_response({"role": "assistant", "content": answer})

    return model


lie_records = []
for case_number, policy in enumerate(["trusting", "checking"] * 4, start=1):
    result = naive_agent_loop(
        "What is 19 + 23?",
        model=make_lie_probe_model(policy),
        tools={"add": lying_add},
        max_turns=3,
    )
    answer = result["answer"]
    lie_records.append(
        {
            "case": case_number,
            "policy": policy,
            "answer": answer,
            "noticed": isinstance(answer, str) and answer.startswith("CONFLICT"),
            "confabulated": answer == "43",
        }
    )
    assert result["status"] == "final"
    assert validate_episode(result["messages"])

for row in lie_records:
    print(
        f"case {row['case']}: {row['policy']:<8} "
        f"noticed={row['noticed']} confabulated={row['confabulated']}"
    )
notices = sum(row["noticed"] for row in lie_records)
confabulations = sum(row["confabulated"] for row in lie_records)
print(f"noticed lie:       {notices}/{len(lie_records)}")
print(f"accepted lie:      {confabulations}/{len(lie_records)}")
assert notices == 4
assert confabulations == 4
case 1: trusting noticed=False confabulated=True
case 2: checking noticed=True confabulated=False
case 3: trusting noticed=False confabulated=True
case 4: checking noticed=True confabulated=False
case 5: trusting noticed=False confabulated=True
case 6: checking noticed=True confabulated=False
case 7: trusting noticed=False confabulated=True
case 8: checking noticed=True confabulated=False
noticed lie:       4/8
accepted lie:      4/8

The loop preserves transport integrity while remaining unable to certify truth. Four cases detect the bad result because the model policy performs an independent check; four cases silently answer 43 because the policy treats an observation as ground truth. The harness did not fabricate the lie, but it also had no semantic guard against a lying tool.

This is a capability-control mismatch in miniature: the system can perform arithmetic, yet its current tool contract does not control whether the reported result is correct. A stronger protocol could add independent checks, provenance, or typed result validation. Those safeguards belong above this deliberately naive loop and should be measured rather than assumed.

Malformed-call recovery

A malformed call is different from a silent lie. The dispatcher can detect that a is a string where add requires an integer, and the loop can return the validator’s message as a tool observation. The correction test asks whether the next model turn improves after seeing that error.

The response fixture always starts with the malformed call {"a": "19", "b": 23}. In the recovery condition, attempt two sends integer arguments after reading the error. In the non-recovery condition, it repeats the malformed call until the hard stop. Six alternating cases make the rate visible while remaining deterministic.

def make_recovery_model(should_recover: bool) -> Callable[[dict[str, Any]], str]:
    def model(request: dict[str, Any]) -> str:
        observations = [m for m in request["messages"] if m["role"] == "tool"]
        if not observations:
            arguments = '{"a": "19", "b": 23}'
        elif observations[-1]["content"].startswith("tool_error"):
            arguments = '{"a": 19, "b": 23}' if should_recover else '{"a": "19", "b": 23}'
        else:
            return raw_response({"role": "assistant", "content": observations[-1]["content"]})

        call = {
            "id": f"recovery-{len(observations) + 1}",
            "type": "function",
            "function": {"name": "add", "arguments": arguments},
        }
        return raw_response({"role": "assistant", "content": None, "tool_calls": [call]})

    return model


def call_arguments(result: dict[str, Any]) -> list[dict[str, Any]]:
    calls = [
        message["tool_calls"][0]
        for message in result["messages"]
        if message["role"] == "assistant" and message.get("tool_calls")
    ]
    return [json.loads(call["function"]["arguments"]) for call in calls]


recovery_records = []
for case_number, should_recover in enumerate([True, False] * 3, start=1):
    result = naive_agent_loop(
        "What is 19 + 23?",
        model=make_recovery_model(should_recover),
        max_turns=3,
    )
    arguments = call_arguments(result)
    tool_messages = [m for m in result["messages"] if m["role"] == "tool"]
    attempt_two_valid = (
        len(arguments) >= 2
        and type(arguments[1]["a"]) is int
        and type(arguments[1]["b"]) is int
    )
    recovered = result["status"] == "final" and result["answer"] == "42"
    error_replayed = bool(tool_messages) and tool_messages[0]["content"].startswith("tool_error")
    recovery_records.append(
        {
            "case": case_number,
            "condition": "recovery" if should_recover else "repeat-error",
            "attempt_2_valid": attempt_two_valid,
            "recovered": recovered,
            "error_replayed": error_replayed,
        }
    )
    assert error_replayed
    assert validate_episode(result["messages"])

for row in recovery_records:
    print(
        f"case {row['case']}: {row['condition']:<13} "
        f"attempt-2-valid={row['attempt_2_valid']} "
        f"recovered={row['recovered']}"
    )
recovered_count = sum(row["recovered"] for row in recovery_records)
valid_second_attempts = sum(row["attempt_2_valid"] for row in recovery_records)
print(f"recovery rate:       {recovered_count}/{len(recovery_records)}")
print(f"valid attempt two:   {valid_second_attempts}/{len(recovery_records)}")
assert recovered_count == 3
assert valid_second_attempts == 3
case 1: recovery      attempt-2-valid=True recovered=True
case 2: repeat-error  attempt-2-valid=False recovered=False
case 3: recovery      attempt-2-valid=True recovered=True
case 4: repeat-error  attempt-2-valid=False recovered=False
case 5: recovery      attempt-2-valid=True recovered=True
case 6: repeat-error  attempt-2-valid=False recovered=False
recovery rate:       3/6
valid attempt two:   3/6

The recovery condition validates on attempt two in three of six cases and reaches a final answer in those same three. The other three cases receive the same correction but repeat it until max_turns, which is correction failure rather than a harness crash. The loop’s useful guarantee is narrower and concrete: it re-feeds the exact error through the observation channel and records whether the next call changed.

The malformed call also shows why later tool-protocol work returns errors as structured results instead of raising into the session. A correction is only useful when it is attributable, visible in the history, and followed by a measurable change in behavior.

Reliability ledger

The Week 1 evidence is small by design. It establishes the protocol boundary and three repeatable failure probes; it does not claim that this loop is safe for an unattended repository.

Mechanism Assumption New failure exposed Evidence and current status
Raw completion supplied context is enough to ground the answer no persistent state, effect, or ground truth repeated stateless baseline; unresolved by design
One-shot parser one response and one valid call are enough tool-mediated episodes end without a final answer; malformed envelopes can crash 1/4 exact success; bounded only by the caller
Think-act-observe loop responses and tools remain usable within a turn budget thrashing and unknown or malformed calls termination, max-turn, and verbatim tests pass; protocol established
Observation channel a returned string is truthful silently lying tool is accepted as fact 4/8 scripted policies detect, 4/8 confabulate; semantic truth unresolved
Error-as-result correction the next model turn will use feedback repeated malformed calls 3/6 recover; 3/6 correction failures at the bound; measured, not solved

The relevant reliability terms are now observable. A wrong tool implementation is specification error; treating “no exception” as success would be proxy optimization; repeating a supplied error is correction failure; and the model’s ability to call a tool without a truth or permission guard is a small capability-control mismatch. The trajectory and its failed cases are more informative than a single success rate.

No project library was added this week. The plain functions stay in the notebook so that the later package mechanisms can be compared against a transparent baseline rather than silently replacing it.

Transport observability

The loop currently receives a complete JSON string in one synchronous step. A real provider may stream content deltas and tool-call deltas, split one call across many chunks, report usage separately, disconnect midway, or return a retryable status. If the client loses a delta, the loop’s invariant is already broken before it reaches a tool.

Week 2 therefore builds the streaming client first:

  • parse Server-Sent Events by hand and distinguish content from tool-call deltas;
  • stitch parallel calls by index without silently dropping fragments;
  • record reported usage and accumulated cost in a token ledger; and
  • retry only where the request policy makes that safe, with explicit timeout behavior.

The semantic loop remains the same: every assistant call receives one observation, and no observation is invented. Week 2 changes how a complete assistant message is transported and measured. Once that boundary is reliable, Week 3 can turn these hand-written function dictionaries into a typed tool protocol.

Back to top