flowchart LR
request["User task"] --> loop["Agent and Session"]
loop --> client["LLMClient"]
client --> endpoint["OpenAI-compatible endpoint"]
endpoint --> client
loop --> registry["ToolRegistry"]
registry --> tools["File, shell, memory, task, and MCP tools"]
tools --> observation["Results and structured errors"]
observation --> loop
loop --> trajectory["Events and trajectory"]
trajectory --> judge["Judge"]
judge --> scorecard["Scorecard"]
policy["Config, context, safety, sandbox, hooks"] -. controls .-> loop
Architecture Overview and Evidence Contract
agents
engineering
reliability
Define the harness boundary, artifact lineage, execution paths, and evidence standard for the course.
A coding agent is not the model endpoint alone. It is a controlled episode in which a model proposes the next message, the harness decides whether a proposed action is well-formed and permitted, a tool produces an observation, and the observation returns to the next model turn. This course studies that boundary as a software system.
The running project is the headless async package in projects/agent-harness. Its thesis is deliberately modest: a useful repository agent is a loop surrounded by contracts. Streaming, tool schemas, session state, context limits, instructions, permissions, sandboxing, hooks, delegation, external protocols, and evaluation each add capability, but each also creates a new way for the system to fail. The course is complete only when those failure modes are observable in a repeatable offline run.
Harness boundary
The course owns the code that turns model messages into a governed episode:
| Outside the harness | Owned by the harness |
|---|---|
| Model weights, sampling, and provider availability | Message history, stream assembly, retries, token accounting, and terminal errors |
| The host operating system and process semantics | Tool schemas, registry lookup, argument validation, result normalization, and dispatch |
| Files, repositories, and subprocesses | Workspace path checks, shell-environment policy, approval decisions, and sandbox selection |
| External Model Context Protocol (MCP) servers and their data | Schema adaptation, transport handling, event visibility, and server-error propagation |
| A task author’s notion of success | Trajectory storage, deterministic checks, judging, aggregation, and scorecard serialization |
The implementation does not pretend to own what it cannot guarantee. A Docker backend can provide a stronger process boundary than host execution, but its security depends on the container runtime and deployment. An MCP adapter can preserve a typed boundary, but it cannot make an untrusted server trustworthy. A judge can apply explicit checks, but it cannot turn an underspecified task into ground truth.
Episode layers
The core path is a repeated state transition: preserve the conversation, request a model response, dispatch each valid tool call, append exactly one observation per call, and continue until a final answer or an explicit budget boundary. The event stream records the same path at a level suitable for evaluation.
The episode invariant introduced in Chapter 01 is structural rather than magical: every assistant turn either finishes with text or contains well-formed tool calls; every tool call receives exactly one tool observation; and the harness does not fabricate an observation. Later layers add policy and evidence without changing that basic contract.
The public shape is intentionally small:
from pathlib import Path
from agent_harness import Agent, Config
config = Config(cwd=Path(".tmp/agent-workspace"))
agent = Agent(config)
async for event in agent.run("Inspect the repository and make the requested change."):
record(event)The model decides what to request. The harness decides what can run, records what did run, and exposes the result to the next turn and to the evaluator.
Artifact lineage
The package grows in the same order as the notebooks. Each stage preserves an interface that later stages import rather than reimplement.
| Stage | Reusable objects | What the stage makes observable |
|---|---|---|
| Transport | LLMClient, StreamEvent, TokenUsage |
Partial text, streamed tool calls, usage, retries, and terminal failures |
| Tool boundary | Tool, ToolRegistry, ToolInvocation, ToolResult |
Name resolution, schema errors, dispatch, and normalized observations |
| Repository action | File and shell tools | Workspace paths, diffs, subprocess output, exit codes, and truncation |
| Agent loop | Session, Agent, EventBus |
Turn transitions, tool execution, final answers, budgets, and lifecycle events |
| Governance | ContextManager, ChatCompactor, Config, prompts, memory |
Context pressure, layered instructions, persistent state, and provenance |
| Safety | ApprovalManager, LoopDetector, sandbox backends |
Approval policy, repeated-command detection, host execution, and Docker execution |
| Extension | HookSystem, SubAgentTool, TaskTool, MCP adapters |
Lifecycle checks, bounded delegation, external schemas, and transport errors |
| Evidence | EvalTask, Trajectory, Judge, Scorecard |
Portable tasks, recorded episodes, independent criteria, and aggregate reports |
The lineage is cumulative but not a claim that every layer makes the agent safer in every setting. The purpose of the experiments is to identify the new guarantee, the new failure surface, and the evidence that distinguishes the two.
Execution paths
The default path is the local offline profile. It uses Python 3.14+, the repository environment, deterministic fake clients, disposable workspaces under .tmp/, and the real package classes. It requires no API key, GPU, network connection, or paid service. This is the source of the course’s repeatable evidence.
The live-provider profile swaps in an OpenAI-compatible endpoint for the client boundary. It requires a credential, network access, and usage budget, and model behavior is no longer deterministic. It is useful for seeing the transport and loop operate against a real service, but it is not required for the core path and does not replace the offline fixtures. Chapter 08 adds Docker as an optional sandbox backend; it is a local execution choice, not a third course profile.
Both paths share message roles, tool schemas, event types, configuration fields, and evaluation interfaces. A profile-specific difference belongs in configuration or in the recorded trajectory, not in an unannounced alternate implementation.
Evidence
An evaluation input is an EvalTask: a prompt plus optional repository or working directory, checks, expected text or files, test command, setup and teardown commands, category, tags, and turn or timeout limits. A Trajectory records the resulting events, response, tool activity, errors, usage, and metadata. This makes the unit of evidence larger than the final assistant string.
The default Judge is deterministic. It can check completion status, expected text, expected files, commands, declared metadata, event presence, tool calls, and small custom checks. An optional model-based criterion is explicit and additive, never silently mixed into the offline score. Scorecard then aggregates per-task results by pass rate, score, duration, turns, tool calls, and category.
A result is therefore stated as a tuple: task definition, harness configuration, model or fake-client behavior, trajectory, criteria, and scorecard. “The agent succeeded” is incomplete unless the task and the evidence that produced that verdict are available.
Build phases
| Phase | Chapter | Artifact or evidence contributed |
|---|---|---|
| Orientation | 00. Overview | Shared architecture, scope boundary, execution profiles, and acceptance contract |
| Part I: The Engine Exposed | 01. Model versus harness | Episode invariant and a measured stateless-versus-loop comparison |
| 02. Streaming client | SSE assembly, retries, usage ledger, and transport regression cases | |
| 03. Tool protocol | Typed tool registry, argument validation, and structured error channel | |
| Part II: Making It a Coding Agent | 04. Coding tools | Workspace-bounded file operations, shell execution, diffs, and edge-case tests |
| 05. Agent loop | Session state transitions, agent events, multi-call turns, and budget stops | |
| Part III: Controlling It | 06. Context management | Token-aware history operations, pruning, compaction, and continuation metadata |
| 07. Instructions and memory | Layered configuration, project instructions, and persistent memory tools | |
| 08. Permissions and sandboxing | Approval decisions, command-risk checks, loop detection, and sandbox backends | |
| 09. Hooks | Lifecycle triggers, subprocess hooks, failure containment, and observability | |
| Part IV: Scaling Out, Then Proving It | 10. Sub-agents and patterns | Role-specific delegation, bounded child sessions, and context isolation choices |
| 11. Model Context Protocol | MCP transport, schema adaptation, tool registration, and external-error handling | |
| 12. Capstone | Offline integration fixture, planted-control failures, trajectory judging, and scorecard report |
Acceptance and non-goals
The course’s final claim is narrow: on deterministic fixtures, the offline harness satisfies the declared contracts, and removing the corresponding control makes at least some planted failures visible.
The capstone accepts the build when:
- the real package completes the scripted repository task and leaves inspectable agent and tool events;
- tool calls, observations, errors, usage, context operations, policy decisions, and external-boundary failures remain distinguishable in the evidence;
- the judge scores the trajectory from declared task criteria rather than from an unexamined final string;
- the scorecard can be serialized, reloaded, and compared across variants; and
- the report names limitations, including boundaries that are evaluated separately rather than enforced automatically by the current loop.
This course does not claim model intelligence, provider reliability, production-grade sandbox security, complete parity with Claude Code, a user interface or CLI, or a benchmark result on a broad task distribution. It also does not train model weights. Those are different experiments with different evidence requirements.