Week 4: Coding Tools with Exact Contracts

A coding agent becomes consequential when a validated call can read and mutate a repository. This chapter exercises the existing file and shell tools inside a disposable workspace, then turns their edge cases into deterministic tests and perturbation experiments. The registry and error channel from Week 3 remain the boundary; Week 5 will carry these results through a multi-turn session.

Tool contracts

A useful coding tool makes its preconditions and postconditions inspectable. The current library exposes the following contracts:

Tool	Important precondition	Observable result
read_file	resolved path stays under cwd; input is a readable text file	line-numbered content and range metadata
write_file	resolved path stays under cwd	created or overwritten file plus a FileDiff
edit	old_string is exact and unique unless replace_all is true	replacement count, line delta, and diff
shell	command and requested cwd stay within the configured workspace	stdout, stderr, exit code, timeout metadata
glob / grep	search root stays under cwd	sorted paths or bounded matches

The package uses names such as read_file and write_file; a wire adapter can expose shorter aliases without weakening these local semantics. The examples use the package names directly so the tests exercise the code that the agent will actually call.

from __future__ import annotations

import asyncio
import os
import shutil
import sys
from pathlib import Path
from typing import Any

PROJECT_SRC = (Path.cwd() / "projects" / "agent-harness" / "src").resolve()
if not PROJECT_SRC.is_dir():
    raise RuntimeError(f"Expected the project source at {PROJECT_SRC}")
sys.path.insert(0, str(PROJECT_SRC))

from agent_harness.config import Config, ShellEnvironmentPolicy
from agent_harness.tools.base import Tool, ToolInvocation, ToolResult
from agent_harness.tools.files import (
    EditTool,
    GlobTool,
    GrepTool,
    ReadFileTool,
    WriteFileTool,
)
from agent_harness.tools.shell import ShellTool

Workspace setup

The tool receives a working directory with the same role as a capability root. Resolve relative and absolute paths, then reject anything whose resolved path is not a descendant. A sibling directory gives us a target for escape tests without touching a real repository.

scratch_root = (Path.cwd() / ".tmp").resolve()
workspace = scratch_root / "agent-harness-week4"
outside_dir = scratch_root / "agent-harness-week4-outside"

for path in (workspace, outside_dir):
    if path.is_dir() and not path.is_symlink():
        shutil.rmtree(path)
    elif path.exists() or path.is_symlink():
        path.unlink()

(workspace / "src").mkdir(parents=True)
(workspace / "src" / "app.py").write_text(
    "def add(a, b):\n"
    "    return a + b\n\n"
    "def main():\n"
    "    print(add(2, 3))\n",
    encoding="utf-8",
)
(workspace / "README.md").write_text("# Fixture\n\nreturn is mentioned here.\n", encoding="utf-8")
outside_dir.mkdir()
(outside_dir / "secret.txt").write_text("outside workspace\n", encoding="utf-8")

config = Config(cwd=workspace)
read_tool = ReadFileTool(config)
write_tool = WriteFileTool(config)
edit_tool = EditTool(config)
shell_tool = ShellTool(config)
glob_tool = GlobTool(config)
grep_tool = GrepTool(config)


async def run_tool(tool: Tool, params: dict[str, Any]) -> ToolResult:
    return await tool.execute(ToolInvocation(params=params, cwd=workspace))


print({
    "workspace": workspace.name,
    "sample_files": sorted(path.relative_to(workspace).as_posix() for path in workspace.rglob("*")),
})
assert (workspace / "src" / "app.py").is_file()

File operations

read_file numbers lines from one and accepts an offset and limit. That convention makes a model’s later edit anchors attributable: the model can cite the exact text it observed. write_file reports whether it created a file and returns a FileDiff, so a caller can preview a mutation before approval.

created = await run_tool(
    write_tool,
    {"path": "notes/session.txt", "content": "first\nsecond\nthird\n"},
)
window = await run_tool(
    read_tool,
    {"path": "notes/session.txt", "offset": 2, "limit": 2},
)

print({
    "created": created.success,
    "created_line_count": created.metadata["lines"],
    "read_window": window.output,
    "diff_available": created.diff is not None,
})
assert created.success is True
assert created.metadata["is_new_file"] is True
assert "     2|second" in window.output
assert "     3|third" in window.output
assert window.metadata["shown_start"] == 2

The adversarial cases are intentionally mundane. A binary file, a directory, a missing path, a symlink, and an absolute path all test different assumptions. The result should name the violated contract, while the workspace check must happen after symlink resolution rather than on the untrusted spelling.

binary_path = workspace / "image.bin"
binary_path.write_bytes(b"PNG\x00binary")
link_path = workspace / "linked"
if link_path.exists() or link_path.is_symlink():
    link_path.unlink()
link_path.symlink_to(outside_dir, target_is_directory=True)

binary_result = await run_tool(read_tool, {"path": "image.bin"})
directory_result = await run_tool(read_tool, {"path": "src"})
missing_result = await run_tool(read_tool, {"path": "missing.txt"})
absolute_result = await run_tool(read_tool, {"path": str(outside_dir / "secret.txt")})
symlink_result = await run_tool(read_tool, {"path": "linked/secret.txt"})

errors = {
    "binary": binary_result.error,
    "directory": directory_result.error,
    "missing": missing_result.error,
    "absolute_escape": absolute_result.error,
    "symlink_escape": symlink_result.error,
}
print(errors)
assert binary_result.success is False and "binary" in binary_result.error.lower()
assert directory_result.success is False and "Not a file" in directory_result.error
assert missing_result.success is False and "File not found" in missing_result.error
assert absolute_result.success is False and "escapes working directory" in absolute_result.error
assert symlink_result.success is False and "escapes working directory" in symlink_result.error
assert (outside_dir / "secret.txt").read_text(encoding="utf-8") == "outside workspace\n"

Exact-match editing

An edit is safe to retry only when the anchor is known. The library counts occurrences before writing: zero matches and multiple matches return errors; one match produces a diff. replace_all=True is an explicit change of contract, not a fallback the model receives silently.

repeat_path = workspace / "repeat.txt"
repeat_path.write_text("same\nsame\n", encoding="utf-8")
ambiguous = await run_tool(
    edit_tool,
    {"path": "repeat.txt", "old_string": "same", "new_string": "changed"},
)
changed = await run_tool(
    edit_tool,
    {
        "path": "src/app.py",
        "old_string": "return a + b",
        "new_string": "return a - b",
    },
)
missing_anchor = await run_tool(
    edit_tool,
    {"path": "src/app.py", "old_string": "return a * b", "new_string": "return a / b"},
)

normalized_diff = changed.diff.to_diff().replace(str(workspace), "<workspace>")
print({
    "ambiguous_error": ambiguous.error,
    "changed": changed.success,
    "changed_metadata": changed.metadata,
    "diff": normalized_diff,
    "missing_anchor_error": missing_anchor.error,
})
assert ambiguous.success is False and "found 2 times" in ambiguous.error
assert changed.success is True and changed.metadata["replaced_count"] == 1
assert "-    return a + b" in normalized_diff
assert "+    return a - b" in normalized_diff
assert missing_anchor.success is False and "not found" in missing_anchor.error

Shell execution

The shell tool pins cwd, starts a new process group, captures stdout and stderr, surfaces the exit code, and kills a timed-out process. Its environment policy removes inherited names matching *KEY*, *TOKEN*, and *SECRET* unless configured otherwise. The blocklist is a baseline check, not a complete command classifier; later hardening will need to reason about compound commands.

failed_command = await run_tool(
    shell_tool,
    {"command": "printf out; printf err >&2; exit 7"},
)
blocked_command = await run_tool(shell_tool, {"command": "rm -rf /"})
timed_out = await run_tool(shell_tool, {"command": "sleep 2", "timeout": 1})
outside_cwd = await run_tool(shell_tool, {"command": "pwd", "cwd": ".."})

print({
    "failed": {
        "success": failed_command.success,
        "exit_code": failed_command.exit_code,
        "output": failed_command.output,
    },
    "blocked": {"success": blocked_command.success, "blocked": blocked_command.metadata.get("blocked")},
    "timed_out": {"success": timed_out.success, "timed_out": timed_out.metadata.get("timed_out")},
    "outside_cwd": {"success": outside_cwd.success, "error": outside_cwd.error},
})
assert failed_command.success is False and failed_command.exit_code == 7
assert "--- stderr ---" in failed_command.output and "Exit code: 7" in failed_command.output
assert blocked_command.success is False and blocked_command.metadata["blocked"] is True
assert timed_out.success is False and timed_out.metadata["timed_out"] is True
assert outside_cwd.success is False and "escapes working directory" in outside_cwd.error

Search tools complete the read side of the contract. glob sorts its results and caps the count; grep returns path, line number, and text while capping matches. Stable output matters because the next model turn may use a returned path as an edit anchor.

python_files = await run_tool(glob_tool, {"pattern": "*.py", "path": "src"})
return_matches = await run_tool(
    grep_tool,
    {"pattern": "return", "path": "src", "include": "*.py"},
)
print({
    "glob": python_files.output,
    "grep": return_matches.output,
    "glob_metadata": python_files.metadata,
    "grep_metadata": return_matches.metadata,
})
assert python_files.success and python_files.output == "src/app.py"
assert return_matches.success and "src/app.py:2:" in return_matches.output

Output truncation

Long output can consume the context window and hide the observation that the model needed. The current shell implementation caps output at 100 KiB and adds a marker, but it does not set the ToolResult.truncated flag. That mismatch is a useful regression observation: consumers must inspect both the marker and metadata until the contract is repaired. A head-and-tail policy is often more useful than a prefix-only policy, because compilers put diagnostics near the end.

def preserve_head_tail(text: str, limit: int) -> str:
    marker = "\n... [middle truncated] ...\n"
    if len(text) <= limit:
        return text
    if limit <= len(marker):
        return marker[:limit]
    available = limit - len(marker)
    head = available // 2
    tail = available - head
    return text[:head] + marker + text[-tail:]


long_text = "BEGIN\n" + ("x" * 5_000) + "\nEND"
head_tail_samples = {
    limit: preserve_head_tail(long_text, limit)
    for limit in (32, 128)
}
actual_capped = await run_tool(
    shell_tool,
    {"command": "python3 -c 'print(\"x\" * 100500)'"},
)
print({
    "head_tail_lengths": {limit: len(value) for limit, value in head_tail_samples.items()},
    "keeps_beginning": head_tail_samples[32].startswith("BEGIN"),
    "keeps_end": head_tail_samples[32].endswith("END"),
    "library_marker_seen": "... [output truncated]" in actual_capped.output,
    "library_truncated_flag": actual_capped.truncated,
})
assert len(head_tail_samples[32]) == 32
assert head_tail_samples[32].startswith("BEGIN") and head_tail_samples[32].endswith("END")
assert "... [output truncated]" in actual_capped.output
assert actual_capped.truncated is False  # an explicit contract gap to fix later

Edit failures and retries

The following experiment uses four small perturbation families. Only the exact anchor succeeds. Indentation shifts, duplicated snippets, and near-miss whitespace return informative errors rather than making a guessed mutation. In a real evaluation, compare this harness failure rate with a model that was given the same observations; otherwise a specification error can be misreported as model incompetence.

perturbations = {
    "exact": ("value = 1\n", "value = 1"),
    "indentation_shift": ("    value = 1\n", "value = 1"),
    "duplicate": ("value = 1\nvalue = 1\n", "value = 1"),
    "near_miss": ("value=1\n", "value = 1"),
}
outcomes: dict[str, bool] = {}
for name, (content, old_string) in perturbations.items():
    path = workspace / f"perturb-{name}.txt"
    path.write_text(content, encoding="utf-8")
    result = await run_tool(
        edit_tool,
        {"path": path.name, "old_string": old_string, "new_string": "value = 2"},
    )
    outcomes[name] = result.success

print(outcomes)
assert outcomes == {
    "exact": True,
    "indentation_shift": False,
    "duplicate": False,
    "near_miss": False,
}
assert sum(outcomes.values()) == 1

The file and shell layers now expose enough evidence for the loop to make a responsible next decision: content with line numbers, a diff for mutations, exit status and stderr for processes, bounded search results, and explicit escape or timeout errors. The remaining reliability problem is capability-control mismatch. This week confines ordinary operations, but its substring blocklist is not a security proof; Week 8 will classify compound commands and add stronger confinement.

Week 3 defined the error-as-result boundary. Week 5 will test whether a model receives these errors faithfully, recovers, and stops when its turn budget is exhausted.