Operations completes the user path. A browser needs an actionable degraded state, an operator needs bounded metrics and dependency probes, and a support bundle must explain failure without exporting tokens or private code. This chapter connects those concerns to the running FastAPI service, SQLite and artifact directories, structured redaction, and a restore drill.
A token can appear in an exception, an HTTP header, an environment dump, or a copied request. Scrub at the logging and bundle boundary, not only at call sites. The scrubber should be tested with seeded secrets and strings that resemble real credentials.
from autocode.ops.logging import SecretScrubber
scrubber = SecretScrubber(["sk-test-secret", "refresh-token"])
raw = "request failed token=sk-test-secret; refresh-token=refresh-token"
clean = scrubber.scrub(raw)
assert "sk-test-secret" not in clean
assert "refresh-token" not in clean
assert clean.count("[REDACTED]") == 3
print(clean)request failed token=[REDACTED]; [REDACTED]=[REDACTED]A fuzzed regression should generate variants of key names, token lengths, JSON nesting, and exception messages. The invariant is simple: seeded secret values never appear in emitted logs or bundles. Redaction can over-match, so inspect usability as a second metric.
/healthz asks whether the process can answer HTTP. /readyz probes whether the session repository and artifact store can serve work. /metrics exposes low-cardinality counters such as sessions created, runs started, and artifacts uploaded. Session ids, raw paths, prompts, and error strings never become metric labels.
The browser should translate a failed request or disconnected socket into a visible offline or degraded state. It should not replace an unavailable API with an empty session list, because that makes dependency failure look like data loss.
from tempfile import TemporaryDirectory
from fastapi.testclient import TestClient
from autocode.runner import DemoAgentRunner
from autocode_service.api import create_app
with TemporaryDirectory() as directory:
app = create_app(database_path=f"{directory}/sessions.db", runner=DemoAgentRunner())
with TestClient(app) as client:
health = client.get("/healthz").json()
ready = client.get("/readyz").json()
client.post("/api/sessions", json={"title": "operations"})
metrics = client.get("/metrics").text
assert health == {"status": "ok"}
assert ready["ready"] is True
assert ready["dependencies"] == {"database": True, "artifacts": True}
assert "sessions_created 1" in metrics
print("health:", health, "readiness:", ready, "metrics:", metrics.strip())health: {'status': 'ok'} readiness: {'ready': True, 'dependencies': {'database': True, 'artifacts': True}} metrics: sessions_created 1The counters identify request volume without exposing a user’s session. Readiness can fail while health remains green, which tells a process supervisor that restart may not repair an unavailable dependency. A deployment should add latency histograms, active WebSockets, queue depth, replay count, and error classes with bounded labels.
A backup is a claim that a future restore will recover coherent browser-visible state. Stop or coordinate writes, copy the SQLite database, journal, and artifact directory, restore into a different location, start the service against that location, and compare session ids, event ids, cursors, and artifact digests through the API. Never overwrite the source during a drill.
from tempfile import TemporaryDirectory
from pathlib import Path
from autocode.ops.backup import backup_directory, restore_directory
with TemporaryDirectory() as directory:
source = Path(directory) / "data"
source.mkdir()
(source / "sessions.db").write_text("session-state")
backup = backup_directory(source, Path(directory) / "backup")
restored = restore_directory(backup, Path(directory) / "restored")
assert (restored / "sessions.db").read_text() == "session-state"
print("restored:", restored)restored: /var/folders/jq/9vsvd9252_349lsng_5gc_jw0000gn/T/tmpt__8z4ec/restoredThe copy helper supports the local drill; production backups need snapshot consistency, encryption, retention, access control, and scheduled restore probes. After restore, load the application through REST and WebSocket replay rather than only checking that files exist. The capstone records both the backup artifact and the browser-visible recovery evidence.
Seed a debug bundle with a token, a private path, and an exception. Add a scrubber test that proves the token is absent from every archive member, then document which fields are intentionally retained for diagnosis and which are dropped.
Choose three fields to retain and three to drop from a support bundle. Add the redaction invariant and a restore check.