Raj's Notes

Harness engineering

Agent contracts, context, MCP and tools, durable state, verification, safety, orchestration, evaluation, and maintenance.

Scope#

How to turn an ambiguous request into a bounded run, connect tools and context through explicit protocol boundaries, preserve state across failures, collect proof, evaluate behavior, and release changes with safety and cost controls.

The sequence defines the model, host runtime, and task contract before it introduces context assembly, MCP, durable state, verification, production authority, orchestration, evaluation, and maintenance. Bundled examples use scripted models and local state, so their programs and focused tests run without a model key or access to another repository. Design traces that are not implemented by those examples say so explicitly.

Reading path#

Note Why it comes here
HE1: Model to harness Separates model proposals, the agent loop, deterministic workflows, host execution, and harness controls.
HE2: Executable intent Defines success, scope, authority, evidence, and stopping before the run receives consequential tools.
HE3: Context architecture Builds one attributable model input from policy, task facts, selected evidence, skills, and working state.
HE4: Tools, MCP, sandboxes Connects proposals to typed capabilities, protocol boundaries, executor identity, isolation, and safe effects.
HE5: State and handoffs Preserves progress, ownership, artifacts, and uncertain effects across processes and failures.
HE6: Verification loops Binds independent evidence to one candidate and uses failed checks to drive bounded repair.
HE7: Safety controls Applies least privilege, approvals, tenant boundaries, containment, and operator stops to the full path.
HE8: Orchestration Splits work among bounded roles only after one run's contracts, state, proof, and safety are explicit.
HE9: Agent evals Measures the resulting system with representative tasks, layered graders, repeated trials, and attack cases.
HE10: Measured maintenance Versions, deploys, observes, rolls back, and eventually retires the complete harness.

Useful background#

AI inference infrastructure is optional background for how a model call is served. This collection treats the model call as one dependency and begins independently with the model-agent-host boundary.

Notes

  1. Model, agent, harness, and failure taxonomy Separate model behavior from the agent loop and its host runtime, then classify failures at the layer that can actually fix them.
  2. Intent and executable contracts Translate a human request into testable outcomes, typed actions, evidence requirements, and stop conditions before the run spends authority.
  3. Context architecture and AGENTS.md Build a context hierarchy that keeps durable instructions, current task facts, retrieved evidence, and tool results distinct and attributable.
  4. Tools, MCP, environments, and sandboxes Treat every tool and MCP server as a capability boundary, then make schema, transport, retry, permission, and isolation semantics explicit.
  5. Durable state, continuity, and handoffs Preserve progress across turns, workers, crashes, and ownership changes without replaying side effects or trusting an opaque transcript.
  6. Verification and feedback loops Turn tool output into evidence, run the cheapest decisive checks first, and bound repair attempts when a candidate fails.
  7. Production safety and control Threat-model the full agent path, apply least privilege, contain prompt injection, and give operators audit, budget, and emergency controls.
  8. Agent orchestration Assign planning, work, and evaluation roles with explicit dependencies, controlled parallelism, and stopping rules.
  9. Evaluation engineering Build representative task suites, layered graders, and trace-based diagnostics that can detect regressions without rewarding shortcuts.
  10. Maintenance and measured change Version the moving parts, improve one real task against a baseline, and operate the harness with before-and-after evidence and a rollback path.