Maintenance and measured change

Working model#

A harness is a changing service, not a finished prompt. Pin what ran, compare one controlled change with a baseline, release behind measurable gates, and keep the old path available until the new one earns trust.

Version the effective system, not only the prompt#

Behavior can change when the model snapshot, sampling settings, instruction chain, tool schema, tool implementation, context retrieval, middleware order, sandbox image, policy, or grader changes. Remote interfaces belong in that list too: an MCP server can negotiate a new protocol version, change its capability catalog, or alter a prompt, resource, and tool definition without a local deploy. Emit an effective configuration manifest with each run so a regression can be tied to the system that produced it.

Treat changes as migrations. Define compatibility for saved checkpoints and handoffs, keep old workflow code available while durable runs may resume, and add version adapters only where their behavior can be tested. Remove a version after its active work is drained and its replay or audit obligation has expired.

Versioning rule. Middleware order determines which attempts a retry counter, usage recorder, transcript check, or error adapter observes. Store that order beside the model, tool, policy, and grader versions so a changed trace can be attributed to the system that produced it.

Change middleware order without losing attribution#

Suppose harness version h17 runs usage accounting outside transient retry, while candidate h18 moves it inside. The same underlying model attempt may now contribute to a different metric count even when user-visible output stays unchanged. The effective manifest must record middleware names and order, model settings, tool and policy versions, context assembler, runtime image, and grader bundle for both candidates.

Run the same versioned cases against h17 and h18, then inspect both accepted outcomes and intermediate spans. If saved workflows can resume across the change, replay tests must cover an h17 checkpoint continuing under the supported path. If the ordering changes persisted behavior, pin active h17 runs to their code or provide a tested patch path. Release notes should state the changed observation semantics instead of describing the move as internal cleanup.

The manifest identifies the system that produced a trace; names alone are placeholders for immutable versions.

harness: h18
middleware_order: [loop_guard, transient_retry, usage, message_repair]
model_config: model-v5
tool_bundle: tools-v12
mcp_protocol: 2025-11-25
mcp_catalog_digest: sha256:catalog-digest
policy: policy-v9
runtime_image: sha256:runtime-digest
grader_bundle: graders-v6
telemetry_schema: gen-ai-schema-version

Measure the same contract before and after one change#

Choose a task family with a known defect or cost. Capture a baseline over the same case versions, environments, permissions, and graders you will use for the candidate. Change one main variable, repeat enough runs to see variance, and compare task success, hard-gate failures, intervention, latency, tokens, and cost per accepted task.

Do not invent improvement numbers or compare unmatched runs. Attach raw run identifiers and report uncertainty. If the candidate improves median latency by skipping a required check, its accepted-task cost and safety score should expose the trade rather than rewarding the shortcut.

• Quality: accepted tasks and failure severity by task family.
• Safety: denied actions, policy breaches, secret exposure, and uncertain effects.
• Efficiency: latency distribution, tokens, tool calls, and cost per accepted task.
• Operations: retries, no-progress stops, human interventions, and recovery success.

Roll out with an owner, a stop rule, and a reversible path#

Start with shadow or read-only runs when the task permits, then a small authorized slice with close trace review. Define promotion gates, maximum exposure, alert ownership, and automatic rollback conditions before rollout. Keep an operator control that can disable the new version without waiting for the model or worker pool.

A maintenance record should name who owns the task suite, tool contracts, policy, environment image, and incident response. Review stale tools and permissions, rotate credentials, refresh cases from incidents, and retire instructions that no longer match the code. Every incident fix should leave behind a test, control, or explicit reason that prevents the finding from disappearing.

Operational readiness. If nobody can say which version is active, which cases guard it, and how to turn it off, the system is not ready for unattended effects.

Assemble the production topology before tuning a worker#

The public examples in this collection run in one local process so each contract stays visible. A production service separates durable authority from replaceable execution. Draw that split before choosing a queue or agent framework.

Keep one identifier chain across those boundaries. A run ID groups the user's contract, an attempt ID names one leased execution, a tool-call ID names one proposal, and an operation ID remains stable across retries of the same intended effect. Receipts point to artifact digests and external resource versions. Trace IDs help debugging, but they don't replace any of these durable identities.

client -> Run API -> run store
                    |
                    v
              scheduler -> queue -> worker lease
                                   |       |
                                   |       +-> model gateway
                                   |       +-> tool broker -> external service
                                   |       +-> sandbox pool -> artifact store
                                   |       +-> verifier
                                   |
                                   +-> versioned checkpoint and receipts

operator stop -> admission gate + scheduler + active leases + tool broker
all components -> traces, metrics, logs, and audit events

Worked interactive platform: keep the control service outside the sandbox#

An interactive coding platform often has a trusted control service and a less-trusted runner. The control service accepts provider events, owns run state, executes integration hooks, selects policy, acquires a sandbox, and handles callbacks. The runner executes the model and tool loop inside the sandbox.

This concrete variant is different from the durable queue-and-lease topology above. It dispatches a run directly to one sandbox, while follow-up delivery, output streaming, post-hooks, and failure hooks use in-process background tasks. The later queue, lease, broker, and verifier trace is a stronger design pattern, not a claim that every component already exists in this interactive path.

provider webhook
  -> public gateway and signature verifier
  -> privileged control service
       -> create WebhookEvent audit row
       -> atomically insert pending AgentRun
       -> link event to the accepted run
       -> await pre-hook
       -> claim a warm sandbox
       -> issue selected run configuration and short-lived credential
       -> route active follow-up prompts
       -> post-hook or failure hook
  -> isolated runner
       -> model and tool loop
       -> workspace and scoped repository access

Pre-hooks, post-hooks, and failure hooks stay in the control service when they need provider bot tokens, workflow-tracking credentials, or clients with broader access than the code-running sandbox should have. The sandbox receives only the selected tier configuration, per-flow overrides, and credentials narrowed to that run, such as a short-lived repository token.

This makes the control service a privileged security boundary. It must authenticate runner start, completion, audit, and prompt callbacks; validate every sandbox-supplied identifier and event; keep hook effects idempotent; and avoid copying long-lived credentials into the run payload, model context, transcript, or workspace. Moving a secret out of the sandbox reduces sandbox blast radius, but it does not reduce the control service's responsibility.

The pre-hook is awaited after the run row is admitted and before sandbox dispatch. Post-hooks and failure hooks are spawned as in-process tasks. A control-service restart can therefore lose a post-hook and strand a row in post_hooks_running. Failure handling persists failed before spawning its failure hook, so failed proves the run status was written but does not prove the failure hook completed. There is no separate failure_hooks_running phase in this state machine. Durable hook execution would need a queue, outbox, or recoverable job record rather than asyncio.create_task alone.

The run state machine separates model execution from integration cleanup#

An enum value does not prove the implementation reaches it. If aborted exists but no transition writes it, treat it as reserved rather than an operational state until code and tests establish its entry rules.

Suppose one external conversation may have only one model turn executing at a time. A read-then-insert sequence is unsafe:

request A: query active run -> none
request B: query active run -> none
request A: insert
request B: insert

Both reads are valid because neither insert existed yet. Put the invariant at the database write boundary. In generic pseudocode:

CREATE UNIQUE INDEX one_active_turn_per_key
ON runs (idempotency_key)
WHERE idempotency_key IS NOT NULL
  AND phase IN ('pending', 'dispatched', 'agent_running');

The partial unique index covers only rows whose phase makes them active. Concurrent insertions race inside the database, one wins, and the other receives a uniqueness conflict that the API translates into “already active.” An application-level precheck can improve the error path but cannot replace the constraint.

The flow defines the key's scope. One flow may key by provider session, another by issue, and another by pull-request head revision and attempt. A flow can return no key and opt out of this database uniqueness rule. PostgreSQL already treats nulls as distinct by default; the IS NOT NULL predicate also keeps unkeyed rows out of the partial index and documents the opt-out.

Excluding post_hooks_running is deliberate. Once the model and tools have stopped, a new user continuation may acquire another sandbox while the previous run finishes provider updates. That overlap is safe only when post-hooks can no longer mutate the old sandbox and their provider writes have explicit ordering or idempotency. Otherwise the state boundary permits two actors to change the same logical result.

The webhook audit event and run admission can also have different lifetimes. If each received delivery is recorded before the run insert, the losing request may leave an audit row without its own run. That row is legitimate evidence that another delivery arrived, but it should be marked deduplicated or linked to the winning run. An unexplained row looks orphaned even when the database correctly prevented duplicate execution.

Stable provider identity routes to an ephemeral sandbox#

Keep the routing chain explicit:

external conversation or issue ID
  -> durable session and run record
  -> active attempt
  -> current sandbox name
  -> current Pod and runner prompt endpoint

The application database owns the durable part. It can retain the external session ID in structured trigger data or a dedicated column and record the sandbox name when dispatch or claim succeeds. Prompt delivery constructs a per-sandbox Service name such as sandbox-7c2.<namespace>.svc.cluster.local; Kubernetes DNS resolves that Service, and its selector routes traffic to the selected runner Pod. The stored sandbox name is therefore an application-to-Service mapping, not a direct name-to-Pod lookup. Do not make a Pod IP or an in-memory process map the only copy of this relationship.

One possible start handshake is:

POST /runs/R204/start
X-Sandbox-Name: sandbox-7c2

The header is a claimed execution identity, not proof by itself. Comparing it with the sandbox name already stored on the run binds the callback to an expected string and prevents an accidental mismatch. It does not stop a compromised sandbox that learns another sandbox's name from copying that value.

A stronger design also binds the request to a source identity, for example authenticated workload identity, mutual TLS, a run-scoped token, or a verified source Pod address under strict network policy. After that check, the control service changes dispatched to agent_running through a guarded transition and returns the selected run configuration. A caller must not be able to impersonate another tenant's sandbox by choosing a header value.

An active follow-up locates the run whose phase still blocks another turn and does not create a second run. The control service then spawns an in-process delivery task. That task reloads the run, waits up to three minutes while it remains pending, dispatched, or agent_running, and retries until a sandbox name and prompt endpoint are available. It finally posts the text to the runner's /prompt endpoint.

Those are three different facts: the active-run lookup found a target, the background task attempted delivery, and the runner accepted the HTTP request. Only the last proves prompt acceptance. A control-service restart can drop the background task because the prompt is not first written to a durable delivery queue. After model execution ends, a later continuation creates a new run and sandbox, restores the selected main transcript, and records a new attempt. Conversation identity remains stable while execution, Service, Pod, and volume identities change.

The concrete Linear path has no DynamoDB issue-to-run mapping. It searches the Postgres AgentRun.trigger_payload JSON for linear_agent_session_input.id; the code records a dedicated indexed column as future work. DynamoDB is available to individual flow hooks for other tracking state, such as CI classification and thread metadata, but it does not route these active Linear prompts.

A separate key-value issue mapping would be a valid alternative when provider payloads do not carry an internal session key. Such a table could map provider, tenant, issue ID, and installation to an internal session or flow. It should remain a routing index, not a second source of truth for run phase, sandbox ownership, or external-effect completion.

Normal trace: one patch run reaches a verified terminal state#

Suppose POST /runs carries caller key fix-842 and a contract that permits a patch only inside packages/parser. The API stores run R204 at checkpoint version 0 before returning 202 Accepted; the response points to status, event, and cancellation endpoints.

The scheduler turns the accepted run into unit U1, writes that decision, then enqueues its identifier. Worker W7 claims U1 under lease epoch 11 and records attempt A3. It assembles context from the pinned checkout, calls the selected model through the gateway, and receives a proposed file read. The tool broker validates the arguments and policy before the sandbox reads anything.

Later, the broker accepts a patch call scoped to the same workspace. The sandbox applies it against base digest B6; the artifact store records patch digest P9, while the broker stores receipt T14. The verifier checks P9 with the required parser test and type check, writes report digest V4, and returns a structured pass. W7 advances the run through a compare-and-set from checkpoint version 6 to 7, linking P9, V4, the effective harness manifest, and the accepted terminal claim. Only then does it acknowledge the queue delivery. The status API reads the run store; it doesn't infer completion from an empty queue or a worker log.

If the queue redelivers U1 after that acknowledgement, a worker sees the terminal checkpoint and exits without repeating the patch or publishing a second result. Queue delivery and effect deduplication remain separate contracts.

Ordered asynchronous output is not automatically durable#

A model loop should not wait on a provider activity API after every useful thought or tool event. It can select user-relevant events, append them to one per-run queue, and let a background sender deliver them in order.

model stream
  -> select useful status events
  -> in-memory ordered queue
  -> one background sender
  -> external activity API

final response -> durable run store
main transcript -> transcript or audit store

One sender preserves order without adding provider latency to every model iteration. The selected status event becomes durable after the external activity API accepts it. Before that acknowledgement, an in-memory queue is volatile; process loss can drop unsent events. This is asynchronous delivery, not a durable outbox.

The final response and full transcript have separate persistence paths. Filtering noisy tool output from the provider's status feed does not imply deleting it from the model transcript or audit record. Conversely, saving a final response does not prove every earlier status event reached the provider.

In the concrete interactive implementation, transport errors, non-200 responses, malformed JSON, and provider-declared failures are logged and the queue item is marked done. There is no retry or durable delivery receipt. Normal runner shutdown waits for the queue to drain, so provider latency is removed from individual model iterations but can still extend shutdown. Only activities accepted by the provider are durable there; a local “done” item means the sender finished handling it, not that the provider stored it.

If progress events must survive restart, first write them to a durable outbox in the same transaction as the run transition, then let a sender claim and acknowledge them. Another option is to reconstruct status from a durable event log. For low-value progress hints, accepting possible loss can be a reasonable latency and complexity choice, but the contract should say so.

Finalize-time transcript upload leaves a crash window#

One transcript adapter in this interactive path buffers main-agent and subagent JSONL entries in runner memory. It writes the main object and separate subagent objects to S3 once during finalize(). Upload errors are logged and deliberately do not fail the run.

That is an audit mirror with a large crash window, not a streaming journal. If the runner dies before finalize(), the entire current transcript can be absent; there is no partial object from which to continue. If a later resume cannot find the prior main transcript, it starts cold. The final response stored with the run row and the transcript object in S3 have separate durability, so either can exist without proving the other was saved.

Reducing the window requires incremental immutable segment objects plus a durable head, followed by a finalization marker. An incomplete S3 multipart upload is not a readable transcript object and does not close this crash window. The resume path must define whether it accepts a partial transcript and how it validates ordering and completeness.

Worker-loss trace: reconcile before another effect#

Now let W7 die after the broker commits operation R204:publish-draft but before checkpoint version 6 records the receipt. Its heartbeat stops, yet the run doesn't become safe to replay merely because the process vanished. After the lease deadline, the scheduler grants epoch 12 to W9; epoch 11 can no longer advance the run.

W9 loads checkpoint version 5 and finds the started operation ID with no recorded outcome. It asks the tool broker to reconcile that ID. If the forge reports draft 913 with the expected artifact digest, the broker stores a reconciled receipt and W9 advances to verification. If the forge reports no matching operation and supports idempotent create under the same key, the broker may retry with that key. If neither fact can be proved, the run enters needs_operator instead of guessing.

Suppose W7 wakes after epoch 12 exists. Its checkpoint compare-and-set includes epoch 11 and fails, even if its local memory says the publish succeeded. A trace for the stale attempt remains useful evidence; it doesn't regain ownership. The queue delivery may happen again, but current checkpoint version, fencing epoch, and effect receipt decide the next safe action.

Stop path: cancellation reaches every source of new work#

An operator stop has to do more than hide the submit button. First, the API closes admission for the affected harness version and stores the stop reason. The scheduler stops creating and leasing units, active workers observe cancellation through the run store, and the model gateway rejects new generations for those runs. The tool broker blocks new effects while still allowing authorized reconciliation and cleanup. Sandboxes receive bounded termination. A run reaches clean cancelled only after current effects are classified as committed or absent; an unresolved effect keeps it in needs_operator or a distinct uncertainty terminal state.

Keep the old version disabled until its active leases end and its uncertain operations have owners. Killing every worker first can erase the only fast path to a clean drain; leaving the tool broker open can let a stale worker create one more effect after the operator thinks the system stopped.

This topology doesn't require separate deployables for every row. A small installation can combine several responsibilities in one service, but it should retain separate state, identities, authorization checks, and evidence at the boundaries above.

A public minimal harness ties the contracts together#

The dependency-free minimal-harness.mjs is the baseline for one measured change. It already supplies a normalized streaming model adapter, two tools, manual schema validation, tenant-scoped authorization, execution receipts, JSON checkpoints, a step budget, uncertain-effect reconciliation, and deterministic verification. No model key, external repository, or network call is needed.

The unchanged baseline runs from the notes repository root:

node notes/05-harness-engineering/examples/minimal-harness.mjs
node --test tests/public-harness.test.mjs

Its output records the causal order. call-read returns a payment error rate and an untrusted instruction. call-bad has valid argument types but fails authorization because it targets another tenant. call-write commits under a stable operation ID, loses its reply, and succeeds only after reconciliation finds the matching argument digest. The final text does not grant success; verification.checks does.

The sections below are a hypothetical candidate design, not output from the unchanged baseline. They show the failure and success traces an implementation would need to produce while preserving tenant policy, stable operation identity, uncertain-result reconciliation, and deterministic verification.

Before: the write has no concurrency precondition#

The baseline read_service result contains current service fields, while set_maintenance requires only tenantId, enabled, and operationId. The operation ID prevents a retry from repeating the same logical effect, but it does not say whether the state observed before the proposal is still current. Another actor can change the service after the read and before the write.

Expected failing trace: a stale decision still commits#

In the hypothetical failing fixture, observable service state has an integer version but writes do not yet enforce it. The model reads shop-42 at version 1. A scripted concurrent actor changes the same record to version 2. Because the old write contract carries no expected version, the proposed maintenance change still commits and advances the record to version 3.

read_service -> { tenantId: "shop-42", maintenance: false, version: 1 }
concurrent update -> { maintenance: false, version: 2 }
set_maintenance without expectedVersion -> committed at version 3

oracle:
  staleWriteRejected = false
  maintenanceUnchanged = false
  effectCount = 1
result: rejected candidate

The existing idempotency and tenant checks both pass in this trace. That is expected: they guard duplicate effects and authority, not stale observations. The failed oracle identifies the missing concurrency contract without weakening either existing control.

Change: bind a new operation to the observed version#

The candidate adds version to each service record and to read_service, then requires an integer expectedVersion in the set_maintenance schema. For a new operation ID, setMaintenance compares the stored and expected versions before mutation. A mismatch returns a structured conflict containing expected and actual versions and leaves the record unchanged. A match changes maintenance state and increments the version in the same store update.

Operation replay is checked before the current-version comparison. If a prior call with the same operation ID and argument digest already committed, a retry returns its recorded result even though that successful write advanced the version. Reusing the operation ID with a different digest still fails. The digest now includes expectedVersion, so reconciliation can distinguish the exact stale or current proposal that produced an uncertain response.

Expected outcomes: conflict, success, and replay remain distinct#

An implementation record would need the original baseline result, the stale-write failure above, a bounded source and test diff, conflict and successful-write receipts, the reconciled lost-response path, and final state-based verification. Validation establishes argument shape, authorization establishes tenant scope, optimistic concurrency protects the observed version, and operation identity protects retries; none substitutes for another.

Resumable approval adds a durable control boundary#

The next hypothetical change adds a human decision between an authorized proposal and execution. The bundled baseline does not implement this approval state. Its invariant is simple: set_maintenance must not execute until the host receives an unexpired approval bound to the proposed call ID and arguments digest.

Before: authorization permits immediate execution#

In the baseline, schema validation, tenant authorization, and the payment-error precondition lead directly to executeTool. Those checks prove that the call is well formed, in scope, and permitted by the standing task contract. They do not prove that a person reviewed this exact effect. In a hypothetical failure fixture, set_maintenance becomes approval-required, no decision is supplied, and the old host still calls the executor once. The candidate is rejected because executorCalls should remain zero while approval is unresolved.

proposal call-write -> schema valid -> tenant allowed -> threshold true
approval decision -> absent
baseline executorCalls -> 1
required executorCalls -> 0
result: rejected candidate

Change: store the reviewed intent before pausing#

The candidate marks set_maintenance as approval-gated in its tool definition. After schema and tenant checks pass, it stores a pending record containing the call ID, tool name, arguments digest, preview, and expiry, then returns a paused status without calling executeTool. A resume decision must match the saved record before execution starts.

Keep approval state separate from the operation receipt. The approval permits a call; the operation record and reconcileWrite still prevent a repeated effect after a lost response.

The pending approval becomes part of resumable state, so the candidate adds a checkpoint schema version. It rejects an unsupported version with a named terminal reason instead of guessing how to resume it. The round-trip evidence proves that approval survives serialization without carrying an executor function, credential, or mutable reference.

Expected fail-closed outcomes#

The expected approval matrix uses the same pending call and resets the executor counter before each resume. A missing decision remains paused. An explicit denial becomes terminal approval_denied. A decision for a changed arguments digest becomes approval_mismatch, and a decision received after expiresAt becomes approval_expired. All four paths leave executorCalls at zero and store a distinct reason.

Expected accepted trace: a fresh host resumes the exact approved call#

A correct accepted path would serialize a versioned checkpoint after the proposal, create a fresh host, load the checkpoint, and apply a decision that matches call-write, its arguments digest, and its expiry. The executor would run once. If its reply were lost after commit, the stable operation ID would still drive reconciliation; approval would not be requested again for the same recorded operation.

checkpoint v4:
  schemaVersion: 2
  status: awaiting_approval
  pending: { callId: call-write, argumentsDigest: d17, expiresAt: T+10m }

fresh host -> load checkpoint v4
decision -> { callId: call-write, argumentsDigest: d17, approved: true, decidedAt: T+2m }
executorCalls -> 1
write reply -> lost after commit
reconciliation -> confirmed_applied for the same operation ID and digest
verification -> accepted from authoritative state

In such an implementation, the saved approval record would prove which proposal a person allowed. Tenant authorization would still constrain who and what may change; optimistic version checking would still reject stale state; the operation record would still deduplicate or reconcile the write; and final verification would still read authoritative state. A verification record would need the pre-change failure, four fail-closed decisions, the checkpoint round trip, the single approved executor call, the reconciled receipt, and the final accepted state.

Retire old paths only after work and evidence have drained#

Keep an inventory of active run versions, resumable checkpoints, tool clients, policy bundles, sandbox images, grader versions, owners, and retirement dates. Before removal, confirm that no supported workflow can resume into the old code, required audit records remain readable, rollback no longer depends on it, and the replacement has passed its release gates. A calendar date alone does not prove those conditions.

Review changes by incident and accepted-task data. Track rollback rate, version skew, checkpoint migration failures, deprecated tool calls, stale permissions, eval drift, cost per accepted task, and time to disable a faulty version. When a metric moves, link it to the effective manifests and case traces before attributing cause. Keep the rollback simple enough to rehearse; an emergency path that requires editing prompts or rebuilding images during an incident is not ready.

Takeaway. Pin the whole effective harness, compare one controlled change, release behind prewritten gates, and remove the old path only after resumable work has cleared it.

Summary#

Maintain the harness as a versioned service whose behavior depends on more than the prompt. Compare a controlled candidate with a matching baseline, release through prewritten gates, and keep rollback viable until active work no longer depends on the old path.

• Record an effective manifest containing model settings, instructions, context assembly, tools, middleware order, runtime image, policy, and graders. Include negotiated MCP and workflow compatibility versions plus the telemetry schema.
• Compare baseline and candidate on the same cases, environments, permissions, budgets, and acceptance rules.
• Change one main variable and repeat enough trials to separate a real effect from run-to-run variance.
• Define rollout owner, maximum exposure, promotion checks, stop conditions, and rollback action before traffic reaches the candidate.
• Keep run state, queue delivery, worker leases, model calls, tool effects, sandbox artifacts, verification, and operator stop as explicit production boundaries.
• Keep privileged provider hooks and long-lived integration credentials in the trusted control service; give the sandbox only selected configuration and narrow run-scoped credentials.
• Enforce active-turn uniqueness in the database. Treat model execution and post-hook cleanup as separate phases, and persist the stable provider-session-to-current-sandbox mapping used for follow-ups.
• Follow-up delivery and post/failure hooks are process-local tasks in the worked interactive path. Active-run lookup or a persisted failure state does not prove prompt or hook completion.
• An in-memory output queue can preserve send order without blocking model iteration, but unsent events remain volatile until a durable outbox or the external activity API accepts them. Logging and dropping a failed send is not a durable receipt.
• A transcript buffered until finalization can vanish on runner failure. The run row, final response, main transcript, and subagent audit objects have separate durability.
• In the worked design, version-checked writes preserve tenant policy, operation deduplication, reconciliation, and deterministic verification while rejecting stale observations.
• Track active run versions and resumable checkpoints so an older workflow cannot wake into missing code.
• Retire a path only after work has drained, audit records remain readable, and rollback no longer requires it.

References#

• Anthropic: Demystifying evals for AI agents
• Google SRE Workbook: Implementing SLOs
• MCP 2025-11-25: Lifecycle and version negotiation
• OpenTelemetry GenAI semantic conventions
• OpenTelemetry: Traces
• DBOS: Workflow recovery
• Minimal harness: Local baseline with scripted models and deterministic tools referenced by the worked designs.
• Durable harness: Local recovery example for checkpoint versions, leases, and reconciliation.