Better agent results¶
Since 2.0 the harness can do more than verify what an agent delivered: it can ask an agent to review the task before work starts, hand the agent the gate it will face, compare failures with the baseline, stop the line when a step is not approved, review the change with a second agent, decompose large tasks, bound the context, govern the budget, learn from corrections and route each call to a model and effort. The motivation for each setting is the evaluation of the thesis (issues #37 to #44 and #52).
Every setting below is optional. A project.yaml without it keeps the 1.0.0 behaviour and its
configuration digest; harness init writes them all (see
the file harness init writes).
harness config validate prints the effective values under agentResults.
Request kinds (provider protocol 1.1)¶
Besides implement, a command provider may receive six read-only request kinds (architecture
since #56, see standards, principles, testing and architecture; locate since issue #55,
see the verification ladder). The request
says "kind", "readOnly": true and carries rendered "instructions"; the response is the 1.0
object (status, summary, optional usage) with a result object:
| Kind | Phase | result |
Setting |
|---|---|---|---|
clarify |
INTENT | {"questions": [{"category", "target", "text"}]} |
intake.ambiguityReview: agent |
acceptance |
SPECIFICATION | {"tests": [{"path", "content"}]} |
verification.acceptanceTests.mode: agent |
plan |
PLANNING | {"subtasks": [{"title", "requirements", "criteria", "constraints"}]} |
planning.decomposition: agent |
review |
INDEPENDENT_REVIEW | {"findings": [{"severity", "rule", "path", "line", "message", "evidence"}]} |
review.agentReview |
locate |
INTENT | {"locations": [{"path", "line", "evidence", "reason"}], "questions": [{"text"}]} |
context.locate (since #55; M and L tasks only, once per task revision) |
architecture (#56) |
INTENT (mode: advise) or DISCOVERY (mode: survey) |
advise: {"options": [{"id", "style", "title", "benefits", "costs", "fit", "recommended", "layers", "allow"}]}; survey: {"style", "summary", "layers", "allow"} |
architecture.mode: agent |
A read-only call that changes the workspace is undone, recorded as a HIGH
agent.read-only-violation finding and its answer is discarded. A result that does not match
the table is a protocol error; the phase that asked is blocked. The request, the provider's
output and the usage it reports are evidence of the run, and each kind may use its own provider,
model and effort (intake.clarifyAgent, verification.acceptanceTests.author,
planning.planner, review.reviewer, context.locate.agent), sent in the request as
routing. Without one, locate takes the router's locate rung, else the bottom rung of the
family's escalation ladder: the cheapest adequate model.
A second attempt for a broken answer (#80)¶
In the 2.0.0 pilot an acceptance call answered with something that was not JSON and
SPECIFICATION blocked at once, while the review panel already retries an invalid answer once on
its fallback provider. runtime.contractRetry, which harness init writes as {mode: once}
since #80, gives every read-only kind that second attempt:
runtime:
contractRetry:
mode: once # once, or off
fallbackProvider: second # optional: an agentProviders entry (or simulated)
An answer breaks its contract when it is a protocol error (no JSON object, no result object)
or when the phase rejects its result (the checks behind intake.agent-review-malformed,
acceptance.malformed, planning.plan-malformed, locate.malformed, architecture.malformed
and review.agent-malformed). The call is then sent once more, to fallbackProvider
when it is set, else to the same provider; the first attempt stays recorded (its request,
output and invocation) and an agent-contract-retry evidence and an agent.call.contract-retry
event say why and where the second went. The phase uses the second answer: a valid one continues
as a valid first answer would, a broken one blocks as before, with one finding. A provider that
did not answer or a budget block is not retried. A call that changed the workspace breaks its
contract too (#92): the write is undone, the HIGH agent.read-only-violation finding of that
attempt stays recorded, its answer is not trusted and the call is sent once more; a second
attempt that writes again blocks the phase, with a finding for each attempt. A
fallbackProvider that is not simulated or an entry of agentProviders is a configuration
error. Without the key the first broken answer blocks, as before. Transient failures of a command
provider keep their own retries (runtime.providerRetries).
A built-in adapter (kind: claude-code, codex, gemini-cli, aider) sends a read-only
request as a prompt (the instructions, then the request as JSON) and reads the result object
from the last JSON object with a result key in the agent's answer. The router's model and
effort become --model and, for Claude Code, --effort; for Codex,
-c model_reasoning_effort="...". These paths are covered by tests against the documented
output formats, not against the live agents.
An implement request keeps the 1.0 form unless an agent-results key adds something to it; then
it is version 1.1 and may carry kind, instructions, workspace, gate, permissions,
routing, budget, contextFiles, lessons and acceptanceTests, as described below. Since issue #55
it may also carry locations (the answer of the locate call) and attachments (a
person's intake attachments), each only when there is something to send. See
verification ladder.
INTENT: ambiguity and completeness (#37)¶
After the deterministic rules (C0 to C3, T1), intake.ambiguityReview: agent asks for
questions about ambiguities, contradictions, boundaries and errors, and about completeness:
missing flows, actors and roles, data and states, edge cases, non-functional requirements,
integrations, acceptance evidence and what is out of scope. Questions are grouped by category
(rule A1); under intake.criteriaPolicy: enforce INTENT blocks until a person answers them
with harness task clarify. The review runs once per task revision (keyed by its digest); an
answer with no question is evidence and the run continues.
intake.validateAnswers checks the answers that produced the current revision: an answer that
cites a requirement id or a document that neither the task nor the workspace contains gets a
question (rule A2) asking to attach or transcribe it, and the agent review of the revised task
checks the answers against the documents they cite.
A review that converges (#79)¶
With the bare ambiguityReview: agent every answered revision gets a new review, and nothing
stops it from raising new questions: in the 2.0.0 pilot two of four governed Haiku runs asked
10, 8 and 8 questions in three rounds and never reached IMPLEMENTATION. The object form, which
harness init writes since #79, makes the review converge:
intake:
ambiguityReview:
mode: agent
maxRounds: 3 # rounds of agent questions a person answers (default 3)
maxQuestions: 8 # agent questions asked at most in one round (default 8)
onExhausted: assume # assume (continue) or block (default when absent)
- The
clarifyrequest carriespreviousQuestions(every question asked about the task in an earlier revision, with its rule, category, target and the answer it got, ornull), theround,maxRoundsandmaxQuestions, and instructions to ask only about blocking ambiguity that the latest revision introduced or left open, never again about a point an answer settles, and nothing when the task can be implemented and verified as it stands. - A question already asked (same rule and normalised text: case, punctuation and spacing do not
count) is dropped, also within a round; a round keeps at most
maxQuestions. The review's evidence (agent-clarify-review) says how many weredropped. - A round is a clarification a person answered in this run that included agent questions (rule
A1). WhenmaxRoundsrounds were answered and the agent still raises points, the review is exhausted (intent.ambiguity.exhausted, with the rounds, the open points and the action): assume: the open points become explicit assumptions of a new task revision (metadata.assumptions:assumptionId, category, target, the question and the assumption that the implementation takes the reading most consistent with the requirements and criteria and states it). The revision is stored (and pinned to the run undergovernance.pinTaskRevision), recorded asambiguity-assumptionsevidence andintent.assumptions.recorded, and the run continues. The assumptions are visible in the task of the implement request, in the operational contract (assumptions, part of its digest), in the gate contract (gate.assumptions) and in the decision brief (asked.assumptionsand one line each under what was not verified).block: the open points are asked again and INTENT stays blocked, as before #79, with the exhaustion recorded.
The bare agent keeps the single-round review and its configuration digest.
More questions for a poorer request (depth: adaptive, 2.0.1)¶
A poor request is never refused: under the configuration of harness init a one-line wish such
as "haz una app como Uber con Uber Eats, viajes y domicilios de comida" is stored with
criteriaPending and INTENT asks for what it is missing. depth: adaptive, which harness init
writes, makes INTENT ask more the less the request says, and start with the basics:
intake:
ambiguityReview:
mode: agent
maxRounds: 3
maxQuestions: 8
onExhausted: assume
depth: adaptive # fixed (or absent): maxRounds and maxQuestions as written
- The measure (
governed_harness.intake.depth.poverty, deterministic, no model) adds, out of 100: the words of the intent (under 25: 35, under 80: 20, under 200: 10), the observable acceptance criteria (those ruleC1does not question; none: 25, one or two: 15, three to five: 5), the requirements (none: 25, one to four: 15, five to fourteen: 5) and the entities, distinct anchors in the intent, requirements and criteria such as numbers, quoted text, code identifiers and file names (under three: 15, under eight: 8). 90 or more isvery-poor, 75 or morepoor, 30 or morepartial, the restdetailed. A task with an observable criterion is at mostpartial: only a request without one is asked the basics deterministically. - The plan, from the configured
maxRoundsandmaxQuestions:
| Level | Rounds | Questions per round | Fundamental topics per round |
|---|---|---|---|
very-poor |
maxRounds + 3 |
maxQuestions + 4 |
3 (deterministic and agent) |
poor |
maxRounds + 1 |
maxQuestions + 2 |
5 (deterministic and agent) |
partial |
maxRounds |
maxQuestions |
all 9 in the first round (agent only) |
detailed |
maxRounds |
maxQuestions |
none |
The plan never exceeds 8 rounds and 16 questions per round, or the configured value when it is
larger. With the init defaults a one-line wish gets 6 rounds of up to 12 questions, a
paragraph without measurable criteria 4 rounds of up to 10.
- The fundamental topics, in this order: users and roles, what each user does, main flows,
platform and stack, data, business rules, limits and edge cases, observable acceptance
criteria, priorities and scope of the first version. Their questions follow intake.language
(Spanish or English).
- Deterministic questions. For very-poor and poor requests INTENT asks first one
question per topic not answered yet (category basics, target task:<topic>, rule C0 when the
task has no criteria, where C0 already asks for the results, and T1 otherwise), then the
other rules. An answer becomes a requirement prefixed with the topic's label, the topic is
recorded in metadata.intakeTopics, and the first answered revision freezes the level in
metadata.intakeDepth so that the answers do not shrink the plan halfway.
- The agent review receives the plan's maxRounds and maxQuestions and a depth object
(level, score, rounds, questions, topicsPerRound, round, topics of the round
with their labels), and its instructions ask it to cover those topics before any detail; once
the topics are covered, it asks about the remaining details.
- Fast lane. A one-line wish looks like size S; under depth: adaptive a very-poor or
poor request keeps the agent review even in the fast lane.
Without the key nothing changes: no topic questions, the configured rounds and questions, the
same clarify request, the fast lane as before and the same configuration digest.
SPECIFICATION: independent, frozen acceptance tests (#52)¶
verification.acceptanceTests.mode: agent asks a separate call for pytest files written from the
criteria only, under verification.acceptanceTests.directory (default tests/acceptance).
SPECIFICATION blocks until a person decides them, bound to their digest:
harness acceptance show --run <runId>
harness acceptance decide --run <runId> --decision APPROVE --digest <digest> --rationale "..."
Approval writes the files, freezes their digests and runs them once on the workspace before the
change (passing there is a MEDIUM acceptance.passes-before finding). Every later VERIFICATION
fails if a frozen file changed (acceptance.modified, HIGH) or the tests do not pass. A proposed
path where a file already exists, for example the frozen test of an earlier run in the same
workspace, is never overwritten: the file is written next to it with the run's suffix
(test_ac_1_<run>.py), and acceptance show lists the new name under renamed (#82).
Write protection and disputes (#94)¶
In the wave 9 pilot the implementing agent rewrote the frozen tests in one run, and in another a
correct change failed three times against approved tests that were wrong (a typo in an expected
value, inputs the clarified task says must be rejected). Two keys of
verification.acceptanceTests, which harness init writes, cover both; without them the
behaviour above is unchanged.
| Key | Absent | init |
Effect |
|---|---|---|---|
protect |
false |
true |
The frozen files are write-protected for the implementing agent. |
disputes |
off |
person |
A frozen test the agent disputes waits for a person instead of only failing the run. |
protect: true. The frozen files join the paths the agent sandbox denies writes to
(runtime.agentSandbox: enforce), next to .harness and .git, and appear under
protectedPaths in the sandbox evidence. After every agent invocation the harness also compares
each frozen file with its approved digest, whatever the sandbox mode: a file that was changed,
deleted or replaced is written back as approved and reported as a HIGH
acceptance.write-undone finding (event acceptance.write.undone), so VERIFICATION runs the
tests a person approved. The implement request says the files are write-protected.
disputes: person. The acceptanceTests block of the implement request tells the agent not
to change or work around a frozen test it believes is wrong, but to list it in its self-report
(provenance.selfReport):
{"selfReport": {"disputedAcceptanceTests": [
{"test": "tests/acceptance/test_ac_1.py::test_ac_1", "reason": "the criterion says 90, the test expects 91"}
]}}
The run waits for a person (BLOCKED in VERIFICATION, exit code 6) when all of this holds:
- VERIFICATION failed and the frozen acceptance tests (
harness.acceptance-tests) are among the mandatory validators that failed; - every frozen file is as it was approved;
- any other mandatory validator that failed locates all its failures in the frozen files (the
project's own test command usually collects them too; this needs
verification.outputParsers), and none is blocked, timed out or in error; - the self-report of the implementation that produced the ChangeSet lists at least one disputed test with a reason.
The same holds in the fast lane of friction (#161): its affected tests, which run first and
stop the attempt on a failure, leave out the frozen files under disputes: person, so the frozen
tests run in harness.acceptance-tests and the full suite before the outcome is decided.
Otherwise the run fails as before. A waiting run is not stopped: governance.stopTheLine leaves
its workspace alone. harness inbox shows an acceptance-dispute entry with the disputed tests,
the reasons and the digest of the dispute (the ChangeSet, the approved tests and what was
disputed); harness acceptance show shows the same under dispute. A person decides, bound to
that digest:
harness acceptance dispute --run <runId> --decision UPHOLD --digest <digest> --rationale "..."
harness acceptance dispute --run <runId> --decision REVISE --digest <digest> --rationale "..."
UPHOLD: the tests stand. VERIFICATION runs again and fails as it would have without the dispute: the run goes on to the correction cycles ofruntime.verificationCorrectionsor stops. The next implement request carriesacceptanceTests.upheld(the tests, the person's rationale). The same ChangeSet is not disputed twice.REVISE: the frozen files are removed and the run returns to SPECIFICATION. The acceptance call runs again with arevisionblock (what was disputed and the person's rationale), and the new proposal waits forharness acceptance decidelike the first one. On approval the harness writes and freezes the new files; the tests approved before stay recorded underprevious(proposal digest, files and their digests, who approved them), andrevisioncounts the rounds. Because the workspace already holds the change, the "fails before" run uses a copy of the workspace with every other changed path as on the baseline. The plan and the implementation stand: the run goes from SPECIFICATION straight back to VERIFICATION of the same change (eventacceptance.revision.resumed), and only a failing VERIFICATION sends it to IMPLEMENTATION again. Rejecting the new proposal leaves the run without frozen tests, as a rejection in the first SPECIFICATION does.
The self-report is the agent's word (quality REPORTED): it can open the wait, it never changes
a test, and only a person (never an agent actor, exit code 5) decides. A run whose provider
gives no self-report (the simulated provider, the embedded session) cannot dispute a test.
Events: acceptance.dispute.opened, acceptance.dispute.decided, acceptance.tests.revision.
PLANNING: decomposition of large tasks (#39)¶
Above planning.threshold requirements (default 12), planning.decomposition: agent asks for
sub-tasks that partition the requirements (every requirement exactly once). PLANNING blocks until
a person approves the plan, bound to its digest:
harness plan show --run <runId>
harness plan decide --run <runId> --decision APPROVE --digest <digest> --rationale "..."
The sub-tasks then run in order on the same workspace, each through IMPLEMENTATION and
VERIFICATION with its own correction budget and its own gate (subtask-<n>), recorded as
subtask.started and subtask.completed events; the task's constraints apply to every sub-task.
A sub-task that does not pass stops the ones after it; the run reaches DECISION when all passed.
REJECT keeps the task whole. Under planning.granularity: adaptive a model listed in
planning.coarseModels starts with the whole task, and the run returns to PLANNING to decompose
only when that attempt fails its corrections. Another model's attempt is not sent back to
PLANNING: when its corrections are spent the run stops in VERIFICATION (exit 6) with a
terminalReason that names the failing validators and says when the last correction changed
nothing (agent.empty-correction).
IMPLEMENTATION: what the agent receives¶
runtime.gateContract: thegateblock lists the validators (command, mandatory or optional), the enabled checks with their policies, the review rules, the blocking severities and the absolute workspace path.harness checkruns the same validators and diff checks on the workspace and records nothing (exit 0 when they pass, 6 otherwise); it reads the configuration and a check state file the harness writes before the call, so it runs inside the agent sandbox. Since #84 the contract suggests only a command the agent may run, by its capability grants (process.execute):checkCommand,harness check --path <workspace> --run <runId>, when they allowharness; otherwisecheckCommands, the validator commands they allow (none when they allow none), and the instructions say to run those, or that the harness runs the gate after the call. Underverification.requirementTraceability(enforceorwarn) the block also hastraceability: the policy, the identifier of each requirement the check looks for in the tests (identifier,requirementId, the text), the requirements it does not check (notChecked: no identifier) and the naming rule it applies (rule: a test file, class or function whose name contains the identifier as whole tokens, or the identifier as a whole word in a test's source or strings). Since #79 it also carries the task revision'sassumptions.governance.phasePermissions:permissions, derived from the capability grants of the agent actor: the filesystem read and write scopes, the commands, network access; no write scope for the read-only kinds. Recorded as evidence of the call.context.manifest: auto:contextFiles, files ranked deterministically (referenced by the task, naming its requirement ids or key terms, changed by earlier steps, declared interface files, paths of active lessons) and capped bycontext.maxFilesandcontext.maxBytes, with paths, sizes, digests and reasons. It is guidance, not a restriction.memory.learnFromFindings: auto:lessons, the approved lessons relevant to the task (see below).budget:budget.remainingper scope, so an adapter can cap its own call.agentRouting:routingwith the model, the effort and the flags for the provider family.
VERIFICATION: deterministic checks (#40, #52)¶
Each enabled check is a validation of the ChangeSet with located findings. Under enforce it is
mandatory and its blocking findings send the run through the correction loop; under warn its
findings are LOW.
| Key | Check |
|---|---|
interface |
A task's declared interface (metadata.interface: {stub, module}): names, parameter order and kinds, defaults, stub bodies. |
architecture |
maxModuleLines, maxFunctionLines, maxComplexity, forbiddenImports (from/to module prefixes), with severity; a limit the baseline already exceeded is LOW. |
securityPatterns |
Unrestricted pickle, marshal, shelve or unsafe YAML loads; stored card numbers or CVC; card numbers that pass Luhn; weak password hashing. HIGH. |
constraints |
Constraints recognised in the task (or listed in metadata.checks): stdlib-only, no-clock-random, annotated-public-api, no-float-money, no-stubs. |
weakenedControls |
Deleted tests or asserts, added skips, suppressions (noqa, nosec, type: ignore), lowered thresholds, broad excepts, errors turned into success. |
testQuality |
Assertion-free tests, declared interface methods no test calls, coverage of the changed lines (diffCoverage, coverage.py), and flakyReruns reruns with another hash seed and the test files in reverse order. |
secrets: context |
Secrets weighed by context: values the task declares, environment assignments, constants, test paths and dummy values; replaces the review's pattern rule. |
sarif |
SARIF reports of external scanners (path, tool, required), with tool and rule versions. |
invariants |
Commands (id, command) that must pass on every VERIFICATION, including each sub-task's. |
riskFactors |
newDependency, authentication, destructiveMigration, publicContract, network, floatMoney, sensitiveLogging, deletedWithoutTests, each block, acknowledge, inform or off. A line the ChangeSet removes and adds again with the same text (other line endings, a move within the file) raises none. |
differential |
A failing mandatory validator also runs on the baseline: shared failures are PREEXISTING_ERROR and do not block; new ones are INTRODUCED_ERROR findings. A test run with no test outcome is not comparable and keeps blocking (#190): pytest exit codes 2 to 5 (interrupted, for example by a collection error, internal error, usage error, no test collected), a pytest output with no passed or failed test, or another runner with no parsed failed test. A shared test failure passes only when every test file the ChangeSet added or modified ran: otherwise tests.changed-not-run (HIGH, the failure stays), or INFO when the output does not list the tests that ran (pytest lists them with -rA or -v). A PREEXISTING_ERROR pass gives no L1 or L2 rung to the certification. |
ratchet |
An optional validator (Ruff, Mypy) must not report more problems than on the baseline. The problems in a frozen acceptance test still as approved are left out: the agent cannot change that file (#183). |
A risk factor with acknowledge must be acknowledged in the decision; the decision brief lists
the factors:
harness gate decide --run <runId> --decision APPROVE --change-set-digest <digest> \
--rationale "..." --acknowledge-risk network
Corrections (#52)¶
runtime.reproduceFirst: a correction attempt that changes nothing is anagent.empty-correctionfinding with the agent's explanation; a correction afterREQUEST_CHANGESmust add or change a test that fails on the code before the correction and passes after it, or a HIGHcorrection.not-reproducedfinding blocks the gate.review.structuredChanges:REQUEST_CHANGESmay carry blocking items,--change-request "description::condition", where the condition istest:<pytest node id>(must pass),absent:<regex>(must not be added) ortext. Every later VERIFICATION checks them.
INDEPENDENT_REVIEW: a second agent (#38)¶
review.agentReview (enforce or warn) sends the ChangeSet and the task to a reviewer. It
runs only after every blocking deterministic check passed and again only when the ChangeSet
digest changed. Under enforce HIGH and CRITICAL findings block the gate and, within the
correction budget, return the run to IMPLEMENTATION with the findings as feedback (trigger
REVIEW_FINDINGS); a reviewer that fails or answers malformed JSON blocks the gate. Under warn
the findings are MEDIUM at most.
Stop the line (#52)¶
governance.stopTheLine: restore keeps the changes of a run that stops without approval
(rejected, cancelled, or failed in IMPLEMENTATION or VERIFICATION) as a quarantined patch and
restores the baseline; block keeps them and refuses new runs in the workspace until a person
runs harness run quarantine --run <runId>. A task that declares ownedPaths gets a HIGH
finding for every changed path outside them. The frozen acceptance tests of a run that can still
be continued (failed in IMPLEMENTATION or VERIFICATION) stay in the workspace when its changes are
restored (keptPaths in the quarantine record, #81); those of a rejected or cancelled run are
restored with the rest.
Under restore the quarantine record also keeps the content of every restored path
(quarantinedContents, each with its digest) and the phase the run stopped in. When
harness run continue resumes a run quarantined in VERIFICATION or a later phase (for example
after installing a missing test dependency), it first puts that content back and records
workspace.quarantine.reapplied, so the run is evaluated on its own ChangeSet again rather than
on the restored baseline (#193). Nothing is written over a new attempt: when a restored path
holds content that is neither its baseline nor its quarantined content (a person or an embedded
session wrote a correction), the quarantine is superseded (workspace.quarantine.superseded) and
the run is evaluated on the workspace as it is. It fails closed: if a stored content is missing
or does not match its digest (a redacted artifact does not), nothing is written,
workspace.quarantine.reapply-refused is recorded and the run is BLOCKED with the paths in its
reason; write the change again (the quarantined patch is on the run's chain) and run continue,
or cancel the run. A run quarantined in IMPLEMENTATION implements again from the baseline.
Budget (#42)¶
budget sets limits per agent call, per task (all its runs) and per run, for costUsd, tokens
and wallSeconds, with warnAt (default 0.8). Usage is what providers report and the wall time
the harness measures. Crossing warnAt records a LOW budget.warning once; crossing a limit
records a HIGH budget.exceeded finding and no further agent call is made until a person raises
the limit. A limit crossed by the last agent call of a run, with no further call to refuse, stops
the run before INDEPENDENT_REVIEW (BLOCKED, exit code 6), or before DECISION when a review
call crossed it (exit code 4, no gate evaluated): the run never reaches the decision as if it were
within its budget (#157):
harness budget show --run <runId>
harness budget raise --run <runId> --scope run --metric costUsd --to 150 --rationale "..."
harness run continue --run <runId>
Lessons (#43)¶
Under memory.learnFromFindings: auto, when a run closes, the findings that caused a correction
are proposed as project memory records lesson:<validator>:<rule> with provenance. A lesson
recurring in memory.recurrenceRuns runs (default 2) is approved automatically only under
memory.autoApproveRecurring; otherwise harness memory approve approves it. Active lessons
relevant to a later task reach its request and their use is recorded (lesson.applied).
Routing (#44)¶
agentRouting.mode: tiered chooses the model and effort of each call with a pure function: the
call kind's rung for clarify, plan, review, acceptance and architecture (locate uses
its own rung or the bottom of the ladder); for implement the rung of the task size (S,
M, L by requirements, owned files and their lines, thresholds; a risk flag raises S to
M; a decomposed task is L). Since #59 a table without the entry of a call kind uses the
implement rung of the task size for it, with the rule fallback:implement:KIND:SIZE, a
warning in the decision and an agent.routing.fallback event, instead of failing the call. A failed verification or blocking review findings move the next
call one rung up the ladder (effort before model), at most maxEscalations; transient failures
retry at the same rung. families maps a provider id to claude-code or codex (otherwise the
command name decides); tables override the starting tables. Every decision is evidence
(agent.routing.decided) with its inputs, rule and policy digest. fixed keeps the provider's
own model.
Trivial implement tasks (#197)¶
Implementation runs on Sonnet by default (S and M rungs of the starting claude-code table,
L on Opus). Only a very easy task goes to Haiku: since #197 an implement call whose task is
trivial uses the family's trivial rung (claude-haiku-5-5 with medium effort in the
claude-code table; the codex table has no cheaper model, so its trivial rung is its S rung;
a table without a trivial entry uses its S rung). A task is trivial only when all of these
hold:
- it is
Sby the thresholds above; - it has at least one and at most
thresholds.trivial.requirementsrequirements (default 1); - it owns at least one and at most
thresholds.trivial.filesfiles (default 1); - those files have at most
thresholds.trivial.loclines (default 150); - it has no risk flag (a public
interface, security words, a path matchingriskPaths) and PLANNING did not decompose it.
The defaults are deliberately conservative starting values to calibrate with
harness routing calibrate, not a measured optimum; thresholds.trivial.requirements: 0 turns
the tier off. The other call kinds and the S/M/L size used by the agent contract, friction
and metrics do not change. The decision records the size trivial and the rule
tier:implement:trivial. When a trivial call fails verification or gets blocking review
findings, its first escalation runs on the S rung (Sonnet, medium) and further escalations
climb the ladder from there (escalation:N:implement:trivial), within maxEscalations. Under
anchored the Haiku rung is kept when the invoking model is Sonnet or Opus, and escalation stops
at the invoking model as for any other call.
agentRouting:
thresholds:
trivial:
requirements: 1
files: 1
loc: 150
The starting tables come from a research note of 2026-10-04 (Claude Code --model/--effort,
Codex -m/model_reasoning_effort); they are a starting point, and whether an account can use
a model is not checked. harness routing calibrate reports the cost per approved task of the
recorded decisions and suggests a table; nothing is applied.
Anchored at the invoking model (#85)¶
With the starting tables tiered ignores the model the person invoked: in the 2.0.0 pilot the
cells invoked with Haiku, Sonnet and Opus routed every call to the same Sonnet and Opus rungs.
agentRouting.mode: anchored, which harness init writes since #85, uses the same tables and
thresholds with the invoking model as the ceiling:
- a rung of a cheaper tier than the invoking model is kept (cheaper rungs for the simple call kinds and sizes);
- a rung of the invoking model's tier runs on the invoking model with the rung's effort;
- a rung above it becomes the top rung allowed (rule
anchored:ceiling:KIND:SIZE); - escalation (
implement,review) climbs the allowed rungs up to the invoking model, never above (anchored:escalation:N:KIND:SIZE).
Claude models are ranked by name (haiku below sonnet below opus); the models of another
family by their first rung in its ladder. A model whose tier cannot be told allows only itself.
With the starting tables: invoked with claude-haiku-4-5-20251001, every call runs on it;
with claude-sonnet-5-5, the plan, review, architecture and size-L rungs run on
Sonnet with high effort and a trivial implement call stays on Haiku (#197); with claude-opus-5-5, the models and efforts are those of tiered.
How the invoking model is determined, for the provider that answers the call:
agentRouting.anchorModelwhen it is set;- else the provider's
model(agentProviders.<id>.model); - else the value of
--model(or-m,--model=...) in itscommandorargs, as a command provider that wraps a CLI is usually invoked; - for the embedded
sessionprovider, whose model the harness cannot see, the same order applied toagentProvider, which answers the read-only calls of a session run (see embedded mode); the session implements with its own model.
Without any of them each call keeps the provider's own model (rule anchored:no-anchor:...). A
call kind's own model or effort (intake.clarifyAgent, planning.planner, ...) and a
reviewer's own models entry still win, as under tiered. Every decision records the anchor.
Since #85 every reviewer of the review panel also records agent.routing.decided (with
reviewer and attempt), and harness routing calibrate counts the reviewers of one run on the
same model once.
Where each setting is tested¶
| Setting | Tests |
|---|---|
Request kinds, intake.ambiguityReview, intake.validateAnswers |
tests/integration/test_agent_clarify_review.py |
intake.ambiguityReview.depth |
tests/unit/test_intake_depth.py, tests/integration/test_intake_depth_flow.py |
| The converging review and its assumptions (#79) | tests/integration/test_ambiguity_convergence.py, tests/unit/test_intent_convergence.py |
runtime.contractRetry (#80) |
tests/integration/test_contract_retry.py |
verification.* checks, risk factors, change requests, differential |
tests/integration/test_verification_checks.py, tests/integration/test_verification_checks_more.py, tests/unit/test_run_outcome.py, tests/unit/test_checks_structure.py, tests/unit/test_checks_diff_quality.py |
verification.acceptanceTests |
tests/integration/test_acceptance_tests.py |
verification.acceptanceTests.protect, disputes (#94) |
tests/integration/test_acceptance_dispute.py |
verification.reverifyOnChange |
tests/integration/test_run_lifecycle.py |
governance.stopTheLine, runtime.gateContract, governance.phasePermissions, harness check |
tests/integration/test_stop_line_and_contract.py, tests/unit/test_gate_contract_terms.py |
runtime.reproduceFirst |
tests/integration/test_reproduce_first.py |
review.agentReview |
tests/integration/test_agent_review.py |
planning |
tests/integration/test_decomposition.py |
context, memory |
tests/integration/test_context_and_lessons.py, tests/unit/test_context_manifest.py |
budget |
tests/integration/test_budget.py |
agentRouting |
tests/integration/test_agent_routing.py, tests/unit/test_routing.py, tests/unit/test_routing_anchored.py, tests/integration/test_review_panel_run.py (reviewer decisions) |
The defaults harness init writes |
tests/integration/test_agent_results_defaults.py and the agent-results flow of scripts/demo_flows.py |