Skip to content

Benchmarks

These are synthetic microbenchmarks of local harness operations, run without warm-up on shared CI runners. They quantify the runtime overhead of the harness mechanisms in the recorded environment. They do not measure developer productivity, code quality or statistical significance.

What is measured

harness benchmark run --iterations N times, in one process:

Operation What it exercises
stateTransition One transition decision of the normative state machine
gateEvaluation One gate evaluation in the positional GateInput form (one passed mandatory input)
canonicalHash Canonical JSON digest (SHA-256) of a small payload
artifactPutDeduplicated Storing identical bytes in the content-addressed artifact store
eventAppend Appending one event to the SQLite hash chain
processDirect / processGoverned Launching python -c pass directly versus through the governed process runner (capabilities, containment, bounded output)

Each operation reports iterations, meanMs, medianMs, p95Ms, operationsPerSecond and totalMs; processGovernanceOverhead reports the absolute and relative median overhead of the governed launch. The process pair uses max(5, min(30, iterations // 50)) samples (20 at 1,000 iterations) instead of iterations.

harness benchmark scenarios --iterations N compares, on generated Python and Node.js projects, a direct patch-and-test path with the complete governed path (all phases, validation, review, gate, an immediate programmatic approval and closure). Human waiting time is excluded.

How CI runs them

benchmarks.yml runs on pushes to develop and main, weekly and on demand:

  1. harness benchmark run --iterations 1000 five times and harness benchmark scenarios --iterations 3 once.
  2. scripts/benchmark_report.py renders one self-contained HTML report: median and p95 per operation (log scale), direct versus governed launch, throughput, dispersion across the five repetitions, environment and methodology, and the thesis reference point. Every chart has an equivalent data table.
  3. The median of each operation is stored on the gh-pages branch by github-action-benchmark (customSmallerIsBetter), which draws the per-commit trend.
  4. A regression above 150 % comments on the commit but does not fail the workflow: shared runners are noisy, and a failed benchmark would block unrelated work.

Published pages: the latest report at /benchmarks/ and the trend at /benchmarks/trend/ of the documentation site.

Thesis reference point

The report draws a reference point from docs/benchmarks/thesis-reference.json, the microbenchmark of the v0.8.0 cut. The evaluation reports only version 2.0.1 (campaign in progress, evaluation/campaign/); results of earlier versions are in the git history and the v0.8.0 tag and are not part of this analysis, so this page lists none of those figures. Environments differ (the thesis machine versus GitHub-hosted runners): never compare exact values.

Run locally

for i in 1 2 3 4 5; do harness benchmark run --iterations 1000 --output bench/micro-$i.json; done
harness benchmark scenarios --iterations 3 --output bench/scenarios.json
python scripts/benchmark_report.py --micro bench/micro-*.json --scenarios bench/scenarios.json \
  --reference docs/benchmarks/thesis-reference.json --output bench/report.html

Before 0.9.0, the microbenchmark failed on macOS with path escapes workspace because the default /var/folders/... temporary directory goes through the /var → /private/var symlink (issue #19).