Matched baseline vs candidate comparison with drumsbench

In one paragraph

This is the record where a Drums number against a VM baseline would eventually go. Today it holds the rules only. drumsbench compare takes two summaries, refuses anything that is not labelled measured and reconciled against a provider receipt, and prints descriptive point estimates with claim: null, always. It does not pick a winner and does not extrapolate to training. Both arms must first exist as measured runs: the baseline has never been provisioned on a real KVM host by this work (EXP-007), and the candidate has never passed a restore (EXP-008, EXP-009). Beyond that, the statistical design that would let a difference be stated is not implemented, and the branch says so in its own words.

Setup

What both runs must satisfy before compare accepts them

• Each run's summary.json carries evidence: measured (not fake_not_evidence, not incomplete_not_evidence), which requires every capture command to have succeeded and every cleanup command to have succeeded.
• Each run is reconciled: drumsbench reconcile summary.json billing.json with a provider-derived receipt whose start and termination times bracket the harness window, covers_all_resources: true, a statement id, and settled gross, credits and tax kept distinct. The cost before this step is rate x hosts x harness window labelled settled: false.
• Same workload hash, same seeds, same checker, equal declared guest_resources; the baseline config's preregistered_sha256 recorded before the run.
• The sameness gate (sharedos_rootfs/sameness.py) must report image sha256, task, evaluator command, guest-server file hash, guest vCPUs and memory and workload hash all known and equal; unknown blocks. The arm's declared fidelity differences (host kernel, Xorg dummy vs virtual GPU, cgroup limits vs vCPUs, overlay vs qcow2, proxy networking) accompany every number.

What compare prints

Descriptive point estimates per arm: attempts, valid attempts, successes, invalid reasons, phase busy seconds, the harness window, the reconciled cost, and ratios between the two. Its claim field is null by construction and its status says that pre-registered statistical review is required. The README: "It never headlines a winner and never extrapolates to training."

What would be needed to state a difference

None of this exists yet; it is listed so the gap is visible:

• A paired design: each seed and task run on both arms under the same host conditions, so the unit of comparison is the pair, and a paired confidence interval on the per-attempt difference in environment time and cost.
• A minimum sample size fixed before the run, from a pilot's variance and the margin below, not chosen after seeing results.
• An equivalence margin for task success: the agent is identical, so success rates should not differ; a pre-registered margin says how much difference would mean the environment changed the task.
• A pre-registered threshold for the headline (cost per valid trajectory) stated before the run, with the note that it was not chosen to be attractive. Any target such as "2x" is a target, never a result.
• Multiple hosts or at least repeated runs on different days, because a single host and a single window confound the arm with the moment.

What ran

Nothing. No measured run exists for either arm; no summary.json, reconciled.json or compare output is on the branch.

Results

None.

Failures and surprises

None recorded.

Limitations

• Scope limits stated on the branch: "one host, one guest, concurrency 1; no statistical design (compare is descriptive)".
• The harness window is not the cloud provision-to-termination window; the harness "does not claim complete billable cost", which is why reconciliation is mandatory.
• The candidate's declared differences from the baseline are structural (shared kernel, no GPU) and will not be removed by more samples; they must be printed next to every number.

Reproduce

python -m drumsbench run --workload W.json --arm baseline --baseline-config B.json --money M.json --out-dir runs/baseline-1 --execute
python -m drumsbench run --workload W.json --arm drums --arm-config D.json --money M.json --out-dir runs/drums-1 --execute
python -m drumsbench reconcile runs/baseline-1/summary.json baseline-billing.json --out runs/baseline-1/reconciled.json
python -m drumsbench reconcile runs/drums-1/summary.json drums-billing.json --out runs/drums-1/reconciled.json
python -m drumsbench compare runs/baseline-1/reconciled.json runs/drums-1/reconciled.json

The statistical design would be added as a pre-registered file hashed into the run before it starts, and compare would be extended to read it; neither change exists.

Sources

• benchmark/drumsbench/compare.py, benchmark/drumsbench/reconcile.py, benchmark/drumsbench/runner.py (summarize)
• benchmark/README.md, "What it enforces" (compare and cost paragraphs), "Not built / known limits"
• benchmark/aws_run/STATUS.md, open item 8 (scope limits)
• benchmark/sharedos_rootfs/sameness.py, benchmark/sharedos_rootfs/CANDIDATE.md section 5
• Commits eb59595, d5247e2 (2026-10-25)