How a number gets onto this site, and what stops a bad one.

Workload selection

• A frozen task set is fixed before any arm is measured, and its hash is recorded. For the Linux arms it is OSWorld task definitions with their external checkers where supported; for the macOS arm it is four scripted tasks (Writer save, browser form, clipboard across apps, visibility) with a verify server that does not trust the agent.
• The smoke task (rename a folder on the desktop, checked by a shell test) is for plumbing only. A smoke pass shows the harness behaved as specified in that run; it says nothing about speed, and nothing about the keystroke script being right until the screenshots are reviewed.
• Representative tasks for experiment 2, illustrative not final: spreadsheet edits, document formatting, file-manager workflows, browser form filling, multi-app clipboard tasks. About 20 to 30 tasks, drawn from benchmark definitions with clean licences (OSWorld Apache-2.0; the CUA-Gym dataset CC-BY-4.0).

Baseline fairness

• The incumbent is tuned, not a strawman. VM baselines run with warm snapshots and pooling on; the KVM accelerator is required and read back from the hypervisor. A cold-boot VM is never the comparison.
• Matched: OS version, applications, tasks, checker, hardware allocation per environment, and reset semantics. The same scripted policy in experiment 1; the same model, prompts and temperature in experiments 2 and 3.
• Disk copy-on-write is not live process fork. A file-level clone restores files; a CRIU restore or a memory snapshot restores processes. Neither is presented as the other.
• Asymmetries are recorded, not hidden: on the Mac, macOS gets the whole machine and the Linux VM gets 40 of 48 GB; a VM has its own kernel, a container does not; a container's network namespace may be owned by the harness rather than the image.
• An ordinary OCI container is a control, not the custom Drums runtime, and is named as such on every page.

Correctness gates

flowchart LR
A[Restore fidelity] --> B[Reset correctness]
B --> C[Isolation with positive control]
C --> D[Throughput at confirmed N]
D --> E[Cost per valid rollout]
E --> F[RL with only the environment changed]

1. Restore fidelity before any performance comparison. The state hash after a restore must equal the hash after the snapshot. On Linux the pre-resume lock census must equal CRIU's dump record, the network namespace must be byte-identical before and after, and the mount contract must hold. The custom runtime currently fails the lock half (EXP-009).
2. Reset correctness. After every reset the work tree must match the golden manifest, the clipboard must be empty, and no application process may be left over. A reset that cannot kill its applications fails rather than proceeding. Reset correctness is a requirement at 100%, not a target.
3. Isolation with a canary and a positive control. A never-touched environment is watched continuously; the eleven isolation tests run per window; a zero is printed only next to a positive control that fired. See verifiers and contamination.
4. Validity gates per attempt. Pre-check, generation check, timeout, verifier availability, isolation. Checker exit 0 is success, 1 is a task failure (still valid), anything else is invalid. A verifier that is unavailable makes the attempt invalid, never a failure.

Cost accounting

usd_per_valid_rollout = host_usd_per_hour / valid_rollouts_per_host_hour

• In: host CPU and RAM hours for the whole window, provisioning and reset time, idle reserve, verifier calls, and, when experiment 2 runs, inference tokens priced at the day's rate. Out, and stated as out: integration engineering, and in experiment 1 all inference (the policy is scripted).
• Three dated price inputs for the Mac arms: owned hardware (invoice over 36 months times 730 hours plus measured watts at a stated kWh price; the invoice figure is a placeholder until the invoice), AWS EC2 Mac M4 Pro about USD 1.97 per hour on demand with a 24-hour minimum (vendor claim via a price mirror, checked 2026-10-20), and Scaleway Mac mini M4 Pro EUR 0.49 per hour converted at the run day's rate (vendor claim, checked 2026-10-20). The Linux arm on the same Mac gets the same three rows.
• Cost is an estimate until reconciled with a provider receipt. The Linux harness labels a run's cost settled: false until a receipt with start and end times and money fields is attached; the comparison command accepts only reconciled summaries.
• Failed and invalid attempts stay in the denominator of attempts and in the numerator of cost. They are not in valid rollouts. Every failed reset, timeout and environment failure gets its own line and is never scored as reward 0.

Statistical uncertainty

• Sustained windows, not bursts. Experiment 1 uses two 15-minute windows per N and a 60-minute soak at the stable N (design values). A burst on pre-warmed resources is reported as a burst.
• N confirmed twice. An N counts as stable only when every gate passed in two separate windows. The stable N is the largest confirmed N.
• Multiple seeds for every agent run; three seeds at minimum for the smoke, more for experiment 3.
• Descriptive point estimates until a pre-registered statistical design exists. Today the comparison command emits no claim. That design must contain, before any data: the primary metric and its direction, the minimum effect of interest, the number of windows and seeds, the test or interval method, the stopping rule, and what counts as a miss.
• Pre-registered thresholds. Gates are committed to git before measurement (the L1 prototype's thresholds were committed 36 minutes before its results); a miss is published as a miss.
• Reproduction within 15%. A second person on a second machine runs one window at N=1 and one at N=4 and must land within 15% on memory per environment, step p95, reset p95 and valid rollouts per host-hour, with contamination 0 next to a fired control.

Model configurations

• Capable closed-weight evaluation agent: a frontier model through Amazon Bedrock's computer-use protocol, used to measure reliability and economics under a capable policy. Its outputs are excluded from training; every record carries that label.
• Open-weight trainable model: the first RL candidate is a small open-weight vision-language model; a stronger open-weight computer-use model follows only when its serving and licence are verified. The serving ladder (context 65536, 49152, 32768, unquantised) is recorded with the largest level that serves the frozen agent configuration.
• Both arms see the same model, the same frozen agent configuration (temperature 0, fixed screenshot size), and the same action allowlist.

Evidence rules

• Raw logs are kept. Nothing deletes a results folder.
• A digest, a summary or a chat transcript is not evidence. A number without a log is author-claimed and says so.
• Label by exactly what it shows: a test pass is software evidence; a probe outcome is a probe outcome; a sandbox measurement names the sandbox.
• No expansion of N, seeds or hosts without the results so far shown.
• Zero contamination is printed only next to a positive control that fired.
• Every number carries a label the first time it appears: measured, author-reported, author-claimed, vendor claim, design value, estimate.

Reproduction

macOS arm

Prerequisites on the Mac: Xcode Command Line Tools with Swift 6, Homebrew, uv. Every host-changing target refuses to run without DRUMS_HOST_OK=1. Set RUN=results/<date>-<host>-mac once and keep it.

A second machine runs make reproduce (metadata, then a sweep at N of 1 and 4 with two confirming runs each, then the report) and prints the four Gate 10 numbers.

Linux arms

The Linux harness runs one user-supplied workload against arms through an adapter contract. Sketch, from the harness README (nothing here has been run against a real VM yet; see EXP-015):

python -m osworld_smoke.make_workload --image /path/to/Ubuntu.qcow2 --out workload.json
python -m drumsbench baseline-hash --workload workload.json --baseline-config baseline.json
python -m drumsbench plan --workload workload.json --arm baseline --baseline-config baseline.json
python -m drumsbench run  --workload workload.json --arm baseline --baseline-config baseline.json \
    --money money.json --out-dir runs/baseline-1 --execute
python -m drumsbench run  --workload workload.json --arm drums --arm-config arm.json \
    --money money.json --out-dir runs/drums-1 --execute
python -m drumsbench verify-receipts runs/baseline-1/receipts.jsonl
python -m drumsbench reconcile runs/baseline-1/summary.json baseline-billing.json --out runs/baseline-1/reconciled.json
python -m drumsbench reconcile runs/drums-1/summary.json drums-billing.json --out runs/drums-1/reconciled.json
python -m drumsbench compare runs/baseline-1/reconciled.json runs/drums-1/reconciled.json

The baseline config hash must be pre-registered before the run; run refuses without --execute and without a money file; compare accepts only measured and reconciled summaries and always emits claim: null. The OCI reference arm and the custom rootfs candidate are driven per slot by their own modules and must pass the sameness gate (same image hash, task, checker and limits) before any number is recorded.

How this site is built

Each experiment is one MDX file with structured frontmatter (id, dates, status, arms, hardware, OS, branch, commit, evidence paths, one limitation line). Each measurement set is one JSON file under data/results/ that names its experiment, arm, host, workload, method and sources, and holds rows with a number, a distribution summary or a pass count. Every content file and every result file is validated against the same schemas before the build; a result file that points at an unknown experiment, or a row without a number, fails the build with the file name. Pages are generated from these files; the benchmark tables are never typed by hand. The site is a static export with no tracking.