QEMU/KVM warm-snapshot baseline harness (drumsbench) and the OSWorld rename scripted smoke, first real run failed 0/0

In one paragraph

drumsbench is the harness we intend to use for every Linux comparison. It takes one user-supplied workload (image, policy command, checker, seeds, ready/warmup/reseed, canary and state-hash commands) and runs it identically against "arms". The implemented arm is a warm QEMU/KVM VM with savevm/loadvm; an external arm such as Drums is reached only through a command you supply. The harness's README says what it is: "This is a harness, not a result. No run has been performed with it". The OSWorld rename smoke is the first workload for it: one pinned OSWorld task, driven through a GUI-only pyautogui channel with no shell path, scored by the task's own checker. The branch records one real attempt of the correct-control seed, which failed: the keystrokes typed the new name into the Files search field and renamed nothing (checker 0 before, 0 after). That statement is author-claimed with no artifact on the branch. The policy was rewritten and is "still unvalidated until the screenshots ... are reviewed". Status is failed because the only run reported missed the task bar; nothing about it is measured here.

Setup

What the harness enforces

• Same workload: one manifest supplies the desktop image (sha256-checked), policy command, checker, seeds, ready/warmup/reseed, canary and state-hash commands. Both arms run these identically.
• Equal guest resources: the baseline config must match guest_resources or it refuses. The design value for this workload is 4 vCPU / 4 GiB.
• Preregistered baseline: preregistered_sha256 must equal the hash of the rest of the baseline config, so flags cannot drift after seeing results. The host description, hourly rate and host count are recorded and shown in compare.
• KVM explicit and read back: the baseline requires accel: "kvm" (TCG rejected), starts QEMU with -machine type=q35,accel=kvm, then query-kvm over QMP must report enabled and present or setup fails.
• Validity gates per attempt: isolation canary (own token present, other attempts' tokens absent), fidelity (state hash after reset equals the hash after snapshot), policy and checker errors. Checker exit 0 is success, 1 is task failure (still a valid attempt), anything else is invalid. No retries; failed and invalid attempts stay in the denominator.
• Hash-chained receipts: receipts.jsonl holds every command with argv, environment, rc, timing, output hashes and tails, plus environment captures and per-attempt records; verify-receipts checks the chain.
• Cost is an estimate until reconciled: cost is rate x hosts x harness window plus declared fixed costs, labelled settled: false. A summary becomes comparison_ready only through reconcile with a provider-derived receipt (start and termination times, covers_all_resources: true, a statement id, settled gross, credits and tax kept distinct). The receipt is attested input; the harness checks shape and bounds, not authenticity.
• Compare is descriptive only: compare accepts only measured plus reconciled summaries and reports point estimates with claim always null; the paired confidence interval, minimum sample and equivalence margin are not implemented. It never headlines a winner and never extrapolates to training.
• Required capture and cleanup: a failing capture command refuses the run; a failing cleanup command marks the run incomplete_not_evidence. Evidence labels are fake_not_evidence, incomplete_not_evidence or measured.
• Runaway protection: per-command timeouts, max_wall_seconds, cost_cap, cleanup always runs, and the sweeper signals a process only on an exact argv match that started after the registry was written.

flowchart TD
A[Provision slot: qemu-img overlay, qemu-system with accel=kvm, QMP query-kvm readback] --> B[ready + warmup commands]
B --> C[savevm bench-clean over QMP]
C --> D[Reference state hash]
D --> E[loadvm bench-clean]
E --> F[reseed + canary plant]
F --> G[Policy command: the task]
G --> H[Checker + canary probe]
H --> I[Reset: loadvm, state hash must equal reference]
I -->|next attempt| E
I --> J[Cleanup, always, even after a crash]
J --> K[summary.json + receipts.jsonl, then reconcile, then compare]

The OSWorld smoke task

The task is bundled from OSWorld commit 04718eab9327856efe8192cc58161acc7a7627e2 and its sha256 (a507ec6c35284c1413efe0d8a325b41a1626522221efc14f153742a865f1c04d) is re-verified by every command.

task id:      e0df059f-28a6-4169-924f-b9623e7184cc
instruction:  I have a directory named "todo_list_Jan_1". Can you help me change its name into "todo_list_Jan_2"?
setup:        echo {CLIENT_PASSWORD} | sudo -S mkdir ~/Desktop/todo_list_Jan_1
              (then a pyautogui click at screen centre)
checker:      func exact_match, result type vm_command_line, command:
              [ -d ~/Desktop/todo_list_Jan_2 ] && [ ! -d ~/Desktop/todo_list_Jan_1 ] && echo 'Directory exists.' || echo 'Directory does not exist.'
expected:     "Directory exists.\n"   (exact match, no stripping)

Three seeds: 1 correct rename to todo_list_Jan_2 (expect checker 0 before, 1 after), 2 wrong name todo_list_Jan_3 (0, 0), 3 no-op control (0, 0). The guest is reached through OSWorld's own server API (/execute, /screenshot, /screen_size, /setup/launch on guest port 5000). ready requires a display server and gnome-shell running, nautilus and sudo installed, a writable ~/Desktop, and the screenshot size equal to the server's screen size. reseed runs the task's own setup and the evaluator must say 0 before any action. The policy has no shell path: only AST-validated pyautogui.click/press/hotkey/write and time.sleep snippets and one fixed launch of nautilus ~/Desktop.

Image: public OSWorld Ubuntu image, zip sha256 b795b6cd...712, extracted qcow2 sha256 6bf667a852b3c307f61d9f09c42559351f45e0607e428b4997becf534cf4d313. The ~11 GB (design value) image was not downloaded into the authoring sandbox.

What ran

• The harness and baseline arm were built and unit-tested against labelled fakes (commits eb59595, d5247e2, a81c9f0, 05:02 to 05:23 UTC). The README states: "No QEMU or KVM exists in the development container, so baseline provisioning has never been run against a real VM; only its command construction is tested."
• The smoke was built at 5db840a (05:16 UTC), "tested against a labelled fake guest only".
• Commit 3b8d6dd (15:04 UTC) and README lines 146 to 153 describe one real run of the correct-control seed on an unnamed host, which failed. No pre-check.json, post-check.json, policy-manifest.json, screenshot or receipts.jsonl from it is on the branch.

Results

No measured result. The author-claimed outcome of the first real correct-control run (no artifact on branch): "pre-check 0, post-check 0". The original keystroke sequence (down, F2, type) "put the new name into the Files search field and renamed nothing".

The rewritten policy (scenarios.py): wait 3 s; Ctrl+L; type /home/user/Desktop; Enter; wait 2 s; Ctrl+A (selects the sole folder); F2; wait 2 s; then it requires the screenshot to differ (a rename dialog) before typing; Ctrl+A; type the new name; Enter. If the screen does not change it stops, keeps every receipt and fails the step. The branch calls this "still unvalidated". Screenshots at each stage (desktop-opened, folder-selected, rename-dialog, name-typed-before-enter, after) are saved for a human to review.

Failures and surprises

• The first GUI script was a guess and the branch said so in advance: "The GUI keystroke script is an unproven guess." It failed on the first real guest.
• The same branch carries two kinds of statement: osworld_smoke/__init__.py says "It has never been run against a real guest", and the README says "The first real run of the correct control FAILED". The later, more specific statement is what we report, labelled author-claimed.
• The harness itself has still never completed a run with receipts on any VM; no receipts.jsonl, summary.json or reconciled.json exists on the branch.

Limitations

• Nothing measured. The failed run has no artifact; the host is not identified.
• The baseline has never been provisioned on a real KVM host by this work; savevm/loadvm under nested virtualization on the planned AWS host is also untested (see EXP-011).
• verdict.py notes that PASS "means the plumbing behaved as specified in this run. It is not proof that the GUI keystroke script is right for the real desktop: review the raw screenshots."
• The process names ready relies on are properties of the Ubuntu GNOME image and were not verified.

Reproduce

python -m osworld_smoke.make_workload --image /path/to/Ubuntu.qcow2 --out smoke-workload.json
python -m drumsbench plan --workload smoke-workload.json --arm baseline --baseline-config B.json   # dry run
python -m drumsbench baseline-hash --workload smoke-workload.json --baseline-config B.json          # write into B.json
python -m drumsbench run --workload smoke-workload.json --arm baseline --baseline-config B.json \
    --money M.json --out-dir runs/baseline-1 --execute --concurrency 1
python -m drumsbench verify-receipts runs/baseline-1/receipts.jsonl
python -m osworld_smoke.verdict runs/baseline-1

Needs qemu-system-x86_64, qemu-img, a readable and writable /dev/kvm, the pinned image and a money file. Templates in benchmark/examples/ fail closed on purpose until filled in.

Sources

• benchmark/README.md, sections "What it enforces", "Baseline mechanics", "OSWorld rename SCRIPTED SMOKE" (lines 104 to 153), "Not built / known limits"
• benchmark/drumsbench/ (runner.py, baseline.py, external.py, compare.py, reconcile.py, contract.py)
• benchmark/osworld_smoke/data/task.json, benchmark/osworld_smoke/scenarios.py, benchmark/osworld_smoke/verdict.py, benchmark/osworld_smoke/__init__.py
• Commits eb59595, d5247e2, 5db840a, a81c9f0, 3b8d6dd (2026-10-25)