DrumsBench

Experiments / EXP-027

Planned: matched incumbent runs on AWS, stock VirtualBox live snapshots on bare metal and a Docker/KVM cold-reset small stage, with the equivalence evidence required before anything is called a comparison

Run the stock OSWorld incumbents (VirtualBox with live snapshots on a bare-metal host; the Docker provider's QEMU/KVM cold reset on a small instance) with the same task, agent and reuse flag as the Drums arm, under fail-closed controls, and produce the first result that could be labelled a comparison only once software equivalence is shown.

plannedWritten and reviewed as code or protocol, but not executed. No numbers.
Date
2026-10-24
Arms
qemu-kvm-warm-snapshot, aws-host
Hardware
planned: one bare-metal instance (96 vCPU class) for VirtualBox; one m7i.xlarge (4 vCPU, 16 GiB) with nested virtualization for Docker/KVM
OS
planned: Ubuntu 24.04 host; guest the OSWorld image (Ubuntu 22.04.3, GNOME 42.9)
Workload
public OSWorld task e0df059f, alternating blocks, concurrency 1, both arms through the public DesktopEnv API
Code
claude/linux-comparison-prep at 9bf0391
Evidence
docs/aws-incumbent-runbook.md
docs/aws-small-stage-plan.md
docs/incumbent-preflight-plan.md
docs/matched-comparison-protocol.md
compare/drums_compare/host_run.py, stages.py, awsctl.py, vbox.py, docker_telemetry.py, imagesize.py
deploy/aws_host/bootstrap.sh, fetch_image.sh, fetch_docker_image.sh, approval.example.json, approval.docker_kvm_small.example.json, host_run.example.json, host_run.docker_kvm.example.json
Limitation
Nothing in this record has run; the host runner's own docstring says it has never run against a real VM, container image or AWS, the approval files say approved false, and every cost figure in the plans is a worst-case arithmetic placeholder that is not published here.

In one paragraph

Every measured number on the Linux side of this site compares Drums with Drums. This record is the plan for the first run against the incumbents OSWorld actually ships, and it is a plan: as of the branch tip no instance has been launched by this code, no VM has booted, no container has run, and compare/drums_compare/host_run.py says of itself "PROTOTYPE: written and unit-tested without a host. It has NEVER run against a real VM, container image or AWS." Two stages are specified. The original route is the stock VirtualBox provider with live snapshots (VBoxManage snapshot take of a running VM), which needs a bare-metal instance and a vCPU quota the runbook records as pending. The small stage is the stock Docker provider, which runs QEMU/KVM inside a container, has no snapshot at all (save_state raises NotImplementedError), and cold-boots a new VM on every reset; it is planned on one m7i.xlarge with nested virtualization, where the Docker guest sits at virtualization level 2 and the Drums sessions at level 1 of the same instance, an asymmetry the plan says is part of what it measures and why the numbers would not transfer to bare metal. Both stages are fail-closed by construction: the runner refuses placeholder prices, stubs, unverified images, a VM not reporting exactly the configured vCPU and RAM, and labels the run NOT_A_COMPARISON unless measured, digest-bound equivalence evidence exists for every software difference between the arms. No cost figure is published here because none is a bill. No account identifiers, image identifiers or credentials from the plan documents appear in this record.

Setup

The two incumbents, from the public OSWorld provider sources at 04718ea:

ProviderReuse semanticsHost it needs
virtualboxsave_state / revert_to_snapshot snapshot a running VM: live state, applications includedbare metal (no nested virtualization documented as supported); VirtualBox not installed anywhere yet; the Extension Pack's separate licence is "NOT ASSUMED"
docker (happysixd/osworld-docker, pinned by digest)save_state raises NotImplementedError; revert_to_snapshot stops and removes the container, so every reset after the first is a full cold boot of a new QEMU/KVM VM; guest budget hard-coded at 4 CPU and 4 GiBany host with /dev/kvm; the provider silently runs without acceleration if /dev/kvm is missing, so the runner refuses unless the container really has it

The Drums arm in both stages: the L0 session type measured in EXP-005 and EXP-023, with cgroup limits matching the guest budget (2 CPU / 4 GiB for VirtualBox's configured VM, 4 CPU / 4 GiB for the Docker guest), reuse OFF on the small stage to match the incumbent's cold reset. Ideally on the image root of EXP-026, which has not reached a dump.

Artifacts pinned by bytes (read from registry metadata and the dataset's LFS pointer, nothing downloaded in the sandbox): the Docker image digest sha256:0e6497a9...baa9 (created 2024-10-17, 84,258,814 compressed bytes); the qcow2 zip at a fixed dataset commit, 12,273,896,463 bytes, sha256 b795b6cd...6712, one member uncompressed 24,460,197,888 bytes; disk budget 120 GiB from measured bytes plus named allowances. The dataset card's License field is blank: "unresolved uncertainty, neither a ban nor permission".

Controls (awsctl.py, "PROTOTYPE, UNEXECUTED ... it has never called AWS"): dry run unless --execute; an owner-written approval file; exactly one tagged host account-wide; a per-stage runtime ceiling enforced by an in-host shutdown timer, an external watcher with a heartbeat receipt, and a one-time managed scheduler deadline created and read back right after launch; deletion of owned volumes verified at teardown; no Elastic IP; SSH from one /32 only, inspected rather than trusted; export through an allowlist with a secret scan. Both example approval files on the branch carry "approved": false.

What ran

Nothing on a host. What exists and was unit-tested with fakes: host_run.py (check, gates, run, export, each refusing unless the previous stage passed for the same config hash), stages.py (its live functions "have never run"), vbox.py ("no real VM has been read yet"), docker_telemetry.py ("never run against a real container"), imagesize.py (bounded HTTP range reads of the zip's central directory, refusing a server that ignores Range), awsctl.py (the worst-case bill arithmetic is reproduced by a unit test), the bootstrap and fetch scripts (refusing root, non-24.04, a virtualized host for the VirtualBox stage, and downloads without the pinned sizes and hash). The sandbox's own preflight probe says go_incumbent: false (no vmx/svm, no VirtualBox) and go_joint_run_on_this_host: false.

Two author-reported statements touch real AWS use and have no artifact on the branch: matched-software-plan.md section 8 says "The earlier actual m7i.xlarge run did use KVM for the stock Docker arm (4 CPU / 4 GiB)", with no numbers, which contradicts host_run.py's "NEVER run" unless that run was manual; and the small-stage plan says the Canonical AMI was "read live", which implies at least one describe call by the owner. We treat both as author claims. The quota status is recorded as: an EC2 vCPU quota increase was pending (the runbook); a later planning sentence assumes it granted; neither has evidence on the branch.

Results

None. The record exists so that the gates are public before the run.

What the first result would and would not be (from the plans, quoted): the small stage's first result "is an ARCHITECTURE MEASUREMENT of cold-reset task execution on one task: not a comparison (software composition is not shown equivalent), not a snapshot or live-state result, no throughput, cost-advantage, training or benchmark claim." The VirtualBox stage's first result "will be an engineering validation on ONE task: no throughput, no cost-advantage, no training and no benchmark claim."

Equivalence evidence required before "comparison" may be used:

  1. An image_manifest.json read from the booted guest (OS, kernel, desktop, browser, LibreOffice, package list), compared with the Drums composition manifest; every difference (openbox versus GNOME, Chromium 141 versus Chrome 130, LibreOffice 24.2 versus 7.3.7.2, different kernels) needs measured, digest-bound evidence that it does not affect the task, or the run stays NOT_A_COMPARISON.
  2. The live-state gate on both arms: a process holding an in-memory token and a file marker survive a snapshot and revert after being killed or deleted post-save; the real evaluator scores correct 1.0, wrong 0.0, correct again 1.0 from the same saved state. An application window with typed text is not covered and is reported as such.
  3. Resource parity: the VM's vCPU and RAM verified from the hypervisor, the Drums cgroup limits verified as applied to every live session, and per-environment capture on both arms. Today "none of that holds" (EXP-023).
  4. The interference canary clean on both arms; at least three alternating blocks; a verdict of WIN only if the bootstrap lower bound of the ratio exceeds 1.0 and the isolation audit, the equivalence pilot and the block count all hold (matched-comparison-protocol.md). The author's own prediction: "I expect the current Drums arm (L0 ...) to tie or lose."

Failures and surprises

Limitations

Reproduce

Not applicable until a run exists. The gate sequence when it does: awsctl plan and awsctl launch --execute with an owner approval file; on the host deploy/aws_host/bootstrap.sh (STAGE=docker or vbox), fetch_docker_image.sh or fetch_image.sh --accept-licence-risk; python -m drums_compare.host_run check, then gates, then run, then export; awsctl teardown must report clean: true. Any refusal ends the attempt; the plans forbid debugging on the clock.

Sources