DrumsBench

Experiments / EXP-031

T1 clean reset, same host and tasks: Drums 0.96 s against OSWorld Docker 20.9 s at each arm's operating point, replicated on a fresh host; the same study lost on cost to tuned KVM

On identical hardware, tasks and policy, compare the Drums release as a customer installs it with the strongest self-hostable incumbents on valid rollouts per dollar, memory per desktop, reset p50 and p99, max sustained concurrency and invalid-reward rate.

verifiedBacked by measurement files we can point to, on named hardware, with the run recorded.
Date
2026-11-08 to 2026-11-09
Arms
drums-api, qemu-tuned-shared, osgym-reimpl, osworld-docker
Hardware
r8i.16xlarge (64 vCPU, 512 GiB, nested virtualization on), on-demand; host A and a fresh host B
OS
Ubuntu 24.04.5, kernel 7.0.0-1014-aws
Workload
55 OSWorld tasks, scripted mock policy (no model), ladder from 8 toward 512 desktops, 300 s per step
Code
sota/t1 at a5b8d66 (private research archive)
Evidence
runbooks/sota/t1/20261108T0700Z/PREREGISTRATION.md and FREEZE.txt (frozen 2026-11-09T03:44:30Z, sha256 9246e879...)
runbooks/sota/t1/20261108T0700Z/REPORT.md, REPRODUCE.md, RUNLOG.md, DEVIATIONS.md
runbooks/sota/t1/20261108T0700Z/summary/tables.md and summary.json
runbooks/sota/t1/20261108T0700Z/raw/ (raw-A-full, raw-B-full, MANIFEST.json, SHA256SUMS)
Limitation
The 0.96 s and 20.9 s are per-rollout resets at different operating points (32 and 16 desktops); a scripted policy, no model; and the Drums build measured here lost valid rollouts per dollar to tuned KVM (0.39x).

In one paragraph

Every arm was installed on a fresh r8i.16xlarge from one command and climbed a concurrency ladder on the same 55 OSWorld tasks with the same scripted policy; the whole run was repeated on a second fresh host. At each arm's operating point the median clean reset was 0.961 s for Drums (32 desktops) and 20.9 s for OSWorld's stock Docker provider (16), CIs [0.954, 0.968] and [20.777, 21.040], and host B agreed within 6% (0.956 s and 20.85 s). At equal load (32 desktops) Docker's reset was 24.2 s with 11% of its rollouts invalid. Two things must be read with it. First, the OSGym-style arm's pre-booted spare reset faster than Drums at p50 (0.006 s) and much slower at p99 (55 s). Second, this Drums build could not bring up its 64th desktop (a per-user pipe-buffer limit), so it lost to tuned KVM on valid rollouts per dollar, 0.39x [0.36, 0.42], on both hosts. The fix and its pre-registered re-run are EXP-032.

Setup

Hosts A and B: r8i.16xlarge, us-east-1a, on-demand, 64 vCPU, 512 GiB, nested virtualization on, gp3 600 GiB at 16k IOPS. Drums: release tarball with drumsd bd43483, drums-lab cb3145d, drums-criu b4fbfddf (the RC1 pins), installed with its own install.sh, hardened profile on, 4 vCPU and 4 GiB per environment, driven through the drumsd HTTP API. Tuned KVM: QEMU 8.2.2, reset by -incoming from one RAM-resident capture. OSWorld Docker: OSWorld 044bc339 DockerProvider unchanged; reset removes the container and cold-boots it. OSGym: our labelled reimplementation of arXiv 2511.11672 (no official code exists). CUA-Sandbox was not run (its OSWorld adapter is not released). Price 4.44544 USD/h on-demand.

What ran

Pass 1 ladders for every arm on host A, a second pass at operating points, and the same on fresh host B touched only by the reproduce script. Policy: oracle, explore, stall, loop and wrong behaviours, 15 rounds, 2.0 s step pause, seed 20261108.

Results

T1: clean reset at each arm's operating point, same host and tasks, host A and fresh host B verified2026-11-09 · hosts A and B

MetricValueUnitnNote
Drums (operating N 32): reset p50, host A0.961s95% CI [0.954, 0.968]; p99 2.338 s; host B p50 0.956, p99 2.351
OSWorld Docker (operating N 16): reset p50, host A20.89s95% CI [20.777, 21.040]; p99 27.182 s; host B p50 20.849, p99 27.519
Equal N = 32: OSWorld Docker reset p5024.21s95% CI [23.962, 24.625]; 11% invalid at N = 32 (21 of 194), not sustained
Tuned KVM (operating N 128): reset p50, host A2.49sp99 7.350 s; at equal N = 32: 1.108 s [1.104, 1.113]
OSGym-style reimplementation (operating N 16): reset p50, host A0.006sa pre-booted spare; p99 55.228 s. Faster than Drums at p50
OSWorld Docker failed resets at N = 161 / 103resets
Same study: Drums / tuned KVM, valid rollouts per USD, host A0.39x5595% CI [0.36, 0.42]; Drums WORSE (64th desktop failed: per-user pipe-buffer limit); host B 0.39 [0.36, 0.42]
Same study: Drums memory per desktop at operating N0.765GiB95% CI [0.697, 0.819]; tuned KVM 2.674, OSWorld Docker 2.649

Host: r8i.16xlarge (64 vCPU, 512 GiB, nested virtualization on), on-demand, us-east-1a, gp3 600 GiB 16k IOPS, Ubuntu 24.04.5, kernel 7.0.0-1014-aws. B is a fresh host touched only by the reproduce script. Workload: 55 OSWorld tasks (tasks55.txt), scripted mock policy (no model): oracle, explore, stall, loop and wrong behaviours, 15 rounds, seed 20261108; ladder from 8 toward 512 desktops, 300 s per step. Method: Pre-registered (frozen 2026-11-09T03:44:30Z, sha256 9246e879...). Reset = per-rollout clean reset as each arm does it (Drums: drumsd reset; OSWorld Docker: remove the container and cold-boot it; tuned KVM: qemu -incoming from a RAM-resident capture). Reset verdict rule: the two arms' 95% CIs do not overlap. Replication tolerance +-20%.. Source: research archive: runbooks/sota/t1/20261108T0700Z/REPORT.md; research archive: runbooks/sota/t1/20261108T0700Z/summary/tables.md (Run A-full, Replication B-full, equal-N table); research archive: runbooks/sota/t1/20261108T0700Z/raw (raw-A-full, raw-B-full, SHA256SUMS).

  • Each arm's reset is at its own operating point (Drums 32, Docker 16). At equal N = 32 the comparison is 0.96 s vs 24.2 s.
  • Drums' 0.96 s here is the RC1 build before the pipe fix, which also lost valid rollouts per dollar to tuned KVM in this study (0.39x). The fixed build is EXP-032.
  • The 0.56 s one-desktop idle-host reset (study T2) is a different measurement on a different host type and must not be paired with Docker's 20.9 s.
  • No model in the loop. One instance type. The OSGym arm is our own reimplementation (no official code exists).

Failures and surprises

Limitations

No model in the loop. One instance type. Resets are each arm's own reset at its own operating point. The 0.56 s Drums reset quoted elsewhere is a different measurement (one desktop, idle m8i.16xlarge host, study T2) and is not paired with Docker's 20.9 s. The Docker valid-per-dollar verdict is underpowered under the 200-rollout rule; that limit applies to cost, not to reset time.

Reproduce

REPRODUCE.md in the study folder (t1-reproduce.sh on a fresh host from the same launch template). Harness commit a5b8d66.

Sources

Pre-registration, report, summary tables and raw archives with checksums, listed under Evidence. Research archive, internal for now.