In one paragraph
Every arm was installed on a fresh r8i.16xlarge from one command and climbed a concurrency ladder on the same 55 OSWorld tasks with the same scripted policy; the whole run was repeated on a second fresh host. At each arm's operating point the median clean reset was 0.961 s for Drums (32 desktops) and 20.9 s for OSWorld's stock Docker provider (16), CIs [0.954, 0.968] and [20.777, 21.040], and host B agreed within 6% (0.956 s and 20.85 s). At equal load (32 desktops) Docker's reset was 24.2 s with 11% of its rollouts invalid. Two things must be read with it. First, the OSGym-style arm's pre-booted spare reset faster than Drums at p50 (0.006 s) and much slower at p99 (55 s). Second, this Drums build could not bring up its 64th desktop (a per-user pipe-buffer limit), so it lost to tuned KVM on valid rollouts per dollar, 0.39x [0.36, 0.42], on both hosts. The fix and its pre-registered re-run are EXP-032.
Setup
Hosts A and B: r8i.16xlarge, us-east-1a, on-demand, 64 vCPU, 512 GiB, nested virtualization on, gp3 600 GiB at 16k IOPS. Drums: release tarball with drumsd bd43483, drums-lab cb3145d, drums-criu b4fbfddf (the RC1 pins), installed with its own install.sh, hardened profile on, 4 vCPU and 4 GiB per environment, driven through the drumsd HTTP API. Tuned KVM: QEMU 8.2.2, reset by -incoming from one RAM-resident capture. OSWorld Docker: OSWorld 044bc339 DockerProvider unchanged; reset removes the container and cold-boots it. OSGym: our labelled reimplementation of arXiv 2511.11672 (no official code exists). CUA-Sandbox was not run (its OSWorld adapter is not released). Price 4.44544 USD/h on-demand.
What ran
Pass 1 ladders for every arm on host A, a second pass at operating points, and the same on fresh host B touched only by the reproduce script. Policy: oracle, explore, stall, loop and wrong behaviours, 15 rounds, 2.0 s step pause, seed 20261108.
Results
T1: clean reset at each arm's operating point, same host and tasks, host A and fresh host B verified2026-11-09 · hosts A and B
| Metric | Value | Unit | n | Note |
|---|---|---|---|---|
| Drums (operating N 32): reset p50, host A | 0.961 | s | 95% CI [0.954, 0.968]; p99 2.338 s; host B p50 0.956, p99 2.351 | |
| OSWorld Docker (operating N 16): reset p50, host A | 20.89 | s | 95% CI [20.777, 21.040]; p99 27.182 s; host B p50 20.849, p99 27.519 | |
| Equal N = 32: OSWorld Docker reset p50 | 24.21 | s | 95% CI [23.962, 24.625]; 11% invalid at N = 32 (21 of 194), not sustained | |
| Tuned KVM (operating N 128): reset p50, host A | 2.49 | s | p99 7.350 s; at equal N = 32: 1.108 s [1.104, 1.113] | |
| OSGym-style reimplementation (operating N 16): reset p50, host A | 0.006 | s | a pre-booted spare; p99 55.228 s. Faster than Drums at p50 | |
| OSWorld Docker failed resets at N = 16 | 1 / 103 | resets | ||
| Same study: Drums / tuned KVM, valid rollouts per USD, host A | 0.39 | x | 55 | 95% CI [0.36, 0.42]; Drums WORSE (64th desktop failed: per-user pipe-buffer limit); host B 0.39 [0.36, 0.42] |
| Same study: Drums memory per desktop at operating N | 0.765 | GiB | 95% CI [0.697, 0.819]; tuned KVM 2.674, OSWorld Docker 2.649 |
Host: r8i.16xlarge (64 vCPU, 512 GiB, nested virtualization on), on-demand, us-east-1a, gp3 600 GiB 16k IOPS, Ubuntu 24.04.5, kernel 7.0.0-1014-aws. B is a fresh host touched only by the reproduce script. Workload: 55 OSWorld tasks (tasks55.txt), scripted mock policy (no model): oracle, explore, stall, loop and wrong behaviours, 15 rounds, seed 20261108; ladder from 8 toward 512 desktops, 300 s per step. Method: Pre-registered (frozen 2026-11-09T03:44:30Z, sha256 9246e879...). Reset = per-rollout clean reset as each arm does it (Drums: drumsd reset; OSWorld Docker: remove the container and cold-boot it; tuned KVM: qemu -incoming from a RAM-resident capture). Reset verdict rule: the two arms' 95% CIs do not overlap. Replication tolerance +-20%.. Source: research archive: runbooks/sota/t1/20261108T0700Z/REPORT.md; research archive: runbooks/sota/t1/20261108T0700Z/summary/tables.md (Run A-full, Replication B-full, equal-N table); research archive: runbooks/sota/t1/20261108T0700Z/raw (raw-A-full, raw-B-full, SHA256SUMS).
- Each arm's reset is at its own operating point (Drums 32, Docker 16). At equal N = 32 the comparison is 0.96 s vs 24.2 s.
- Drums' 0.96 s here is the RC1 build before the pipe fix, which also lost valid rollouts per dollar to tuned KVM in this study (0.39x). The fixed build is EXP-032.
- The 0.56 s one-desktop idle-host reset (study T2) is a different measurement on a different host type and must not be paired with Docker's 20.9 s.
- No model in the loop. One instance type. The OSGym arm is our own reimplementation (no official code exists).
Failures and surprises
- Drums' operating point stopped at 32 because every hardened environment shares one host uid, which hits the per-user pipe-buffer limit at the 64th desktop. An exploratory probe (not pre-registered) removed the limit and 256 desktops were sustained.
- OSWorld Docker: 1 of 103 resets failed at 16 desktops; 21 of 194 rollouts invalid at 32.
Limitations
No model in the loop. One instance type. Resets are each arm's own reset at its own operating point. The 0.56 s Drums reset quoted elsewhere is a different measurement (one desktop, idle m8i.16xlarge host, study T2) and is not paired with Docker's 20.9 s. The Docker valid-per-dollar verdict is underpowered under the 200-rollout rule; that limit applies to cost, not to reset time.
Reproduce
REPRODUCE.md in the study folder (t1-reproduce.sh on a fresh host from the same launch template). Harness commit a5b8d66.
Sources
Pre-registration, report, summary tables and raw archives with checksums, listed under Evidence. Research archive, internal for now.