DrumsBench

Experiments / EXP-034

Prime Intellect hosted sandboxes against Drums at 32 concurrent: Drums about 3.1x the valid rollouts per dollar, paired over 55 tasks, replicated; Prime's fork keeps state but takes 11 s

Measure Prime Intellect's hosted sandboxes as an incumbent on fork, clean reset and valid rollouts per dollar, tuned per Prime's own docs, against Drums from the same driver.

verifiedBacked by measurement files we can point to, on named hardware, with the run recorded.
Date
2026-11-10 to 2026-11-11
Arms
prime-intellect-hosted, drums-api
Hardware
driver m7i.2xlarge; Drums on r8i.16xlarge on-demand; Prime sandboxes 4 vCPU / 8 GiB (cost run) and 4 vCPU / 4 GiB (fork and reset)
OS
Ubuntu 24.04 (driver, Drums host); Prime guest xfce4 on Xvfb with mutter
Workload
55-task OSWorld suite, T1 scripted policies, N = 32, one 1,200 s rung per arm; fork(4) of a desktop with Chrome and Calc open; clean reset from a pause checkpoint
Code
sota/t1 at d2d0acb (private research archive)
Evidence
runbooks/sota/prime/20261110T0400Z/PREREGISTRATION.md and FREEZE.txt (frozen 2026-11-10T05:16:19Z, sha256 dba6ad3b...)
runbooks/sota/prime/20261110T0400Z/DEVIATIONS.md (D7, the cost amendment, pre-registered 2026-11-11T04:21Z)
runbooks/sota/prime/20261110T0400Z/REPORT.md
runbooks/sota/prime/20261110T0400Z/raw/m7/out/m7-analysis-main.json and raw/m7/out-repl/m7-analysis-replication.json
runbooks/sota/prime/20261110T0400Z/raw/main/tables.md, raw/replication/tables.md, MANIFEST.json, SHA256SUMS
Limitation
Paired over tasks with the same driver and N, but different hardware under the desktops and a different price basis (Prime list price against AWS on-demand list price); a scripted policy, no model.

In one paragraph

From the same driver host, with the same harness, 55 tasks, policies, 32 concurrent desktops and window, Prime's hosted sandboxes gave 200.7 valid rollouts per dollar and Drums 624.0. Paired over tasks, the Prime/Drums ratio is 0.32, 95% CI [0.19, 0.48], so Drums gives about 3.1x the valid rollouts per dollar; on fresh hosts it replicated at 0.32 [0.19, 0.49] (Prime +0.1%, Drums -0.2%). The rule was pre-registered as amendment D7 before any cost measurement. Prime is strong on one thing Drums' cold fork is not fast at: its new memory fork kept Chrome and Calc state in 280 of 280 children. The price is time: fork(4) took 11.0 s p50 because the parent must be paused first (about 8 s), against Drums' cold 2.35 s; clean reset was 2.18 s (replication-qualified) against Drums' 0.56 s. Those two are unpaired.

Setup

Arm label: Prime Intellect hosted sandboxes, prime-sandboxes 0.5.9 (released the same day, adding pause() and create(checkpoint_id, restore_memory=True)). Cost run: the osworld-e2b d762ce2 recipe built on Prime's builder, 4 vCPU / 8 GiB / 10 GiB, egress denied, clean state = pause checkpoint, reset = delete plus create with restore_memory. Drums: the T1-fixed release (drumsd a8d118b) on a fresh r8i.16xlarge, driven from the same m7i.2xlarge through an ssh tunnel (API 1.3 to 2.5 ms). Cost basis: Prime list price on sandbox-seconds clipped to the rung window; Drums the host's on-demand list price (4.44544 USD/h) on the same window; both also on the whole-ledger basis.

What ran

2026-11-10: fork and reset phases, main run (N = 40) and a fresh driver (N = 30). 2026-11-11: the D7 cost run at N = 32 for both arms, then both again on fresh hosts; Drums also at N = 128 (its own operating point, secondary) and Prime at N = 64 (secondary).

Results

Valid rollouts per USD at 32 concurrent: Prime Intellect hosted sandboxes against Drums, same remote driver, paired over 55 tasks verified2026-11-11 · driver + Drums host + Prime cloud

MetricValueUnitnNote
Prime / Drums valid-per-USD ratio, main run0.32x5595% CI [0.19, 0.48]; about 3.1x more valid rollouts per USD on Drums
Prime / Drums valid-per-USD ratio, replication (fresh driver, fresh Drums host)0.32x5595% CI [0.19, 0.49]; Prime +0.1%, Drums -0.2%
Prime N = 32, valid rollouts per USD (main / replication)200.7per USDreplication 201.0; 569 of 580 valid (98.1%); whole-ledger 186.5
Drums N = 32, valid rollouts per USD (main / replication)624per USDreplication 623.0; 1,097 of 1,097 valid; whole-ledger 609.6
Prime N = 32, valid rollouts per hour1,180per hDrums N = 32: 2,774 per h
Drums at N = 128 (its own operating point, single run), valid per USD1,496.8per USDnot replicated; driver-bound (D13); no ratio or CI is computed against Prime's N = 32
Checker-correct rollouts (an aside, not the metric): Prime / Drums at N = 320 to 22rolloutsPrime 22 of 580, Drums 0 of 1,097 through the tunnel; needs a look before anyone quotes a correct rate

Host: driver m7i.2xlarge (us-east-1a); Drums on r8i.16xlarge on-demand (4.44544 USD/h); Prime sandboxes 4 vCPU / 8 GiB / 10 GiB, region 'us', Ubuntu 24.04 (driver and Drums host); Prime guest xfce4 on Xvfb with mutter. Drums driven through an ssh tunnel from the same driver (API 1.3 to 2.5 ms). Workload: 55-task OSWorld suite with the T1 scripted policies (oracle, explore, stall, loop, wrong), seed 20261108, reps 40, OSWorld checkers on the driver, one 1,200 s rung per arm at N = 32. Method: Amendment D7, pre-registered 2026-11-11T04:21Z before any M7 measurement. Paired claim rule: per-task bootstrap over 55 tasks, 10,000 resamples, seed 20261108, 95% CI of Prime/Drums excluding 1, at least 24 paired tasks and 200 rollouts per arm. Cost: Prime list price on sandbox-seconds in the rung window; Drums the host's on-demand list price on the same window. Replication: both arms on fresh hosts, tolerance +-25%.. Source: research archive: runbooks/sota/prime/20261110T0400Z/REPORT.md (section 9); research archive: runbooks/sota/prime/20261110T0400Z/DEVIATIONS.md (D7, D8, D13); research archive: runbooks/sota/prime/20261110T0400Z/raw/m7/out/m7-analysis-main.json; research archive: runbooks/sota/prime/20261110T0400Z/raw/m7/out-repl/m7-analysis-replication.json.

  • Paired over tasks with the same driver, N and window, but different hardware under the desktops and a different price basis (Prime list price vs AWS on-demand list price). Not same-host.
  • Valid means the harness's validity gates passed (a rollout a trainer could use); it is not task success. No model was in the loop.
  • The Drums host is paid at its full hourly rate at N = 32, well below its operating point.
  • The Prime desktop runs mutter, not GNOME Shell (D8); Prime's 11 invalid rollouts are all one task (5bc63fb9).
  • The 'about 8.4x at own operating points' figure is not in the finished report and is not used here.

Prime Intellect hosted sandboxes: fork(4) and clean reset of a running desktop (unpaired against Drums' same-host T2 numbers) verified2026-11-10 · driver + Prime cloud

MetricValueUnitnNote
Memory fork(4): pause, then 4 creates with restore_memory, until all 4 usable11.04s4095% CI [10.811, 11.597]; p99 18.136; replication 10.934 s (-0.9%)
Children with Chrome and Calc state intact (memory fork)280 / 280children160/160 main, 120/120 replication; the filesystem fork kept 0/280
Clean reset, memory: main run2.93s40p99 33.8 s; two resets took 33.8 and 29.4 s; outside tolerance
Clean reset, memory: replication (the replication-qualified headline)2.18s30p99 2.888 s; clean 30/30
Drums reference (study T2, one desktop, same-host client): fork(4) / clean reset0.56 to 2.35sfork(4) p50 2.349 s, reset p50 0.560 s; unpaired

Host: driver m7i.large on-demand (us-east-1c); Prime sandboxes 4 vCPU / 4 GiB, Prime guest xfce4 on Xvfb. internet path on every call; placement not observable. Workload: Fork a desktop with Chrome and LibreOffice Calc open and typed text into 4 copies; clean reset = new sandbox from the clean pause checkpoint, to first screenshot. Method: Pre-registered (FREEZE 2026-11-10T05:16:19Z, sha256 dba6ad3b...). N = 40 main, N = 30 replication on a fresh driver; tolerance +-25%; replication-qualified headline where the two runs differ.. Source: research archive: runbooks/sota/prime/20261110T0400Z/REPORT.md; research archive: runbooks/sota/prime/20261110T0400Z/raw/main/tables.md (rows 2a, 4a); research archive: runbooks/sota/prime/20261110T0400Z/raw/replication/tables.md (rows 2a, 4a).

  • Unpaired: vendor cloud vs our host, different hardware, guest and network position. Every Prime/Drums ratio is labelled unpaired.
  • Prime's fork is pause-then-create; the parent is paused about 8 s and unusable for about 13 s around each fork. The memory-fork primitive was released by Prime hours before measurement.
  • Prime keeps a running desktop's state in every child; Drums' cold fork does too but takes 2.35 s, above the 250 ms bar.

Failures and surprises

Limitations

Different hardware and price basis: label this "paired over tasks, unpaired on hardware". The Drums host is paid in full at 32 desktops, far below its operating point. Prime's desktop runs mutter, not GNOME Shell (D8), and its 11 invalid rollouts are all one task. Prime's own-operating-point comparison (Drums 1,497 at N = 128 against Prime 201 at N = 32) is a single run with no ratio or interval and is not a claim.

Reproduce

T1 harness sota/t1 at d2d0acb on an m7i.2xlarge driver; REPRODUCE.md and the D7 amendment in the study folder.

Sources

Pre-registration, the D7 amendment, the report, and the analysis JSON and raw tables under Evidence. Research archive, internal for now.