In one paragraph
From the same driver host, with the same harness, 55 tasks, policies, 32 concurrent desktops and window, Prime's hosted sandboxes gave 200.7 valid rollouts per dollar and Drums 624.0. Paired over tasks, the Prime/Drums ratio is 0.32, 95% CI [0.19, 0.48], so Drums gives about 3.1x the valid rollouts per dollar; on fresh hosts it replicated at 0.32 [0.19, 0.49] (Prime +0.1%, Drums -0.2%). The rule was pre-registered as amendment D7 before any cost measurement. Prime is strong on one thing Drums' cold fork is not fast at: its new memory fork kept Chrome and Calc state in 280 of 280 children. The price is time: fork(4) took 11.0 s p50 because the parent must be paused first (about 8 s), against Drums' cold 2.35 s; clean reset was 2.18 s (replication-qualified) against Drums' 0.56 s. Those two are unpaired.
Setup
Arm label: Prime Intellect hosted sandboxes, prime-sandboxes 0.5.9 (released the same day, adding pause() and create(checkpoint_id, restore_memory=True)). Cost run: the osworld-e2b d762ce2 recipe built on Prime's builder, 4 vCPU / 8 GiB / 10 GiB, egress denied, clean state = pause checkpoint, reset = delete plus create with restore_memory. Drums: the T1-fixed release (drumsd a8d118b) on a fresh r8i.16xlarge, driven from the same m7i.2xlarge through an ssh tunnel (API 1.3 to 2.5 ms). Cost basis: Prime list price on sandbox-seconds clipped to the rung window; Drums the host's on-demand list price (4.44544 USD/h) on the same window; both also on the whole-ledger basis.
What ran
2026-11-10: fork and reset phases, main run (N = 40) and a fresh driver (N = 30). 2026-11-11: the D7 cost run at N = 32 for both arms, then both again on fresh hosts; Drums also at N = 128 (its own operating point, secondary) and Prime at N = 64 (secondary).
Results
Valid rollouts per USD at 32 concurrent: Prime Intellect hosted sandboxes against Drums, same remote driver, paired over 55 tasks verified2026-11-11 · driver + Drums host + Prime cloud
| Metric | Value | Unit | n | Note |
|---|---|---|---|---|
| Prime / Drums valid-per-USD ratio, main run | 0.32 | x | 55 | 95% CI [0.19, 0.48]; about 3.1x more valid rollouts per USD on Drums |
| Prime / Drums valid-per-USD ratio, replication (fresh driver, fresh Drums host) | 0.32 | x | 55 | 95% CI [0.19, 0.49]; Prime +0.1%, Drums -0.2% |
| Prime N = 32, valid rollouts per USD (main / replication) | 200.7 | per USD | replication 201.0; 569 of 580 valid (98.1%); whole-ledger 186.5 | |
| Drums N = 32, valid rollouts per USD (main / replication) | 624 | per USD | replication 623.0; 1,097 of 1,097 valid; whole-ledger 609.6 | |
| Prime N = 32, valid rollouts per hour | 1,180 | per h | Drums N = 32: 2,774 per h | |
| Drums at N = 128 (its own operating point, single run), valid per USD | 1,496.8 | per USD | not replicated; driver-bound (D13); no ratio or CI is computed against Prime's N = 32 | |
| Checker-correct rollouts (an aside, not the metric): Prime / Drums at N = 32 | 0 to 22 | rollouts | Prime 22 of 580, Drums 0 of 1,097 through the tunnel; needs a look before anyone quotes a correct rate |
Host: driver m7i.2xlarge (us-east-1a); Drums on r8i.16xlarge on-demand (4.44544 USD/h); Prime sandboxes 4 vCPU / 8 GiB / 10 GiB, region 'us', Ubuntu 24.04 (driver and Drums host); Prime guest xfce4 on Xvfb with mutter. Drums driven through an ssh tunnel from the same driver (API 1.3 to 2.5 ms). Workload: 55-task OSWorld suite with the T1 scripted policies (oracle, explore, stall, loop, wrong), seed 20261108, reps 40, OSWorld checkers on the driver, one 1,200 s rung per arm at N = 32. Method: Amendment D7, pre-registered 2026-11-11T04:21Z before any M7 measurement. Paired claim rule: per-task bootstrap over 55 tasks, 10,000 resamples, seed 20261108, 95% CI of Prime/Drums excluding 1, at least 24 paired tasks and 200 rollouts per arm. Cost: Prime list price on sandbox-seconds in the rung window; Drums the host's on-demand list price on the same window. Replication: both arms on fresh hosts, tolerance +-25%.. Source: research archive: runbooks/sota/prime/20261110T0400Z/REPORT.md (section 9); research archive: runbooks/sota/prime/20261110T0400Z/DEVIATIONS.md (D7, D8, D13); research archive: runbooks/sota/prime/20261110T0400Z/raw/m7/out/m7-analysis-main.json; research archive: runbooks/sota/prime/20261110T0400Z/raw/m7/out-repl/m7-analysis-replication.json.
- Paired over tasks with the same driver, N and window, but different hardware under the desktops and a different price basis (Prime list price vs AWS on-demand list price). Not same-host.
- Valid means the harness's validity gates passed (a rollout a trainer could use); it is not task success. No model was in the loop.
- The Drums host is paid at its full hourly rate at N = 32, well below its operating point.
- The Prime desktop runs mutter, not GNOME Shell (D8); Prime's 11 invalid rollouts are all one task (5bc63fb9).
- The 'about 8.4x at own operating points' figure is not in the finished report and is not used here.
Prime Intellect hosted sandboxes: fork(4) and clean reset of a running desktop (unpaired against Drums' same-host T2 numbers) verified2026-11-10 · driver + Prime cloud
| Metric | Value | Unit | n | Note |
|---|---|---|---|---|
| Memory fork(4): pause, then 4 creates with restore_memory, until all 4 usable | 11.04 | s | 40 | 95% CI [10.811, 11.597]; p99 18.136; replication 10.934 s (-0.9%) |
| Children with Chrome and Calc state intact (memory fork) | 280 / 280 | children | 160/160 main, 120/120 replication; the filesystem fork kept 0/280 | |
| Clean reset, memory: main run | 2.93 | s | 40 | p99 33.8 s; two resets took 33.8 and 29.4 s; outside tolerance |
| Clean reset, memory: replication (the replication-qualified headline) | 2.18 | s | 30 | p99 2.888 s; clean 30/30 |
| Drums reference (study T2, one desktop, same-host client): fork(4) / clean reset | 0.56 to 2.35 | s | fork(4) p50 2.349 s, reset p50 0.560 s; unpaired |
Host: driver m7i.large on-demand (us-east-1c); Prime sandboxes 4 vCPU / 4 GiB, Prime guest xfce4 on Xvfb. internet path on every call; placement not observable. Workload: Fork a desktop with Chrome and LibreOffice Calc open and typed text into 4 copies; clean reset = new sandbox from the clean pause checkpoint, to first screenshot. Method: Pre-registered (FREEZE 2026-11-10T05:16:19Z, sha256 dba6ad3b...). N = 40 main, N = 30 replication on a fresh driver; tolerance +-25%; replication-qualified headline where the two runs differ.. Source: research archive: runbooks/sota/prime/20261110T0400Z/REPORT.md; research archive: runbooks/sota/prime/20261110T0400Z/raw/main/tables.md (rows 2a, 4a); research archive: runbooks/sota/prime/20261110T0400Z/raw/replication/tables.md (rows 2a, 4a).
- Unpaired: vendor cloud vs our host, different hardware, guest and network position. Every Prime/Drums ratio is labelled unpaired.
- Prime's fork is pause-then-create; the parent is paused about 8 s and unusable for about 13 s around each fork. The memory-fork primitive was released by Prime hours before measurement.
- Prime keeps a running desktop's state in every child; Drums' cold fork does too but takes 2.35 s, above the 250 ms bar.
Failures and surprises
- The main-run memory reset was outside the replication tolerance (-25.7%): resets 6 and 7 took 33.8 and 29.4 s to reach RUNNING and later ones sat at 2.5 to 3.4 s, which looks like a placement change on Prime's side. The headline is the replication-qualified 2.18 s, with the 2.93 s main run next to it.
- Checker-correct counts through the tunnel were 0 of 1,097 for Drums against 22 of 580 for Prime. Validity gates passed on both, and correct rate is not the metric, but it needs a look before anyone quotes a correct rate.
- Thirteen numbered deviations; D1 to D9 recorded before the phase they affect.
Limitations
Different hardware and price basis: label this "paired over tasks, unpaired on hardware". The Drums host is paid in full at 32 desktops, far below its operating point. Prime's desktop runs mutter, not GNOME Shell (D8), and its 11 invalid rollouts are all one task. Prime's own-operating-point comparison (Drums 1,497 at N = 128 against Prime 201 at N = 32) is a single run with no ratio or interval and is not a claim.
Reproduce
T1 harness sota/t1 at d2d0acb on an m7i.2xlarge driver; REPRODUCE.md and the D7 amendment in the study folder.
Sources
Pre-registration, the D7 amendment, the report, and the analysis JSON and raw tables under Evidence. Research archive, internal for now.