In one paragraph
A real agent (ScaleCUA, a 9B open-weight model fine-tuned for OSWorld) ran the same 55 OSWorld tasks on five stacks that shared one environment host and one 8-GPU host. Cost counts both hosts for the whole wall time, failed attempts included; the metric is verified (training-eligible) rollouts per dollar, with successes per dollar beside it. Under the frozen report tool Drums gave 2.97x OSWorld Docker and 2.97x Firecracker, 1.15x our OSGym-style reimplementation, and 0.90x tuned KVM: a loss, CI [0.881, 0.914]. Amendment A1 re-ran the Drums arm on the release build (with the Chrome memory-protection-keys fix): 2,132 of 2,142 rollouts verified, and the result did not change. A1 also found that the frozen primary charged every arm about two hours of an attempt invalidated by a vLLM outage; charging each arm only its valid attempt gives 4.9x against Docker and Firecracker and a tie with tuned KVM (0.995, CI [0.965, 1.021]). That recomputation was chosen after the outcome, so the frozen numbers stay the pre-registered result.
Setup
One environment host (r8i.16xlarge, 64 vCPU, 512 GB, 4.44544 USD/h on-demand) and one GPU host (p4d.24xlarge, 8 x A100, spot 16.9631 USD/h). The model is served by vLLM 0.19 with data parallelism 8. Each stack first climbed its own concurrency ladder; its operating point is the highest level it sustained: Drums 128, tuned KVM 128, OSGym-style 64, OSWorld Docker 16, Firecracker 16. Cua's open parts sustained no level. The OSGym arm is our reimplementation from the paper, because no official code exists.
What ran
The pre-registered primary phase dp8 (all 8 GPUs serving). A vLLM outage (deviation D3, 2026-10-31 06:45 to 16:29Z) invalidated every operating-point run, and each was re-run. A1 (frozen 2026-11-03T00:41:24Z) changed only the Drums build: runtime rc1 cb3145d, CRIU rc1 with the pkeys fix. Its first attempt never made a model call (tokenizer load, A1-D1) and is kept; the valid run went from 20:38:52Z to 21:33:45Z.
Results
Real-model throughput, frozen primary (dp8): verified rollouts per USD, Drums against four incumbents on one host preliminary2026-10-31 · env host + GPU host
| Metric | Value | Unit | n | Note |
|---|---|---|---|---|
| Drums / OSWorld Docker, verified rollouts per USD | 2.97 | x | 55 | 95% CI [2.830, 3.150]; Drums better |
| Drums / Firecracker, verified rollouts per USD | 2.97 | x | 55 | 95% CI [2.826, 3.139]; Drums better |
| Drums / OSGym-style reimplementation, verified rollouts per USD | 1.15 | x | 55 | 95% CI [1.118, 1.193]; the OSGym arm is our own reimplementation |
| Drums / tuned KVM, verified rollouts per USD | 0.897 | x | 55 | 95% CI [0.881, 0.914]; Drums WORSE than tuned KVM under the frozen rule |
| Drums / tuned KVM, successes per USD | 0.947 | x | 55 | 95% CI [0.855, 1.072]; inconclusive, point estimate worse |
| Drums (128 desktops), USD per 1,000 verified rollouts | 8.89 | USD | 2,129 verified, 797 successes, 3,183 s | |
| tuned KVM (128), USD per 1,000 verified rollouts | 8.86 | USD | 2,127 verified, 754 successes | |
| OSGym-style (64), USD per 1,000 verified rollouts | 12.26 | USD | 2,064 verified, 681 successes | |
| OSWorld Docker (16), USD per 1,000 verified rollouts | 44.31 | USD | 979 verified, 373 successes | |
| Firecracker (16), USD per 1,000 verified rollouts | 43.77 | USD | 996 verified, 325 successes |
Host: env host r8i.16xlarge (64 vCPU, 512 GB, 4.44544 USD/h); GPU host p4d.24xlarge (8 x A100), spot 16.9631 USD/h, Ubuntu 24.04, kernel 7.0.0-1014-aws. every arm on the same env host and the same GPU host. Workload: 55 OSWorld tasks; each stack at the highest concurrency it sustained in its own ladder (Drums 128, tuned KVM 128, OSGym-style 64, OSWorld Docker 16, Firecracker 16). Model: ScaleCUA-qwen3.5-osworld-sft rev ff2508f6, vLLM 0.19, data-parallel 8, temperature 0.8, 15 steps max. Method: Pre-registered (frozen 2026-10-31T01:00:11Z, sha256 996bb736...). Cost = wall time x (GPU + env host price), every failed attempt included. Verified = training-eligible rollouts; successes = eligible rollouts the OSWorld checker scored 1. Paired task-clustered bootstrap over 55 tasks, 10,000 resamples. Numbers from the frozen report tool, unchanged.. Source: research archive: runbooks/evidence/repro-check-dp8-20261103.json (phases.dp8.pairs, phases.dp8.arms.<arm>.op); research archive: runbooks/evidence/EVIDENCE-2026-11-03.md; research archive: runbooks/realmodel-throughput/20261031T0100Z/summary/summary-dp8-interim.json.
- The Drums build in this phase lacked the Chrome memory-protection-keys fix; amendment A1 (next file) re-ran the Drums arm on the release build.
- The frozen primary charges every arm about 2 h of an operating-point attempt invalidated by a vLLM outage (deviation D3). This compresses the ratios; see A1.
- Cua: no level sustained. One host type, one model, one task set; no fresh-account replication yet.
- Verified means the attempt passed the validity gates and is eligible for training; it does not mean the task was solved. Successes are reported separately.
Amendment A1: the Drums arm on the release build, and every arm charged only its valid attempt (post-hoc accounting) preliminary2026-11-03 · env host + GPU host
| Metric | Value | Unit | n | Note |
|---|---|---|---|---|
| A1 Drums release build: verified rollouts of all rollouts | 2132 / 2142 | rollouts | 764 successes in 3,132 s; 128 desktops, 0 bring-up errors | |
| A1 under the frozen report tool: USD per 1,000 verified | 8.74 | USD | parent's pre-pkeys build 8.89: the fix does not change the result | |
| A1, one rule for all arms: USD per 1,000 verified, Drums release build | 9.18 | USD | tuned KVM 9.14, OSGym-style 12.81, OSWorld Docker 44.95, Firecracker 45.02 | |
| A1 rule: Drums (release) / OSWorld Docker, verified per USD | 4.9 | x | 55 | 95% CI [4.648, 5.193]; post-hoc |
| A1 rule: Drums (release) / Firecracker, verified per USD | 4.9 | x | 55 | 95% CI [4.643, 5.186]; post-hoc |
| A1 rule: Drums (release) / OSGym-style, verified per USD | 1.4 | x | 55 | 95% CI [1.340, 1.448]; post-hoc |
| A1 rule: Drums (release) / tuned KVM, verified per USD | 0.995 | x | 55 | 95% CI [0.965, 1.021]; a tie |
| A1 rule: Drums (release) / tuned KVM, successes per USD | 1.01 | x | 55 | 95% CI [0.835, 1.193]; a tie |
Host: env host r8i.16xlarge (4.44544 USD/h); GPU host p4d.24xlarge (8 x A100), priced at the parent's spot 16.9631 USD/h, Ubuntu 24.04, kernel 7.0.0-1014-aws. same hosts and harness as the parent. Workload: Same 55 tasks, order, policy, sampling, gates and stop rule as the parent's dp8 operating point; Drums at 128 desktops. Model: ScaleCUA-qwen3.5-osworld-sft rev ff2508f6 (served files byte-identical to the parent's), vLLM 0.19, DP 8. Method: PREREGISTRATION-A1 frozen 2026-11-03T00:41:24Z (sha256 8139f82b...). Run 20:38:52Z to 21:33:45Z. Table B of the report: a1_matched.py charges each arm only the bring-up and teardown of its valid operating-point attempt, same bootstrap (seed 20261031). This rule was chosen after the outcome.. Source: research archive: runbooks/realmodel-throughput/20261031T0100Z/amendment-a1/REPORT-A1.md; research archive: runbooks/realmodel-throughput/20261031T0100Z/amendment-a1/summary/summary-a1-matched-gpu16.9631.json; research archive: runbooks/realmodel-throughput/20261031T0100Z/amendment-a1/summary/summary-a1.json.
- The one-rule-for-all accounting is a post-hoc correction, not the pre-registered result. The frozen 2.97x stays the pre-registered claim; publicly, about 3x (conservative) until a fresh-account replication pre-registered with the valid-attempt rule.
- Drums ties tuned KVM under either accounting's interval on successes, and is worse on verified per USD under the frozen rule (0.897).
- The release run used the hardened profile off, as in the parent; page-sharing clones failed under the hardened profile (A1-D3, a release defect).
- Full raw runs (about 4 GB) are on the env host's disk, not in the archive; summaries, ledger and provenance are.
Failures and surprises
- The frozen accounting rule billed each arm for its valid attempt plus about two hours of the outage-invalidated one. A fixed two hours added to Drums' 53-minute run inflates its cost about 3.4x and to Docker's two-hour run about 2x, which compresses the ratios from about 4.8x to 2.97x. Nobody chose this; it is a side effect of D3.
- Page-sharing clones failed under the hardened profile (A1-D3), so A1 ran with hardening off, as the parent did. That is a release defect.
- Tuned KVM is the strongest incumbent here. Drums does not beat it on cost per verified rollout with a model in the loop.
Limitations
One host type, one model, one task set. The GPU dominates the bill at this scale, which narrows every environment difference. "Verified" is not "solved": successes are reported separately. No fresh-account replication yet; the public line is "about 3x (conservative)" against OSWorld Docker and Firecracker until one exists.
Reproduce
The harness is on branch realmodel-throughput at 7a8b8c6; A1's driver is labbench/tprun-a1.sh, and amendment-a1/summary/a1-analyze.sh plus a1_matched.py regenerate both tables from the ledger. Full raw runs (about 4 GB) are on the environment host's disk.
Sources
Pre-registration, deviations, the frozen summary, the reproduction check of the frozen table, the A1 pre-registration and report, and the A1 summaries, all listed under Evidence. Research archive, internal for now.