In one paragraph
We trained ScaleCUA 9B with GRPO for 8 hours of trained time on Drums (128 desktops) and the offline band success on the 18 trained tasks rose from 0.623 to 0.863 (+0.240, CI [+0.154, +0.328]): the pre-registered Phase A criterion was met. Then the identical trainer and config ran on OSWorld's stock Docker provider at its sustained 16 desktops. Its first checkpoint at or above 0.8634 was step 20 (0.895) after 27,658 s of trained time, against Drums' 27,832 s: 0.99x in time and in dollars. Eight times the desktops gave 7% more training episodes per trained hour and no faster learning, because at this trainer and GPU budget the GPU side (reference pass and update) set the pace. This is a tie, and the result we expected to show an environment advantage did not.
Setup
GPU host p4d.24xlarge (8 x A100 40 GB: 4 actor GPUs at TP4, 4 rollout GPUs as two vLLM engines at TP2); environment host c8i.24xlarge in another region, about 65 ms per call on both arms. Trainer SCALE-CUA 3929e2f with the same patches on both arms (sha256-checked). The two configs differ only in train.concurrency (128 or 16) and the group name. All evaluations of both arms ran on Drums slots, as pre-registered. Prices for the ratio: 21.96048 USD/h on both arms.
What ran
Phase A: step-0 evaluations, 480 minutes of training, the same evaluations of step 16. Phase B: the same config with 16 desktops, censored at 57,622 s trained; every 4th checkpoint archived and evaluated offline. Drums runtime: drums-lab rl-learn at ac60ee3; Phase B scripts on monorepo branch rl-learn-phase-b (60dfeee, d19ffb4), none touching the trainer, configs or evaluator.
Results
L1: GRPO on ScaleCUA, Drums (128 desktops) against OSWorld Docker (16): same success in the same trained time preliminary2026-11-08 · G1 + env host
| Metric | Value | Unit | n | Note |
|---|---|---|---|---|
| Phase A (Drums): band success, step 0 to step 16 | 0.623 to 0.863 | success rate | 18 | +0.240, 95% CI [+0.154, +0.328]; pre-registered criterion met |
| Phase B (OSWorld Docker): first checkpoint at or above 0.8634 | 0.895 | band success | 18 | step 20; crossed, not censored |
| Trained time to target, OSWorld Docker / Drums | 0.99 | x | 1 | 27,658 s / 27,832 s; one pair, no interval; no Drums advantage shown |
| Dollars to target at equal prices, OSWorld Docker / Drums | 0.99 | x | 1 | 168.71 / 169.78 USD |
| Wall time to target (evaluation waits included), OSWorld Docker / Drums | 1.05 | x | 1 | 30,505 / 29,089 s; descriptive |
| Training episodes per trained hour, Drums (128) vs Docker (16) | 461 to 495 | episodes/h | Drums 495, Docker 461: 8x the desktops gave 7% more episodes | |
| Held-out (22 tasks, K = 10), Drums step 16 minus step 0 | 0.033 | success rate | 22 | 95% CI [-0.012, +0.080]; no held-out gain on either arm |
Host: GPU p4d.24xlarge spot (8 x A100 40 GB; 4 actor TP4 + 2 vLLM engines TP2); env c8i.24xlarge on-demand (96 vCPU, 190,292 MiB), Ubuntu 24.04, kernel 7.0.0-1014-aws. GPU host and env host in different regions, about 65 ms per call on both arms. Workload: 18-task learnable OSWorld band (K = 9 offline evaluation), 53 in-distribution tasks (K = 3), 22 held-out (K = 10); GRPO n 8, 16-group pulls, lr 3e-6, 15 turns. Model: ScaleCUA-qwen3.5-osworld-sft rev ff2508f6 (9B). Method: Phase A frozen 2026-11-04T08:30:04Z (sha256 a1f0f1fe...), Phase B frozen 2026-11-07T03:18:29Z (sha256 d9f63594...). Configs differ only in train.concurrency (128 / 16). Paired task bootstrap, 10,000 resamples, seed 20261104. Crossing = first incumbent checkpoint with band success >= 0.8634 (Drums' last point estimate). Equal prices on both arms: 21.96048 USD/h.. Source: research archive: runbooks/rl-learn/20261103T0050Z/REPORT-L1.md; research archive: runbooks/rl-learn/20261103T0050Z/analysis/learn_analysis.json; research archive: runbooks/rl-learn/20261103T0050Z/analysis/phaseb/learn_phaseb_crossing.json.
- One pair, one host pair, three days apart. No interval bounds the ratio.
- Drums' time to target is an upper bound (its steps 4 to 12 were not kept); the incumbent's is known only to a 4-update grid, so its true ratio lies in (0.81, 0.99].
- 687 untimed incumbent training episodes during excluded evaluation waits favour the incumbent.
- At this trainer and GPU budget the GPU side (reference pass and update) set the pace, not the environment.
Failures and surprises
- Phase A attempt 1 crashed when Ray's spill filled the disk; declared invalid and restarted with a fresh budget (A-D2).
- Hosts were left idle for 46 hours after Phase A (about 1,010 USD of operator error, not charged to an arm).
- Phase B: spot capacity refused the GPU host 7 times; 35 Docker reset faults, all recovered.
Limitations
One pair, three days apart. Drums' crossing time is an upper bound because its intermediate checkpoints were not kept; the incumbent's is known only to a 4-update grid, so its ratio lies in (0.81, 0.99]. 687 untimed incumbent episodes during excluded evaluation waits favour the incumbent. The case where the environment is the bottleneck (more GPUs, heavier desktops, more desktops than Docker can sustain) is not tested.
Reproduce
On branch rl-learn-phase-b, experiments/experiments/rl-ab: orchestrator/learn_arm.sh docker ..., the Phase B chain, and analysis/learn_phaseb_crossing.py with seed 20261104 and censor 57,622 s. About 25 h of both hosts, about 544 USD.
Sources
Both pre-registrations, the report, the run log, and the analysis and raw folders under Evidence. Research archive, internal for now.