DrumsBench

Experiments / EXP-029

L1: GRPO raises ScaleCUA 9B's success on Drums, and OSWorld's stock Docker stack reaches the same success in the same trained time (0.99x, one pair)

Phase A: show a GRPO setup in which in-distribution success rises within 8 h of trained time on Drums. Phase B: with identical training on OSWorld's Docker provider, measure trained time and dollars until the incumbent reaches Drums' final success.

preliminaryA real observation that needs more runs, a positive control, or an independent check before it counts.
Date
2026-11-04 to 2026-11-08
Arms
drums-rootfs-criu, osworld-docker
Hardware
GPU p4d.24xlarge spot (8 x A100 40 GB); env c8i.24xlarge on-demand (96 vCPU)
OS
Ubuntu 24.04
Model
ScaleCUA-qwen3.5-osworld-sft rev ff2508f6 (9B), full GRPO
Workload
18-task OSWorld band (K = 9), 53 in-distribution tasks (K = 3), 22 held-out (K = 10); Drums 128 desktops, Docker 16
Code
rl-learn-phase-b at 60dfeee (private research archive)
Evidence
runbooks/rl-learn/20261103T0050Z/PREREGISTRATION-L1.md (frozen 2026-11-04T08:30:04Z) and PREREGISTRATION-L1-PHASEB.md (frozen 2026-11-07T03:18:29Z)
runbooks/rl-learn/20261103T0050Z/REPORT-L1.md and RUNLOG-L1.md
runbooks/rl-learn/20261103T0050Z/analysis/learn_analysis.json
runbooks/rl-learn/20261103T0050Z/analysis/phaseb/learn_phaseb_crossing.json and band_vs_trained_time.svg
runbooks/rl-learn/20261103T0050Z/raw/ and arm-docker/ (with SHA256SUMS)
Limitation
No Drums advantage in time or dollars to the learning target is shown: one pair, no interval on the ratio, and no held-out gain on either arm.

In one paragraph

We trained ScaleCUA 9B with GRPO for 8 hours of trained time on Drums (128 desktops) and the offline band success on the 18 trained tasks rose from 0.623 to 0.863 (+0.240, CI [+0.154, +0.328]): the pre-registered Phase A criterion was met. Then the identical trainer and config ran on OSWorld's stock Docker provider at its sustained 16 desktops. Its first checkpoint at or above 0.8634 was step 20 (0.895) after 27,658 s of trained time, against Drums' 27,832 s: 0.99x in time and in dollars. Eight times the desktops gave 7% more training episodes per trained hour and no faster learning, because at this trainer and GPU budget the GPU side (reference pass and update) set the pace. This is a tie, and the result we expected to show an environment advantage did not.

Setup

GPU host p4d.24xlarge (8 x A100 40 GB: 4 actor GPUs at TP4, 4 rollout GPUs as two vLLM engines at TP2); environment host c8i.24xlarge in another region, about 65 ms per call on both arms. Trainer SCALE-CUA 3929e2f with the same patches on both arms (sha256-checked). The two configs differ only in train.concurrency (128 or 16) and the group name. All evaluations of both arms ran on Drums slots, as pre-registered. Prices for the ratio: 21.96048 USD/h on both arms.

What ran

Phase A: step-0 evaluations, 480 minutes of training, the same evaluations of step 16. Phase B: the same config with 16 desktops, censored at 57,622 s trained; every 4th checkpoint archived and evaluated offline. Drums runtime: drums-lab rl-learn at ac60ee3; Phase B scripts on monorepo branch rl-learn-phase-b (60dfeee, d19ffb4), none touching the trainer, configs or evaluator.

Results

L1: GRPO on ScaleCUA, Drums (128 desktops) against OSWorld Docker (16): same success in the same trained time preliminary2026-11-08 · G1 + env host

MetricValueUnitnNote
Phase A (Drums): band success, step 0 to step 160.623 to 0.863success rate18+0.240, 95% CI [+0.154, +0.328]; pre-registered criterion met
Phase B (OSWorld Docker): first checkpoint at or above 0.86340.895band success18step 20; crossed, not censored
Trained time to target, OSWorld Docker / Drums0.99x127,658 s / 27,832 s; one pair, no interval; no Drums advantage shown
Dollars to target at equal prices, OSWorld Docker / Drums0.99x1168.71 / 169.78 USD
Wall time to target (evaluation waits included), OSWorld Docker / Drums1.05x130,505 / 29,089 s; descriptive
Training episodes per trained hour, Drums (128) vs Docker (16)461 to 495episodes/hDrums 495, Docker 461: 8x the desktops gave 7% more episodes
Held-out (22 tasks, K = 10), Drums step 16 minus step 00.033success rate2295% CI [-0.012, +0.080]; no held-out gain on either arm

Host: GPU p4d.24xlarge spot (8 x A100 40 GB; 4 actor TP4 + 2 vLLM engines TP2); env c8i.24xlarge on-demand (96 vCPU, 190,292 MiB), Ubuntu 24.04, kernel 7.0.0-1014-aws. GPU host and env host in different regions, about 65 ms per call on both arms. Workload: 18-task learnable OSWorld band (K = 9 offline evaluation), 53 in-distribution tasks (K = 3), 22 held-out (K = 10); GRPO n 8, 16-group pulls, lr 3e-6, 15 turns. Model: ScaleCUA-qwen3.5-osworld-sft rev ff2508f6 (9B). Method: Phase A frozen 2026-11-04T08:30:04Z (sha256 a1f0f1fe...), Phase B frozen 2026-11-07T03:18:29Z (sha256 d9f63594...). Configs differ only in train.concurrency (128 / 16). Paired task bootstrap, 10,000 resamples, seed 20261104. Crossing = first incumbent checkpoint with band success >= 0.8634 (Drums' last point estimate). Equal prices on both arms: 21.96048 USD/h.. Source: research archive: runbooks/rl-learn/20261103T0050Z/REPORT-L1.md; research archive: runbooks/rl-learn/20261103T0050Z/analysis/learn_analysis.json; research archive: runbooks/rl-learn/20261103T0050Z/analysis/phaseb/learn_phaseb_crossing.json.

  • One pair, one host pair, three days apart. No interval bounds the ratio.
  • Drums' time to target is an upper bound (its steps 4 to 12 were not kept); the incumbent's is known only to a 4-update grid, so its true ratio lies in (0.81, 0.99].
  • 687 untimed incumbent training episodes during excluded evaluation waits favour the incumbent.
  • At this trainer and GPU budget the GPU side (reference pass and update) set the pace, not the environment.

Failures and surprises

Limitations

One pair, three days apart. Drums' crossing time is an upper bound because its intermediate checkpoints were not kept; the incumbent's is known only to a 4-update grid, so its ratio lies in (0.81, 0.99]. 687 untimed incumbent episodes during excluded evaluation waits favour the incumbent. The case where the environment is the bottleneck (more GPUs, heavier desktops, more desktops than Docker can sustain) is not tested.

Reproduce

On branch rl-learn-phase-b, experiments/experiments/rl-ab: orchestrator/learn_arm.sh docker ..., the Phase B chain, and analysis/learn_phaseb_crossing.py with seed 20261104 and censor 57,622 s. About 25 h of both hosts, about 544 USD.

Sources

Both pre-registrations, the report, the run log, and the analysis and raw folders under Evidence. Research archive, internal for now.