DrumsBench

Experiments / EXP-036

D1: eight hours of full-parameter GRPO on Qwen3.5-27B with every desktop on Drums Cloud raised held-out success from 0.54 to 1.00 on one self-made template

Starting from the public Qwen3.5-27B weights, test whether 480 minutes of full-parameter GRPO on 64 variants of one drums-verifiers template, every desktop served by Drums Cloud, raises success on 32 held-out variants (paired variant bootstrap, 95% CI excluding 0).

verifiedBacked by measurement files we can point to, on named hardware, with the run recorded.
Date
2026-11-16 to 2026-11-17
Arms
drums-cloud
Hardware
Modal 8 x H200 141 GB; Drums Cloud worker c8i.24xlarge (96 vCPU, 192 GiB)
OS
worker Ubuntu 24.04, kernel 7.0.0-1014-aws, drumsd v0.2.1
Model
Qwen/Qwen3.5-27B rev fc05daec18b0a78c049392ed2e771dde82bdf654, bf16, full-parameter
Workload
d1_files_collect: 64 training variants, 32 held-out, zero overlap; up to 128 desktops in flight for training, 48 for evaluation
Code
templates/d1 at 42e1e60 (private research archive)
Evidence
runbooks/rl-l2/20261116T0426Z/PREREGISTRATION.md (frozen 2026-11-16T13:09:28Z, sha256 6819b607...) and analysis/prereg-frozen.txt
runbooks/rl-l2/20261116T0426Z/REPORT.md, RUNLOG.md, AUDIT.md
runbooks/rl-l2/20261116T0426Z/analysis/verdict-heldout-FINAL.json, verdict-train.json, d1_analysis.py
/evidence/d1-2026-11-16/ on this site (frozen pre-registration)
Limitation
One template, one app, one run, saturated by step 5: a working training loop and a within-template transfer, not a general capability claim.

In one paragraph

The full write-up is the research note First training run; this record puts it in the experiment log with its result file. Pre-registered, one look: held-out success went from 0.5417 to 1.0000, a paired difference of +0.4583, 95% CI [0.3750, 0.5458] over 32 unseen variants, both evaluations complete. The run made only 6 optimizer steps in 8 hours because the policy saturated the 64 training variants by step 5. The run doubled as an audit of our own product and found 37 issues, the largest being that one 128-desktop run took the control-plane database, the console and every gateway down for about 3.5 hours. D2 (EXP-037) replicated the result on the fixed product.

Setup

prime-rl 167d4c3: FSDP2 full-parameter trainer on 6 GPUs, vLLM TP2 on 2 GPUs, GRPO groups of 8, batch 128, lr 3e-6, no reference KL. Desktops: one Drums Cloud worker installed from the public drumsd v0.2.1 release, reached through the gateway. Agent: 1280 x 720 screenshots, 5 kept in context, 15 turns, 1,536 completion tokens, thinking off. Task family from drums-verifiers 0.2.1 plus branch templates/d1 at 42e1e60; the checker compares the target tree with the oracle's, and 4 near-miss solutions per variant must score 0.

What ran

Template choice by a step-0 scan, a one-step smoke, the freeze at 13:09:28Z, then training on Modal for W = 480 minutes of trained time, then offline evaluation of step 0 and step 6 on the held-out (K = 6) and training (K = 2) variants.

Results

D1: eight hours of full-parameter GRPO on Qwen3.5-27B, every desktop through Drums Cloud, held-out success verified2026-11-16 · Modal GPUs + Drums Cloud worker

MetricValueUnitnNote
Held-out success, step 0 to step 6 (last within W)0.542 to 1success rate32+0.4583, 95% CI [0.3750, 0.5458]; criterion met
Held-out valid episodes, step 0 / step 6189 / 192episodesstep 6: 192 of 192; step 0 had 3 no_app_crash invalid
Optimizer steps in 8 h of trained time6stepssaturated by step 5: 93% then 98% of groups discarded as zero-advantage
GPU and host spend (estimate from timestamps)504USDabout 84 GPU-hours on Modal plus the host; an estimate, not a bill

Host: Modal 8 x H200 141 GB (6 trainer, 2 vLLM TP2); Drums worker c8i.24xlarge (96 vCPU, 192 GiB) on drumsd v0.2.1, worker Ubuntu 24.04, kernel 7.0.0-1014-aws. one 50-minute stint on 8 x B200 before a preemption. Workload: d1_files_collect (one self-made drums-verifiers template): 64 training variants (seed 611161), 32 held-out (seed 611162), zero overlap; 15 turns, 1280 x 720, 5 screenshots in context. Model: Qwen/Qwen3.5-27B rev fc05daec, full-parameter GRPO, prime-rl 167d4c3. Method: Pre-registered (frozen 2026-11-16T13:09:28Z, sha256 6819b607...). Held-out K = 6, step 0 vs last checkpoint within W = 480 min trained; paired variant bootstrap, 10,000 resamples, seed 20261116; met when the lower bound is above 0. One look.. Source: research archive: runbooks/rl-l2/20261116T0426Z/REPORT.md; research archive: runbooks/rl-l2/20261116T0426Z/analysis/verdict-heldout-FINAL.json; research archive: runbooks/rl-l2/20261116T0426Z/analysis/verdict-train.json.

  • One template, one app, one run: a working loop and a within-template transfer, not a general capability claim.
  • The first step-0 evaluation was incomplete (a gateway restart) and was repeated once in full, as pre-registered.
  • The run took the control-plane database, the console and every gateway down for about 3.5 h; it continued by talking to drumsd directly. Six operator actions were needed.

Failures and surprises

Limitations

One template, one app, one run. Saturation means more steps would not have taught more on this pool. Spend about 504 USD is an estimate from timestamps, not a bill.

Reproduce

The pre-registration lists every pin; the analysis is analysis/d1_analysis.py on the evaluation records.

Sources

Pre-registration, report, run log, audit and verdict files under Evidence; the frozen pre-registration is also published on this site.