DrumsBench

Research notes / 2026-11-17

First training run: eight hours of GRPO on a 27B model, every desktop on Drums

Pre-registered. Qwen3.5-27B, full-parameter GRPO, one self-made task family. Held-out success 0.54 to 1.00 (+0.46, 95% CI 0.38 to 0.55) over 32 unseen variants. Six optimizer steps; the policy saturated the 64 training variants by step 5. One template, one app, one run.

This page is the evidence behind the blog post Our first real training run on Drums. The pre-registration was frozen before the launch; its sha256 is below and the frozen text is linked unchanged. The run log and the 37-finding audit of our own product are internal for now.

Setup

itemvalue
modelQwen/Qwen3.5-27B, revision fc05daec18b0a78c049392ed2e771dde82bdf654, bf16
trainerprime-rl commit 167d4c3, full-parameter GRPO, FSDP2 on 6 GPUs, vLLM (TP2) on 2 GPUs, lr 3e-6, groups of 8, 128 samples per step, temperature 0.8
GPUsModal, 8 x H200 (one 50-minute stint on 8 x B200 before a preemption; resumed from the last checkpoint)
budgetW = 480 min of trained time (trainer wall from the start of its loop; evaluation waits do not count)
task familyd1_files_collect from drums-verifiers 0.2.2: collect every file whose name starts with a prefix from a folder tree into a new folder, renaming each copy; the checker reads the files
variants64 training (seed 611161), 32 held-out (seed 611162), zero overlap on instruction, file set and parameters
desktopsone Drums Cloud worker host (c8i.24xlarge), up to 128 desktops in flight for training, 48 for evaluation
agent1280 x 720 screenshots, 5 kept in context, 15 turns maximum, 1,536 completion tokens, thinking off

Pre-registered criterion

Offline evaluation of the start weights and of the last checkpoint written within W on the 32 held-out variants, K = 6 episodes each (192). Per variant, success = share of valid episodes the checker scored 1.0. Paired variant bootstrap of last minus step 0 (10,000 resamples, seed 20261116, percentile 95% interval); met when the lower bound is above 0. Completeness: at least 90% of the 192 episodes valid and every variant valid once; an incomplete evaluation is repeated once in full and the repeat is used. One look.

Frozen 2026-11-16T13:09:28Z, sha256 6819b60727060129b8923d49b987e9861472c11c92f872e31a80c7461a4a8e24: PREREGISTRATION-frozen.md.

Result

evaluationstep 0step 6 (last)
held-out, 32 variants, K = 6102 of 189 valid; mean over variants 0.5417192 of 192; 1.0000
training, 64 variants, K = 274 of 125 valid; 0.6016128 of 128; 1.0000

Held-out paired difference over all 32 variants: +0.4583, 95% CI [0.3750, 0.5458]. Criterion met. Training variants: +0.3984, CI [0.3047, 0.4922].

At step 0, 30 of the 32 held-out variants were mixed (sometimes solved, sometimes not), 1 was never solved and 1 always solved.

Episode shape under evaluation (median): step 0 used 15 turns (the cap), 13 actions and 161 s; step 6 used 8 turns, 7 actions and 62 s.

The training curve

stepbatch rewardwall for the batchepisodes discarded (whole group same outcome)
10.6418 minfew
20.64814 minfew
30.75815 minsome
40.84416 min59% (184 of 312)
50.8591 h 24 min93% (1,808 of 1,936)
60.8753 h 37 min98% (7,744 of 7,872)

GRPO learns from groups whose members disagree. By step 5 almost every group of 8 attempts on a training variant all succeeded, so the trainer threw them away and waited for the rare mixed group. Eight hours bought six optimizer steps; the last one cost about 7,900 desktop episodes. About 9,000 rollouts in all, 0 failed on the host.

What happened along the way

What this shows and does not show

Shows: the loop works end to end for a client's shape of job, and the gain transfers to unseen variants of the same template. Does not show: anything comparative (no other environment ran), transfer across apps or templates, or robustness across seeds (one run). The 1.00 is on a template the policy had saturated.

Files

Reproduce: python3 d1_analysis.py --step0 step0-heldout-k6r.jsonl --last last-heldout-k6.jsonl prints the verdict JSON above.