DrumsBench

Experiments / EXP-037

D2: the D1 training run again on the fixed product, through the gateway only: held-out 0.50 to 0.99, zero operator actions, about 389 USD of GPU from the billing report

With D1's model, trainer, task pool, budget and agent held fixed, test whether the post-D1 fixes removed the product gaps a client hit, what they did to throughput and cost, and whether the held-out result replicates.

verifiedBacked by measurement files we can point to, on named hardware, with the run recorded.
Date
2026-11-18
Arms
drums-cloud
Hardware
Modal 8 x H200 (6 trainer + 2 inference); Drums Cloud worker c8i.24xlarge
OS
worker Ubuntu 24.04, kernel 7.0.0-1014-aws
Model
Qwen/Qwen3.5-27B rev fc05daec18b0a78c049392ed2e771dde82bdf654, full-parameter
Workload
the same 64 training and 32 held-out d1_files_collect variants (identical bytes); 3 full screenshots of 6 in context, downscale 0.5; in-flight floor 96 of 128 desktops
Code
main at 3ef6f60 (private research archive)
Evidence
runbooks/rl-d2/20261118T0500Z/PREREGISTRATION.md (frozen 2026-11-18T05:4xZ, sha256 b5dfadd6...)
runbooks/rl-d2/20261118T0500Z/REPORT.md and RUNLOG.md
runbooks/rl-d2/20261118T0500Z/analysis/verdict-heldout.json, verdict-train.json, d1_analysis.py (unchanged from D1)
runbooks/rl-d2/20261118T0500Z/modal/billing-final.txt
Limitation
A replication with a changed observation (fewer full screenshots), not an identical rerun; the same single saturated template.

In one paragraph

D2 reran D1 on the fixed product, launched the way a client would launch it (the bring-your-own-Modal template, drums preflight, everything through the gateway with no direct drumsd path). Held-out success went from 0.4979 to 0.9948, a paired difference of +0.4969, 95% CI [0.3917, 0.5990]: the criterion is met and D1's +0.46 [0.38, 0.55] replicates. The product side moved on every measure we pre-registered: 0 operator actions (D1: 6), 9 optimizer steps in 8.25 h of wall time (D1: 6 in 8 h of trained time), about 389 USD of GPU from Modal's billing report (D1: about 504 USD estimated). One new problem appeared: 21 episodes lost to gateway 503s during four 1 to 2 s database blips (A-45).

Setup

As D1 (EXP-036), with three pre-registered differences: the client path (drums-sdk 0.4.0, drums-verifiers 0.3.0 and its Modal template), the gateway build from monorepo main at 3ef6f60 with the data-path cache and usage spool on, and the throughput settings (3 full screenshots of 6, downscale 0.5, stride 3). The budget clock is wall time from container start, set to 8.25 h, stricter than D1's trained time.

What ran

One pre-start relaunch (no 8-GPU container in 20 minutes; the template cancelled cleanly by design), then training from scratch on 8 x H200, then step-0 and step-9 evaluations.

Results

D2: the D1 run again on the fixed product, through the gateway only, with a changed observation verified2026-11-18 · Modal GPUs + Drums Cloud worker

MetricValueUnitnNote
Held-out success, step 0 to step 9 (last within W)0.498 to 0.995success rate32+0.4969, 95% CI [0.3917, 0.5990]; criterion met; D1 +0.46 [0.38, 0.55]
Training variants (K = 2), step 0 to last0.664 to 1success rate64+0.3359, 95% CI [0.2422, 0.4297]
Operator actions during training0actionsD1: 6
Optimizer steps in the budget9steps1.09 steps per hour (D1 0.75)
GPU spend, Modal billing report389USDabout 389: train 366.17, smoke 12.33, evals, export and setup about 11; about 43 USD per optimizer step
Episodes lost to gateway 503s (four 1 to 2 s database blips, A-45)21episodes

Host: Modal 8 x H200 (6 trainer + 2 inference); same Drums Cloud worker c8i.24xlarge, worker Ubuntu 24.04, kernel 7.0.0-1014-aws. launched with the bring-your-own-Modal template; drums-sdk 0.4.0. Workload: Same 64 training and 32 held-out d1_files_collect variants (identical bytes); 3 full screenshots of 6 in context, downscale 0.5 (changed from D1). Model: Qwen/Qwen3.5-27B rev fc05daec, full-parameter GRPO, prime-rl 167d4c3. Method: Pre-registered (frozen 2026-11-18T05:4xZ, sha256 b5dfadd6...). D1's analysis script unchanged. Budget 8.25 h wall from container start (stricter than D1's trained-time clock).. Source: research archive: runbooks/rl-d2/20261118T0500Z/REPORT.md; research archive: runbooks/rl-d2/20261118T0500Z/analysis/verdict-heldout.json; research archive: runbooks/rl-d2/20261118T0500Z/analysis/verdict-train.json; research archive: runbooks/rl-d2/20261118T0500Z/modal/billing-final.txt.

  • A replication with a changed observation (fewer full screenshots), not an identical rerun.
  • D1's cost is an estimate from timestamps; D2's is the billing report. Compare with care.
  • Saturation again: a 64-variant single-template pool is solved in under 10 steps. One template, one app.

Failures and surprises

Limitations

The observation changed, so this is not an identical rerun. D1's cost is an estimate and D2's is a billing report. One template, one app; the next run needs a bigger pool, not more hours.

Reproduce

The pre-registration lists every pin; the analysis is D1's d1_analysis.py, unchanged, fed by per_task_to_records.py.

Sources

Pre-registration, report, run log, verdict files and the billing report under Evidence. Research archive, internal for now.