DrumsBench

Experiments / EXP-030

rl-branch: forking a running desktop at turn 6 for tree GRPO did not cut GPU cost; it doubled updates per hour and learned less (pre-registered negative)

Test whether fork-at-decision-point tree GRPO reaches the L1 control's band success (0.8634) in less GPU time and fewer dollars than plain GRPO.

failedRan and missed a correctness, isolation, reset or pre-registered performance bar. Kept on the record.
Date
2026-11-08 to 2026-11-10
Hypothesis
GPU-side time and dollars to reach band K = 9 success of at least 0.8634 are lower with tree GRPO than with plain GRPO.
Arms
drums-rootfs-criu
Hardware
GPU p4d.24xlarge spot (8 x A100 40 GB); env c8i.24xlarge
OS
Ubuntu 24.04
Model
ScaleCUA-qwen3.5-osworld-sft rev ff2508f6 (9B)
Workload
L1 Phase A tasks, evaluator and budget (480 min trained); 2 trunks x 4 forked children per group, CRIU fork after turn 6
Code
rl-branch at 8055129 (private research archive)
Evidence
runbooks/rl-branch/20261108T0656Z/PREREGISTRATION.md (frozen 2026-11-08T07:56:20Z)
runbooks/rl-branch/20261108T0656Z/REPORT.md, REPORT.interim-20261109.md, RUNLOG.md, REPRODUCE.md
runbooks/rl-branch/20261108T0656Z/analysis/crossing, analysis/branch, analysis/util
runbooks/rl-branch/20261108T0656Z/raw/ (timed, dryrun, MANIFEST.json, SHA256SUMS)
Limitation
One pair at one setting (2 trunks, 4 children, fork at turn 6); a throughput gain without a learning gain, not proof that tree rollouts cannot help.

In one paragraph

The idea: fork a running desktop with CRIU at a decision point, train the shared prefix once, and get more learning per GPU-hour. It worked as engineering: 1,460 forks at about 6.5 s each with isolation and state gates passing, twice the updates per trained hour (28 against 16) and 59% more kept trajectories per GPU-second. It did not work as learning. No checkpoint reached the control's 0.8634 within the control's budget, so the time ratio is censored at at least 1.035x (worse). At equal trained time the branch arm scored 0.654 against 0.863, a difference of -0.209 (CI [-0.339, -0.074]). Children share a 6-turn prefix and 63% of forks gave every child the same reward, so most continuation advantages were zero. Branching stays an internal mechanism; the claim stays "cheaper rollouts", not "cheaper learning".

Setup

Control: the L1 Phase A plain-GRPO Drums arm (EXP-029), same GPU and environment instances, same prices (21.96048 USD/h), same evaluator scripts. Branch: trunks in slots 0 to 31, fork children in 32 to 127, in-loop evaluation in 128 to 143; fork capture on a 48 GiB tmpfs. Tree advantage: trunk segments trained once with weight C; child continuations get (r - V_m) / sigma; zero-advantage items are not trained.

What ran

One timed run, resumed once: the environment disk filled during step 21 (D8), the disk was grown and training resumed from step 20 with the void interval (6,870.4 s) removed from the clock, a rule fixed before any evaluation. Seven band evaluations (steps 4 to 28).

Results

rl-branch: forking a running desktop at turn 6 for tree GRPO did not make training cheaper failed2026-11-10 · G1 + env host

MetricValueUnitnNote
Checkpoints whose band evaluation reached 0.8634 within 28,811 s0 / 7checkpointscensored: target not reached within the control's budget
GPU hours and dollars, branch / control (lower bound)1.04x1at least 1.035x (175.75 vs 169.78 USD); lower bound, no CI
Equal trained time: band success, branch step 28 minus control step 16-0.209success rate180.654 vs 0.863, 95% CI [-0.339, -0.074]; branch worse
Held-out (K = 10), branch step 28 minus control-0.092success rate220.445 vs 0.538, 95% CI [-0.195, -0.003]
Updates in the budget, control vs branch16 to 28updatescontrol 16, branch 28
Kept trajectories per GPU-second, control vs branch0.092 to 0.145per s+59% for branch: a throughput gain without a learning gain
Forks: request time6.5s1460p90 7.6 s; 1,460 forked trees; 100 children failed
Forks with at least two children whose children all got the same reward911 / 1446forks0.630 unanimous: zero continuation advantage

Host: GPU p4d.24xlarge spot (8 x A100 40 GB); env c8i.24xlarge (fork capture on a 48 GiB tmpfs), Ubuntu 24.04, kernel 7.0.0-1014-aws. same instances as the L1 control (EXP-029). Workload: Same tasks, evaluator and budget (W = 480 min trained) as L1 Phase A; tree GRPO with 2 trunks x 4 forked children per group, fork after turn 6. Model: ScaleCUA-qwen3.5-osworld-sft rev ff2508f6 (9B). Method: Pre-registered 2026-11-08T07:56:20Z (sha256 e095e7b2...). Primary: trained time at the first complete band K = 9 evaluation >= 0.8634, censored at 28,811 s; ratio = t_branch / 27,832 s. Prices 21.96048 USD/h on both arms. Paired task bootstrap, 10,000 resamples, seed 20261104.. Source: research archive: runbooks/rl-branch/20261108T0656Z/REPORT.md; research archive: runbooks/rl-branch/20261108T0656Z/analysis/crossing/learn_phaseb_crossing.json; research archive: runbooks/rl-branch/20261108T0656Z/analysis/branch/branch_analysis.json; research archive: runbooks/rl-branch/20261108T0656Z/analysis/util/util.json.

  • One pair, run four days apart. The run is in two segments (disk-full resume, D8; the void interval of 6,870.4 s is removed from the clock).
  • The control's 27,832 s is an upper bound. Checkpoint-to-checkpoint noise looks like about +-0.1 (the step 24 dip).
  • Only one setting was tested (2 trunks, 4 children, fork at turn 6). This does not show that tree rollouts cannot help with other settings.

Failures and surprises

Limitations

One pair, four days apart, in two segments. The control's 27,832 s is an upper bound. The step 24 dip suggests checkpoint noise of about 0.1 at N = 18 tasks. More trunks, later forks or value-targeted forks were not tested. Study spend about 483 USD (estimate).

Reproduce

REPRODUCE.md in the study folder; analysis from raw/ only with analysis/branch_crossing.py, branch_analysis.py and util_analysis.py (commands in the report). A full rerun is about 15 to 17 h of both hosts, about 350 USD on spot.

Sources

Pre-registration, report, run log and analysis outputs under Evidence. Research archive, internal for now.