Experiment 3, RL training A/B with only the environment infrastructure changed

In one paragraph

Experiments 1 and 2 are about the environment on its own. Experiment 3 is the question the project exists for: if the only thing you change is the environment infrastructure, does an RL run reach the same place faster or cheaper? The design is an A/B with everything else pinned: the same open-weight model (candidate: a Qwen3-VL-4B-Instruct class model, chosen because it is small enough to train on the hardware we have and is not a frontier model under evaluation-only terms), the same algorithm, the same rewards, the same task split. The pre-registered outcome is a target success rate on held-out tasks; we report the time and cost each arm took to reach it, and "never reached target" honestly if one did not. As a quality check we also compare learning curves at equal numbers of valid trajectories, which should coincide if the environment only changes throughput. Nothing has run and no training code exists on the branch.

Setup

Design

• Model: identical weights, tokenizer and vision preprocessing on both arms, from a pinned revision with verified file hashes (the same discipline as EXP-014).
• Algorithm and rewards: identical, pinned by source hash; the reward is the task checker's outcome plus whatever shaping is chosen, frozen before the run. Frontier-model outputs are not used anywhere in training (the Bedrock route's NO_TRAINING_USE rule).
• Tasks: the frozen set from Experiment 2 (EXP-013), split in advance into training tasks and held-out tasks; the held-out set is never touched by the trainer.
• Arms: the QEMU/KVM warm-snapshot baseline against the Drums candidate, both under drumsbench's validity gates, so an invalid or corrupted trajectory is dropped from training on both arms by the same rule and counted.
• Pre-registered target: a success rate on the held-out tasks, written down with its evaluation protocol (number of episodes, seeds, max steps) before the first training step and hashed into the run. It is a bar, not a prediction, and not chosen to be attractive.
• Outcome per arm: wall-clock time, environment cost (reconciled), inference cost, number of valid trajectories consumed, and the number of invalid trajectories discarded, at the moment the target is first reached. If the target is not reached within the budget, the record says "never reached target" with the final held-out rate and the budget spent.
• Seeds: multiple training seeds per arm where budget permits; with one seed per arm the record says so and claims nothing about variance.
• Quality check: the learning curve (held-out success against valid trajectories consumed) for both arms. If the environment only changes throughput, these curves should lie on top of each other; a gap means the environment changed the data, which is a correctness finding that outranks any speed result.

Prior art to reproduce

Two external results are the reference points for whether the setup is sane, as recorded in our plan:

• CUA-Sandbox reported SFT-to-RL experiments on OSWorld (author-reported, a paper); reproducing their open-weight training setup on the baseline arm first is the sanity check before any A/B.
• ScaleCUA's released stack, with license caveats as we understand them: the trainer is MIT-licensed, while the task pool and the environment worker carry no license. We would reuse the trainer only, and build the task pool from OSWorld definitions and the environment worker from drumsbench. These license facts come from our plan and must be re-verified against the upstream repositories before any code is reused.

What ran

Nothing. No trainer, reward, model download or task split exists on this branch.

Results

None.

Failures and surprises

None recorded.

Limitations

• The Drums arm has no passing restore (EXP-008, EXP-009), so there is nothing to train on yet; the first A/B may have to be baseline versus the OCI control, which would compare a VM with an ordinary container and say nothing about Drums.
• A 4B-class model may not learn the task set at all within budget on either arm; that outcome is reported as "never reached target" for both, which is informative about the task set, not about the environment.
• Training cost is dominated by GPU time, which is identical by design across arms; the environment's share must be reported on its own.
• One seed per arm is likely under budget; any difference seen then is a single draw.

Reproduce

Not reproducible yet. The preconditions, in order: a Drums restore that passes its gates; Experiment 2 receipts on both arms with a frozen task set; a served open-weight model (EXP-014 for the inference side); a pinned trainer and reward; a pre-registered target file hashed into the run. Then one training run per arm per seed under drumsbench, with the held-out evaluation run as a separate measured workload.

Sources

• benchmark/drumsbench/ (validity gates that decide which trajectories count), benchmark/bedrock_cua/__init__.py (TRAINING_USE exclusion)
• benchmark/cua_adapter/, benchmark/model_serving/ (the open-weight inference side)
• The Experiment 3 design, the candidate model class and the prior-art notes (CUA-Sandbox on OSWorld; ScaleCUA's trainer MIT, task pool and env worker unlicensed) come from the Drums research plan, not from a file on this branch, and the license facts are to be re-verified upstream.