# D1 pre-registration: eight hours of full-parameter GRPO on one self-made task template, through Drums Cloud, raises Qwen3.5-27B's success on unseen variants of that task

Status: FROZEN 2026-11-16 at the training launch (v4 + pins; sha256 in RUNLOG.md). Was: DRAFT v4 for the owner. Supersedes the L2 draft of 2026-11-16T04:26Z (kept in RUNLOG.md) after the owner's decisions of
2026-11-15/16 EST: the model is Qwen3.5-27B, there is no fallback, the run is about eight hours on one task we create ourselves
through dynamic task (template) creation, the desktops come from Drums Cloud through the gateway (our production path, with a
second production-grade host bound to our workspace), the training is full-parameter (no LoRA), the GPU is on Modal credits, the run is performed the way a client performs it
from our dashboard with our whole stack (console, gateway, SDK, MCP computer tools, drums-verifiers, prime-rl), and the run
doubles as an audit of the whole loop (section 10). Values marked **[PIN]** are filled in at launch; **[GATE]** must pass before the freeze;
**[OWNER]** needs a decision. At launch the text above the freeze line is hashed into RUNLOG.md BEFORE any training step; later
changes are deviations with UTC times.

## 1. Question

Starting from the public Qwen3.5-27B weights, do eight hours of full-parameter GRPO trained time on variants of ONE task template,
generated by the drums-verifiers task factory and run on Drums Cloud desktops through the gateway, raise success on held-out
variants of the same template (parameter draws the policy never saw), with a 95% confidence interval that excludes 0?

This is the customer loop end to end: write a task template, let the factory make variants with checkers, train, measure on fresh
variants. It is not a claim about other tasks, other apps or OSWorld; section 9 says what it does and does not show.

## 2. What "a task we create" means here

A template is a `drums_tasks` family: a parametrised instruction, a programmatic checker, a scripted solution (the oracle) and
near-miss solutions. The factory draws parameters from a seed and emits one drumsd task JSON per variant; `drums_tasks.validate`
proves every variant on a live host (oracle passes, near-misses fail). One template gives as many distinct variants as we want.

- Three candidate templates are written for D1 (one per app, so the step-0 scan can pick a learnable one): **[OWNER: name the
  task, or accept the candidates]**
  - T-files: "collect every file whose name matches `<pattern>` from `<folder tree>` into `<target>` and name the copies
    `<scheme>`" (files app; checker compares the target tree to the oracle's).
  - T-calc: "in `<workbook>` compute `<aggregate>` of column `<c>` where `<condition>` and write it in `<cell>`" (LibreOffice
    Calc; checker reads the saved cell).
  - T-writer: "in `<document>` find every `<pattern>` and apply `<style change>`, save as `<name>`" (LibreOffice Writer;
    checker inspects the saved document).
- Split: 64 training variants and 32 held-out variants per template, drawn from two disjoint seeds **[PIN]**. Held-out variants
  differ in every parameter from every training variant (checked by `analysis/variant_overlap.py`, zero overlap **[GATE]**).
- The template chosen for the run is the one whose step-0 scan (section 6) shows the most learning signal; the other two are
  reported as not run.

## 3. Design

| item | value | why |
|---|---|---|
| model | `Qwen/Qwen3.5-27B` (dense 27.8B VLM, bf16 55.6 GB, Apache-2.0) **[PIN: revision, shard sha256 list]** | the owner's choice; a base chat model, not a computer-use SFT |
| adaptation | **full parameter** (every language-model weight; vision tower frozen as in L1) | the owner's choice (2026-11-16): the customer claim is full fine-tuning |
| trainer | **prime-rl at commit 167d4c3** (2026-11-15, tagged v0.10.1.dev238; prime-rl is not on PyPI, so the commit is the pin; FSDP2 trainer, `qwen3_5` multimodal modeling, GRPO) with **drums-verifiers 0.2.1** as the environment package: `DrumsEnv` drives each desktop through the Drums **MCP computer tools** (the same 34 tools the gateway serves at `/mcp`), `DRUMS_URL=https://beta.drums.run/gw` and a workspace key; rewards are the lab verifier's checker plus the episode's validity gates **[PIN: prime-rl commit, drums-verifiers commit, the D1 TOML sha256]** | the owner's choice: our public integration path, nothing lab-only; prime-rl supports Qwen3.5 VLMs natively, so the lab trainer is not needed |
| GPU | **Modal**, one container with 8 GPUs, H200 141 GB or B200 192 GB (the run took 8 x H200: B200 had no capacity, see pins) | one container with 8 x B200 192 GB (the owner's 1,000 USD Modal credits) running `uv run rl`: 2 GPUs for vLLM inference (TP2) and 6 GPUs for the FSDP2 trainer (full parameter; about 450 GB of weights, gradients and optimizer state over 1,152 GB); fallback if Blackwell kernels fail: 8 x H200 141 GB with the same split; weights and checkpoints on Modal Volumes `d1-weights` and `d1-runs`; **no region pin** (a pin to us-east costs 1.75x: 93.7 vs 53.6 USD/h for 8 x B200, which does not fit the credits); the region Modal placed the container in is recorded at launch **[PIN: GPU type, region, price per hour, image digest]** | a full-parameter 27B does not fit 4 x H100; one container keeps inference and trainer on the same node |
| environment | **Drums Cloud through the gateway** (`https://beta.drums.run/gw`, an API key of the drums-labs workspace with `computers:read/write`, `runs:read`, `usage:read`): a second production-grade worker host (c8i.24xlarge, 96 vCPU, 190 GiB, us-east-1) installed from the public release tarball the normal way, approved and **bound to the drums-labs workspace in the console**, serving 128 training desktops + 8 evaluation slots; drums-rc1 keeps serving every other workspace and is not used **[PIN: host id, bind record, admission line]** | the owner's choice: the same path a customer takes (gateway, keys, quotas, spend cap, audit log); rc1 alone holds about 8 desktops, one group, which is one or two updates an hour |
| group size, groups per pull | 8, 16 (128 desktops) **[PIN after 6.4]** | pulls must fill; 8 h is about 25 to 50 updates |
| loss and optimisation | prime-rl's shipped GRPO loss at 167d4c3: IPO probability band (eps 0.3) in place of a ratio clip, **no reference-KL term** (prime-rl has none); AdamW lr 3e-6; temperature 0.8; max 15 turns; thinking mode OFF for the 27B (renderer qwen3.5, thinking disabled); exact resolved config pinned from `uv run rl --dry-run` **[PIN sha256]** | L1's "KL 1e-4, clip 0.2 / 0.28" are SCALE-CUA knobs that do not exist in prime-rl; the shipped loss is used and stated, not emulated |
| sampling | DynamicSampler from the step-0 scan of the 64 training variants (sigma 0.15, target 0.5) **[PIN: file sha256]** | keeps the pull in the informative range |
| step-limit successes | trained as 1.0 | as L1 |
| budget W | **480 min of trained time** (trainer wall from START minus evaluation waits; saves count), hard-capped by the Modal balance | the owner's decision of 2026-11-16 ("ok do 8 hours, go"): about 25 to 50 updates, enough to see a +0.10 gain |
| in-loop evaluation | none | the hour is for training; the measurement is offline |
| offline evaluation | step 0 and the last checkpoint: the 32 held-out variants at K = 6 (192 episodes, PRIMARY); the 64 training variants at K = 2 (128 episodes, secondary) | the decision measurements, same served-policy path at both ends |
| checkpoints | every update as a prime-rl DCP checkpoint on the Modal Volume (about 333 GB each, estimate), `keep_last = 2`; an HF bf16 export (56 GB) of step 0 (the start weights) and of every 8th step plus the last, made by `export_hf` for the offline evaluations and the curve | a per-update curve would cost about 333 GB per step of Volume; every 8th update is the curve's resolution |

## 4. Primary metric and success criterion

- PRIMARY: offline evaluation of the start weights and of the last checkpoint written within W, K = 6 episodes on each of the 32
  held-out variants (tags `step0-heldout-k6`, `last-heldout-k6`). Per variant: success share over its valid episodes (success =
  checker 1.0 whatever ended the episode; task errors are not episodes).
- Criterion: paired variant bootstrap of last minus step 0 over the held-out variants valid in both (10,000 resamples, seed
  20261116, percentile 95% CI); met when the lower bound is above 0.
- Completeness: at least 90% of the 192 episodes valid and every variant valid at least once; an incomplete evaluation is repeated
  once in full (tag `<tag>r`) and the repeat is used.
- One look for the criterion. The per-update held-out curve is descriptive and is evaluated after the verdict is recorded.
- Power **[GATE: computed from the actual step-0 scan before the freeze]**: with 32 variants whose step-0 rates are spread over
  0.1 to 0.9, K = 6 per variant detects a +0.15 mean gain with about 0.8 power; a +0.10 gain with about 0.5. The report states the
  realised power.

## 5. Secondaries and descriptive

- SECONDARY 1: the 64 training variants at K = 2, step 0 vs last, same test (did it learn what it trained on).
- DESCRIPTIVE: the per-update held-out curve (many looks); successes per policy version; informative-group share; trained time,
  wall time, evaluation time and dollars; environment metrics (desktops held, resets, task errors, refusals); the two templates not
  run, with their step-0 scan numbers.
- Comparison with the 9B runs: descriptive only (different model, task and budget).

## 6. Pre-flight gates (results and UTC times in RUNLOG.md before the freeze)

1. **prime-rl trains Qwen3.5-27B full-parameter on Modal with drums-verifiers as the environment [GATE].** A Modal image with
   prime-rl 0.10.0 and drums-verifiers 0.2.1 installed from PyPI (plus the D1 template), the weights on a Modal Volume, 8 x B200:
   `uv run rl` with the D1 TOML completes one step end to end (inference up, one rollout group through the MCP tools against the
   gateway, one optimizer step with peak memory recorded, one weight sync into the inference engines, one checkpoint). If B200
   fails (CUDA or kernel support), 8 x H200 is tried. Smoke budget about 60 USD of Modal credits. If neither runs the model within
   two working days, the owner decides between a smaller Qwen3.5 size and waiting for a prime-rl fix; either is a design change
   recorded before the freeze, not a deviation.
2. **The production path holds, as a client would use it [GATE].** The second host is installed from the public tarball by the
   documented procedure, approved and bound to the drums-labs workspace **in the console**; the API key is created in the console
   with the documented scopes; the trainer reaches desktops only through the gateway (`/v1` and `/mcp`); 128 + 8 desktops are
   admitted; quotas and the spend cap are set in the console so the study fits inside them and a refusal is a finding, not a
   crash; the live desktops are visible on the console's Computers page and the screenshot viewer during the run; usage and the
   audit log show the study's calls afterwards. drums-rc1 is never addressed. Where the console has no screen for a step (it
   does not launch training runs in this version), the step is done with the public CLI or SDK and the gap is an audit finding.
3. **Templates validate [GATE].** All 96 variants of each candidate template pass `drums_tasks.validate` on the env host (oracle
   passes, near-misses fail); failures are fixed in the template or the variant is dropped and counted.
4. **Variant overlap [GATE].** Zero parameter overlap between training and held-out variants per template.
5. **Step-0 scan and signal [GATE].** With the untouched 27B: K = 2 on the 64 training variants of each candidate template (384
   episodes in all, the evaluation path). A template qualifies when at least 16 of its 64 training variants are mixed (some
   success, some failure) and the pooled rate is between 0.10 and 0.80. The template with the most mixed variants is run. If none
   qualifies, the run does not start: the 27B has no success signal on these tasks and the owner is told (a short SFT warm-up or
   an easier template would be the next step; neither is part of D1).
6. **Power [GATE].** From the chosen template's step-0 profile; section 4 numbers replaced by the computed ones.
7. **Credits [GATE].** GPU on Modal credits (balance read before launch **[PIN]**); AWS pays the second worker host only (about
   55 USD; balance about 400 USD on 2026-11-16, of which drums-rc1 uses about 16 USD a day).

## 7. Owner decisions **[OWNER]**

1. The task: one of the three candidates, or a template you describe (one sentence and what "done" means is enough).
2. (settled 2026-11-16: full parameter.)
3. (settled 2026-11-16: Modal for the GPU; Drums Cloud through the gateway for the desktops, with a second bound host.)
4. (settled 2026-11-16: W = 480 min trained.) Note: at the 27B's speed that is about 2 to 6 updates; if the step-0 scan shows a shallow slope
   you may prefer 120 min (about 2x the GPU line below). Default stays 60.

## 8. Spend estimate (an estimate; this model has not been trained on this stack)

| item | basis | USD |
|---|---|---|
| GPU, Modal 8 x B200 192 GB | 53.6 USD/h unpinned (Modal `billing rates` 2026-11-16: 6.25 USD/GPU/h + CPU and RAM) x about 13.5 h (image build and gate 1 about 2.5 h, step-0 scans about 1.5 h, training 8 h plus waits, offline evaluations and the curve about 1.5 h) | about 725 of Modal credits (H200: about 560); the remaining balance is the retry margin |
| second worker host, c8i.24xlarge, us-east-1 on-demand | 4.50 USD/h x about 20 h (install and bind, validation, scans, training, evaluations) | about 90 |
| template work | no compute | 0 |
| EBS, transfer, Modal Volume | model download 56 GB, up to 6 HF exports | about 15 |
| **D1** | | **about 725 Modal + about 105 AWS** (the Modal agent's own estimate with caps: 510 to 600) |

Everything is stopped when the data are pulled; the ledger goes in the report.

## 9. What D1 will and will not show

Will show, if met: that a customer-defined task template, run through the factory and one hour of full-parameter GRPO on Drums
Cloud desktops through the gateway, improves a public 27B model on unseen variants of that task, with the step-0 baseline, the
per-update curve and the dollar cost.

Will not show: anything about other tasks or apps (one template); generalisation to OSWorld; a comparison with another stack;
the best achievable 27B; a gain under about +0.10 (power at this size).

## 10. The loop audit (second deliverable)

D1 is run as a customer would run it, and every stage is logged as findings: what happened, what it cost us in time, the fix,
the owner, and whether a public user would have hit it too. `AUDIT.md` beside the report, one table per stage:

| stage | what is recorded |
|---|---|
| templates and tasks | time to write each template; what the factory rejected; variants failing validation and why; checker or oracle bugs found |
| verifiers | checker disagreements between the lab verifier and a redundant checker; episodes marked invalid, by gate; isolation-detector false positives (known from the Cua work) |
| environments | ready, reset and fork times from the gateway's metrics; quarantines; refused starts; anything that looked like a desktop fault to the policy |
| gateway and console | quota and spend-cap behaviour at 128 concurrent desktops; audit-log completeness for the study's calls; key scopes that got in the way; request ids on every error |
| trainer integration | what it took to run prime-rl with drums-verifiers against the gateway on Modal, from the public packages and docs only; every TOML knob we had to discover; the same work a prime-rl user does |
| GPUs | per update: seconds in rollout wait, reference pass, update and weight sync; GPU busy fraction on the actor and rollout GPUs; peak memory and headroom at full parameter; dollars per update; minutes the GPUs sat idle waiting for desktops and minutes desktops sat idle waiting for the GPUs (L1: the GPU set the pace); what Modal cost versus the same hours on AWS |
| open source and integrations | everything installed from the public artefacts only (drumsd release tarball, drums-sdk and drums-verifiers from PyPI, prime-rl from PyPI); the MCP computer tools are the action path for every episode, so every tool-level rough edge is logged; the console screens used and the ones missing (no training-run screen in this version); every place we needed private knowledge is listed as a documentation or packaging gap |
| benchmarks | the run's own desktops per host, reset time and cost per valid episode next to what bench.drums.run states for the same host class |

The audit has no pass or fail. It is the list of what to fix next, ranked by how often a customer would hit it.

## 11. Pins (filled 2026-11-16 before the freeze)

| pin | value |
|---|---|
| model | `Qwen/Qwen3.5-27B` revision `fc05daec18b0a78c049392ed2e771dde82bdf654`; shard sha256 list = `d1-weights:/SHA256SUMS` (24 entries, written 2026-11-16T07:36Z) |
| trainer | prime-rl commit `167d4c3` (v0.10.1.dev238, not on PyPI); `modal/d1.toml` sha256 `0f41747e8abddb9b59ca46388926070bd33959adeb7ab1608aa164b66aa702ed` (Qwen3.5-27B, 2 inference + 6 trainer GPUs, GRPO/IPO eps 0.3, no reference KL, group 8, batch 128 = 16 groups, lr 3e-6 AdamW constant, max_norm 1.0, 15 turns, seq_len 32768, full activation checkpointing, ckpt every step keep_last 2); `modal/smoke.toml` sha256 `9beae23af584f749f42e362970f135e3bc19a1d2425ce3150d172b804e776fef`; no trainer patches |
| drums-verifiers | 0.2.1 from PyPI in the image + the D1 template branch `templates/d1` at commit `42e1e60` (families d1_files_collect, d1_calc_aggregate, d1_writer_style; split.py; ACTION_CHUNK 8000; validate retries 429); GENERATOR_VERSION `t6-factory-2`; drums-sdk 0.3.0 |
| sampler | uniform over the 64 training variants (prime-rl default); no curriculum (section 3's DynamicSampler has no prime-rl equivalent; decided 2026-11-16T11:32Z before the freeze) |
| console records | workspace `00000000-0000-4000-8000-000000000001` (drums-labs); client API key created by the owner in the console 2026-11-16T06:1xZ (id in the console's Keys page; never recorded here); host row `00000000-0000-4000-8000-000000000103` drums-prod-2, bound active 2026-11-16T05:48:48Z; workspace_limits max_live_computers 150, max_spend_usd 500, max_parallel_runs 4; price_versions 203 (4.49808 USD/h) |
| templates and seeds | template **d1_files_collect**; training seed 611161 (64 variants); held-out seed 611162 (32 variants); `pool.json` sha256 `c0ccca4f7a549d7f3e797f50ff4c3d8a119c51905e7862a46d9845e193b40504` (overlap checks instruction/file/params: 0 collisions); train `index.json` `9cab00540b037b055ebace6a1a58bb6f81eb5bf8b5b907985c9b8094908a8bc7`, train `tasks.jsonl` `b22899d9e38e973eeae2ea66b1f2506ed84a009a349a3dd29b72f50a66b6f545`; held-out `index.json` `06aca9bb30048b6d6cb235332fca0aed4fa9f1f8d1d550835c9e2a520aff6585`, held-out `tasks.jsonl` `c7af68d868d90be9f878a951e761fa5ab421cea5e09f19bd72654dcdd504877c`; lab verifier `drums_t6.py` `27adcd4c0a41283a03a5339016c5f3b39d25bd5c9a40de9587fdaf2080016af7` (installed in /etc/drums/verifiers/ on drums-prod-2) |
| step-0 scan | `modal/pull/scan/scan/scan-results.json`; scan-eval.toml sha256 `c39bab74bf3b4cc5b681c752f54e63d5a025e05c8635c913a1d7a3bc26493dde`, scan-inference.toml `c300095ff07635d8a92bc010f4bdedeb26ebea4e49295dc0b3b4f4947a51dd26`; files: 27/64 mixed, pooled 0.629 |
| offline evaluation configs | the scan-eval template with K = 6 on the held-out set (tag `step0-heldout-k6` / `last-heldout-k6`) and K = 2 on the training set; the filled TOMLs' sha256 go in RUNLOG.md when written (same keys as the scan: thinking off, 48 in flight, temperature per scan-eval.toml) |
| bootstrap seed | 20261116 |
| desktops | 128 training (d1.toml max_inflight 128, env.max_concurrent_agents 128) + up to 48 evaluation on drums-prod-2 (max_desktops 136; the evaluation runs after training, never concurrently); admission line: drumsd `max_creates_in_flight` 16, `max_concurrent_restores` 36 |
| GPUs, prices and hosts | Modal workspace drums-labs, no region pin, Function `train_any` gpu = ["H200:8", "B200:8"] (the smoke ran on **8 x H200 141 GB**, driver 610.57.04; B200 had no capacity in 3 h, section 6.1 fallback); list prices 4.54 USD/GPU/h H200, 6.25 B200; image `d1-prime-rl:167d4c3` (Dockerfile sha256 `137e43f354c8e87bf9a756492f41c2ee055ee85ab2b59394f5e84ffad6dc2fa9`, build 5, nvidia/cuda:13.0.1 base, torch 2.14.0+cu130, vLLM per prime-rl's lock); Volumes `d1-weights`, `d1-runs`; worker host drums-prod-2 = `i-0e3c45e7c79df693e` c8i.24xlarge us-east-1b, EIP 3.226.227.165 (`lab.drums.run`), release tarball `drums-release-80ff10e07568.tar.gz` sha256 `f2c4e9782692c1701d236ec05dcfde7ebba89a9a34b71bae1aa957b1203c2e7f`, drumsd commit `80ff10e07568`, kernel 7.0.0-1014-aws; gateway build 20261116053522-f469254; gateway key = the rc1 key set (A-02), id not recorded here |
| CRIU on the env host | sha256 `c06001dd0bbb7a828c634de5582341c2ee0f5c283d187af92ada9559c433dcf7` |
| Modal balance | not readable from the CLI; the owner's dashboard holds it. Spend before the training launch (estimates at list price): scan 1.2 GPU-h H200 about 5.5 USD; smoke 8 x H200 x 19 min about 11.5 USD; CPU jobs < 1 USD |
| gate 6.1 smoke | `ap-GutlnBVOGlbxktgzZzdpbU`, 2026-11-16T12:28Z to 12:46Z: one step, 8/8 trainable samples, reward 0.875, trainer step 5m16s (3m45s after the batch arrived), **peak memory 77.4 GiB per trainer GPU**, checkpoint step_1 written in 5m22s, NCCL startup broadcast to the 2 inference ranks OK |

