Training a computer-use agent by reinforcement learning is a loop. A model looks at a screen, chooses an action, the computer changes, and at the end something checks whether the task got done. The check becomes a reward and the reward updates the model. Infrastructure sits in the middle of every turn of that loop, so infrastructure mistakes become training signal. This note explains the loop and the one number we think matters.
The parts
- Policy. The model being trained (or, in an infrastructure experiment, a scripted oracle that always does the right thing so that any failure is the environment's).
- Environment. A computer in a known starting state: the OS, the applications, a document, a browser profile.
- Observation. A screenshot, and sometimes an accessibility tree. In our harness every screenshot carries the generation it was taken under and a stale flag.
- Action. A click, a key, typed text, a scroll. Every action carries a generation.
- Verifier. Code that inspects the final state (files, database rows, a server-side form row) and returns success or failure. The verifier is not the agent and must not trust it.
- Trainer update. The optimiser step that turns rewards into a new policy. In published systems one update consumes tens to hundreds of rollouts.
Three outcomes, not two
Published systems often score a rollout that crashed as reward zero (ComputerRL states this explicitly). That is the pattern a valid-rollout metric exists to replace. We count three outcomes:
| Outcome | What happened | Reward | Counts in valid rollouts | Counts in attempts |
|---|---|---|---|---|
| valid, task succeeded | reset verified, all actions applied under the right generation, verifier ran and said done | 1.0 | yes | yes |
| valid, task failed | same, but the verifier said not done | 0.0 | yes | yes |
| invalid (environment failure) | dirty reset, stale generation applied, timeout, black screenshot, verifier unreachable, a contamination event | none | no | yes |
A failed task is still real training experience: the agent tried something and the world said no. An invalid rollout is noise with a reward label attached, and it must be removed from training and from the denominator. The definition is in what Drums is.
Why cost per valid verified rollout
Throughput numbers in the literature divide by different things: configured replicas, logical capsules, trajectories in a burst, steps per second. None of them is the quantity a training run pays for. A training run pays for valid rollouts with a trustworthy reward, and everything else (failed resets, retries, idle reserve, verifier calls, inference) is cost that must be attributed to those rollouts.
usd_per_valid_rollout = host_usd_per_hour / valid_rollouts_per_host_hour
That is the whole formula in experiment 1 (host cost only; inference is excluded and stated as excluded). The fuller definition from the 2026-10-15 baseline: cost per 1,000 verified rollout attempts equals 1,000 times total attributable cost divided by attempts with a trustworthy final evaluator result, where total attributable cost includes host CPU and RAM hours, GPU and inference, storage and network, provisioning and reset, idle reserve, verifier calls, and amortised integration and operations.
Three facts about this metric from the prior-art read:
- No published source reports environment cost per valid rollout (checked across the map in prior art). The nearest are OSGym's USD 0.23 per replica-day (hosting only, price regime unstated, author-reported) and CUA-Gym's about USD 0.32 of VM time per synthesised task (author-reported).
- The fraction of a real training budget that goes to environments rather than inference is not stated in any source. ScaleCUA's 64 H800 GPUs against 600 CPU environments is the best ratio evidence and gives no dollar split (author-reported). If environments are fraction e of spend and a shared runtime halves that component, the gross saving is e/2. At e = 1% the ceiling is 0.5%; at e = 20% it is 10%. We intend to measure e, not pick a flattering value.
- "Valid trajectories" is not defined in OSGym's paper; "failed rollout scores 0" is ComputerRL's rule. These are reasons the metric is worth publishing, not evidence that ours is better.
The differentiating experiment: branch from a prepared state
The one experiment we think separates a shared runtime from a tuned VM pool is this. An agent opens LibreOffice, loads a document, and reaches an intermediate state after k steps. From that exact state, create many independent continuations and evaluate each. Compare how efficiently each runtime creates those continuations, how faithful they are to the prepared state, and what each valid continuation costs.
What the experiment needs before it can be run, in order: a restore that is faithful (locks included), a reset that is correct, isolation with a positive control that fired, then throughput at a confirmed N, then cost, and only then an RL run where nothing changes but the environment. The order is the gate order on methodology.