In one paragraph
Experiment 1 asks whether an environment is correct (isolation, reset, no contamination). Experiment 2 asks what it costs to get valid agent experience out of it. The plan is to run the same frontier agent, Claude Opus 5.5 through the Bedrock adapter of EXP-012, with the same frozen configuration, on the same frozen task set, against each environment arm, and to count what comes out: trajectories that completed, trajectories that failed the task but are still valid experience, trajectories that are invalid or corrupted by the environment, and the time and money spent on inference, on the environment, on verification and on recovery. The point is not whether Opus 5.5 is good at desktops. The point is whether the environment wastes its steps. Nothing has run. This record is the protocol and the preconditions.
Setup
What Experiment 2 measures
Per arm, per task, per seed, from the harness's receipts and the adapter's episode records:
| quantity | where it comes from |
|---|---|
| completed trajectories | adapter ending done_claimed or max_steps and the external checker returning success |
| failed-but-valid trajectories | checker returns task failure (exit 1) with every validity gate passed |
| invalid or corrupted trajectories | isolation canary violated, fidelity hash mismatch after reset, policy or checker error, backend_error, protocol_error, action_rejected, any non-zero checker exit other than 1 |
| task success rate | completed over all valid trajectories, reported per arm with its denominator |
| inference time and cost | the endpoint's own usage fields and api_seconds per call, summed per trajectory; price from the provider's published rate, settled only by reconciliation |
| environment time and cost | environment_seconds (execute, sleep, observe) plus the harness phases restore, reseed, reset and state hash, times the reconciled host rate |
| verification cost | time of the checker, canary probe and state-hash phases per attempt |
| recovery overhead | time and attempts spent on failed restores or resets and on re-provisioning a slot after an invalid attempt |
The headline per arm is cost per valid trajectory (environment plus verification plus recovery, inference reported separately because it is identical by design across arms) and the invalid rate. Both come from the same receipts the harness already hash-chains.
Rules fixed in advance
- Failed tasks can still be valid RL experience. A trajectory where the agent did the wrong thing but the environment behaved (isolation held, reset was exact, the checker ran) counts in the valid denominator. Only environment faults make a trajectory invalid. This is the harness's existing rule (checker exit 1 is "task failure, still valid").
- Frontier-model outputs are not used for training. The Bedrock route carries
NO_TRAINING_USEon every record; Experiment 2 produces measurements, not a dataset. Experiment 3 (EXP-018) uses an open-weight model for that reason. - Same model, same frozen config hash, same system prompt, same effort (
max), same append-only history on every arm. The config is frozen once and refused if it differs. - Same workload hash, same seeds, same checker on every arm; the sameness gate must report every identity fact known and equal before any comparison.
- No retries; failed and invalid attempts stay in the denominator.
comparestays descriptive withclaim: nulluntil a statistical design exists (EXP-015).
The frozen workload
The idea: 20 to 30 desktop tasks that every arm can support (GNOME Files, a text editor, a browser where the image has one), taken from OSWorld task definitions with their external checkers wherever the task's checker is a command or file inspection the harness can run from outside. Tasks whose checkers need an OSWorld component the arms do not expose are excluded, not approximated. The one task wired so far is the folder rename of EXP-007; the set is not frozen.
What ran
Nothing. No preflight, no episode, no workload freeze.
Results
None. No numbers will appear here until a run with receipts and episode records exists.
Failures and surprises
None recorded. The nearest observations are in EXP-012: the real endpoint rejected a trimmed history, and earlier runs ended as endpoint_error.
Limitations
- The Drums candidate arm cannot be in this experiment until it has a restore that passes its own gates (EXP-008, EXP-009). Until then the arms are the QEMU/KVM baseline and the OCI control, which only compares a VM with an ordinary container.
- Inference cost dominates environment cost for a frontier model; the environment numbers must be reported on their own, not folded into a total that hides them.
- Opus 5.5 outputs cannot feed training, so Experiment 2 says nothing about learning; it measures throughput and corruption.
- Append-only history means later steps are more expensive than earlier ones; cost per trajectory depends on step count, which depends on the model's behaviour, so per-arm comparisons must hold step counts side by side.
Reproduce
The preflight that must pass first, bound to the frozen config:
python -m bedrock_cua freeze CONFIG.json
python -m bedrock_cua preflight --config FROZEN --screenshots DIR_OF_1920x1080_PNGS --out preflight-1 --turns 6
Then, per arm, with the adapter as the workload's policy command and the harness owning reseed, checker, canary and reset:
python -m drumsbench plan --workload W.json --arm baseline --baseline-config B.json
python -m drumsbench run --workload W.json --arm baseline --baseline-config B.json --money M.json --out-dir runs/exp2-baseline --execute
python -m drumsbench run --workload W.json --arm drums --arm-config D.json --money M.json --out-dir runs/exp2-oci --execute
python -m drumsbench reconcile ... && python -m drumsbench compare ...
Each run directory with its receipts.jsonl, summary.json, reconciled.json, the episode.json files and the NO_TRAINING_USE marker is the evidence this record is waiting for.
Sources
benchmark/bedrock_cua/BEDROCK.md,benchmark/bedrock_cua/preflight.py,benchmark/bedrock_cua/config.pybenchmark/drumsbench/runner.py,compare.py,reconcile.py(validity rules, cost estimate, descriptive compare)benchmark/osworld_smoke/data/task.json(the one task wired so far)benchmark/sharedos_rootfs/sameness.py- The Experiment 2 definition (quantities, frozen workload, training exclusion) comes from the Drums research plan, not from a file on this branch.