DrumsBench

Experiments / EXP-021

Branching a real task setup state: a LibreOffice Writer launch restores 3.6x faster than cold plus setup, and the snapshot-API bridge preserves and branches task state on real sessions

Validate, against thresholds fixed before the run, that the post-setup state of one real public task setup can be checkpointed and restored as private siblings faster than redoing the setup, and that OSWorld's public snapshot API driven through Drums preserves and branches task state under the real evaluator path.

verifiedBacked by measurement files we can point to, on named hardware, with the run recorded.
Date
2026-10-24
Hypothesis
The author gave LibreOffice checkpointing about a 40% chance and expected the saving to be small because launch-only setups are cheap.
Arms
l1-criu-container
Hardware
4 vCPU, 16 GiB, cgroup v1, no KVM (dev-sandbox)
OS
Ubuntu 24.04, kernel 6.18 (sandbox)
Workload
Setup of public task e1fc0df3 (launch libreoffice --writer, LibreOffice 24.2.7.2) via the real SetupController in a Drums L0 session, checkpointed with CRIU 4.2; bridge on public task e0df059f through the real DesktopEnv, 6 rollouts per reuse mode, plus a 3-sibling mid-task branch
Code
claude/linux-comparison-prep at b8d1aad
Evidence
docs/data/l1-criu/task_branch_probe.json
docs/data/l1-criu/task_branch_probe.log
docs/data/l1-criu/task_branch_probe_run1_T3_dialog_harness_fail.json
docs/data/l1-criu/diagnosis/README.md and the four PNGs
docs/data/l1-criu/matched_reuse/run3_final.json
docs/data/l1-criu/matched_reuse/run1_restore_race_and_EXDEV.json
docs/data/l1-criu/matched_reuse/run2_after_redirect_dir_with_24s_sudo_hostname_stall.json
docs/data/l1-criu/matched_reuse/findings_before_after.json
docs/data/l1-criu/matched_reuse/chained_checkpoint_whiteout_finding.txt
docs/l1-shared-runtime-design.md (sections 18 to 21)
Limitation
Both arms are Drums; this is harness validation, not a comparison with any incumbent, and the 3.6x is on a setup of about 3.1 s, so the saving is about 2.25 s per rollout after the first of a group.

In one paragraph

EXP-005 showed that restoring an empty warm session saves about a second and misses its target once hygiene is added. The recurring cost in training is per rollout of the same task: reset the guest and replay the task's setup. So we tested branching a state after a real setup. Measured: the setup of public task e1fc0df3 is exactly launch libreoffice --writer; run by OSWorld's real SetupController in a Drums L0 session, cold session plus setup reaches agent-ready in p50 3.12 s; restoring a CRIU checkpoint taken after that setup reaches it in p50 0.87 s, ratio 3.6x. The checkpoint dumped in 0.35 s to a 236 MiB image; two siblings restored in 0.73 and 0.80 s; both showed Untitled 1 — LibreOffice Writer; a token typed into one sibling appeared in its clipboard and not the other's; the sharing audit under revision 2a passed with only /dev/null shared (cold pair the same); memory per added session was 238.5 MiB in both arms. All six registered thresholds pass. The first measured run failed threshold 3 for both siblings because LibreOffice's first-run "Tip of the Day" modal, part of the preserved state, swallowed the keystrokes; a cold control did the same, so it was a harness defect, not a restore failure, and the failed run is kept. The second half of this record is the snapshot-API bridge: OSWorld's public _save_state / revert_to_snapshot implemented as CRIU checkpoint and restore, validated on real sessions for task e0df059f (6 valid rollouts in each reuse mode, scores 0.0 then 1.0 on all twelve) and a mid-task branch into three siblings that scored 1.0 / 0.0 / 1.0 with the agent's own process state identical and the infrastructure reseeded. Both arms are Drums; this is not a comparison. "Drums" means the L0 prototype: namespaces, overlayfs on the sandbox's own Ubuntu 24.04 root, Xvfb, openbox; no shared-runtime (L1) result exists.

Setup

Host. One development sandbox: 4 vCPU, 16 GiB, kernel 6.18, cgroup v1, no KVM; hostname and provider not recorded. LibreOffice 24.2.7.2 from the Ubuntu 24.04 archive with only the gen VCL plugin (no GTK plugin, so no AT-SPI exposure).

prepared state(checkpoint)env01env02env03reset: back to the checkpointreset = kill apps, clear pasteboard,swap ~/work for a fresh clone, then g+1env02 generationg=1g=2g=3action, g=1appliedaction, g=1 (late)409 stale_gen, not appliedaction, g=3applied
Reset versus branch. A cold reset replays the task setup every rollout; a branch restores a checkpoint taken after the setup (or, in V2, after an agent prefix), so rollouts 2 to G of a group share one setup. The same shape exists for VM snapshots; nothing here measures an incumbent.

Part 1, task-state branching. Registered in section 18.4 (commit 1e2f3eb, 08:42 UTC) with six thresholds: feasibility (LibreOffice dumps and restores, no workaround that drops it from the checkpoint); state preserved (Writer window with the same title in each sibling); in-app state works and is private (a unique token typed into A through the real pyautogui path is readable from A and not B, and the reverse); the C3 audit under revision 2a with a cold-pair control; latency BRANCH p50 at most 0.50 x SETUP p50; memory at most 1.05x. Amendment before the measured run (c4488e2, 08:55 UTC): the registered accessibility-tree readiness could not be evaluated because this LibreOffice build exposes nothing to AT-SPI (the a11y tree stayed at 541 bytes while 99.6% of pixels changed), so readiness became "wmctrl -l lists a window whose title contains LibreOffice Writer AND the screenshot differs from blank by more than 10%", and threshold 3 reads the clipboard through xclip after select-all and copy. Arms: SETUP = cold friendly session plus the real setup; BRANCH = restore of the post-setup checkpoint, declared not clean (browser and application state preserved), with hostname and server-RNG reseed. 10 starts per arm interleaved; memory over 3 trials to 12 sessions (not 20, to stay within host memory).

Part 2, the snapshot-API bridge. Registered in section 20 (3526b6a, 09:19 UTC). Both arms get the same reuse through the same code path: OSWorld's public snapshot API on DesktopEnv. The Drums provider implements save_state as "CRIU-checkpoint this session" and revert_to_snapshot as "destroy and restore from that checkpoint". V1: 6 rollouts of task e0df059f at concurrency 1 with reuse="none" and with reuse="post_setup", a scripted correct action, the interference canary on. V2: after the setup an agent prefix writes a unique token to a file and starts a long-lived Python process that seeds random; the state is saved as agent_prefix; three siblings are restored: A renames correctly (expected 1.0), B renames wrongly (expected 0.0), C repeats A (expected 1.0).

What ran

envd_linux.l1proto.criu_task_probe ran twice: run 1 (kept as task_branch_probe_run1_T3_dialog_harness_fail.json) and run 2 (task_branch_probe.json, commit eb571a9, 09:10 UTC). The diagnosis scripts and screenshots were committed in 93d3842. tests/test_matched_reuse_real.py ran three times: run 1 (restore race and EXDEV rename failure), run 2 (scores fixed, a 24 s sudo stall), run 3 final (run3_final.json, commit b8d1aad, 09:40 UTC). envd_linux.l1proto.findings_evidence produced findings_before_after.json. A chained-checkpoint test found and fixed a whiteout defect (chained_checkpoint_whiteout_finding.txt, commit f37be36).

Results

Branching a real task setup state (LibreOffice Writer launch) and the snapshot-API bridge V1/V2 on real Drums L0 sessions verified2026-10-24 · dev-sandbox

MetricValueUnitnNote
run 2 (final): SETUP, cold session plus real LibreOffice launch, to agent-ready3.12 (p95 3.24)s10
run 2 (final): BRANCH, restore of the post-setup checkpoint, to agent-ready0.868 (p95 0.917)s10
run 2 (final): SETUP p50 / BRANCH p503.6xdesign target was at least 2x; PASS; about 2.25 s saved per rollout after the first of a group
T1 checkpoint of the running LibreOffice: dump time0.346s1image 236.48 MiB; window title 'Untitled 1 - LibreOffice Writer'
T1b restore of two siblings: restore_total_s0.726 to 0.8s2
run 2 (final): host memory per added session, COLD with setup238.5MiB33 trials to 12 sessions, not 20, to stay within host memory
run 2 (final): host memory per added session, RESTORE238.51MiB3ratio 1.00005; cold minus restore 95% interval [-10.35, +6.70] MiB; bar at most 1.05x: PASS
run 2 (final): thresholds passed (T1 feasibility, T2 state, T3 private in-app state, T4 audit 2a, latency, memory)6 / 6thresholds
run 1: thresholds passed5 / 6thresholdsT3 failed for BOTH siblings (own_a false, own_b false): the preserved first-run 'Tip of the Day' modal swallowed keystrokes; a cold control did the same; a harness defect, not a restore failure
run 1: SETUP p50 / BRANCH p503.6xSETUP p50 3.267 / p95 3.592 s; BRANCH p50 0.908 / p95 1.115 s; memory 235.13 vs 235.46 MiB
V1 reuse=none: valid rollouts6 / 6rolloutsscores 0.0 before / 1.0 after on every rollout; canary 6 clean; wall 28.062 s
V1 reuse=none: per-rollout reset2.17 (0.834 to 2.5)s6the first rollout (0.834 s) runs on a fresh environment with no revert; later ones revert to the initial state
V1 reuse=post_setup: valid rollouts6 / 6rolloutsphases setup+save then 5 x revert; all setup_only; wall 20.894 s
V1 reuse=post_setup: first rollout (setup AND save)1.86s1
V1 reuse=post_setup: per-rollout revert0.82 (0.789 to 0.843)s5
V2 mid-task branch: save_branch1.16s1
V2 mid-task branch: restore_branch0.962s1
V2 scores: A correct / B wrong / C correct again0 to 1score1.0 / 0.0 / 1.0; agent_next_draw identical across siblings (task state preserved), hostnames and server draws differ (infrastructure reseeded)
V2 gates passed (i task state, ii reseed, iii divergence, iv isolation, v audit 2a)5 / 5gates

Host: 4 vCPU, 16 GiB, cgroup v1, no KVM, Ubuntu 24.04, kernel 6.18 (sandbox). one development sandbox; hostname and provider not recorded; not a VM or cloud host. Workload: Task branching: the setup of public task e1fc0df3 (exactly 'launch libreoffice --writer', LibreOffice 24.2.7.2, gen VCL plugin) run by OSWorld's real SetupController in a Drums L0 session, checkpointed with CRIU 4.2 and restored as two siblings. Bridge: public task e0df059f (rename a directory; shell getter + exact_match) through the real DesktopEnv with reuse none vs post_setup (V1, 6 rollouts each) and a mid-task branch into three siblings (V2).. Method: SETUP = cold friendly session plus the real setup, timed to agent-ready (wmctrl lists a window titled LibreOffice Writer AND the screenshot differs from blank in more than 10% of pixels; the registered a11y-tree criterion was replaced before the measured run because this LibreOffice build exposes nothing to AT-SPI). BRANCH = restore of the post-setup checkpoint, declared not clean, with hostname and server-RNG reseed. 10 starts per arm interleaved; memory slope over 3 trials to 12 sessions. Registration 1e2f3eb, amendment c4488e2, results eb571a9; bridge registration 3526b6a, results b8d1aad. Both arms are Drums; this is harness validation, not a comparison.. Source: branch claude/linux-comparison-prep: docs/data/l1-criu/task_branch_probe.json (stages, latency, memory_summary); branch claude/linux-comparison-prep: docs/data/l1-criu/task_branch_probe_run1_T3_dialog_harness_fail.json (stages.T3, latency, memory_summary); branch claude/linux-comparison-prep: docs/data/l1-criu/diagnosis/README.md; branch claude/linux-comparison-prep: docs/data/l1-criu/matched_reuse/run3_final.json (v1_none, v1_post_setup, v2); branch claude/linux-comparison-prep: docs/data/l1-criu/matched_reuse/findings_before_after.json; branch claude/linux-comparison-prep: docs/l1-shared-runtime-design.md sections 18 to 21.

  • Both arms are Drums L0 sessions; this is harness validation, NOT a comparison with any incumbent and not a throughput result.
  • The run1/run2 bridge timings (about 24 to 26 s resets, a sudo hostname-resolution stall) are invalid and are not quoted; they are kept on the branch as evidence of how an environment defect can fake a reuse benefit.
  • Three of the four diagnosis PNGs are byte-identical (sha256 47fbd307...db98); they must not be presented as three distinct captures.
  • The task e1fc0df3 itself was not evaluated (its evaluator needs the internet); only its launch setup was branched.
  • Evaluator path for e0df059f is a shell getter plus string match; librosa is stubbed and unreached; the stub list was not saved for these runs.
  • Drums here is the L0 prototype restored by CRIU; no shared-runtime (L1) result exists.

Part 1: branching the LibreOffice launch state (measured, run 2)

ThresholdMeasuredVerdict
1 feasibilityT1 dump 0.346 s, image 236.48 MiB, title at checkpoint Untitled 1 — LibreOffice Writer; T1b two siblings restored, restore_total_s 0.800 and 0.726PASS
2 state preservedboth siblings list the window with the same titlePASS
3 in-app state works and is privateown_a: true, own_b: true, a_in_b: false, b_in_a: falsePASS
4 sharing audit, revision 2a10 processes each; namespaces distinct; shared_character_devices: ["chr:6:3:/dev/null"], the cold pair the same; no pipe, socket, lock or writable file shared; PASS_revision_2a: truePASS
5 latencySETUP p50 3.122 / p95 3.243 s; BRANCH p50 0.868 / p95 0.917 s; ratio 3.595 (author-reported bootstrap interval 3.4 to 3.8); latency_faults: []PASS (bar 2x)
6 memoryCOLD with setup 238.50, RESTORE 238.51 MiB per added session; ratio 1.00005; interval [-10.35, +6.70] MiBPASS (bar 1.05x)

Run 1 passed thresholds 1, 2, 4, 5 and 6 (ratio 3.599; memory 235.13 vs 235.46 MiB) and failed 3 for both siblings: own_a: false, own_b: false, PASS: false.

What the saving is worth, by the author's arithmetic, not a measurement: about 2.25 s per rollout after the first of a group of G, so 2.25 s x (G-1)/G (1.97 s for G = 8): about 3% of a 60 s rollout and 0.7% of a 300 s one. The author's conclusion: "On this evidence, branching the setup state of launch-only tasks is a small saving."

Part 2: the snapshot-API bridge on real sessions (measured, run3_final.json)

GateMeasuredVerdict
V1 reuse="none"6 of 6 valid; scores 0.0 before and 1.0 after on every rollout; canary 6 clean, 0 interference; reset_s 0.834 then 2.44, 2.49, 2.39, 2.39, 2.50; wall 28.062 sPASS
V1 reuse="post_setup"6 of 6 valid; same scores; phases setup+save then 5 x revert; reset_s 1.859 then 0.814, 0.835, 0.789, 0.817, 0.843; wall 20.894 s; all setup_onlyPASS
V2 (i) task state preservedboth siblings hold the same prefix token (test string 931e4c96e699), the agent process alive, and its next draw identical: 0.8624508153567833 in A and BPASS
V2 (ii) infrastructure reseededhostnames differ; guest-server draws differ (0.877 vs 0.800)PASS
V2 (iii) divergent outcomesA 1.0, B 0.0, C 1.0 (PASS_iii_divergent: true)PASS
V2 (iv) isolationbefore acting B saw todo_list_Jan_1 and no todo_list_Jan_2; after A acted B's desktop held only its own todo_list_Jan_3; C after restore saw todo_list_Jan_1 againPASS
V2 (v) audit 2ashared /dev/null only, control the same; no lock, pipe or socket; no write visible acrossPASS

Mid-task save_branch_s 1.161, restore_branch_s 0.962. The first-rollout work (setup and save, 1.86 s) is inside the 20.9 s and inside the cost; with 6 rollouts it is a large share, so these per-rollout figures are not steady state.

Failures and surprises

Limitations

Reproduce

PYTHONPATH=<OSWorld checkout>:envd-linux:compare DRUMS_CRIU_HOME=<private criu dir> \
  <osworld venv python> -m envd_linux.l1proto.criu_task_probe --out probe.json \
  --task <e1fc0df3.json> --starts 10 --trials 3 --max-n 12
DRUMS_OSW_PYTHON=<venv python> DRUMS_OSW_SRC=<OSWorld checkout> pytest envd-linux/tests/test_matched_reuse_real.py

Needs root, overlayfs, Xvfb, openbox, LibreOffice, the OSWorld kit and CRIU 4.2. Compare latency.ratio_setup_over_branch_p50 and stages.T3_* in the probe output, and v1_*.valid, v2.PASS_iii_divergent in the bridge output.

Sources