Benchmarks
Measured numbers only. Every table names its hardware, workload, method and sample size.
Where a cell is missing, it has not been measured. Hypothetical, estimated or unmatched numbers never appear on this page as results.
- verified
- Backed by measurement files we can point to, on named hardware, with the run recorded.
- preliminary
- A real observation that needs more runs, a positive control, or an independent check before it counts.
- failed
- Ran and missed a correctness, isolation, reset or pre-registered performance bar. Kept on the record.
- planned
- Written and reviewed as code or protocol, but not executed. No numbers.
What will be compared
| Arm | Slug | What it is | Role |
|---|---|---|---|
| QEMU/KVM warm VM snapshots | qemu-kvm-warm-snapshot | One VM per environment, restored from a warm memory snapshot; KVM required and read back | primary baseline |
| OCI desktop with CRIU | oci-runc-criu | The same desktop image under runc 1.5.2 with CRIU 4.3 checkpoint and restore | reference arm; a control, not Drums |
| Custom Drums runtime | drums-rootfs-criu | The same image mounted read-only with a private overlay per slot in its own namespaces, CRIU restore with a root contract | candidate arm; no performance number until its restore fidelity gate passes |
| macOS sessions | macos-sessions | One macOS user session per environment on one Mac, no VMs | candidate arm on macOS |
| Linux VM with N virtual displays | linux-vm-xvfb-on-mac | One Ubuntu VM on the same Mac, one Xvfb display per environment | comparison arm on the Mac |
| Optimised published implementations | per record | OSGym, CUA-Sandbox or CUA-Lite style implementations, where reproducible and licensed | optimised incumbents, where permission allows |
Equivalence rules: the same OS version and applications, the same frozen task set, the same checker, the same hardware allocation per environment (for example 4 vCPU and 4 GiB on the Linux arms, read back from the cgroup or the VM), and the same reset semantics. Baselines run with pooling and snapshots on, never as a cold-boot strawman. When perfect equivalence is impossible (the Mac arms: macOS gets the whole machine, the VM gets 40 of 48 GB; a VM has its own kernel, a container does not) the difference is recorded on the record and in the result file's caveats, and the host's own memory pressure is logged so that swapping cannot penalise one arm unseen.
Metrics
| Metric | Unit | What it needs |
|---|---|---|
| Prepare and restore latency | s, p50 and p95 | a faithful restore (locks, network, files) |
| Reset latency and reset correctness | s, p50 and p95; manifest matches out of resets | a manifest check after every reset |
| CPU, RAM and storage per environment | cores, MiB private dirty memory (never RSS sums), MiB | host sampling under load, not per-process sums |
| Stable N | environments | the largest N that passes every gate in two separate windows |
| Valid verified rollouts per hour | rollouts per host-hour | sustained windows, not bursts |
| Cost per valid verified rollout | USD | three dated price inputs, inference stated as in or out |
| Isolation | contamination events over environment-hours, next to a fired positive control | a positive control that fired |
| Failure rates | environment failures over attempts, lost checks | every failed reset and timeout on its own line |
| Learning curves and cost to a fixed success rate | success rate vs rollouts and vs USD | experiment 3, identical models and rewards |
What exists today
As of 2026-10-25 no two arms have been measured on the same workload. No comparison exists. The harness's own comparison command is descriptive and emits no claim by construction. The files listed below are the only measured numbers this project has, and none of them is a performance result: they are harness test passes on named hardware, a host baseline with no environment running, a pre-registered negative on a Linux prototype, and probe outcomes. Author-claimed numbers from branch text without a log (dump seconds, image sizes, restore times, a five-call agent success) are not in any file here; they are quoted as claims on their experiment records.
The tables under "All measurement files" are generated from data/results/. Each file passes a schema at build time, names its experiment, arm, host, workload, method and sources, and carries its caveats next to its rows. Nothing on this page is typed in by hand.
All measurement files
12 files across 6 arms: macos-sessions, l1-fork-prototype, l1-criu-container, l1-shared-chromium, l1-ksm, sharedos-spike. Each links back to its experiment record.
Harness test suites and envd release build on studio-a verified2026-10-22 · studio-a
| Metric | Value | Unit | n | Note |
|---|---|---|---|---|
| python tests passed | 813 | tests | pytest -W error, Python 3.12.14; parametrized tests expand, so this is above the 723 test functions in the source | |
| python tests skipped | 1 | tests | the envd-linux real X round trip; needs a real X display | |
| python suite wall time | 479 | s | test runtime, not a performance figure for the harness | |
| swift tests passed | 105 / 105 | tests | Swift 6.1.2 | |
| swift suite wall time | 79 | s | printed as 1m19s; test runtime, not a performance figure | |
| envd release build wall time | 36.6 | s | swift build -c release; a build time, not a performance figure |
Host: Mac Studio M4 Max, 16 cores, 48 GB, macOS 15.7.5 (24G624). An institution-managed Mac with MDM and endpoint protection. Xcode Command Line Tools only (Swift 6.1.2, no full Xcode). Python 3.12.14 in a uv virtual environment.. Workload: pytest with -W error across drumsctl, envd-linux, metrics, report, scripts and verify-server; swift test for envd (EnvdCore); swift build -c release for envd. Method: Run once over SSH on 2026-10-22 at repository head f0d01ce after copying the repository by git bundle. Every Python test runs against drumsctl/tests/fake_envd.py or in process; the Swift tests are unit tests. Counts and wall times were read from the run logs kept in the operator account's home on the Studio.. Source: studio-a: drums-pytest.log (operator account home, not on the branch); studio-a: drums-swift-test.log (operator account home, not on the branch); studio-a: drums-build.log (operator account home, not on the branch); branch exp1-sessions-per-studio, head f0d01ce.
- Software evidence, not a benchmark result. The rows show that the code runs on this build; they say nothing about sessions, isolation, density or cost.
- Every Python test runs against fakes or in process; no test touched a real environment session, a real envd LaunchAgent or a real screen capture.
- The run logs live in the operator account's home on the Studio and are not on the branch; the same counts were last measured on the development laptop on the final tree.
- The envd binary sizes (release build and ad-hoc-signed install) differ and are not published as figures.
Record: EXP-001. File: data/results/exp-001-studio-test-suites.json.
Host baseline with no environment session logged in (pilot folder) preliminary2026-10-22 · studio-a
| Metric | Value | Unit | n | Note |
|---|---|---|---|---|
| memory used | 16.04 (15.99 to 16.07) | GB | 12 | of 48 GB; mem_used_bytes / 1e9; no environment session logged in |
| swap used | 0 | GB | 12 | 0 in every sample |
| WindowServer CPU | 12.9 to 21 | % | 12 | ps decaying average, not instantaneous |
| memory pressure normal | 12 / 12 | samples | memory_pressure reported normal in every sample |
Host: Mac Studio M4 Max, 16 cores, 48 GB, macOS 15.7.5 (24G624). An institution-managed Mac with MDM and endpoint protection whose agents were running throughout. One 1080p display attached. The console session held a terminal. No environment session was logged in.. Workload: idle host: no environment session, no envd running, no task; the metrics sampler only. Method: make meta, then make baseline-sample at about 02:47 Eastern on 2026-10-22, into the pilot run folder results/2026-10-22-studio-a-mac-pilot2. The sampler ran as the operator (not root) for about 60 s and wrote 12 host.jsonl lines. Memory in decimal gigabytes from mem_used_bytes; CPU percentages are ps decaying averages, not instantaneous readings.. Source: results/2026-10-22-studio-a-mac-pilot2/host.jsonl (pilot folder on studio-a; copy in the internal runbook archive); results/2026-10-22-studio-a-mac-pilot2/meta.json (same; its host field is not published).
- A pilot folder, not the published run folder.
- The 16 GB includes the institution's MDM and endpoint-protection agents and a console session with a terminal; no environment session existed. It is a baseline input the report subtracts from later windows, not a per-environment number and not an idle cost of the harness.
- The design's idle-session cost (the footprint of a session's processes after login) was never taken, and the post-login baseline (step 9 of the reproduce table) does not exist.
- Watts are null: powermetrics needs root and the sampler did not run as root.
- Total CPU swung between 39.0 and 388.9 percent across the 12 samples (ps decaying averages, with setup work close in time) and is left off this table on purpose.
Record: EXP-002. File: data/results/exp-002-host-baseline-0env.json.
Positive control probe P on studio-a: six attempts, none ran to completion failed2026-10-23 · studio-a
| Metric | Value | Unit | n | Note |
|---|---|---|---|---|
| probe P attempts | 6 | attempts | all Gate P: FAIL, all exit 2; five with app=Writer route=tap, one with app=TextEdit | |
| attempts that ran to completion | 0 / 6 | attempts | five stopped at no_focus (could not put LibreOffice Writer in front; the tap route types only into a front instance); one stopped at ax_not_trusted (the terminal running it lacked Accessibility) | |
| typing half fired | 0 / 6 | attempts | never started on any attempt; typing_cross was not measured | |
| pasteboard half fired | 0 / 6 | attempts | never started on any attempt; 0 of 0 reads; pb_cross was not measured |
Host: Mac Studio M4 Max, 16 cores, 48 GB, macOS 15.7.5 (24G624). An institution-managed Mac with MDM and endpoint protection. LibreOffice 26.8.1 installed. Run from the console session over SSH; the terminal had no Accessibility grant; no env session logged in.. Workload: probes/probe_p_positive_control.sh with probes/positive_control.swift: two Writer instances (or the TextEdit fallback) forced into one login session, typed into at once with disjoint alphabets and racing on one pasteboard. Method: Six attempts between 2026-10-23 02:58Z and 06:17Z, each writing one Gate P verdict row to probes.md in a pilot results folder set by DRUMS_RESULTS_DIR. Exit 2 is the program's could-not-run code (no Accessibility, app missing, no window, bad settings); it is neither a detector pass nor a detector fail. Rows below count attempts.. Source: results/probes-pilot-2-sessions/probes.md on studio-a (pilot results folder; not on the branch).
- A failure to run at its preconditions (window focus, Accessibility), not a detector verdict. Nothing here says anything about isolation.
- The positive control has not fired on any machine on its route of record (tap). Until it does, no isolation zero may be published from this harness.
- The pid-route numbers from the author's laptop (pb_cross 20 of 20, typing_cross 0; macOS 27.0, TextEdit, 6 s) are a different route, app, machine and OS and are not a result.
- The two error classes are no_focus (5 attempts) and ax_not_trusted (1 attempt); both have known fixes that need a person at the machine.
Record: EXP-003. File: data/results/exp-003-probe-p-attempts.json.
Mechanism F: fork the OSWorld guest server from a warm zygote (C0, C1, F arms) failed2026-10-23 · dev-sandbox
| Metric | Value | Unit | n | Note |
|---|---|---|---|---|
| host memory per added session, C0 (as shipped) | 280.96 (279.87 to 282.2) | MiB | 4 | per-trial slopes; n = trials of 20 sessions |
| host memory per added session, C1 (module entry, cold) | 231.15 (230.17 to 232.59) | MiB | 4 | the fair baseline for F |
| host memory per added session, F (forked) | 216.83 (215.69 to 218.08) | MiB | 4 | |
| C1 minus F | 14.32 (13.13 to 15.55) | MiB | 4 | min/max are the bootstrap 95% interval over trials (excludes 0) |
| saving F vs C1 | 6.2 | % | F/C1 = 0.938 | |
| pre-registered memory bar (a) | 15 | % | design value, commit 84a9241: F slope at most 0.85 x C1 slope; FAILED | |
| server-ready latency, C0 | 1.53 (p95 1.62) | s | 20 | |
| server-ready latency, C1 | 1.08 (p95 1.18) | s | 20 | |
| server-ready latency, F | 0.862 (p95 0.93) | s | 20 | F p50 / C1 p50 = 0.80; bar (b) needed at most 0.70; FAILED on the median, p95 condition held |
| Chromium-ready latency, C1 | 1.43 (p95 1.53) | s | 20 | |
| Chromium-ready latency, F | 1.21 (p95 1.33) | s | 20 | bar (c) F p95 below C1 p95: PASSED; the gain is the server's, the Chromium part is the same in every arm |
| create() latency, C1 | 2 (p95 2.09) | s | 20 | |
| create() latency, F | 1.74 (p95 1.88) | s | 20 | |
| zygote preload time | 0.221 | s | 1 | paid once per host; zygote RSS 56152 KiB, PSS 51448 KiB |
| forkable Python memory per session (guest server + init, C1 private) | 51.7 | MiB | 1 | 42184 + 10708 KiB private at N = 20, trial 0; the share bound: at most about 22% of 231 MiB even if wholly shared |
| Chromium private memory per session (10 processes, C1) | 90.9 | MiB | 1 | 93100 KiB; exec'd in every arm, never forked |
| pre-registered bars passed (a, b, c, d) | 2 / 4 | bars | a memory FAIL, b server-ready median FAIL, c Chromium-ready PASS, d sibling hygiene PASS |
Host: 4 vCPU, 16 GiB, cgroup v1, no KVM, Ubuntu 24.04, kernel 6.18 (sandbox). one development sandbox; hostname and provider not recorded; not a VM or cloud host. Workload: Drums L0 session (namespaces + overlayfs + Xvfb + openbox) with the real OSWorld guest server and Playwright Chromium 141 (--no-sandbox) on a blank data-URL page; 20 sessions per arm for latency, 4 trials of 20 concurrent sessions per arm for memory. Method: Memory: slope of (MemTotal - MemAvailable) against concurrent session count at 1, 4, 8, 12, 16, 20 sessions, 4 trials per arm; the 95% interval of (C1 - F) is a bootstrap over trials. Latency: 20 sequential sessions per arm after 2 discarded warm-ups, arm order rotated. C0 = guest server as OSWorld ships it (python main.py, debug reloader); C1 = fresh interpreter with the same module entry F uses; F = guest server forked from a warm zygote. Thresholds fixed in commit 84a9241 before measurement; results in dce0211.. Source: branch claude/linux-comparison-prep: docs/data/l1-prototype/bench.json (memory_summary, latency_summary, breakdown, zygote); branch claude/linux-comparison-prep: docs/data/l1-prototype/hygiene.json; branch claude/linux-comparison-prep: docs/data/l1-prototype/hazard.jsonl; branch claude/linux-comparison-prep: docs/l1-shared-runtime-design.md sections 8 and 9.
- Only the Python guest server is forked (fork point before import pyautogui); Chromium is exec'd in every arm.
- One development sandbox, one kernel; intervals are over 4 trials on that host, not over hosts.
- Drums here is the L0 prototype; no shared-runtime (L1) result exists; drums_level = L0 in every run.
- No incumbent was measured; nothing here compares Drums with a VM or container baseline.
- The hazard test forked a stand-in process with live X, D-Bus, socket and lock state, not Chromium itself; the Chromium conclusion is an inference.
Record: EXP-004. File: data/results/exp-004-fork-prototype.json.
Mechanism C: CRIU 4.2 checkpoint and restore of a warm Drums L0 session with Chromium, without and with the clone-hygiene reseed failed2026-10-24 · dev-sandbox
| Metric | Value | Unit | n | Note |
|---|---|---|---|---|
| no hygiene: COLD time to agent-ready | 2.05 (p95 2.19) | s | 20 | |
| no hygiene: RESTORE_FULL time to agent-ready | 0.98 (p95 1.22) | s | 20 | preliminary: a restore that skips hygiene is not a usable restore (clones share RNG state, cookies, hostname) |
| no hygiene: RESTORE_SERVER_ONLY time to agent-ready (partial arm) | 1.22 (p95 1.28) | s | 20 | |
| no hygiene: COLD p50 / RESTORE_FULL p50 | 2.09 | x | design target 2x (RESTORE p50 at most 0.50 x COLD p50); passes the point rule narrowly | |
| no hygiene: host memory per added session, COLD | 229.4 | MiB | 3 | |
| no hygiene: host memory per added session, RESTORE | 221.88 | MiB | 3 | ratio 0.9672; cold minus restore 95% interval [6.09, 8.94] MiB |
| server-only dump (R3) | 0.21 | s | 1 | image 74.78 MiB |
| Chromium dump (R4) | 0.806 | s | 1 | image 253.09 MiB; a one-time cost per checkpoint |
| Chromium restore call (R4), total | 0.763 | s | 1 | of which criu restore 0.319 s; browser title drums-ready present |
| C2 screenshot pixels changed vs blank desktop | 0.998 | fraction | threshold 0.10; PASS | |
| C3 sibling audit, literal pre-registered wording | 0 / 1 | checks | FAIL: two host character devices shared, /dev/null and /dev/urandom; the cold-pair control shares the same two | |
| C3 sibling audit, revision 2a (exactly those two devices, cold control, nothing else) | 1 / 1 | checks | PASS; the original FAIL stays on the record | |
| C1 task e0df059f score from restored sessions, before acting | 0 | score | ||
| C1 task e0df059f score from restored sessions, after the scripted action | 1 | score | shell getter + exact_match path only; librosa stubbed and unreached; stub list not saved for this run | |
| page cache dropped (NOT pre-registered): COLD time to agent-ready | 2.59 (p95 2.74) | s | 10 | |
| page cache dropped (NOT pre-registered): RESTORE_FULL time to agent-ready | 1.38 (p95 1.47) | s | 10 | ratio 1.874; the 2x target is not met with cold cache |
| reseeded: COLD time to agent-ready | 2.1 (p95 2.24) | s | 20 | |
| reseeded: RESTORE_FULL time to agent-ready | 1.19 (p95 1.29) | s | 20 | the usable restore: hostname, server RNG and browser context reseeded |
| reseeded: RESTORE_SERVER_ONLY time to agent-ready (partial arm) | 1.24 (p95 1.37) | s | 20 | |
| reseeded: COLD p50 / RESTORE_FULL p50 | 1.76 | x | below the 2x target: FAIL | |
| reseeded: host memory per added session, COLD | 229.59 | MiB | 3 | |
| reseeded: host memory per added session, RESTORE | 238.32 | MiB | 3 | ratio 1.0380; cold minus restore 95% interval [-9.86, -7.42] MiB (restore costs more); passes the 1.05x bar, fails 'no more than cold' |
| reseed cost per sibling | 0.198 to 0.215 | s | 2 | clone_hygiene.json reseed_s |
| reseeded: C1 task score before / after | 0 to 1 | score | clone_task_semantics.json: reseeded true, sessions_were_restored true, session clock minus host clock -0.185 s |
Host: 4 vCPU, 16 GiB, cgroup v1, no KVM, Ubuntu 24.04, kernel 6.18 (sandbox). one development sandbox; hostname and provider not recorded; not a VM or cloud host. Workload: Drums L0 session (namespaces + overlayfs full root + Xvfb + D-Bus + AT-SPI + openbox) with the real OSWorld guest server and Playwright Chromium 141 (--no-sandbox) on a data-URL page titled drums-ready; public OSWorld task e0df059f for the correctness check. Method: COLD = create the session, launch the browser, load the page. RESTORE_FULL = criu restore of a checkpoint taken from a COLD session in that ready state, into a fresh network and mount namespace. RESTORE_SERVER_ONLY = restore of a checkpoint without the browser, then a cold browser start (partial arm). Time to agent-ready ends when a screenshot differs from blank in more than 10% of pixels and the page title is present. 20 starts per arm, interleaved, page cache warm (10 starts with the page cache dropped). Memory: slope of host used memory against session count at 1, 4, 8, 12, 16, 20, 3 trials. Private CRIU 4.2 extracted from Ubuntu archive packages. Thresholds fixed in 8f5ed06; no-hygiene results in ca07330; C3 revision 2/2a registered in 080a893 and fc1d88d; reseeded results in ba21252.. Source: branch claude/linux-comparison-prep: docs/data/l1-criu/criu_probe.json (criu, stages.R3_server_only_dump, stages.R4_chromium_dump, stages.R4_chromium_restore, stages.C2_screenshot_shows_browser, stages.C3_sibling_audit, stages.C1_public_task_from_restored_session, latency, memory_summary); branch claude/linux-comparison-prep: docs/data/l1-criu/probe_reseeded.json (latency, memory_summary, stages.C3_sibling_audit.PASS_revision_2a, stages.C1_public_task_from_restored_session); branch claude/linux-comparison-prep: docs/data/l1-criu/criu_probe_dropcache.json (latency, stages); branch claude/linux-comparison-prep: docs/data/l1-criu/clone_hygiene.json; branch claude/linux-comparison-prep: docs/data/l1-criu/clone_task_semantics.json; branch claude/linux-comparison-prep: docs/l1-shared-runtime-design.md sections 14 to 17.
- Drums here is the L0 prototype restored by CRIU; no shared-runtime (L1) result exists; no container, OCI image, VM or incumbent is in any of these files.
- The 2.09x figure is a restore with no hygiene and is labelled preliminary; the usable (reseeded) restore is 1.76x and misses the 2x target.
- The 1.87x drop-cache figure was not pre-registered (10 starts).
- Residual identical state across clones: Python string-hash seed and address-space layout; a checkpoint not declared clean keeps identical cookies and V8 Math.random sequences by design.
- One task, one checkpoint image per arm, --no-sandbox Chromium, one host and kernel; restore on another host untested.
- The CRIU POSIX-lock-after-restore failure recorded elsewhere on the site stays unresolved; this probe's --file-locks flag is a dump-time flag, not a repair.
Record: EXP-005. File: data/results/exp-005-criu-restore.json.
Mechanism K: kernel same-page merging across Drums L0 sessions, K1 (ASLR on) and K2 (ASLR off), with the cross-session timing side channel failed2026-10-24 · dev-sandbox
| Metric | Value | Unit | n | Note |
|---|---|---|---|---|
| host memory per added session, OFF | 229.73 (227.57 to 232.33) | MiB | 3 | |
| K1 (merging, ASLR on): host memory per added session | 212.7 (210.59 to 214.24) | MiB | 3 | 0.926 x OFF; OFF minus K1 95% interval [14.23, 19.84] MiB; saving 7.4%; bar M-K1 (at most 0.80 x OFF, design value): FAIL |
| K1 (merging, ASLR on): ksmd CPU fraction | 0.895 | vCPU | 3 | bar M-K2 at most 0.25 vCPU (design value): FAIL |
| K1 (merging, ASLR on): create latency | 2.01 (p95 2.68) | s | 60 | |
| K2 (merging, ASLR off): host memory per added session | 135.36 (134.67 to 136.41) | MiB | 3 | 0.589 x OFF; OFF minus K2 95% interval [92.21, 96.87] MiB; saving 41.1%; bar M-K1: PASS, but only with ASLR off in every session process (security regression, not shippable) |
| K2 (merging, ASLR off): ksmd CPU fraction | 0.706 | vCPU | 3 | bar M-K2 at most 0.25 vCPU (design value): FAIL |
| K2 (merging, ASLR off): create latency | 1.92 (p95 2.28) | s | 60 | bar M-K4 (K2 p95 at most 1.10 x OFF p95): PASS; the author does not claim the faster K2 creation as a benefit |
| OFF: create latency | 2.41 (p95 3.2) | s | 60 | |
| K2 semantics (M-K3): guest-contract and real-server failures, namespaces distinct, ASLR really off, task e0df059f 0.0 then 1.0 | 1 / 1 | checks | ksm_semantics.json PASS true; stubbed_modules [librosa] recorded for this run; Xvfb first mapping identical in both sessions (555555554000) | |
| side channel probe 1: median first-write latency, merged vs unique page | 345 to 3,854 | ns | 8192 | 11.17x; held-out balanced accuracy 0.99951; capacity 0.994 bit/page; about 203.3 bit/s over a 40.0 s round |
| side channel probe 2: median first-write latency, merged vs unique page | 342 to 2,973 | ns | 8192 | 8.69x; held-out balanced accuracy 0.99890; capacity 0.988 bit/page; about 202.1 bit/s |
| re-run with 2 vCPU / 4 GiB session cgroups: OFF / K1 / K2 memory per added session | 136.57 to 229.68 | MiB | 2 | OFF 229.68, K1 214.68, K2 136.57; ksmd 0.898 / 0.709 vCPU; a readiness check, not a separate result |
Host: 4 vCPU, 16 GiB, cgroup v1, no KVM, Ubuntu 24.04, kernel 6.18 (sandbox). one development sandbox; hostname and provider not recorded; not a VM or cloud host. Workload: Drums L0 sessions (namespaces + overlayfs + Xvfb + openbox) each with the real OSWorld guest server and its own Playwright Chromium 141; OFF = no merging; K1 = every session process marked mergeable (prctl PR_SET_MEMORY_MERGE), ASLR on; K2 = K1 plus personality(ADDR_NO_RANDOMIZE) for every process exec'd in the session. Method: Host KSM knobs set for all arms: run=1, pages_to_scan=20000, sleep_millisecs=2, use_zero_pages=1 (prior 0/100/20/0, restored on close). Memory: slope of host used memory against sessions at 1, 4, 8, 12, 16, 20 after the merge counters settle, 3 trials per arm interleaved (2 trials in the re-run with 2 vCPU / 4 GiB session cgroups). M-K2: ksmd CPU time over trial wall time. M-K4: create latency from 60 creations per arm. M-K5: two cooperating probes in two sessions, 8192 shared-content and 8192 unique-content private pages each, first-write latency classified with a threshold fitted on even pages and scored on odd pages. Thresholds fixed in 8062ff0; results in 1218b2c.. Source: branch claude/linux-comparison-prep: docs/data/l1-ksm/ksm_bench.json (memory_summary, create_summary, knobs_set, knobs_prior); branch claude/linux-comparison-prep: docs/data/l1-ksm/ksm_bench.log; branch claude/linux-comparison-prep: docs/data/l1-ksm/ksm_bench_limits.json (memory_summary); branch claude/linux-comparison-prep: docs/data/l1-ksm/ksm_semantics.json; branch claude/linux-comparison-prep: docs/data/l1-ksm/ksm_sidechannel.json (probes); branch claude/linux-comparison-prep: docs/l1-shared-runtime-design.md sections 12 and 13.
- No K arm is shippable (author's verdict, commit d9de4ad): K1 fails memory and CPU; K2 passes memory only with ASLR off for every process in the session, which is a security regression, and fails CPU.
- The timing side channel is an upper bound from two cooperating probes with known page content: it confirms whether a guessable 4 KiB page exists in another session; it does not read memory and is not an attack on real task data.
- The M-K3 isolation audit ran on K2 only and does not cover the timing channel; it was measured separately.
- Scan knobs were set aggressively and not re-tuned; a slower scan's saving was not measured.
- Drums here is the L0 prototype plus a kernel feature; no shared-runtime (L1) result exists; incumbents' VMs can use KSM too, so nothing here compares against an incumbent.
- One host, one kernel, 3 trials; intervals are over trials.
Record: EXP-020. File: data/results/exp-020-ksm.json.
Branching a real task setup state (LibreOffice Writer launch) and the snapshot-API bridge V1/V2 on real Drums L0 sessions verified2026-10-24 · dev-sandbox
| Metric | Value | Unit | n | Note |
|---|---|---|---|---|
| run 2 (final): SETUP, cold session plus real LibreOffice launch, to agent-ready | 3.12 (p95 3.24) | s | 10 | |
| run 2 (final): BRANCH, restore of the post-setup checkpoint, to agent-ready | 0.868 (p95 0.917) | s | 10 | |
| run 2 (final): SETUP p50 / BRANCH p50 | 3.6 | x | design target was at least 2x; PASS; about 2.25 s saved per rollout after the first of a group | |
| T1 checkpoint of the running LibreOffice: dump time | 0.346 | s | 1 | image 236.48 MiB; window title 'Untitled 1 - LibreOffice Writer' |
| T1b restore of two siblings: restore_total_s | 0.726 to 0.8 | s | 2 | |
| run 2 (final): host memory per added session, COLD with setup | 238.5 | MiB | 3 | 3 trials to 12 sessions, not 20, to stay within host memory |
| run 2 (final): host memory per added session, RESTORE | 238.51 | MiB | 3 | ratio 1.00005; cold minus restore 95% interval [-10.35, +6.70] MiB; bar at most 1.05x: PASS |
| run 2 (final): thresholds passed (T1 feasibility, T2 state, T3 private in-app state, T4 audit 2a, latency, memory) | 6 / 6 | thresholds | ||
| run 1: thresholds passed | 5 / 6 | thresholds | T3 failed for BOTH siblings (own_a false, own_b false): the preserved first-run 'Tip of the Day' modal swallowed keystrokes; a cold control did the same; a harness defect, not a restore failure | |
| run 1: SETUP p50 / BRANCH p50 | 3.6 | x | SETUP p50 3.267 / p95 3.592 s; BRANCH p50 0.908 / p95 1.115 s; memory 235.13 vs 235.46 MiB | |
| V1 reuse=none: valid rollouts | 6 / 6 | rollouts | scores 0.0 before / 1.0 after on every rollout; canary 6 clean; wall 28.062 s | |
| V1 reuse=none: per-rollout reset | 2.17 (0.834 to 2.5) | s | 6 | the first rollout (0.834 s) runs on a fresh environment with no revert; later ones revert to the initial state |
| V1 reuse=post_setup: valid rollouts | 6 / 6 | rollouts | phases setup+save then 5 x revert; all setup_only; wall 20.894 s | |
| V1 reuse=post_setup: first rollout (setup AND save) | 1.86 | s | 1 | |
| V1 reuse=post_setup: per-rollout revert | 0.82 (0.789 to 0.843) | s | 5 | |
| V2 mid-task branch: save_branch | 1.16 | s | 1 | |
| V2 mid-task branch: restore_branch | 0.962 | s | 1 | |
| V2 scores: A correct / B wrong / C correct again | 0 to 1 | score | 1.0 / 0.0 / 1.0; agent_next_draw identical across siblings (task state preserved), hostnames and server draws differ (infrastructure reseeded) | |
| V2 gates passed (i task state, ii reseed, iii divergence, iv isolation, v audit 2a) | 5 / 5 | gates |
Host: 4 vCPU, 16 GiB, cgroup v1, no KVM, Ubuntu 24.04, kernel 6.18 (sandbox). one development sandbox; hostname and provider not recorded; not a VM or cloud host. Workload: Task branching: the setup of public task e1fc0df3 (exactly 'launch libreoffice --writer', LibreOffice 24.2.7.2, gen VCL plugin) run by OSWorld's real SetupController in a Drums L0 session, checkpointed with CRIU 4.2 and restored as two siblings. Bridge: public task e0df059f (rename a directory; shell getter + exact_match) through the real DesktopEnv with reuse none vs post_setup (V1, 6 rollouts each) and a mid-task branch into three siblings (V2).. Method: SETUP = cold friendly session plus the real setup, timed to agent-ready (wmctrl lists a window titled LibreOffice Writer AND the screenshot differs from blank in more than 10% of pixels; the registered a11y-tree criterion was replaced before the measured run because this LibreOffice build exposes nothing to AT-SPI). BRANCH = restore of the post-setup checkpoint, declared not clean, with hostname and server-RNG reseed. 10 starts per arm interleaved; memory slope over 3 trials to 12 sessions. Registration 1e2f3eb, amendment c4488e2, results eb571a9; bridge registration 3526b6a, results b8d1aad. Both arms are Drums; this is harness validation, not a comparison.. Source: branch claude/linux-comparison-prep: docs/data/l1-criu/task_branch_probe.json (stages, latency, memory_summary); branch claude/linux-comparison-prep: docs/data/l1-criu/task_branch_probe_run1_T3_dialog_harness_fail.json (stages.T3, latency, memory_summary); branch claude/linux-comparison-prep: docs/data/l1-criu/diagnosis/README.md; branch claude/linux-comparison-prep: docs/data/l1-criu/matched_reuse/run3_final.json (v1_none, v1_post_setup, v2); branch claude/linux-comparison-prep: docs/data/l1-criu/matched_reuse/findings_before_after.json; branch claude/linux-comparison-prep: docs/l1-shared-runtime-design.md sections 18 to 21.
- Both arms are Drums L0 sessions; this is harness validation, NOT a comparison with any incumbent and not a throughput result.
- The run1/run2 bridge timings (about 24 to 26 s resets, a sudo hostname-resolution stall) are invalid and are not quoted; they are kept on the branch as evidence of how an environment defect can fake a reuse benefit.
- Three of the four diagnosis PNGs are byte-identical (sha256 47fbd307...db98); they must not be presented as three distinct captures.
- The task e1fc0df3 itself was not evaluated (its evaluator needs the internet); only its launch setup was branched.
- Evaluator path for e0df059f is a shell getter plus string match; librosa is stubbed and unreached; the stub list was not saved for these runs.
- Drums here is the L0 prototype restored by CRIU; no shared-runtime (L1) result exists.
Record: EXP-021. File: data/results/exp-021-task-branching.json.
Crash fault windows: SIGKILL at 22 registered points during restore, checkpoint, teardown and sweep, plus supervisor-crash orphan cleanup verified2026-10-24 · dev-sandbox
| Metric | Value | Unit | n | Note |
|---|---|---|---|---|
| dev run (harness defect): windows passing | 0 / 8 | windows | the test counted its own decoy mount as unledgered (O1 false on all 8); P1 to P6 held; a harness defect, not a product result | |
| run 1: windows passing | 6 / 22 | windows | W1 to W3 pass; W4_30 is the first real failure (P1: a forked criu restorer survived the sweep); W4_80 onward fail as a cascade; W8 to W12 carry no criteria (not run) | |
| run 2: windows passing | 0 / 22 | windows | INVALID as measurements: a criu restorer left by run 1 still held a veth, so every restore hit 'Can't restore link: -17 File exists' | |
| run 3 (after the stray-process fix): windows passing | 22 / 22 | windows | run3_final.log: 22 passed in 165.40 s | |
| run 4 (after the survivors change): windows passing | 22 / 22 | windows | ||
| run 5 (after the path-component boundary fix): windows passing | 22 / 22 | windows | ||
| orphan cleanup: run processes alive after supervisor SIGKILL (the regression) | 4 | processes | 4 tagged processes, 3 veth links, 7 mounts survived the kill because CRIU-friendly sessions clear the kernel's kill-on-parent-exit | |
| orphan cleanup: processes reaped by the ownership sweep | 4 | processes | ||
| orphan cleanup: mounts unmounted by the sweep | 7 | mounts | plus 1 nftables table and 3 X11 links removed; afterwards 0 processes, 0 mounts, 0 links, 0 ledger files | |
| orphan cleanup: decoys untouched (foreign-tag process, unrecorded tmpfs) | 2 / 2 | decoys |
Host: 4 vCPU, 16 GiB, cgroup v1, no KVM, Ubuntu 24.04, kernel 6.18 (sandbox). one development sandbox; hostname and provider not recorded; not a VM or cloud host. Workload: A driver process builds a cold CRIU-friendly Drums L0 session, takes a checkpoint, then arms a test-only fault hook (DRUMS_FAULT_AT) for one operation; the Supervisor (and in W4 the top criu process first) SIGKILLs itself at the named point; the parent records a snapshot, runs the ownership sweep and checks criteria P1 to P6 and O1. Method: 22 window instances: W1, W2, W3_30/80/150/250, W4_30/80/150/250, W5 to W16 (restore, checkpoint, destroy, failed-restore teardown, and the sweep itself killed at three points and rerun). A window passes when after the sweep no run process or criu remains (P1), no mount (P2), no veth/nftables/X11 link (P3), ledger gone (P4), decoys untouched (P5), a further sweep is a no-op (P6), and every mount was ledgered before the sweep (O1). Registered in b9f8a67 before any run; run1 d25cf84; fix and runs 3/4 cc3a560; path-component correction and run5 8b3bfd2. Orphan cleanup registered 080a893, measured ba21252.. Source: branch claude/linux-comparison-prep: docs/data/l1-criu/fault_windows/dev_run_harness_defect_decoy_counted_as_unledgered.jsonl; branch claude/linux-comparison-prep: docs/data/l1-criu/fault_windows/run1_raw_with_cascade.jsonl; branch claude/linux-comparison-prep: docs/data/l1-criu/fault_windows/run2_all_invalid_leftover_criu_from_run1.jsonl; branch claude/linux-comparison-prep: docs/data/l1-criu/fault_windows/run3_final_22_pass.jsonl; branch claude/linux-comparison-prep: docs/data/l1-criu/fault_windows/run4_after_survivors_change_22_pass.jsonl; branch claude/linux-comparison-prep: docs/data/l1-criu/fault_windows/run5_after_component_boundary_fix.jsonl; branch claude/linux-comparison-prep: docs/data/l1-criu/fault_windows/run3_final.log; branch claude/linux-comparison-prep: docs/data/l1-criu/orphan_cleanup.json; branch claude/linux-comparison-prep: docs/l1-shared-runtime-design.md sections 16.2, 17.2, 24, 24b, 24c.
- Bounded to these 22 window instances on this host and kernel; not covered: a whole-machine crash, CRIU killed as a whole process group, other signals, a second concurrent Supervisor, sessions of a different run than the checkpoint's, real hosts.
- Residual stated by the author: a task forked in the instant after the sweep's last scan is not seen by that sweep.
- Orphan cleanup is demonstrated for idle restored and cold friendly sessions; cleanup works only for the run that made the checkpoint.
- Drums here is the L0 prototype restored by CRIU; no shared-runtime (L1) result exists; this is software evidence about cleanup, not a benchmark.
Record: EXP-022. File: data/results/exp-022-fault-windows.json.
First-check driver smoke on real Drums L0 sessions (drums_post_setup vs drums_no_reuse) and per-session cgroup limits read back from the kernel preliminary2026-10-24 · dev-sandbox
| Metric | Value | Unit | n | Note |
|---|---|---|---|---|
| original smoke: drums_post_setup valid rollouts | 6 / 6 | rollouts | 2 blocks, 0 faults, 0 interference, canary clean | |
| original smoke: drums_no_reuse valid rollouts | 6 / 6 | rollouts | 2 blocks, 0 faults, 0 interference, canary clean | |
| original smoke: drums_post_setup arm wall time (6 rollouts) | 23.39 | s | includes the first rollout's setup and save; raw ratio to the other arm is labelled NOT_A_CLAIM in the file | |
| original smoke: drums_no_reuse arm wall time (6 rollouts) | 27.81 | s | ||
| original smoke: drums_post_setup first-rollout work (setup and save, both blocks) | 3.65 | s | 2 | per block 1.791 and 1.858 s |
| original smoke: external preparation (pool construction before the driver clock) | 0.45 | s | the pool creates sessions lazily, so this understates preparation | |
| original smoke: drums_post_setup per-rollout revert reset | 0.79 to 0.858 | s | 4 | block means 0.824 and 0.815 s |
| original smoke: drums_no_reuse per-rollout reset after the first | 2.35 to 2.46 | s | 4 | first rollout of each block 0.815 / 0.814 s on a fresh environment |
| limits variant (first per-session attempt): valid rollouts | 12 / 12 | rollouts | INVALID for CPU and memory: v2 files read on a v1 host (cpu_s 0.0, mem_peak 0.0 mean 'not measured'); only pids.max 4096 and procs_max 9 are real | |
| v1 smoke (corrected): valid rollouts, both arms | 12 / 12 | rollouts | ||
| v1 smoke: drums_post_setup arm wall time | 23.99 | s | first_rollout_work 3.709 s | |
| v1 smoke: drums_no_reuse arm wall time | 29.36 | s | ||
| v1 smoke: session CPU quota read back from the kernel | 2 | vCPU | cpu.cfs_quota_us 200000 / cpu.cfs_period_us 100000 | |
| v1 smoke: session memory limit read back from the kernel | 4 | GiB | memory.limit_in_bytes 4294967296; memsw the same (no swap) | |
| v1 smoke: session pids.max read back from the kernel | 4,096 | pids | ||
| v1 smoke: per-session peak memory (kernel), drums_post_setup blocks | 93.82 to 93.84 | MiB | 2 | about 2% of the 4 GiB limit on this trivial task |
| v1 smoke: per-session peak memory (kernel), drums_no_reuse blocks | 94.62 to 94.94 | MiB | 2 | |
| v1 smoke: per-session CPU, sampled lower bound, drums_post_setup blocks | 1.05 to 1.97 | s | 2 | sum of utime+stime of pids seen in the cgroup; cpuacct not in the cgroup on v1 |
| v1 smoke: per-session CPU, sampled lower bound, drums_no_reuse blocks | 2.96 to 4.02 | s | 2 | |
| v1 smoke: processes per session, maximum seen | 9 to 10 | processes | empty_samples 0 in every block |
Host: 4 vCPU, 16 GiB, cgroup v1, no KVM, Ubuntu 24.04, kernel 6.18 (sandbox). one development sandbox; hostname and provider not recorded; not a VM or cloud host. Workload: Public OSWorld task e0df059f three times per block, 2 blocks per arm in ABBA order, concurrency 1, through the real DesktopEnv; drums_no_reuse = a Drums L0 session destroyed and recreated per rollout; drums_post_setup = the same session type with OSWorld's public snapshot API implemented as a CRIU checkpoint after the first setup and restore for later rollouts. Both arms are Drums; no incumbent ran.. Method: compare/drums_compare/first_check.py driver (engineering validation only; computes no verdict). Original smoke 3c87319 with host-wide memory and CPU only; first per-session attempt 4bc9afc read cgroup v2 files on a v1 host (INVALID for CPU and memory); corrected v1 smoke d6a9e8a applies 2 CPU / 4 GiB / 4096 pids per session and reads the limits back from the kernel, sampling at 0.2 s. Prices are a placeholder 3600 USD/hour so cost equals wall seconds; no cost or valid-per-hour figure is published here.. Source: branch claude/linux-comparison-prep: docs/data/l1-criu/first_check_smoke/summary.json (arms.*.rollouts, valid, faults, wall_s, first_rollout_work_s, external_prep_s); branch claude/linux-comparison-prep: docs/data/l1-criu/first_check_smoke/block0_drums_post_setup.json, block1_drums_post_setup.json, block0_drums_no_reuse.json, block1_drums_no_reuse.json (rows[].reset_s, summary.reuse); branch claude/linux-comparison-prep: docs/data/l1-criu/first_check_smoke/pins.json, preflight_probe.json; branch claude/linux-comparison-prep: docs/data/l1-criu/first_check_smoke_limits/README.md, summary.json; branch claude/linux-comparison-prep: docs/data/l1-criu/first_check_smoke_v1/summary.json (arms.*.wall_s, per_environment_blocks[].sessions.*.limits_as_read_by_kernel, mem_peak_mib_kernel, cpu_s_lower_bound, procs_max, empty_samples); branch claude/linux-comparison-prep: docs/incumbent-preflight-plan.md sections 8b and 8c.
- NOT a comparison: both arms are Drums L0 sessions; the incumbent adapter has run against fakes only.
- Prices are a placeholder (3600 USD/hour, 'not a price'); no cost per valid or valid per hour is published.
- Host-wide peak_delta_per_env_mib values (106.3, 67.3, -27.4, 18.2) are host noise and are not reported as per-session figures.
- The limits variant is invalid for CPU and memory; the v1 smoke's CPU is a sampled lower bound.
- resource_parity_claimed is false in every summary; per-session capture exists on the Drums arm only.
- After the two limit-enabled smokes one empty session cgroup directory per run was left behind (open finding, not diagnosed).
- Drums here is the L0 prototype; no shared-runtime (L1) result exists.
Record: EXP-023. File: data/results/exp-023-first-check-smoke.json.
OSWorld task setup survey, Drums rootfs coverage of the 369 public tasks, the manual audit of 13 candidates, and the train/held-out split verified2026-10-24 · dev-sandbox
| Metric | Value | Unit | n | Note |
|---|---|---|---|---|
| public OSWorld tasks surveyed (test_all.json) | 369 | tasks | ||
| tasks with a setup config | 358 | tasks | ||
| tasks whose setup downloads files | 260 | tasks | 458 files, all from huggingface.co | |
| tasks whose setup launches an application | 175 | tasks | google-chrome 79 (each with a socat helper), code 28, thunderbird 25, gimp 23, vlc 20, nautilus 11, libreoffice 10 | |
| tasks whose setup opens a file | 152 | tasks | ||
| tasks with browser tabs, logins or Google Drive in setup | 54 | tasks | ||
| tasks whose setup needs nothing from the network | 73 | tasks | 46 of those launch or open something | |
| explicit setup sleeps, mean per task | 0.084 | s | 369 | no task sleeps 10 s or more; setup cost is not written into the configs as waits |
| smallest-tier candidates by the static classifier | 13 | tasks | os 8, multi_apps 4, libreoffice_impress 1; 3.5% of 369 | |
| tasks blocked by 'setup downloads files' | 260 | tasks | ||
| tasks blocked by 'evaluator getter needs the network / cloud' | 182 | tasks | ||
| tasks blocked by 'launches google-chrome (not on the Drums rootfs)' | 79 | tasks | ||
| tasks blocked by 'GNOME-dependent step or getter' | 28 | tasks | ||
| tasks ever run on Drums | 1 | tasks | e0df059f (rename a directory; shell getter + exact_match) | |
| manual audit of the 13 candidates: usable unchanged | 1 | tasks | e0df059f; docs/task-audit-13.md | |
| manual audit of the 13 candidates: conditional | 1 | tasks | 2cd43775 (Impress autosave): HOME path mismatch /home/user vs the session home; design only | |
| manual audit of the 13 candidates: blocked | 11 | tasks | apt installs, downloads, GNOME gsettings, systemd/timedatectl, PulseAudio, AT-SPI on gnome-terminal, Spotify install | |
| split pool (test_nogdrive.json) | 361 | tasks | 8 Google Drive tasks used on neither side | |
| evaluator-signature groups in the pool | 147 | groups | largest 55, 109 singletons; a whole group goes to one side | |
| training-eligible tasks (seed 0, 40% of groups held out) | 211 | tasks | from docs/training-comparison-plan.md section 13; the split output file is not committed; deterministic from osworld_split.py | |
| held-out tasks (seed 0) | 150 | tasks | domain skew: held-out libreoffice_calc 45 vs train 2; gimp 19 vs 7 |
Host: 4 vCPU, 16 GiB, cgroup v1, no KVM, Ubuntu 24.04, kernel 6.18 (sandbox). one development sandbox; hostname and provider not recorded; not a VM or cloud host. Workload: Static analysis of the public xlang-ai/OSWorld task JSON files at commit 04718ea (369 tasks in test_all.json, 361 in test_nogdrive.json) against the Drums L0 rootfs composition (Ubuntu 24.04, openbox, Playwright Chromium 141, LibreOffice 24.2.7.2); nothing ran in a session for the survey, coverage or split. Method: osworld_setup_survey.json counts setup step types, downloads, launches and sleeps per task. task_coverage.json applies a heuristic classifier (reasons a task is not a smallest-tier candidate: downloads, network getters, apps absent on the Drums rootfs, GNOME steps, proxies, Chrome state, accessibility-tree evaluators, fixed IPs, infeasible-type tasks). The 13 candidates were then read by hand against the task JSON and evaluator source (docs/task-audit-13.md). The split groups the 361 tasks by evaluator-function signature and holds out 40% of groups with seed 0 (osworld_split.py; docs/training-comparison-plan.md section 13; the split output file is not committed).. Source: branch claude/linux-comparison-prep: docs/data/l1-criu/osworld_setup_survey.json (tasks, with_config, with_downloads, download_files, download_hosts, with_launch, with_open, with_browser_or_login, explicit_sleep_s, launch_commands, no_network_needed_by_setup); branch claude/linux-comparison-prep: docs/data/l1-criu/task_coverage/task_coverage.json (tasks, smallest_tier_candidates, candidates_by_domain, reason_counts, validated_on_drums, rootfs_apps_present, rootfs_apps_absent); branch claude/linux-comparison-prep: docs/task-audit-13.md (section 2 table); branch claude/linux-comparison-prep: docs/training-comparison-plan.md section 13 (the split); branch claude/linux-comparison-prep: compare/drums_compare/osworld_split.py.
- Static analysis only; the classifier is heuristic and missed apt and network use inside command steps, downloads in postconfig, GNOME hotkeys and hard-coded /home/user paths, which the manual audit caught.
- 'No known blocker' in the coverage file meant a coarse static filter, never compatibility; the audit supersedes it: 13 candidates become 1 unchanged, 1 conditional, 11 blocked.
- librosa is MagicMock-stubbed in every DesktopEnv import on the branch; the e0df059f evaluator path never reaches it; no run is called unmocked.
- The split is the project's own benchmark, not comparable to published OSWorld headlines; grouping by judge signature reduces contamination but does not prove there is none.
- The split output file is not on the branch; the 211/150 counts are quoted from the plan document and are reproducible from the deterministic script.
Record: EXP-025. File: data/results/exp-025-osworld-coverage.json.