Experiments
Every run, in order, including the ones that failed.
One record per experiment: what we set out to show, what hardware and code it ran on, the measurements, and the honest limitation.
- verified
- Backed by measurement files we can point to, on named hardware, with the run recorded.
- preliminary
- A real observation that needs more runs, a positive control, or an independent check before it counts.
- failed
- Ran and missed a correctness, isolation, reset or pre-registered performance bar. Kept on the record.
- planned
- Written and reviewed as code or protocol, but not executed. No numbers.
Each record has a fixed identifier (EXP-001 and up) that never changes or gets reused, a start date, a status, the arm it is about, the hardware and OS it ran on, the git branch and commit, and the evidence paths behind it. Records are listed in date order; the ID order follows the date the work started, not the date it was written up. A record whose status is failed is written with the same sections and the same care as one that passed: what we expected, what happened, what we think it means, what we do not know. A record whose only evidence is text without a log says so in its body and is never above preliminary.
To cite a record, give its ID, its commit and the site path, for example "Drums EXP-004, commit dce0211, bench.drums.run/experiments/EXP-004". Result tables on a record are generated from files under data/results/, and each file names the raw logs it summarises.
- EXP-001macOS sessions-per-Studio harness: design, build on fakes, test suites pass on the StudioverifiedBuild the Experiment 1 harness (envd, drumsctl, verify server, sampler, probes, report) and show that its two test suites pass on the target Mac Studio build.2026-10-20 to 2026-10-22Mac Studio M4 Max, 16 cores, 48 GB (studio-a)macos-sessions
- EXP-002Studio Day 0: host setup passes 1 and 2, and the 0-env host baselinepreliminaryPrepare studio-a for the Experiment 1 probes (tools, settings, envd, env accounts, golden tree) and take the 60-second host baseline with no environment session logged in.2026-10-22Mac Studio M4 Max, 16 cores, 48 GB (studio-a)macos-sessions
- EXP-003Positive control probe P on the Studio: six attempts, failed to runfailedMake the typing and pasteboard detectors fire on studio-a with two Writer instances forced into one session on the tap route (Gate P requires typing_cross above 0 and pb_cross above 0).2026-10-23Mac Studio M4 Max, 16 cores, 48 GB (studio-a)macos-sessions
- EXP-004Mechanism F: forking the OSWorld guest server from a warm zygote saved 6.2% of memory against a pre-registered 15% barfailedDecide, against thresholds fixed before measurement, whether forking the Python guest server from a pre-warmed zygote cuts host memory per session by at least 15% and server-ready median latency by at least 30%, and record what breaks when a live GUI process is forked.2026-10-234 vCPU, 16 GiB, cgroup v1, no KVM (dev-sandbox)l1-fork-prototype
- EXP-024L0 spikes: a real Chromium task through envd-linux, the real OSWorld guest server inside a session, one public task scored 0.0 then 1.0, and the systemd-as-PID-1 incidentpreliminaryRecord what the Linux L0 session prototype demonstrably ran on 2026-10-23 (single samples, test-backed), what it did not, and the full-suite runs that followed as software evidence.2026-10-23 to 2026-10-244 vCPU, 16 GiB, cgroup v1, no KVM (dev-sandbox)l1-criu-container
- EXP-005Mechanism C: CRIU checkpoint and restore of a warm Drums L0 session with Chromium is 2.09x faster than cold without hygiene, 1.76x with the required reseed, below the 2x barfailedDecide, against thresholds fixed before measurement, whether a warm Drums L0 session with its browser can be checkpointed with CRIU and restored as isolated siblings at least 2x faster than a cold start, without costing more than 1.05x the memory, with a public task still scoring correctly.2026-10-244 vCPU, 16 GiB, cgroup v1, no KVM (dev-sandbox)l1-criu-container
- EXP-019Mechanism B: one shared Chromium with a profile per session cut memory 54% and failed isolation: a sibling read another session's secret and ran script in its pagefailedMeasure, against thresholds fixed before the run, how much memory and latency one shared browser with per-session profiles saves, and whether any cross-session observation or influence path exists.2026-10-244 vCPU, 16 GiB, cgroup v1, no KVM (dev-sandbox)l1-shared-chromium
- EXP-020Mechanism K: kernel page merging saves 7% with ASLR on and 41% only with ASLR off, costs 0.7 to 0.9 vCPU against a 0.25 bar, and opens a cross-session timing channelfailedMeasure, against thresholds fixed before the run, whether kernel same-page merging (KSM) across Drums L0 sessions cuts per-session memory at an acceptable CPU cost, with and without address-space randomisation, and whether it introduces a cross-session side channel.2026-10-244 vCPU, 16 GiB, cgroup v1, no KVM (dev-sandbox)l1-ksm
- EXP-021Branching a real task setup state: a LibreOffice Writer launch restores 3.6x faster than cold plus setup, and the snapshot-API bridge preserves and branches task state on real sessionsverifiedValidate, against thresholds fixed before the run, that the post-setup state of one real public task setup can be checkpointed and restored as private siblings faster than redoing the setup, and that OSWorld's public snapshot API driven through Drums preserves and branches task state under the real evaluator path.2026-10-244 vCPU, 16 GiB, cgroup v1, no KVM (dev-sandbox)l1-criu-container
- EXP-022Crash fault windows: a SIGKILL at each of 22 registered points during restore, checkpoint, teardown and sweep, and orphan cleanup after a supervisor crashverifiedEstablish, for 22 registered fault windows on this host, that ownership-based cleanup leaves no process, mount, link, table or ledger of a run behind after the supervisor (and in some windows CRIU or the sweep itself) is killed mid-operation, while sparing resources that are not the run's.2026-10-244 vCPU, 16 GiB, cgroup v1, no KVM (dev-sandbox)l1-criu-container
- EXP-023First-check driver smoke: Drums post_setup versus Drums no_reuse, 12 of 12 valid, and per-session cgroup limits read back from the kernel (not a comparison)preliminaryShow that the alternating-block comparison driver runs end to end on real Drums L0 sessions with canary, accounting and telemetry, and that per-session cgroup limits in the incumbent's shape (2 CPU, 4 GiB) are applied and read back by the kernel.2026-10-244 vCPU, 16 GiB, cgroup v1, no KVM (dev-sandbox)l1-criu-container
- EXP-025OSWorld task survey and coverage: 13 of 369 public tasks pass a static filter for the Drums rootfs, a manual audit leaves 1 usable and 1 conditional, and a 211/150 train/held-out split by evaluator signatureverifiedEstablish from the public task files which OSWorld tasks could run on the Drums L0 root filesystem without downloads, absent applications or GNOME, how many survive a manual audit, and how the 361-task pool is split for any later training comparison.2026-10-244 vCPU, 16 GiB, cgroup v1, no KVM (dev-sandbox)l1-criu-container
- EXP-026Drums full-root sessions on the OSWorld image root with GNOME Shell 42.9: seven host attempts, two simultaneous GNOME sessions reported working, a CRIU 4.3 dump that failed on configfs, a black pre-dump capturefailedBoot a Drums session whose root filesystem is the incumbent's own OSWorld image (Ubuntu 22.04.3, GNOME Shell 42.9, uid 1000) on a real Ubuntu 24.04 host, prove a usable desktop, and dump and restore it with CRIU.2026-10-24 to 2026-10-25real Ubuntu 24.04 host (owner-operated; instance type and hardware not stated in the branch; cgroup v2)drums-rootfs-criu
- EXP-027Planned: matched incumbent runs on AWS, stock VirtualBox live snapshots on bare metal and a Docker/KVM cold-reset small stage, with the equivalence evidence required before anything is called a comparisonplannedRun the stock OSWorld incumbents (VirtualBox with live snapshots on a bare-metal host; the Docker provider's QEMU/KVM cold reset on a small instance) with the same task, agent and reuse flag as the Drums arm, under fail-closed controls, and produce the first result that could be labelled a comparison only once software equivalence is shown.2026-10-24planned: one bare-metal instance (96 vCPU class) for VirtualBox; one m7i.xlarge (4 vCPU, 16 GiB) with nested virtualization for Docker/KVMqemu-kvm-warm-snapshotaws-host
- EXP-006Shared-OS spike in an authoring sandbox, namespaced Xvfb tree launched and swept, CRIU absentpreliminaryCheck whether a private-namespace Xvfb plus xterm plus probe-app tree can be launched, probed through the process table and the X wire, and swept clean, as the first gate of a shared-OS checkpoint diagnostic.2026-10-25unknown container (authoring sandbox), no cgroup limitssharedos-spike
- EXP-007QEMU/KVM warm-snapshot baseline harness (drumsbench) and the OSWorld rename scripted smoke, first real run failed 0/0failedBuild a black-box harness that runs one frozen desktop workload against a warm QEMU/KVM snapshot baseline and an external arm under the same validity gates, and smoke-test its plumbing on one pinned OSWorld task.2026-10-25unidentified host for the author-claimed run; no QEMU or KVM in the authoring sandboxqemu-kvm-warm-snapshot
- EXP-008Custom rootfs runtime (the Drums candidate arm on Linux), CRIU dump and restore attempts of the GNOME treefailedRun the pinned OSWorld image's own userland (dbus, Xorg, GNOME, guest server) as a shared-kernel slot and checkpoint and restore it with CRIU 4.3 under the same validity gates as the VM baseline.2026-10-25operator's root Linux host, not identified on the branch (kernel reported as 7.0-aws in PROTOCOL.md)drums-rootfs-criu
- EXP-009CRIU POSIX lock lost on a mapped file after restore, probe and patch candidate, unresolvedpreliminaryIsolate why a restored GNOME slot holds only one of its two Nautilus POSIX read locks at pre-resume, with a minimal probe that needs no desktop, and find out whether CRIU 4.3 can restore such a lock at all.2026-10-25authoring sandbox for the recorded probe runs (kernel reported as 6.18.44); the operator host has not run the probe per the branchdrums-rootfs-criu
- EXP-010OCI reference arm (runc 1.5.2 + CRIU 4.3), restore with a broken network namespace, its cause, and a holder fix proven on a minimal containerpreliminaryRun the same OSWorld userland, task and limits under a maintained container runtime (runc) driving the same CRIU, so that a checkpoint failure in the custom runtime can be attributed to our code or to the method.2026-10-25operator's root Linux host, not identified on the branch (kernel reported as 7.0-aws)oci-runc-criu
- EXP-011First paid AWS host launch terminated by its own heartbeat backstopfailedLaunch one bounded, self-cleaning EC2 host under the aws_run gates to run the baseline smoke, with an AWS-side terminate backstop that cannot leave a host running unattended.2026-10-25c8i.2xlarge (8 vCPU / 16 GiB, nested virtualization enabled), us-east-1aws-host
- EXP-012Bedrock Claude Opus 5.5 computer-use adapter, early real-endpoint calls failed, the reported five-call success has no artifactpreliminaryDrive a slot's desktop through Amazon Bedrock's computer-use protocol with Claude Opus 5.5 under a frozen, hashed configuration and a closed GUI-only action allowlist, recording every call and never running the checker inside the adapter.2026-10-25operator's host for the slots; Bedrock endpoint in us-east-1bedrock-opus-cua
- EXP-013Real-agent rollouts on matched arms through Bedrock (Experiment 2 of the plan)plannedMeasure, with a frontier agent driving real desktop tasks, how many valid trajectories each environment arm yields per unit of time and money, and how often each arm corrupts a trajectory, under identical workload, model and frozen config.2026-10-25bedrock-opus-cuaqemu-kvm-warm-snapshotoci-runc-criu
- EXP-014Self-hosted UI-TARS-1.5-7B serving ladder for an open-weight agentplannedServe the pinned UI-TARS-1.5-7B model unquantized on one GPU host, behind a keyless loopback endpoint shared by every arm, at the largest context that provably serves the frozen agent configuration without shrinking screenshots.2026-10-251x NVIDIA L4, 22888 MiB usable (operator-reported); a g6.2xlarge-class host is named as operator-sideuitars-serving
- EXP-015Matched baseline vs candidate comparison with drumsbenchplannedCompare the QEMU/KVM warm-snapshot baseline with the custom rootfs candidate on the same workload, seeds, checker and declared guest resources, from two measured and reconciled runs, with a pre-registered statistical design.2026-10-25qemu-kvm-warm-snapshotdrums-rootfs-criu
- EXP-016macOS probes A, B and C, gates, sweep, soak and cost per valid rollout (the Experiment 1 headline)plannedFind N_max, the largest number of background macOS sessions on one Mac Studio at which every pre-registered gate holds over two 15-minute windows, then price a valid rollout at three host prices.2026-10-25Mac Studio M4 Max, 16 cores, 48 GB (studio-a)macos-sessions
- EXP-017Linux VM with N Xvfb displays on the same Mac (the comparison arm)plannedRun the same four tasks, eleven isolation tests, sampler and report inside one Ubuntu VM on the same Mac Studio, with one Xvfb display per environment, so the macOS row has a Linux row next to it with the asymmetries stated.2026-10-25Mac Studio M4 Max, 16 cores, 48 GB (studio-a); guest: Ubuntu 24.04 arm64 under UTM, 14 vCPUs, 40 GBlinux-vm-xvfb-on-mac
- EXP-018Experiment 3, RL training A/B with only the environment infrastructure changedplannedTrain the same open-weight computer-use model with the same algorithm and rewards on two environment arms that differ only in infrastructure, and compare the time and cost to reach a pre-registered success rate on held-out tasks.2026-10-25training-ab