Custom rootfs runtime (the Drums candidate arm on Linux), CRIU dump and restore attempts of the GNOME tree

In one paragraph

This is the arm that is supposed to become Drums on Linux: not a VM and not an OCI container, but a process tree on the host kernel built from an overlay of the OSWorld image, running the image's own GNOME desktop and guest server, checkpointed by calling CRIU directly. Over about six hours on 2026-10-25 the author took it through a series of real dumps and restores on an operator host. Each attempt failed at a new gate, each failure was fixed in source, and the last one reached a restore that CRIU itself called successful but that our lock gate rejected because a POSIX lock on a mapped file was missing (EXP-009). Every number and error line in this record is author-claimed, no log on branch. The runtime has never completed a passing snapshot and restore cycle. It is not yet a working Drums runtime, and nothing here compares it with anything.

Setup

What the slot runtime is

Mechanism, from the branch README and CANDIDATE.md:

• Image: qemu-nbd read-only attach of the pinned Ubuntu.qcow2 (sha256 6bf667a8...313), mounted read-only as the lower layer.
• Per slot: an overlayfs (lower = the image, upper = per-slot disk state) in private pid, mount, net, ipc and uts namespaces, under cgroup v2 limits equal to the baseline guest (design values: CPUQuota=400% for 4 vCPU, MemoryMax=4096M for 4 GiB, memory.swap.max 0). Before a slot is declared ready the arm reads the kernel's view back (cgroup path, cpu.max, memory.max, memory.swap.max, init in cgroup.procs, descendants in the same cgroup) and sweeps the slot if anything differs.
• Tree init in two stages. Stage A (host Python, pid 1 of the slot) calls setsid() first, builds the overlay and the private mounts (a tmpfs /dev with native mknod nodes, a private devpts, proc, sysfs), then pivot_root(".", ".") into the overlay and detaches the old root, so the namespace root is the overlay and no host mount remains. It then execs the image's own python3 with a stdlib-only supervisor.py as -c text, so no host interpreter or library is mapped inside the tree.
• Image-side supervisor: starts the image's own dbus-daemon --system, the image's real /usr/lib/xorg/Xorg with a private dummy config, the GNOME session (gnome_x11 profile, with session variables derived only from the image's own files) and the OSWorld guest server. A dead dbus, display, GNOME shell or guest server is a failed slot, never a tolerated one.
• Display driver staged, not installed: the real image has Xorg and GNOME but no Xvfb and no dummy driver, so one pinned jammy package (xserver-xorg-video-dummy 1:0.3.8-2build1, 9578 bytes, sha256 re-verified) is unpacked into the slot's upper layer, only dummy_drv.so, after checking the xorg-video-abi-25 ABI against the image's own dpkg status. The image is never written to. This is a declared fidelity difference from the VM's virtual GPU.
• Network: the slot's own namespace, created by the tree with lo up, is part of the CRIU image; a netns-aware proxy exposes the slot's guest server on a host port.
• Snapshot: CRIU dump (tree stopped) plus a copy of the upper layer plus an immediate restore. Reset: stop, put the saved upper back, CRIU restore. criu restore runs inside a fresh systemd-run --scope with the same CPU, memory and swap properties, and with --root pointing at a fresh overlay mount made inside unshare --mount --propagation private, so the host mount namespace is never changed.
• Gates around it (candidate.py): ownership markers on every verb, a source hash of the runtime files that must match at restore, a mount contract read from the dumped init's own mountinfo (no external mounts, dump refused otherwise), network images (netns/netdev/ifaddr) must exist after the dump, cgroup readback plus mount-table and network-state equality after the restore and before the proxy, and the file-lock contract described in EXP-009.
• Sameness gate (sameness.py): the arms may be compared only if image sha256, task, evaluator command, guest-server file hash, guest vCPUs and memory and workload hash are all known and equal; unknown blocks.

Declared differences from the OCI control

Shared with the control, so not a differentiator: the same pinned image, overlayfs, the same supervisor and services, the same cgroup limits, the same CRIU 4.3, the same acceptance gates. CANDIDATE.md says what is left to compare if both arms ever pass: "overhead (dump/restore time, image size, idle cost without a holder or runc), network fidelity and the dependency footprint. Nothing about that is measured yet, and neither arm is a Drums result until the full workload, CUA and resource accounting exist."

What ran

A sequence of real attempts on the operator host, 14:55 to 18:49 UTC, reconstructed from the README, CANDIDATE.md and commit messages. All author-claimed; no artifact on the branch for any of them.

1. First real tree dump failed: mnt:403:./dev/tty doesn't have a proper root mount. CRIU's validate_mounts rejects a bind of one host device file. Fix (dc6c015): a plain private tmpfs /dev with native nodes.
2. Next dump failed: criu/mount.c: The root task has another root than mntns: .../merged. A chroot inside the host's mount namespace is not the namespace root. Fix (b8605b8): pivot_root and the image-side supervisor.
3. Next dump failed: criu/cr-dump.c: A session leader of 1510 (1) is outside of its pid namespace. The launcher's setsid ran before unshare. Fix (114726b): stage A calls setsid() first and fails closed unless pid == sid == pgid.
4. Readiness failure inside the chroot: LookupError: unknown encoding: idna from a lazy host import. Fix (c0f4fe5): numeric readiness probe, preloaded modules. Also found: the Xorg log directory was unwritable after Xorg dropped privileges, and the slot user had lost its sudo group because supplementary groups were not taken from the image's /etc/group (both fixed in 97b8727, dc6c015).
5. First successful dump, then a restore crash (CANDIDATE.md section 1, run on 114726b): dump succeeded in 137 s producing 453 MiB of images (author-claimed, no log on branch); the immediate criu restore died in 0.118 s (author-claimed) after listing host mounts (securityfs, cgroup2, pstore, efivarfs, bpf, debugfs, tracefs), then mount.c:48 BUG, then SIGSEGV. The restore argv had no --root. Source reading: without --root, CRIU's mount-v2 engine tries to unmount the mount table it inherited from CRIU's own (host) namespace and hits BUG_ON(!mi->plain_mountpoint). Fix (846cc7c): the --root restore contract in a private mount namespace.
6. Candidate arm run (846cc7c): fresh GNOME provision, mount and network readback and the task checker passed before the dump (author-claimed; this is the only place on the branch where a GNOME slot of this runtime is said to have passed the task checker). The dump then failed after 123 s (author-claimed) with Error (criu/file-lock.c:112): Some file locks are hold by dumping tasks! You can try --file-locks to dump them. No restore was reached. CRIU's refusal is a safety check, not a bug. Fix (a84dde4): a pre-dump lock census that adds --file-locks to both dump and restore only when every lock is an advisory lock on a file of the slot's own overlay or tmpfs and no outside process holds or waits on it.
7. --file-locks run (a84dde4 on 0e16bc5): the census found two Nautilus POSIX READ locks (.../nautilus/tags/meta.db range 1073741826 to 1073742335 and meta.db-shm byte 128), no outside holder. The dump passed in 135 s (author-claimed). The explicit --root restore reported "Restore finished successfully. Tasks resumed." and the mount.c:48 crash did not recur. The strict lock gate then failed: expected 2 READ locks, post-resume census [].
8. Pre-resume gate run (fd39055): the dump log recorded both locks, the restore log reported applying both ((posix) fd 8 start 0x40000002 len 0x1fe and fd 10 start 0x80 len 1), filelocks.img unchanged, yet CRIU's own pre-resume hook saw only the meta.db lock. The gate was right to fail. This is the open failure recorded in EXP-009.

Results

No measured result. The author-claimed figures, each with no log on the branch: dump 137 s and 453 MiB (step 5), restore crash at 0.118 s (step 5), dump refused after 123 s (step 6), --file-locks dump 135 s (step 7). We do not show these in a result table and nothing should be computed from them.

What the sequence establishes, if the author's text is taken at face value: the tree can be provisioned to a GNOME desktop that passes the OSWorld task checker before any snapshot; CRIU 4.3 accepts the whole tree for dump once /dev, pivot_root, session ownership and --file-locks are right; a --root restore completes from CRIU's point of view. What it does not establish: a restored slot that holds every lock it held at dump time, passes the mount and network equality gates, serves the task again and resets. No restore has passed the arm's gates.

Failures and surprises

• Six distinct real failures in one afternoon, each at the next gate: device bind, chroot root, session leader, lazy import, missing --root, file locks, and then the lock itself. The branch's own prediction held: "Whether CRIU now accepts the whole tree is unproven until the next real dump: the overlay root, devpts, proc/sysfs mounts and open files are the next gates."
• The dump that was refused for file locks is the same mechanism that makes the OCI control look clean: CANDIDATE.md notes that runc forwards --file-locks with default false, so "the OCI control passing means no lock was held there, not that locks are ignored".
• Several docstrings on the branch still say "NOTHING HERE HAS BEEN RUN AGAINST THE REAL IMAGE" while later docs in the same package describe these live runs. We report the later, specific statements as author-claimed.

Limitations

• No artifact on the branch for any live run: no dump.log, restore.log, images-manifest.json, locks-census.json or criu-evidence-* directory. The host is not identified.
• The runtime's docstrings and launch.py say its CRIU flags "were NOT executed against a real CRIU for this tree"; the live text in CANDIDATE.md postdates them. Both are on the branch.
• Likely blockers listed by the branch as inferred and unproven: nbd on cloud kernels, gnome-shell under software GL without a seat, whether the non-root slot user can start the image's Xorg with the dummy config, systemd-run needing a usable cgroup v2 hierarchy.
• No cost, no matched comparison, no CUA agent has run on this arm.

Reproduce

On a root Ubuntu 24.04 host with the image, from benchmark/ (commands from the README; nothing in the sandbox can run them):

sudo ./sharedos_rootfs/host/install_criu.sh --repo "$PWD/.." --evidence /var/lib/drumsbench/criu-install
sudo modprobe nbd max_part=16
sudo python3 -m sharedos_rootfs.image attach --image /data/Ubuntu.qcow2 --expected-sha256 6bf667a8...313 \
    --mountpoint /mnt/osworld --state /var/lib/drumsbench/image-mount.json
python3 -m sharedos_rootfs.rootfs_inspect /mnt/osworld --out /var/lib/drumsbench/inspection.json --mount-state /var/lib/drumsbench/image-mount.json
export SHAREDOS_ROOTFS_CONFIG=/var/lib/drumsbench/rootfs-config.json
sudo -E python3 -m sharedos_rootfs.candidate environment
sudo -E python3 -m sharedos_rootfs.candidate provision --slot 0
sudo -E python3 -m sharedos_rootfs.candidate snapshot --slot 0      # dump + immediate restore, with every gate
sudo -E python3 -m sharedos_rootfs.candidate reset --slot 0
sudo -E python3 -m sharedos_rootfs.candidate cleanup

CANDIDATE.md section 6 asks for a new state directory, the slotN/ evidence plus restore logs to be collected, and "Any failure after the root fix is reported with its log line, not worked around with flags."

Sources

• benchmark/README.md, sections "Rootfs slot runtime", "OCI reference arm (maintained runc) and the paused custom restore", "Drums candidate arm"
• benchmark/sharedos_rootfs/CANDIDATE.md (sections 1, 2, 3, 3b, 4, 5, 6)
• benchmark/sharedos_rootfs/ (tree.py, supervisor.py, launch.py, criu_root.py, candidate.py, locks.py, sameness.py, display_stage.py, host/install_criu.sh)
• Commits 15c1a47, 08f8a16, 58fe8ae, 52f66b3, 97b8727, c0f4fe5, dc6c015, b8605b8, 114726b, 846cc7c, a84dde4, fd39055 (2026-10-25)