DrumsBench

Experiments / EXP-010

OCI reference arm (runc 1.5.2 + CRIU 4.3), restore with a broken network namespace, its cause, and a holder fix proven on a minimal container

Run the same OSWorld userland, task and limits under a maintained container runtime (runc) driving the same CRIU, so that a checkpoint failure in the custom runtime can be attributed to our code or to the method.

preliminaryA real observation that needs more runs, a positive control, or an independent check before it counts.
Date
2026-10-25
Hypothesis
runc's checkpoint and restore path restores the container's loopback network as it was at dump time.
Arms
oci-runc-criu
Hardware
operator's root Linux host, not identified on the branch (kernel reported as 7.0-aws)
OS
host Ubuntu 24.04; runc 1.5.2 (signature-verified release binary) driving CRIU 4.3 built from source; container userland OSWorld Ubuntu 22.04.3
Workload
GNOME container on the pinned image for the first restore; /bin/sleep containers for the netprobe and holder checks
Code
claude/ecstatic-archimedes-456go6 at a9b57b9
Limitation
Every live result here is author-claimed with no runc or CRIU log, netprobe-result.json or holder.json on the branch; the "successful" 0.225 s restore held no file lock and had no working loopback, and the holder fix has only been shown on a sleep container, not on the GNOME slot.

In one paragraph

This arm is an ordinary container control, not the Drums runtime. It takes the same pinned OSWorld image, task, 4 vCPU / 4 GiB limits and 1920x1080 screen as the custom runtime, but lets runc 1.5.2 build the namespaces, mounts, devices and cgroup from an OCI config.json and drive CRIU 4.3 through its own checkpoint and restore commands. Its job is to answer "is the failure ours or the method's?". On the operator host, runc restore of the GNOME container reported success in 0.225 s (author-claimed, no log on branch), but inside the restored namespace the loopback interface was down with no address, and the restore log said No netns-10.img image. A layered probe, then a reading of runc's source, found the cause: runc always tells CRIU to treat the network namespace as empty, so CRIU neither dumps nor restores it; that is runc's design, not a CRIU or kernel fault. The documented way around it is for the harness to own the namespace through a small holder process, and a minimal sleep container was checkpointed and restored that way with the namespace identical before and after (author-claimed). The GNOME container has not been run on the holder path.

Setup

What ran

All author-claimed, from PROTOCOL.md sections 6 to 9 and commit messages, no artifact on the branch.

  1. Live restore of the GNOME container (operator, ee208df, 16:59 UTC): "runc restore reported "Restore finished successfully. Tasks resumed." in 0.225 s; init pid identity and the cgroup/limits readback passed; but in the restored private netns lo is DOWN with no 127.0.0.1 (the image server listens on 0.0.0.0:5000, a namespace-local curl says "Network unreachable"); the restore log contains No netns-10.img image." At that point the doc said "No causal claim has been made."
  2. Operator read-only host check (section 7): no CRIU config file, CRIU_CONFIG_FILE empty, no external or empty-ns message, yet every network image is absent, and the dump log stops after Dump NET namespace info 10 via 1661 / net: Dumping netns links with no LD: Got link. That ruled out the config-file explanation.
  3. Layered netprobe (091af65, ab51daf): three phases, each removing one layer. kernel: in a fresh unshare --net send CRIU's exact RTM_GETLINK request and decode replies the way CRIU does. criu: a plain child with its own netns and lo up, dumped with plain criu dump --leave-running -v4. oci: the same sleep as a runc container on the read-only image, checkpoint, delete, restore. Result (section 8): "plain CRIU (own netns, lo up) dumps all network image types, logs LD: Got link ..., and runs ip addr save; the minimal runc container (same CRIU, same rootfs pin, /bin/sleep, no GNOME, no overlay) reproduces the full failure: no network images, restore rc 0, lo DOWN, no addresses. Kernel raw request passes."
  4. Minimal external-netns check (0deada5, section 9): "checkpoint 0, restore 0, an external-key netns image present (26 bytes), namespace before/after identical (lo UP, 127.0.0.1/8, ::1/128), the restored container in the holder's namespace, cleanup clean."
  5. The holder was wired into the driver (a9b57b9, 17:49 UTC). The next operator step in the doc, "One GNOME provision -> snapshot -> restore (and reset), then pixels, input and reset verification", is not recorded as done.

Results

No measured result. The author-claimed numbers: restore reported in 0.225 s (step 1), a 26-byte external-key netns image (step 4). Not shown in a table.

The source-grounded cause

From PROTOCOL.md section 8, read at runc v1.5.2 and CRIU commit 2b7ac603: runc's checkpoint.go criuOptions(), used by both runc checkpoint and runc restore, "begins // runc doesn't manage network devices and their configuration. nsmask := unix.CLONE_NEWNET and ends opts.EmptyNs = uint32(nsmask): runc always asks CRIU for empty_ns = CLONE_NEWNET, whether or not --empty-ns is given." On the CRIU side, dump_net_ns then skips the whole links, addresses, routes, rules, iptables and netns-conf branch and only runs dump_nf_ct(), which is the function that logs Dumping netns links; image files are opened lazily so nothing is created. On restore, prepare_net_ns_first/second_stage return early when empty_ns has the NET bit, so the namespace is a fresh one with lo down. "So the "missing" images and the DOWN loopback are runc's documented-by-code design (the caller owns the container's network), not a CRIU, kernel or host fault, and no CRIU/kernel change would alter it. This also explains why the custom runtime (which called CRIU directly) is unaffected." A falsifiable check on the preserved logs is given: the runc dump log should contain exactly two Dumping netns links lines and no LD: Got link after Dump NET namespace info; the plain-CRIU dump log should contain four.

The holder path

runc's own mechanism for a caller-owned namespace: if config.json names a network namespace by path, runc passes --external net[inode]:extRootNetNS at checkpoint and --inherit-fd fd[N]:extRootNetNS at restore, and CRIU records only the key. The harness therefore runs one holder process per slot (holder.py).

sequenceDiagram
participant D as Driver (oci_reference)
participant H as Holder (per slot)
participant R as runc 1.5.2
participant C as CRIU 4.3
D->>H: start at provision, before the bundle is written
H->>H: unshare net, bring lo up, setsid, wait with no expiry
H-->>D: handshake pid, start ticks, netns inode
D->>D: verify identity, write slotN/holder.json, set config.json network path to proc/PID/ns/net
D->>R: runc create and start (container joins the holder's namespace)
D->>R: runc checkpoint
R->>C: external net[inode]:extRootNetNS (CRIU records only the key)
D->>D: re-verify holder, record ip -j link/addr/route
D->>R: runc restore
R->>C: inherit-fd fd[N]:extRootNetNS (same namespace must exist)
D->>D: gate: lo UP, 127.0.0.1/8, readback byte-equal to the checkpointed state
D->>H: cleanup: container proven gone, then holder stopped through a pidfd
The harness-owned network namespace path. The holder outlives the provisioning command and must be the same process (same start ticks, argv, source hash and namespace inode) at every later verb; a lost holder means a fresh provision, never a recreated namespace.

Holder rules from section 9: the holder creates the namespace with unshare (no ip netns add, no host mount) and brings lo up itself; the driver proves the child's identity before writing holder.json and re-verifies it before any nsenter, snapshot, restore or ready check; a missing or changed holder fails closed before any destructive step; a failed restore keeps the holder so the restore can be retried; after a host restart the slot is unusable by design.

Failures and surprises

Limitations

Reproduce

Netprobe, from PROTOCOL.md section 7 (root, new evidence directory, cheaper phases first):

sudo python3 -m oci_reference.netprobe --evidence /var/lib/drumsbench/netprobe-1 --phases kernel,criu --strace
sudo python3 -m oci_reference.netprobe --evidence /var/lib/drumsbench/netprobe-2 --phases oci --runc /opt/runc/runc-1.5.2 --rootfs <read-only mounted image dir>
sudo python3 -m oci_reference.netprobe --evidence <NEW dir> --phases oci --external-netns --runc /opt/runc/runc-1.5.2 --rootfs <read-only image dir>

Reference arm: host/install_runc.sh, then python -m oci_reference VERB [--slot N] with its own state directory (never the custom rootfs-state), provision, snapshot, restore, reset, cleanup. Publishing netprobe-result.json, the CRIU evidence directories and holder.json from a fresh-state GNOME cycle would move this record forward.

Sources