Crash fault windows: a SIGKILL at each of 22 registered points during restore, checkpoint, teardown and sweep, and orphan cleanup after a supervisor crash

In one paragraph

CRIU-friendly sessions (EXP-005) clear the kernel's kill-on-parent-exit, so if the supervisor dies nothing kills the sessions but our own sweep. This record is about whether that sweep can be trusted when the death happens in the middle of a restore, a checkpoint, a teardown or the sweep itself. Twenty-two fault-window instances were registered before any run (commit b9f8a67). Run 1 passed W1 to W3 and failed from W4_30 on: CRIU's forked restorer tasks, also named criu and not direct children of the supervisor, survived the sweep untagged and held a half-created veth. Run 2 was entirely invalid because that leftover restorer made every later restore fail with -17 File exists. After the fix (ownership by recorded paths, not by name; a ledger entry is kept while any of its processes could not be reaped) runs 3, 4 and 5 passed 22 of 22 (run 3: 22 passed in 165.40s). Separately, killing a supervisor with one cold and two restored sessions live left 4 processes, 3 veth links and 7 mounts behind; the ownership sweep reaped exactly those and spared a decoy process and an unrecorded mount. This is verified software evidence about cleanup on one host, not a benchmark. "Drums" here means the L0 prototype (namespaces, overlayfs on the sandbox's own Ubuntu 24.04 root, Xvfb, openbox) restored by CRIU 4.2; no shared-runtime (L1) result exists.

Setup

Host. One development sandbox: 4 vCPU, 16 GiB, kernel 6.18, cgroup v1, no KVM; hostname and provider not recorded.

Why this matters for an RL environment. A rollout farm restarts sessions thousands of times. A supervisor that dies mid-restore and leaves a half-restored process tree, a mounted overlay or a veth behind poisons the next run in ways that look like flaky environments rather than crashes. Design doc section 2 and EXP-024 record exactly that happening once: a killed test run left sessions, an nftables table and ip_forward=1, and the next run failed 18 tests with "File exists".

Method (section 24, registered before implementation): a test-only hook DRUMS_FAULT_AT=<window> (a no-op when unset) makes the Supervisor SIGKILL itself at a named point; a timed variant kills at 30, 80, 150 or 250 ms after CRIU starts. A driver process builds a cold friendly session, takes a checkpoint, then arms the fault for one operation. After the death the parent test snapshots the host, runs the ownership sweep (envd_linux.sweep) and checks the criteria. Every window's raw snapshot is kept.

Windows. W1 restore before overlay; W2 restore after overlay, CRIU not started; W3_30/80/150/250 Supervisor killed N ms after CRIU started; W4_30/80/150/250 the same instants with CRIU's top process killed first; W5 after CRIU returned, veth not yet addressed; W6 after register, readiness not checked; W7 mid-reseed; W8 checkpoint after dump; W9 after root unmount; W10 after layer bind; W11 destroy after kill; W12 destroy after root unmount; W13 teardown of a failed restore; W14, W15, W16 the sweep itself killed after kill, after unmount, after links, then rerun.

Criteria, all must hold. P1 no process of the run (matched by argv --tag drums:<run>, never by substring) and no criu left; P2 no mount under any run directory or /tmp/.criu.*; P3 no dvh* link, nftables table or X11 link of the run; P4 ledger file gone; P5 a decoy process with another run's tag and a tmpfs at an unrecorded path untouched; P6 a further sweep is a no-op; O1, checked before the sweep, every mount under the run's directories was in the ledger.

What ran

Six raw files under docs/data/l1-criu/fault_windows/, in order: a development run (8 windows), run 1 (22, commit d25cf84, 10:33 UTC), run 2 (22, invalid), run 3 and run 4 (22 each, commit cc3a560, 10:49 UTC), run 5 after the path-component correction (22, commit 8b3bfd2, 11:05 UTC). Orphan cleanup: tests/test_criu_orphan_cleanup.py (registered in 080a893, measured in ba21252).

Results

Per window, run 1 versus runs 3 to 5 (measured)

The run 1 row for W4_30 reads op: restore, point: restore.during_criu_both:30, PASS: false. Run 2's 22 rows all show P1 and P3 false (or no criteria): invalid as measurements, kept as evidence of the cascade. Run 3's log ends 22 passed in 165.40s (0:02:45).

Orphan cleanup after a supervisor SIGKILL (measured, orphan_cleanup.json)

The author expected about 75% that this passes first time; it passed on the first complete run. An unplanned second demonstration: a probe run killed with SIGKILL mid-measurement was also reaped by the sweep.

Failures and surprises

• W4, the first real failure. Killing the supervisor's direct criu child and then the supervisor interrupted the restore but left CRIU's forked restorer tasks running: they are also named criu, are not direct children, carried no --tag (a session's command line is restored last) and held the half-created veth. So CRIU and the supervisor were not "killed together" as a process group. At 150 and 250 ms the restore had usually already completed (3 tagged processes alive), so those windows reproduce W3; the genuinely interrupted instants are 30 and 80 ms.
• The fix. Ownership by recorded paths, never by name alone: a criu process whose command line names the dead run's state directory or one of its recorded mounts, and any process in a foreign mount namespace holding a mount under the run's state directory, is reaped with the tagged ones; and a ledger entry is no longer removed while any of its processes could not be reaped (they are reported as survivors), because an entry removed over a living stray leaves it unowned for good, which is what made run 2 invalid.
• A second correction after review (24c). The stray matcher first matched owned paths by substring, so a run at /tmp/run1 would have matched a different run's /tmp/run10. Corrected to path components; discovery repeats until a pass finds nothing new (at most five passes); tests cover a prefix-colliding criu process surviving. Run 5 reran all 22 windows after it.
• Harness lessons kept. The development run counted its own decoy mount as unledgered (O1 false on all 8 windows while P1 to P6 held). A leftover from one window silently invalidated the next 22; the test now refuses to run a window on an unclean host and records it as INVALID.
• An earlier harness finding (section 17.2): a killed-but-unreaped supervisor still looks alive to the sweep (the test waits for it), and a restored session carries the checkpointing run's tag, so cleanup works only for the run that made the checkpoint.

Limitations

• Not covered, in the author's words: "a crash of the whole machine, a CRIU killed as a whole process group, other signals, a second concurrent Supervisor, sessions of a different run than the checkpoint's, real hosts."
• Residual, stated: a task forked in the instant after the sweep's last scan is not seen by that sweep; the entry is kept only while a victim already found is still alive.
• Orphan cleanup is demonstrated for idle restored and cold friendly sessions and now for these 22 mid-operation windows; a crash during the first restore of a run with no checkpoint yet was not a registered window.
• The sweep deletes only resources carrying an ownership record; a leak made by a process that never wrote a ledger entry is not swept, by design.
• One host, one kernel, cgroup v1; the ledger does not record cgroup directories (see EXP-023's open finding).
• This is software evidence about cleanup, not a benchmark result, and says nothing about an incumbent.

Reproduce

DRUMS_CRIU_HOME=<private criu dir> pytest envd-linux/tests/test_criu_fault_windows.py -q   # writes one JSONL row per window
pytest envd-linux/tests/test_criu_orphan_cleanup.py -q
python -m envd_linux.sweep --ledger <dir>        # the standalone sweep

Needs root, overlayfs, Xvfb, nft and CRIU 4.2; the windows must not run concurrently (veth names and the nftables table are host-global). Check every row's criteria and PASS.

Sources

• docs/data/l1-criu/fault_windows/dev_run_harness_defect_decoy_counted_as_unledgered.jsonl
• docs/data/l1-criu/fault_windows/run1_raw_with_cascade.jsonl (d25cf84, 2026-10-24 10:33:52 UTC)
• docs/data/l1-criu/fault_windows/run2_all_invalid_leftover_criu_from_run1.jsonl
• docs/data/l1-criu/fault_windows/run3_final_22_pass.jsonl, run3_final.log, run4_after_survivors_change_22_pass.jsonl (cc3a560, 10:49:54 UTC)
• docs/data/l1-criu/fault_windows/run5_after_component_boundary_fix.jsonl (8b3bfd2, 11:05:08 UTC)
• docs/data/l1-criu/orphan_cleanup.json (ba21252, 08:41:03 UTC)
• docs/l1-shared-runtime-design.md sections 16.2, 17.2, 24 (registration, b9f8a67, 10:24:36 UTC), 24b, 24c
• envd-linux/envd_linux/sweep.py; envd-linux/tests/test_criu_fault_windows.py, test_criu_orphan_cleanup.py, test_sweep.py
• Branch claude/linux-comparison-prep; result file data/results/exp-022-fault-windows.json on this site