In one paragraph
Before claiming anything about "OSWorld on Drums" we needed to know how many of the 369 public tasks could even run on the Drums L0 root filesystem (the sandbox's Ubuntu 24.04 with openbox, Playwright Chromium and LibreOffice 24.2, but no Google Chrome, GIMP, VLC, Thunderbird, VS Code, Nautilus or GNOME). Measured from the files: 358 tasks have a setup config; 260 download files in setup (458 files, all from huggingface.co); 175 launch an application (Chrome 79, VS Code 28, Thunderbird 25, GIMP 23, VLC 20, Nautilus 11, LibreOffice 10); 152 open a file; only 73 need nothing from the network in setup. A static classifier found 13 "smallest-tier" candidates (3.5%: os 8, multi_apps 4, libreoffice_impress 1). A manual audit of those 13 against the task JSON and evaluator source left 1 usable unchanged (e0df059f, the rename task every scored run on the branch used), 1 conditional (2cd43775, an Impress autosave task blocked by a /home/user path), and 11 blocked (apt installs, downloads, GNOME gsettings, systemd timedatectl, PulseAudio, AT-SPI on gnome-terminal, a Spotify install). The audit also records that librosa is MagicMock-stubbed in every DesktopEnv import on the branch, that only one run recorded its stub list, and that the e0df059f evaluator path never reaches it: "No run is called unmocked." Separately, the 361-task test_nogdrive pool is split deterministically by evaluator-function signature (147 groups) into 211 training-eligible and 150 held-out tasks with seed 0, with the author's own warning that this split "is OUR benchmark, not comparable to published headlines". Status verified as static analysis; nothing in this record ran in a session. "Drums" here means the L0 prototype; no shared-runtime (L1) result exists.
Setup
Inputs. The public xlang-ai/OSWorld checkout at 04718ea (Apache-2.0): evaluation_examples/examples/<domain>/<id>.json, desktop_env/evaluators/{metrics,getters}/*; the Drums composition manifest (drums_composition_manifest.json: Ubuntu 24.04 noble, kernel 6.18, LibreOffice 24.2.7.2, xvfb 2:21.1.12, openbox 3.6.1, Chromium 141.0.7390.37, Python 3.12.3, the guest server copied unmodified from 04718ea).
Rootfs applications (task_coverage.json): present chromium, gnome-terminal, libreoffice, socat, soffice; absent code, gimp, google-chrome, nautilus, thunderbird, vlc.
Three analyses.
| Analysis | Tool | Commit (UTC) |
|---|---|---|
| setup survey | envd_linux.l1proto.setup_survey | 1e2f3eb, 2026-10-24 08:42 |
| coverage classifier | envd_linux.l1proto.task_coverage | 6ece94a, 11:27 |
| manual audit of the 13 | desk reading plus one local import | 2612350, 13:14 |
| train/held-out split | compare/drums_compare/osworld_split.py | bd01ec8, 2026-10-23 12:34 |
What ran
Two scripts read every task JSON and wrote osworld_setup_survey.json and task_coverage/task_coverage.json. The audit read the 13 candidates' JSON and evaluator source, ran which for required binaries and one import to confirm librosa is missing from the OSWorld venv; nothing was run in a session, installed, downloaded or logged in. The split script groups test_nogdrive.json by evaluator.func signature and holds out 40% of groups with seed 0; its output file is not committed, so the 211/150 counts are quoted from training-comparison-plan.md section 13 and are reproducible from the deterministic script.
Results
OSWorld task setup survey, Drums rootfs coverage of the 369 public tasks, the manual audit of 13 candidates, and the train/held-out split verified2026-10-24 · dev-sandbox
| Metric | Value | Unit | n | Note |
|---|---|---|---|---|
| public OSWorld tasks surveyed (test_all.json) | 369 | tasks | ||
| tasks with a setup config | 358 | tasks | ||
| tasks whose setup downloads files | 260 | tasks | 458 files, all from huggingface.co | |
| tasks whose setup launches an application | 175 | tasks | google-chrome 79 (each with a socat helper), code 28, thunderbird 25, gimp 23, vlc 20, nautilus 11, libreoffice 10 | |
| tasks whose setup opens a file | 152 | tasks | ||
| tasks with browser tabs, logins or Google Drive in setup | 54 | tasks | ||
| tasks whose setup needs nothing from the network | 73 | tasks | 46 of those launch or open something | |
| explicit setup sleeps, mean per task | 0.084 | s | 369 | no task sleeps 10 s or more; setup cost is not written into the configs as waits |
| smallest-tier candidates by the static classifier | 13 | tasks | os 8, multi_apps 4, libreoffice_impress 1; 3.5% of 369 | |
| tasks blocked by 'setup downloads files' | 260 | tasks | ||
| tasks blocked by 'evaluator getter needs the network / cloud' | 182 | tasks | ||
| tasks blocked by 'launches google-chrome (not on the Drums rootfs)' | 79 | tasks | ||
| tasks blocked by 'GNOME-dependent step or getter' | 28 | tasks | ||
| tasks ever run on Drums | 1 | tasks | e0df059f (rename a directory; shell getter + exact_match) | |
| manual audit of the 13 candidates: usable unchanged | 1 | tasks | e0df059f; docs/task-audit-13.md | |
| manual audit of the 13 candidates: conditional | 1 | tasks | 2cd43775 (Impress autosave): HOME path mismatch /home/user vs the session home; design only | |
| manual audit of the 13 candidates: blocked | 11 | tasks | apt installs, downloads, GNOME gsettings, systemd/timedatectl, PulseAudio, AT-SPI on gnome-terminal, Spotify install | |
| split pool (test_nogdrive.json) | 361 | tasks | 8 Google Drive tasks used on neither side | |
| evaluator-signature groups in the pool | 147 | groups | largest 55, 109 singletons; a whole group goes to one side | |
| training-eligible tasks (seed 0, 40% of groups held out) | 211 | tasks | from docs/training-comparison-plan.md section 13; the split output file is not committed; deterministic from osworld_split.py | |
| held-out tasks (seed 0) | 150 | tasks | domain skew: held-out libreoffice_calc 45 vs train 2; gimp 19 vs 7 |
Host: 4 vCPU, 16 GiB, cgroup v1, no KVM, Ubuntu 24.04, kernel 6.18 (sandbox). one development sandbox; hostname and provider not recorded; not a VM or cloud host. Workload: Static analysis of the public xlang-ai/OSWorld task JSON files at commit 04718ea (369 tasks in test_all.json, 361 in test_nogdrive.json) against the Drums L0 rootfs composition (Ubuntu 24.04, openbox, Playwright Chromium 141, LibreOffice 24.2.7.2); nothing ran in a session for the survey, coverage or split. Method: osworld_setup_survey.json counts setup step types, downloads, launches and sleeps per task. task_coverage.json applies a heuristic classifier (reasons a task is not a smallest-tier candidate: downloads, network getters, apps absent on the Drums rootfs, GNOME steps, proxies, Chrome state, accessibility-tree evaluators, fixed IPs, infeasible-type tasks). The 13 candidates were then read by hand against the task JSON and evaluator source (docs/task-audit-13.md). The split groups the 361 tasks by evaluator-function signature and holds out 40% of groups with seed 0 (osworld_split.py; docs/training-comparison-plan.md section 13; the split output file is not committed).. Source: branch claude/linux-comparison-prep: docs/data/l1-criu/osworld_setup_survey.json (tasks, with_config, with_downloads, download_files, download_hosts, with_launch, with_open, with_browser_or_login, explicit_sleep_s, launch_commands, no_network_needed_by_setup); branch claude/linux-comparison-prep: docs/data/l1-criu/task_coverage/task_coverage.json (tasks, smallest_tier_candidates, candidates_by_domain, reason_counts, validated_on_drums, rootfs_apps_present, rootfs_apps_absent); branch claude/linux-comparison-prep: docs/task-audit-13.md (section 2 table); branch claude/linux-comparison-prep: docs/training-comparison-plan.md section 13 (the split); branch claude/linux-comparison-prep: compare/drums_compare/osworld_split.py.
- Static analysis only; the classifier is heuristic and missed apt and network use inside command steps, downloads in postconfig, GNOME hotkeys and hard-coded /home/user paths, which the manual audit caught.
- 'No known blocker' in the coverage file meant a coarse static filter, never compatibility; the audit supersedes it: 13 candidates become 1 unchanged, 1 conditional, 11 blocked.
- librosa is MagicMock-stubbed in every DesktopEnv import on the branch; the e0df059f evaluator path never reaches it; no run is called unmocked.
- The split is the project's own benchmark, not comparable to published OSWorld headlines; grouping by judge signature reduces contamination but does not prove there is none.
- The split output file is not on the branch; the 211/150 counts are quoted from the plan document and are reproducible from the deterministic script.
Setup survey (measured from the files)
| Quantity | Count |
|---|---|
| tasks | 369 (chrome 46, gimp 26, libreoffice_calc 47, libreoffice_impress 47, libreoffice_writer 23, multi_apps 101, os 24, thunderbird 15, vlc 17, vs_code 23) |
| with a setup config | 358 |
| setup downloads files | 260; 458 files, all from huggingface.co |
| setup launches an application | 175 (google-chrome 79 each with socat, code 28, thunderbird 25, gimp 23, VLC 20, nautilus 11, libreoffice 10) |
| setup opens a file | 152 |
| browser tabs, logins or Google Drive | 54 |
| explicit sleeps | mean 0.084 s, max 5.0 s, none 10 s or more |
| setup needs nothing from the network | 73 (46 of those launch or open something) |
The downloads dominate per domain: libreoffice_calc 47 of 47, libreoffice_writer 23 of 23, thunderbird 15 of 15, impress 45 of 47, multi_apps 85 of 101, gimp 20 of 26; chrome 1 of 46, os 7 of 24.
Coverage (measured, heuristic classifier)
Reasons a task is not a smallest-tier candidate (a task can carry several): setup downloads 260; evaluator getter needs the network or cloud 182; launches google-chrome 79; asks for a proxy 56; browser/login setup step 55; evaluator reads Chrome state 41; GNOME-dependent step or getter 28; launches code 28; infeasible-type evaluator 26; thunderbird 25; gimp 23; vlc 20; evaluator uses the accessibility tree 15; nautilus 11; fixed IP 8. Candidates: 13 (os 8, multi_apps 4, libreoffice_impress 1); by evaluator check_include_exclude 5, check_list 3, exact_match 2, others 1 each. validated_on_drums: ["e0df059f"]. The file's own label: "Candidates only: coverage by static dependency. Only one task has been RUN on Drums. Heuristic classifier: a task can be wrongly kept or dropped; read rows."
The audit of the 13 (desk reading, task-audit-13.md)
| Verdict | Tasks | Deciding facts |
|---|---|---|
| usable unchanged | e0df059f | shell getter plus exact_match; already run on Drums |
| conditional | 2cd43775 | Impress autosave; evaluator reads /home/user/.config/libreoffice/4/user/registrymodifications.xcu while the session home differs; design only (make the session home match, or a labelled variant of the task) |
| blocked | 2373b66a, 3c8f201a, e1fc0df3, f8369178, 13584542, 28cc3b7e, 3ce045a0, 5812b315, 5ced85fc, 94d95f96, b6781586 | apt installs (sysstat, expect), downloads inside command steps or postconfig, an internet extension install, GNOME gsettings, Ctrl+Alt+T under openbox, vm_terminal_output needing AT-SPI on gnome-terminal, PulseAudio pactl, a Spotify install, timedatectl needing systemd and D-Bus |
The classifier missed these because it scans only config commands and getter types. "No known blocker" in task-coverage.md meant a coarse static filter, never compatibility; the audit supersedes it.
The librosa stub (checked)
import desktop_env.desktop_env imports the whole metrics package, whose vlc.py imports librosa, which is not installed in the OSWorld venv. Every driver on the branch loops on ModuleNotFoundError and inserts a MagicMock. Recorded stub lists: K2 (ksm_semantics.json: ["librosa"]) only; the V1/V2 bridge runs, the I3 smokes and the CRIU probes did not save theirs ("historical unknown"; the author infers librosa alone from venv file times, and labels that an inference). The e0df059f scored path (setup execute steps, get_vm_command_line, exact_match) reaches no librosa code. Supported, narrowly: those scores are a string comparison of one shell command's output. Not supported: that the full evaluator stack works.
The split (specified; author-reported counts)
Pool test_nogdrive.json, 361 tasks (8 Google Drive tasks on neither side); 147 evaluator-signature groups (largest 55, 109 singletons); a whole group goes to one side; 40% of groups held out with seed 0: 211 training-eligible, 150 held out. Costs the author lists: the held-out set is not the standard benchmark; worst-case binomial 95% half-width about 8.0 points at n = 150 versus 5.2 at 361; domain skew (held-out libreoffice_calc 45 vs train 2, gimp 19 vs 7); residual leakage through shared source pages (a union over source would collapse 242 of 361 tasks into one group, so it cannot be enforced); a 211-task pool is small; whether the released ScaleCUA SFT checkpoint saw the public tasks is not established.
Failures and surprises
- The static filter over-counted by an order of magnitude: 13 candidates became 1 usable. The failure modes were ordinary (apt and network use inside command strings,
postconfigsteps, hard-coded home paths) and the author chose not to change the frozen classifier, recording the audit instead. - The second-best task is blocked by a home-directory path, which means "same task on both arms" needs either a session home of
/home/useror a labelled custom variant; neither was implemented. huggingface.cowas unreachable from the sandbox and the file-cache dataset's licence is unresolved, so the 260 download tasks cannot be run there and their files were not used.
Limitations
- Static and desk analysis only; the one task ever run on Drums is
e0df059f. - Verdicts marked "reasoned" in the audit (for example the
TypeErrorpath when the autosave file is missing) were read from code, not run. - The split's output file is not on the branch; the counts are from the plan document, deterministic from the committed script. The split has not been audited for near-duplicates the way ScaleCUA's pool was.
- Nothing here says anything about an incumbent or about training; the training comparison itself is EXP-018 (planned).
Reproduce
python -m envd_linux.l1proto.setup_survey --osw-src <OSWorld checkout at 04718ea> --out osworld_setup_survey.json
python -m envd_linux.l1proto.task_coverage --osw-src <checkout> --manifest docs/data/l1-criu/drums_composition_manifest.json --out task_coverage.json
python -m drums_compare.osworld_split --pool <checkout>/evaluation_examples/test_nogdrive.json --seed 0 --heldout 0.4 --out split.json
Expect 369 / 358 / 260 / 175 / 152 from the survey, 13 candidates from the coverage file, and 211 / 150 from the split.
Sources
docs/data/l1-criu/osworld_setup_survey.json(1e2f3eb, 2026-10-24 08:42:56 UTC)docs/data/l1-criu/task_coverage/task_coverage.json(6ece94a, 11:27:44 UTC)docs/task-coverage.md;docs/task-audit-13.md(2612350, 13:14:03 UTC)docs/training-comparison-plan.mdsection 13 (bd01ec8, 2026-10-23 12:34:26 UTC);compare/drums_compare/osworld_split.pydocs/l1-shared-runtime-design.mdsections 18.2 and 25docs/osworld-contract-spike.mdsection 8 (getter and evaluator counts over the 361-task pool)envd-linux/envd_linux/l1proto/setup_survey.py,task_coverage.py- Branch
claude/linux-comparison-prep; result filedata/results/exp-025-osworld-coverage.jsonon this site