None of these is settled. Where a source gives a measurement that would settle one, it is named.
About the field
- How did CUA-Sandbox actually run OSWorld (shared desktop session vs VM per environment), and how were display, input and clipboard split per desktop session? Will the OSWorld adapter land, and under what licence?
- How does MAI-Gym's 20,000 peak split across phone, desktop and browser; what hypervisor and VM size; what did "concurrent attempts interfere" mean in practice?
- What is a "sandbox" in EvoCUA's 100,000 figure: a booted VM or a scheduled slot?
- How does OSGym reset a VM, how long does it take, what price regime is behind USD 0.23, and how are "valid trajectories" counted?
- What CPU hosts and RAM ran ScaleCUA's 600 QEMU VMs; was the snapshot path used in the main runs; how many failures were VM flake vs judge error?
- What fraction of total training dollars and critical-path time is environment vs inference on any real workload? This is the single most important missing number.
- How much of OSWorld 2.0's 30.6% to 42.6% "scoring/environment mismatch" is verifier error vs agent error? Can a non-VM backend plug into its state and event APIs; do its website images run on ARM64?
- Will CUA-Lite, MAI-Gym and OSWorld-web get licences; will OSGym's code appear; will Trace2World's code appear?
About macOS
- The hard cap on concurrent GUI sessions per Mac; per-session WindowServer and GPU cost; whether background sessions get GPU compositing. Settled by logging in accounts until the first refusal.
- Whether a posted key from one session can ever reach another session; modifier state across sessions. Settled by the keyboard-storm and modifier-leak tests at N of four or more, next to a fired positive control.
- Privacy grants: one click per Mac or per user; prompt cadence on macOS 26 and 27; whether a monthly re-prompt blacks out one environment mid-run.
- Whether Chromium, Electron and LibreOffice keep painting in a switched-out session, and which flags are required. Settled by probe C: at least 55 distinct frames of 60.
- Screen capture behaviour at 16 to 32 captures per second over hours; silent stream stalls.
- Reliable teardown of a live session; programmatic login without a human; headless operation without a monitor.
- Real-profile clone and reset times for Chrome and LibreOffice (only a synthetic 20,000-file tree was timed); whether a sandboxed app's container can be swapped while quit; daemons holding stale values after a swap.
- The Linux guest cap under Apple's virtualisation framework; per-virtual-display memory; ARM64 Chromium vs Chrome breakage on OSWorld tasks.
- The licence questions in desktop isolation on macOS. For counsel, not engineering.
About the shared-runtime design generally
- Quiescence for native apps: CUA-Sandbox counts HTTP requests and background jobs because its apps are web servers; a GUI app has no request boundary to hook. What is the readiness barrier for a native app?
- The per-app state contract: how much hand-written adapter work each supported app needs (CUA-Sandbox's gate refuses any task touching an unreviewed resource), and the engineering cost per newly supported application.
- Cross-contamination detection as a first-class metric: which checks, what positive controls, how to attribute a leak to input, pasteboard, files, preferences or observation.
- Whether desktop state branching inside a training loop (branch-served rollouts sharing a prefix) changes GRPO economics at all, and at what fidelity.
- Sub-second GUI-ready desktop reset: achievable on any OS? The published desktop bar is 37.2 s p50 (MAI-Gym) and 5.84 s prepare (CUA-Sandbox), both author-reported.
- Whether the Chromium-dominated per-environment memory (about 600 MB on macOS, measured on a laptop; about 875 MiB browser plus agent on Linux per CUA-Sandbox, author-reported) can be shared with per-environment profiles. That would be the next win after runtime sharing.
- The macOS analogue of Linux primitives: APFS clonefile for XFS reflink; nothing for D-Bus; no OverlayFS; no KSM.
- How many CUA-Gym desktop tasks run on macOS apps without edits, and the re-verification cost per ported task.
- On Linux: whether a lock-preserving restore of POSIX locks on mapped files is possible with CRIU 4.3's ordering, or needs the patch candidate or an upstream change (EXP-009). Settled by the prescribed order: stock control on the operator's host, then patched, then one GNOME gate.
The eight falsification questions
Re-answered each week. The two columns are the 2026-10-15 and 2026-10-23 answers.
| # | Question | 2026-10-15 answer | 2026-10-23 answer |
|---|---|---|---|
| 1 | Why might sharing not matter? | Low environment share of spend, cheap existing density, inference and stragglers dominating | Stronger. MAI-Gym reached an author-reported 20,000 concurrent environments without sharing. CUA-Sandbox's desktop gains were modest (3.2x memory, 3.08x throughput at 8-way) next to its web gains (up to 51x and 504x, author-reported), suggesting desktop workloads have proportionally less shareable runtime. Our own L1 prototype measured only a 6.2% memory saving from sharing the server process. The prize may be small on the desktop workloads that matter |
| 2 | What solves it already? | Containers, tuned VMs, snapshot pools, page sharing and reclamation, workload-specific simulators | Much more, now verified by full reads. CUA-Lite (VM-free desktop, 0.6 GB per environment author-reported, code released, no licence), CUA-Sandbox (shared runtime plus branch, published, web code MIT), ScaleCUA (full RL stack, trainer MIT), OSGym-class KVM plus CoW at an operating point, MAI-Gym's isolation engineering. The "nothing exists" version of the Drums story is dead |
| 3 | Why might labs not buy? | Internal stacks, correctness and integration cost, insufficient dollars saved; no willingness-to-pay evidence | Unchanged and reinforced. Every lab with scale in the reviewed sources built in-house. Training-side infrastructure is now sold by GPU-hour by general hosts. No willingness-to-pay evidence exists in any reviewed source |
| 4 | Does simulation remove the need? | Credible for tool, web and synthetic GUI subsets; not proved sufficient for all native workflows | Pressure increased. CUA-Gym-Hub (94 session-scoped mock apps), MAI-Gym clone-to-real transfer (28.8% to 48.1%), Trace2World (94.0% sim-to-real retention), all author-reported, none reproduced, all pointing the same way. Credible for web-shaped and synthesisable tasks; unproven for stateful desktop-native work, which is where Drums must win |
| 5 | Does inference overwhelm environment cost? | Plausible, not quantified for an actual customer | Still the single most important missing number. ScaleCUA's 64 GPUs against 600 CPU environments is the best ratio evidence available, and the paper gives no dollar split. Unresolved |
| 6 | Are KVM, Firecracker and Apple-native stacks good enough? | Real substitutes must be measured; Apple-specific data missing | Reframed by CUA-Lite. The Linux "VM tax" is smaller than assumed (0.6 GB container vs 3.8 GB VM, author-reported), so the question is no longer VM vs no-VM but whether even the VM-free baseline leaves economic room for shared runtimes. Untested. macOS evidence remains thin |
| 7 | Is the macOS wedge tiny? | Unknown: no measured count of macOS RL buyers, budget or rollout volume | Still unknown. One vendor sells macOS desktops per seat; the macOS coverage in the one open agent harness is four tasks. The only macOS RL training evidence found is two papers from one group. Not a current thesis input either way |
| 8 | Does isolation kill sharing? | Plausible for unscoped state; CUA-Sandbox explicitly retreats to private backends | Sharpest version yet. MAI-Gym states sharing is impossible and engineered isolation; CUA-Sandbox states sharing works with state capsules and published web evidence; our own hazard tests show five concrete breakages when forking live GUI processes. Honest read: sharing works only within carefully scoped state boundaries, and nobody has published desktop RL-scale evidence either way. That gap is the bet, and it may fail |
None of these unknowns validates Drums.