DrumsBench

Experiments / EXP-019

Mechanism B: one shared Chromium with a profile per session cut memory 54% and failed isolation: a sibling read another session's secret and ran script in its page

Measure, against thresholds fixed before the run, how much memory and latency one shared browser with per-session profiles saves, and whether any cross-session observation or influence path exists.

failedRan and missed a correctness, isolation, reset or pre-registered performance bar. Kept on the record.
Date
2026-10-24
Hypothesis
The author expected the memory bar to be a coin flip, the latency bar to fail, storage isolation to pass, and the shared debugging endpoint to fail the isolation check.
Arms
l1-shared-chromium
Hardware
4 vCPU, 16 GiB, cgroup v1, no KVM (dev-sandbox)
OS
Ubuntu 24.04, kernel 6.18 (sandbox)
Workload
Drums L0 sessions with the real OSWorld guest server; A0 one Chromium per session on its own Xvfb; B one shared Chromium on a host Xvfb with one profile per session; 20 profiles for the isolation checks
Code
claude/linux-comparison-prep at ee312d3
Evidence
docs/data/l1-branch/bench.json
docs/data/l1-branch/isolation.json
docs/l1-shared-runtime-design.md (sections 10 and 11)
docs/mechanism-b-shared-browser.md
Limitation
The 54% saving is real and is disqualified: with the simplest path a session needs to reach its own profile, it could list, read and drive every sibling's pages through the shared debugging endpoint, and the browser is not in the session's own screenshot.

In one paragraph

This is the clearest record on the site of why isolation is the thing that matters, not the saving. Today every Drums session starts its own browser, about ten processes. Mechanism B starts one Chromium for the whole machine and gives each session its own profile (cookies, storage, cache) and its own window inside it. Measured: host memory per added session fell from 229.76 MiB (A0, own browser) to 104.61 MiB (B), a 125 MiB or 54% saving, with a 175 MiB one-time cost for the shared browser; at 20 sessions the host used 2281 MiB instead of 4596. Browser-ready latency improved too (p50 0.205 s vs 0.320 s). Cookies, storage and cache stayed separate across 20 profiles. And then the isolation check failed: from inside one session, through a plain TCP forwarder to the shared debugging endpoint, the probe listed all 21 windows, read a value that existed only in a sibling's page memory (the test string SECRET-IN-P2), and executed script in the sibling that changed its title to HIJACKED, which the host then saw. One of 60 concurrent attaches also landed in the wrong profile, and a session's own screenshot did not contain its browser at all. The author's verdict, which we adopt: not a path. "Drums" here means the L0 prototype (namespaces, overlayfs on the sandbox's own Ubuntu 24.04 root, Xvfb, openbox) plus this one sharing mechanism; no shared-runtime (L1) result exists; no incumbent was measured.

Setup

Host. One development sandbox: 4 vCPU, 16 GiB, kernel 6.18, cgroup v1, no KVM; hostname and provider not recorded.

Why isolation is the whole question. A computer-use RL environment has to promise two things about session k: nothing another session does is visible to k, and nothing k does reaches another session. The usual unit of that promise is a process tree in its own namespaces. Mechanism B changes the unit to a Chromium profile, which is weaker by construction: the browser main process, the GPU process, the network service (sockets, DNS), the storage service and the remote-debugging endpoint are shared by every session. The experiment was designed to find out whether Chromium's own profile separation is enough.

Arms (identical sessions, real OSWorld guest server, own Xvfb, openbox, D-Bus, AT-SPI, overlay, network namespaces, template profile):

ArmBrowser
A0each session execs its own Chromium, headful on the session's Xvfb (the existing arm)
Bthe sessions run no browser; one shared headful Chromium on a host Xvfb with remote debugging and one --user-data-dir; session k is attached by chrome --user-data-dir=<shared> --profile-directory=Slot<k> <url>, which hands off to the running browser and opens a window of that profile
flowchart TB
subgraph A0["A0: one browser per session"]
  S1["session 1: own Xvfb + own Chromium"]
  S2["session 2: own Xvfb + own Chromium"]
end
subgraph B["B: one shared browser"]
  BR["ONE Chromium: browser, GPU, network, storage, debug endpoint, ONE display"]
  PA["profile window Slot1"]
  PB["profile window Slot2"]
  BR --- PA
  BR --- PB
  T1["session 1: own Xvfb, guest server, no browser"] -. "needs a path to its profile" .-> BR
  T2["session 2: own Xvfb, guest server, no browser"] -. "needs a path to its profile" .-> BR
end
What is shared in arm B. Every dotted path is also a path to every other session's window, because there is one debugging endpoint for the whole browser.

Pre-registered bars (design values, commit d116f92, 2026-10-24 05:19 UTC): M1 B slope at most 0.80 x A0 slope and the interval of (A0 minus B) excludes 0; M2 B p95 below A0 p95 and B p50 at most 0.7 x A0 p50; M3 zero foreign tags across cookies, localStorage, IndexedDB and cache for 20 profiles and zero cross-profile cache hits; M4 20 distinct targets and contexts, 60 of 60 concurrent attaches correct; M5 "any cross-profile observation or influence path counts as a leak: the isolation check FAILS"; M6 and M7 reported.

What ran

envd_linux.l1proto.branch_bench measured memory (4 trials, 1 to 20 attached sessions) and latency (20 per arm, 2 warm-ups, interleaved). envd_linux.l1proto.branch_isolation ran M3 to M7 with 20 profiles attached at once: each profile wrote a unique tag into four stores of the same origin and every profile read back; 60 concurrent attaches were verified through chrome://version; from inside a session's network namespace the probe tried the debugging endpoint directly and then through a plain TCP forwarder on the session's gateway; the session's own screenshot was hashed before and after attach; renderers, the GPU process and the browser main process were killed in turn. Results committed in ee312d3 (06:04 UTC).

Results

Mechanism B: one shared Chromium with one profile per session (A0 vs B), memory, latency and the isolation check failed2026-10-24 · dev-sandbox

MetricValueUnitnNote
host memory per added session, A0 (own Chromium per session)229.76 (227.09 to 231.4)MiB4
host memory per added session, B (shared Chromium, profile per session)104.61 (102.31 to 106.51)MiB4
A0 minus B125.15 (122.89 to 127.31)MiB4min/max are the bootstrap 95% interval over trials
saving B vs A0 per added session54.5%ratio 0.455; pre-registered bar M1 was at most 0.80 (design value): PASS
fixed cost of the shared browser (zero profiles)175.31MiB
total host memory at 20 sessions, A04,595.79MiB4
total host memory at 20 sessions, B (incl. fixed cost)2,281.42MiB4
browser-ready latency, A00.32 (p95 0.38)s20
browser-ready latency, B (attach incl. profile copy)0.205 (p95 0.233)s20profile copy p50 0.0143 s, handoff p50 0.0926 s; bar M2 (B p50 at most 0.7 x A0 p50): PASS at 0.64
M3 foreign observations across 20 profiles x 3 stores0countPASS; 0 cross-profile cache hits, 0 requests with a foreign cookie header
M4 distinct target ids / browser-context ids for 20 profiles20countboth 20 of 20
M4 concurrent attaches landing in the intended profile59 / 60attachesbar was 60 of 60: FAIL by one (attach C2x19); cause not diagnosed
M5b sibling targets visible from a session with a forwarder to the shared endpoint21targetsa sibling's page secret (test string SECRET-IN-P2) was read and script executed in the sibling (title became HIJACKED)
M5 isolation check0 / 1checksVERDICT isolation_check FAIL: shared_debug_channel_leak true; storage_isolation_leak false
M6 session's own screenshot unchanged after its profile attached1 / 1checksthe browser is NOT in the session's display; 24 windows sit in the one shared host Xvfb
M7 renderer kills that lost exactly one profile8 / 8killsGPU-process kill lost none; browser main-process kill took the endpoint down (every profile lost)
shared processes under 20 profiles: browser 1, zygote 2, GPU 1, network service 1, storage service 1, renderers 2228processes

Host: 4 vCPU, 16 GiB, cgroup v1, no KVM, Ubuntu 24.04, kernel 6.18 (sandbox). one development sandbox; hostname and provider not recorded; not a VM or cloud host. Workload: Drums L0 sessions (namespaces + overlayfs + Xvfb + openbox) with the real OSWorld guest server; A0 = each session execs its own Chromium on its own Xvfb; B = one shared headful Chromium on a host Xvfb, session k attached as profile Slot k; 20 profiles for the isolation checks. Method: Memory: slope of host used memory against attached sessions at 1, 4, 8, 12, 16, 20, 4 trials per arm; the shared browser's fixed cost measured separately. Latency: browser-ready, 20 per arm, 2 warm-ups discarded, interleaved. M3 storage isolation across cookies, localStorage, IndexedDB and HTTP cache; M4 target routing with 60 concurrent attaches; M5 shared-channel exposure from inside a session's network namespace; M6 the session's own screenshot before and after attach; M7 blast radius of process kills. Thresholds fixed in d116f92; results in ee312d3.. Source: branch claude/linux-comparison-prep: docs/data/l1-branch/bench.json (memory_summary, latency_summary, template_profile_mib); branch claude/linux-comparison-prep: docs/data/l1-branch/isolation.json (M3_storage_isolation, M4_targets, M4_concurrent_attach, M5a_as_built, M5b_with_forwarder_on_the_gateway, M5_process_inventory, M6_display_and_observation, M7_blast_radius, VERDICT); branch claude/linux-comparison-prep: docs/l1-shared-runtime-design.md sections 10 and 11.

  • Isolation FAILED: a session with any path to the shared debugging endpoint can list, read and drive every sibling's pages; the per-session filtering proxy that would be needed was not built or audited.
  • The memory saving is real but bought with a weaker isolation unit (a profile, not a process) and a different observation path (the browser is not in the session's screenshot).
  • As built the endpoint listened on 127.0.0.1 only; the leak needed a plain TCP forwarder on the session's gateway, which any real design would need anyway so that a session can reach its own profile.
  • Drums here is the L0 prototype plus one sharing mechanism; no shared-runtime (L1) result exists; no incumbent was measured.
  • One host, one kernel, 4 trials; no real OSWorld task ran through the shared browser.
BarMeasuredVerdict
M1 memoryA0 229.76, B 104.61 MiB per added session (per-trial B 105.3 / 106.5 / 104.3 / 102.3); ratio 0.455; A0 minus B 125.15 MiB, interval [122.89, 127.31]; fixed cost 175.31 MiB; totals at 20 sessions 4595.79 vs 2281.42 MiBPASS (the author expected 0.65 to 0.85)
M2 latencyA0 p50 0.320 / p95 0.380 s; B p50 0.205 / p95 0.233 s (profile copy 0.014 s, handoff 0.093 s); B p50 = 0.64 x A0PASS (the author expected a FAIL)
M3 storage isolation0 foreign observations in 60 store checks; 20 cache requests for 20 profiles, 0 cross-profile hits; 0 requests with a foreign cookie header; each profile's second fetch served from its own cachePASS
M4 targets and routing20 distinct target ids, 20 distinct browser-context ids, every profile path matched; concurrent attaches 59 of 60 correct (C2x19 failed verification; cause not diagnosed)FAIL by one
M5 shared-channel exposuresee belowFAIL
M6 display and observationsession_screenshot_unchanged_after_attach: true; 24 windows in the one shared host Xvfbreported: the browser is not in the session's observation
M7 blast radius8 renderer kills each lost exactly 1 profile; GPU-process kill lost none; browser main-process kill: endpoint down, every profile lostreported

M5, the leak, step by step (measured, isolation.json)

  1. As built, nothing leaked, by accident. Chromium bound the debugging port on 127.0.0.1 only (the requested --remote-debugging-address=0.0.0.0 is ignored in headful mode). From session 0: URLError: Connection refused. This is not a design property: any real design must give a session a path to its own profile.
  2. With the simplest such path, a plain TCP forwarder on the session's gateway address, the endpoint was reachable: true and showed targets_visible: 21: 19 About Version tabs, about:blank, and the sibling's page ready-P2.
  3. sibling_secret_read: "SECRET-IN-P2": a value that existed only in the sibling's page memory was read from the other session.
  4. sibling_script_executed: "HIJACKED" and sibling_title_seen_by_host_after: "HIJACKED": script ran inside the sibling's page and changed what the host observed; sibling_was_modified_by_a_session: true.
  5. process_info_entries: 26 of types GPU, browser, NetworkService, renderer, StorageService: process information for the whole browser is readable from a session.
  6. Reading the sibling's cookies through the browser-level Storage.getCookies returned an empty list (not conclusive; the other channels suffice).
  7. No TCP connection was shared between profiles (the test server saw different client ports for P3 and P4).

VERDICT: {storage_isolation_leak: false, shared_debug_channel_leak: true, isolation_check: "FAIL"}. The weakness is the control channel, not the storage. A per-session filtering proxy exposing only that session's targets could narrow it; it was not built, and it would have to be audited too.

Failures and surprises

Limitations

Reproduce

python -m envd_linux.l1proto.branch_bench --out bench.json --latency-n 20 --warmup 2 --trials 4 --max-n 20
python -m envd_linux.l1proto.branch_isolation --out isolation.json --profiles 20

Needs root, overlayfs, Xvfb, the OSWorld guest-server kit and Playwright Chromium. Check VERDICT.isolation_check in isolation.json; any cross-profile observation or influence path is a FAIL under the registered rule.

Sources