macOS sessions-per-Studio harness: design, build on fakes, test suites pass on the Studio

In one paragraph

Over 2026-10-20 and 2026-10-21 we built the harness for Experiment 1, "how many independent computer-use environments does one Mac Studio run with no VMs, at what cost per valid rollout, with zero observed cross-contamination next to a positive control that fired". An environment is one macOS user account logged in to its own GUI session in the background; a small Swift agent, envd, runs inside each session and drumsctl drives every envd from the console session. Everything was built on a laptop against fakes. On 2026-10-22 the repository at head f0d01ce was copied to the target machine, studio-a, and both test suites passed there: 813 Python tests passed with 1 skipped in 479 s, and 105 of 105 Swift tests passed in 1 min 19 s (both measured). That is software evidence that the code runs on macOS 15.7.5 with Command Line Tools only. It is not a benchmark. No environment has been logged in, no isolation test has run against a real session, and no density, throughput, isolation or cost number exists for the Mac.

Setup

What an environment is on macOS

One local macOS user account (envNN, home /Users/envNN) logged in to its own GUI session in the background through fast user switching (MultipleSessionEnabled). No virtual machines. Inside that session one Swift agent, envd, runs as the LaunchAgent run.drums.envd and serves an HTTP API on 127.0.0.1:7700+N. The apps under test (LibreOffice Writer, Google Chrome) run as that user with their profiles inside ~/work.

Why a session and not a process: per-pid event routing inside one session (CGEventPostToPid) shares the pasteboard, focus and dialogs with everything else in that session, so it is not an environment. The design uses it only as a positive control.

Why not VMs: Apple's macOS license and the Virtualization framework cap macOS guests at two per host (vendor claim; see Limitations). The results table is to list "macOS VMs: cap 2 per host, not measured" as a row.

Shared and not shared

The isolation model document says it plainly: "This is not a security boundary. An app in one session is one user account away from another session, not one machine away."

Components, at head f0d01ce

• envd/ Swift, no dependencies: screenshot (ScreenCaptureKit, screencapture -x fallback), click, type, key, scroll, keydown, keyup through CGEvent.post(tap: .cgSessionEventTap), pasteboard get, set and clear, launch, kill, windows, accessibility text, reset, log, health; plus one-shot modes for the probes.
• envd-linux/ Python, the same API over one Xvfb display (xdotool, scrot, xclip, wmctrl).
• drumsctl/ Python host side: client, token, golden, tasks, rollout, verify, isolation, canary, sweep, gates, hosts, events, cli.
• verify-server/ a small sqlite HTTP service: /form, /vis, /vis/count, /lease, /audit, /rows.
• metrics/ the host sampler (vm_stat, memory_pressure, swap, footprint per uid, WindowServer and replayd CPU, powermetrics watts; on Linux MemAvailable, PSS and Xvfb CPU).
• report/ builds table.md, cost.json and RESULTS.md; holds prices.yaml and not_claimed.md.
• probes/ the Day 1 probes A, B, C and P.
• scripts/mac/ host setup; tasks/ four task YAMLs plus key maps; baseline-linux/ the Ubuntu VM arm; docs/ the API, isolation model, trajectory format, task coordinates and license note.
• License: Apache-2.0.

The envd API

Every route runs a local guard first: any Origin header gets 403 forbidden_origin, and Host must be 127.0.0.1:<port> or localhost:<port>, else 403 forbidden_host. A web page loaded in Chrome on the same machine therefore cannot drive another environment over DNS rebinding.

One-shot modes (macOS only, for the probes; no server, no lock): --oneshot-screenshot, --oneshot-type, --oneshot-axtext, --oneshot-hold. The input modes check Accessibility first and exit 2 with ax_not_trusted when the grant is missing, because macOS would otherwise drop the events in silence.

Reset semantics and the generation counter

• Generation counter. One integer per environment. Every mutating call carries the generation it was made for. The check and the effect are one atomic step (an actor in Swift, one lock in Python), so a reset cannot land between them. A refused call does nothing and is not logged.
• 409 stale_gen. A call with an old generation gets 409 stale_gen with the current generation. Isolation test 10 fires resets mid-step and requires that no old-generation action is ever applied.
• APFS clone swap. Reset order: the golden manifest must be readable; release held keys; kill every app and wait up to 5 s; clear the pasteboard; make sure ~/work.spare exists; rename ~/work to ~/work.old.<ms> and ~/work.spare to ~/work; delete the old tree in the background; verify ~/work against ~/golden/manifest.sha256; clone the next spare in the background (cp -c -R on APFS, cp --reflink=always on XFS, plain copy as the fallback); one last look for apps started meanwhile; release held keys again; only then advance the generation. The answer carries gen, ready_ms, manifest_ok and leftover_pids.
• Manifest check. manifest_ok: false means the swapped-in tree differs; drumsctl treats that as a reset error. A reset that cannot kill the apps answers 500 apps_still_running and leaves the generation unchanged.
• Not reset: Keychain, TCC, Launch Services and preferences outside the manifest.

The canary, env00

env00 never receives an action. Before the first window of a sweep or soak, drumsctl opens an empty TextEdit document there. Every 2 s a watcher reads the focused text through /axtext, TextEdit's layer-0 window list and the pasteboard hash, and writes a line to canary.jsonl. Any change inside a window's span counts as one cross-delivery, and because the watcher's flag is sticky every rollout that ends after that moment in that window becomes env_fail with reason isolation. An unreachable canary at sweep start is an error unless --no-canary is given, in which case the table says Canary: off. The console session can be watched the same way (author claim: that recipe has not been run on a machine yet).

The verify server

A local sqlite HTTP service on port 7600 that never trusts the agent under test. The tasks post to it: /form (a coloured page with a Token field and Submit), /vis (a page that counts frames only while visible and posts every 5 s), /vis/count, /lease, /audit. At the end of every sweep window and of the soak drumsctl asks /audit for rows whose token names another environment and writes them as audit cross rows. Recorded deviation: drumsctl takes no /lease in this version, so the generation half of the audit never fires; the token half is what counts.

One rollout

sequenceDiagram
participant C as drumsctl (console)
participant E as envd (env session)
participant A as Writer or Chrome
participant V as verify server
C->>E: POST /gen
C->>E: POST /launch (gen)
E->>A: spawn as the env user
loop each step
  C->>E: POST /action (gen)
  E->>A: CGEvent on the session tap
  C->>E: GET /screenshot, poll until changed
end
A->>V: form post with the token (browser tasks)
C->>V: ask for the verdict
C->>E: POST /reset (gen_next)
E->>A: kill apps, clear pasteboard, swap work tree
E-->>C: gen, ready_ms, manifest_ok

Trajectories

Each rollout writes results/<run>/trajectories/<rollout_id>.jsonl: one header line (env, gen, rollout id, task, token, n, os, poll 200 ms and step timeout 5000 ms, both design values), one step line per step reached (the action as sent; the observation as hash, optional png, stale, black and capture-failed flags; status; latency; step_ms, the time from the action send to the first changed region poll; shot counts), and one footer (outcome valid, task_fail or env_fail; reason; verdict; reward 1.0, 0.0 or null; step p95; reset ready time). The rollout id is D<env>-G<gen>-R<rollout>; the token is the rollout id plus 8 random hex characters, and every typed string holds the token so a verifier can tell whose text it reads. Flat companions sit next to the trajectories: rollouts.jsonl, events.jsonl, isolation.jsonl, canary.jsonl, pb.jsonl, windows.jsonl, host.jsonl, power.jsonl, sweep.json, soak.json and meta.json. Screenshots are not saved by default; only hashes are kept.

A rollout is valid when the pre-check passed (work tree equals the golden manifest, pasteboard empty, no leftover app windows), every step applied under the right generation with no timeout, the verifier returned a verdict, every isolation check active during the rollout stayed clean, and the verdict was ok. task_fail means the verifier said not done; because the policy is a scripted oracle, a task failure counts against the valid fraction. env_fail means the environment was unusable (reward null, never 0), with reasons such as precheck:work_dirty, stale_gen, timeout, black_screenshot, verifier_unavailable or isolation.

The four tasks

All coordinates ship as null, and drumsctl refuses a task with a null coordinate. They are measured once per OS on the target machine; that step has not happened.

The eleven isolation tests

Rows go to isolation.jsonl as clean, cross or lost. A lost row is never silent: the gate isolation_unverified fails any window with one. Trial counts below are design values.

1. positive_control (the gate). Runs the pasteboard round and the keyboard-storm round on two environments that share one session, or on a fake with a leak wired in, and reports whether the detector fired. If it does not fire, no zero may be published. On the Studio only its pasteboard half can fire, because drumsctl cannot start a second Writer in one session; probe P is the control of record for the typing half (see EXP-003).
2. pasteboard. All N environments set a tagged string at the same moment, then read back; 50 trials per window, 10 in the soak, plus after every W and C rollout. Catches a shared or leaking pasteboard.
3. keyboard_storm. All N environments type 10 distinct tokens (about 220 characters) within 5 s into the Writer the sweep opened; each document is saved and read from the host; 10 trials. Catches keystrokes landing in another session.
4. modifier_leak. Environment A holds Shift, then Cmd, 5 s each, while B types 12 lowercase letters; B's document must hold exactly the lowercase text (capitals mean Shift leaked, missing text means Cmd leaked); 3 trials, rotating. Catches system-wide modifier state.
5. dialog_steal. B opens Save As for 5 s while A types a token; the other environments' screenshot p95 before and during is compared; B's Save As field is read through Accessibility against a baseline, and only new text with a piece of the token counts; a bystander more than 20% and more than 50 ms slower is a cross row; 3 trials.
6. canaries. env00 as above, and optionally the console; rows go to canary.jsonl.
7. observation_privacy. For 120 s only odd environments act; even environments' screenshots below the clock, hashed every 2 s, must never change, and no screenshot may hold 8 or more sampled pixels of another environment's form-page colour; 1 trial per window.
8. files_and_prefs. One environment writes a token file into its own ~/work and a preference through its own envd; every other environment's work dir is walked, defaults read runs in every other environment, and every log in the run folder is searched for the token; 3 trials, rotating.
9. reset_during_write. Set the pasteboard, type, save, reset 200 ms later; 50 ms after the reset returns, check the tree against the manifest and the pasteboard; 20 trials; on demand.
10. stale_generation. Per trial, 4 old-generation clicks at the same moment as a reset; after it returns one more old-generation click must get 409; the log must show no old-generation action applied after the reset; a new-generation click must work; 100 trials in the soak, 20 in Gate 2; applied_stale must be 0.
11. crash_blast_radius. Kill A's apps, then A's envd through A's own envd (the operator may not signal another user's process); A's health must fail or come back at a lower generation within 5 s; other environments' screenshots keep being timed; wait up to 60 s for launchd to restart envd; reset A; the pasteboard must be empty; 1 trial; on demand.

Tests 2, 3, 4, 5 and 7 run after each sweep window; the soak runs 10 and then 2; tests 1, 8, 9 and 11 run on demand; test 6 runs the whole time. Every detector has a unit test with a deliberate leak that makes it fire. That proves each detector reads its inputs. It does not prove those inputs would carry a leak on the Studio.

What ran

On the laptop, 2026-10-20 evening to 2026-10-21 22:45: the whole harness, built task by task against fakes (a fake envd, an in-process verify server, made-up sampler data), with a whole-branch review and fix rounds at the end. Head f0d01ce, 2026-10-21 22:45 -0400. Nothing pushed; main untouched.

On studio-a, 2026-10-22, over SSH: the repository copied by git bundle, a uv virtual environment on Python 3.12.14, then make test (both suites) and make build-envd.

The same counts had last been measured on the laptop on the final tree. The static count is lower (723 Python test functions at head; measured by grep) because parametrized tests expand into several cases.

Results

• The harness exists on branch exp1-sessions-per-studio at f0d01ce and its two test suites pass on macOS 15.7.5 (24G624) with Command Line Tools only (measured, 2026-10-22).
• envd builds in release mode on that machine (measured, 36.6 s).
• Nothing else. There is no N_max, no valid-rollouts-per-host-hour, no USD per valid rollout at any price, no contamination count, no clicks-per-boot count, no per-env memory, no step, reset or screenshot p95, no probe A, B or C verdict and no Linux row. The repository's own results block reads "Status: nothing measured yet."

Failures and surprises

• uv picked Python 3.14.7 first on the Studio; the suite was stopped and re-run on 3.12.14, the version the laptop had tested. The first estimate of 45 minutes for the Python suite was wrong; it took about 8 minutes, and only the first test_cli tests are slow.
• Real screen capture through envd could not be validated on the laptop (no Screen Recording grant for the terminal there; the route returned 500 capture_failed). It is validated by synthetic image injection only. Real 1x and 2x capture and the screencapture fallback remain unverified.
• Defects found and fixed before the Studio, all against fakes: a verifier-seen leak used to be scored as a task failure, not contamination; isolation tests 3, 4 and 5 ran with no Writer open; the crash test could report clean without a crash; the shared scratch folder followed symlinks; memory per env counted idle sessions; the did-not-fire side of the positive control had no test; step latency left out the wait for the screen to change; dialog_steal would have flagged the Save As file name as a leak.
• The Studio Day 0 findings (CGSession missing on this build, signing trust needing an on-screen dialog, and the rest) are recorded in EXP-002.

Limitations

• Every test runs against fakes or in process. Isolation tests 1 to 11 are proven to fire only in unit tests with a deliberate leak. No real-session row exists.
• No environment has been logged in on any machine. No envd LaunchAgent has been loaded. No TCC grant exists for envd in any env session.
• The positive control has not fired on any machine on its route of record (EXP-003). Until it does, the word "zero" may not appear next to any isolation count.
• This is not a security boundary. One kernel, one WindowServer, one root.
• macOS license, in plain words: Apple's license caps macOS virtual machines at two per Mac, which is why this experiment uses login sessions and lists macOS VMs as "cap 2 per host, not measured". The same license has a time-sharing clause and a one-user-at-a-time clause written for humans and remote desktops; whether many scripted sessions on a Mac you own fall under them is an open question that only Apple can answer, and legal review is needed before anyone runs this beyond their own measurement on their own Mac. Leasing a Mac to others is restricted by the license to certain developer services in 24-hour minimum blocks, which is why hosted Mac prices are per day and the price inputs are whole-host hourly rates. Hosting this for other people is not supported by this result. Nothing here says any use is inside or outside the license. Linux has none of these limits, which is one reason the comparison arm (EXP-017) exists.

What we will not claim

Copied word for word from report/report/not_claimed.md:

• Not a security boundary. One kernel, one WindowServer, one root. We measure accidental cross-talk over the stated env-hours.
• Not model quality. The policy is scripted; inference cost is excluded.
• Not automatic session creation. Each account took one human login and the recorded TCC clicks.
• Not any app. Writer and the Chromium family only. No Electron, Finder, drag-and-drop, hover, IME, games.
• Not full state reset. Keychain, TCC, Launch Services and prefs outside the manifest are not reset.
• Not long-run stability. Longest window is 60 min; monthly Screen Recording re-prompts and replayd decay are untested.
• Not headless unless the no-monitor rerun was done. Not one macOS version: numbers are for the build in meta.json on one M4 Max 48 GB.
• Not a license opinion. Run on a Mac we own; hosting for others is not supported by this result.
• Not the best Linux. One documented VM config on the same Mac, asymmetry stated.
• Not default Chrome behavior if throttling flags were required; the table says which flags each OS needed.
• Zero means zero observed by the listed checks, next to a positive control that fired.

A further sentence, accepted on 2026-10-22 and to be added to that file: the positive control fires the typing and pasteboard detectors with interference forced inside one session. It shows those two detectors can fire. It does not show that a leak between sessions, or through another channel, would be caught.

Reproduce

Prerequisites at the Studio: Xcode Command Line Tools with Swift 6, Homebrew, brew install uv. Every host-changing script refuses to run without DRUMS_HOST_OK=1. Set RUN once (export RUN=results/<date>-<host>-mac) and keep it for the whole run. Steps 0 and the two suites are what this record covers; the rest is the full table from the README, listed so the reader can see where this record stops.

1. git clone <repo>; uv sync; make test; make build-envd (both suites; envd release build). This record ends here.
2. DRUMS_HOST_OK=1 make setup-host (Homebrew, casks, pmset, MultipleSessionEnabled, /usr/local/drums, the signing identity; make signing-identity redoes that step alone).
3. DRUMS_HOST_OK=1 make create-users (env00 to env32, hidden standard users).
4. DRUMS_HOST_OK=1 make install-launchagent.
5. make golden (the note must be empty).
6. DRUMS_HOST_OK=1 make seed-env-homes.
7. make gate0 (FileVault, user count, CGSession, console session state, env00 home group).
8. make meta, then make baseline-sample (60 s of host samples before any env login).
9. make verify-server in a second terminal with the same RUN.
10. DRUMS_HOST_OK=1 make probes (A, B, C, P in order; stops at the first FAIL).
11. By hand: log in the remaining accounts, grant Accessibility, take the first capture per session, count clicks and minutes per login.
12. make health, make gate1; after the coordinates are measured, make gate2 COORDS=....
13. By hand: measure the six task coordinates in env01's session; pass as COORDS= from here on.
14. DRUMS_HOST_OK=1 make park-console (by hand on builds without CGSession).
15. make health and stat -f %Su /dev/console.
16. make sweep COORDS=....
17. make soak N=<N_max> COORDS=....
18. make positive-control COORDS=..., then make isolation-extra N=<N_max> COORDS=....
19. make report EUR_USD=<rate> STUDIO_USD=<invoice> (make power-sample as root during the sweep and soak for watts).
20. make baseline-linux (prints the Linux steps; run inside the Ubuntu guest with OS=linux PY=python3).
21. make results RUNS="results/<mac run> results/<linux run>" LINUX_HOST=....

Sources

• Repository branch exp1-sessions-per-studio, head f0d01ce (2026-10-21 22:45 -0400): README.md, docs/envd-api.md, docs/isolation-model.md, docs/trajectory-format.md, docs/task-coordinates.md, docs/license-note.md, report/report/not_claimed.md, report/report/prices.yaml, probes/README.md, tasks/*.yaml, Makefile.
• Studio run logs of 2026-10-22 in the operator account's home on studio-a: drums-pytest.log, drums-swift-test.log, drums-build.log (not on the branch).
• Result file: data/results/exp-001-studio-test-suites.json.
• Related records: EXP-002 (Day 0 host setup and 0-env baseline), EXP-003 (positive control attempts), EXP-016 (the planned probes, sweep, soak and cost), EXP-017 (the Linux comparison arm).