A verifier is the code that decides whether a task got done. A contamination check is the code that decides whether one attempt touched another. Both produce the numbers a training run trusts, and both can be wrong in ways that look like success. This note describes ours, names the public reports of verifier error we rely on, and explains the rule that no zero is printed without a positive control.
Verifiers
A verifier inspects the final state (files, database rows, application state) and returns a result. It must not trust the agent. Two kinds appear on this site:
- OSWorld checkers. Each OSWorld task ships an evaluator: a command run in the guest and a comparison rule. The smoke task used on the Linux arms is one of them: the instruction asks to rename a folder on the desktop, the setup creates the folder, and the evaluator is a shell test for the new name and the absence of the old one compared with
exact_matchagainst an expected string. Our harness runs the task's evaluator command verbatim and applies OSWorld's rule with no stripping. The checker's exit code maps to outcomes: 0 success, 1 task failure (still valid), anything else invalid. - Our verify server. A small local SQLite-backed HTTP service that "never trusts the agent under test". Tasks post to it: a form page with a token field, a page that counts frames only while visible, a lease endpoint and an audit endpoint. At the end of every window the harness asks the audit endpoint for rows whose token names a different environment, and writes them as cross rows. A verifier that cannot be reached makes the rollout invalid (
verifier_unavailable), never a task failure.
Verifier false positives are a public, measured problem
- OSWorld 2.0 reports that 30.6% to 42.6% of runs were tagged "scoring/environment mismatch", and its case studies include runs that scored 1.0 by bypassing the UI or injecting a script (author-reported, arXiv 2606.29537).
- ScaleCUA's generated judges agree with humans 82.5% of the time (author-reported).
- UI-TARS-2 describes its reward model as having a "relatively high false positive rate" (author-reported).
- A public issue on the OSWorld tracker (issue 599) reports checker false positives. We cite it as a public report; we have not independently reproduced it.
- CUA-Gym's rewards check final state only, so a leaking environment could pass a reward it should fail; that filter would not catch it (our reading of the published method).
The consequence for infrastructure: a reward can be wrong because the verifier is wrong, because the environment leaked, or because the agent cheated. Only the second is ours to measure, and it is the one nobody publishes.
Tokens: how cross-delivery is detected
Every rollout gets a token made from its rollout id plus eight random hex characters. Every string the policy types contains the token. So any document, clipboard, form row or log line can be attributed to exactly one environment, and a token found anywhere it should not be is a cross-delivery event with a known source and a known destination. The isolation rows record exactly that: a test name, the trial, the environment it came from, the environment it reached, and a result of clean, cross or lost. A lost row (the check could not read its inputs) is never silent: it fails the window.
The canary
One environment, env00, never receives an action. Before the first window the harness opens an empty text document there. Every two seconds a watcher reads the focused text through Accessibility, the window list and the pasteboard hash, and writes a line. Any change inside a window's span is a contamination event, and because the flag is sticky, every rollout that ends after that moment in that window becomes an environment failure with reason isolation. The parked console can be watched the same way (that recipe is an author claim; it has not run on a machine).
The positive control, and why zero is meaningless without it
A positive control is a run where the protection is deliberately removed so the detector must fire. If it does not fire, the detector is broken, and a zero measured by it is a statement about the detector, not about the system.
In experiment 1 the control is two LibreOffice Writer instances in one login session, driven by the per-process input path that we know shares the pasteboard and the input stream. The typing half types disjoint alphabets into both at once and counts runs of foreign letters; the pasteboard half has each side set the pasteboard within 100 ms of the other and counts reads that return the other side's string. The gate is that both counts are above zero.
A fired control is also narrow. It shows the typing and pasteboard detectors can fire with interference forced inside one session. It does not show that a leak between sessions, or through another channel, would be caught. That sentence is in the experiment's not-claimed list.
The eleven isolation tests (macOS harness)
| # | Test | What it does | When it runs |
|---|---|---|---|
| 1 | positive_control | the gate above; returns detector-ok only if cross-delivery was observed | on demand, before any zero is published |
| 2 | pasteboard | all N environments set a tagged pasteboard string at the same moment, then read back | 50 trials per window, 10 in the soak, and after every writer and clipboard rollout |
| 3 | keyboard_storm | all N environments type ten distinct tokens within five seconds into an open Writer document; each document is saved and read from the host | 10 trials per window |
| 4 | modifier_leak | one environment holds Shift, then Cmd, for five seconds while another types lowercase letters; capitals mean Shift leaked, missing text means Cmd leaked | 3 trials per window, rotating |
| 5 | dialog_steal | one environment opens Save As while another types a token; the dialog field is read through Accessibility against a baseline; bystanders' screenshot latency is timed before and during | 3 trials per window |
| 6 | canaries | env00 and optionally the parked console, watched continuously | the whole time |
| 7 | observation_privacy | for 120 s only odd environments act; even environments' screenshots below the clock line must never change; no screenshot may contain another environment's form-page colour | 1 trial per window |
| 8 | files_and_prefs | one environment writes a token file and a preference; every other environment's work tree and preferences are searched, and every log in the run folder | 3 trials, on demand |
| 9 | reset_during_write | set the pasteboard, type, save, reset 200 ms later; check tree and pasteboard 50 ms after the reset returns | 20 trials, on demand |
| 10 | stale_generation | four old-generation clicks at the same moment as a reset; afterwards one more old-generation click must be refused and the log must show none applied | 100 trials in the soak, 20 at Gate 2 |
| 11 | crash_blast_radius | kill one environment's apps and agent; it must come back at a lower generation within the limit; other environments' screenshots keep being timed | 1 trial, on demand |
Every detector has a unit test with a deliberate leak wired in that makes it fire. That proves the detector reads its inputs. It does not prove those inputs would carry a leak on the real machine; only the positive control does that.
What a published isolation number will look like
Contamination events over environment-hours, as 0 of X, printed next to the positive control's Y of X with Y above zero, with the list of checks that were active and the macOS build. Nothing less, and nothing yet.