DrumsBench

Experiments / EXP-035

E2B hosted service: fork(4) 5.15 s and clean reset 2.00 s with state kept, replicated; valid rollouts per dollar 28 to 156 against Drums' 1,612 from another study, unpaired and not replicated

Measure E2B's hosted service as an incumbent, tuned per E2B's own docs and paid from credits only, on fork, clean reset and valid rollouts per dollar.

preliminaryA real observation that needs more runs, a positive control, or an independent check before it counts.
Date
2026-11-10
Arms
e2b-hosted
Hardware
drivers m7i.large and m7i.2xlarge (us-east-1); E2B desktop template xfce 8 vCPU / 8 GiB; osworld-e2b template 4 vCPU / 8 GiB
OS
E2B guests (desktop template; osworld-e2b GNOME Shell on Xvfb)
Workload
fork(4) of a desktop with Chrome and Calc open; clean reset from a full snapshot; 55-task OSWorld suite with the T1 scripted policy at N = 32, one rung
Code
sota/e2b at 119d5e6 (private research archive)
Evidence
runbooks/sota/e2b/20261110T0300Z/PREREGISTRATION.md and FREEZE.txt (frozen 2026-11-10T03:03:18Z, sha256 3d074747...)
runbooks/sota/e2b/20261110T0300Z/REPORT.md, DEVIATIONS.md, RUNLOG.md, REPRODUCE.md, SHA256SUMS
runbooks/sota/e2b/20261110T0300Z/raw/m5/out/m5main/m5-analysis.json and m5-sensitivity.json
Limitation
The per-dollar figures are unpaired and cross-study (E2B at 32 from a remote driver against Drums at 128 with a local client), the E2B rung ran once and was not sustained by the T1 rule, and 156 per dollar is a sensitivity written after the outcome.

In one paragraph

Arm label: E2B hosted service. On fork and reset E2B is solid and replicated: fork(4) of a running desktop with Chrome and Calc open took 5.15 s p50 with 160 of 160 copies keeping their state, and a clean reset took 2.00 s p50, both within 10% on a fresh driver; against Drums' same-host T2 numbers (2.35 s and 0.56 s) the verdicts are "E2B slower", labelled unpaired. On valid rollouts per dollar the result is weaker evidence. On the pre-registered basis E2B gave 28.2 per dollar (only 40% of rollouts valid, almost all because one GNOME daemon, gvfsd-metadata, exits on its own after a reset and the frozen alive gate counts that as invalid). The most favourable sensitivity, written after the outcome (first 1,200 s only and that daemon forgiven), gives 156.1. Drums' 1,612 to 1,616 comes from a different study (EXP-032) at 128 desktops with a local client. So the range is "about 10x to 57x below Drums depending on the basis", and it is unpaired and not replicated.

Setup

Fork and reset (M1 to M4): E2B's desktop template via e2b 2.63.0 and e2b-desktop 2.7.2, driven from an m7i.large in us-east-1; E2B's backend region is not disclosed. Cost (M5): the T1 harness unchanged except three labelled adapter changes to its e2b-cloud arm (D3), the third-party e2b-dev/osworld-e2b template at d762ce2 built in our team, N = 32 because of the 100-sandbox limit and the budget.

What ran

Pre-registered phases M1 to M6 on 2026-11-10 between 03:03 and 04:24Z, with fork, reset and step replicated on a fresh driver; the M5 cost rung once.

Results

E2B hosted service: fork(4) and clean reset of a running desktop (replicated, unpaired against Drums' same-host T2 numbers) verified2026-11-10 · driver + E2B cloud

MetricValueUnitnNote
E2B fork(4) until all 4 usable5.15s4095% CI [4.90, 5.44]; p99 8.12; replication 4.67 s (-9%); about 2.2x Drums' 2.35 s (unpaired)
E2B fork copies with state intact160 / 160copiesreplication 120/120
E2B clean reset2s4095% CI [1.77, 2.12]; p99 10.03; replication 1.91 s (-4%); 40/40 clean; Drums 0.56 s (T2, unpaired)

Host: driver m7i.large (us-east-1); E2B desktop template, xfce 8 vCPU / 8 GiB, E2B desktop template. vendor numbers hold for the named template builds on 2026-11-10 03:03 to 04:24Z. Workload: fork(4) of a desktop with Chrome and LibreOffice Calc open and typed state, until all 4 copies answer a screenshot; clean reset = new sandbox from a clean full snapshot. Method: Pre-registered (frozen 2026-11-10T03:03:18Z). N = 40 main, N = 30 on a fresh driver; tolerance +-25%. Verdict rule (unpaired): ratio CI excludes 1.. Source: research archive: runbooks/sota/e2b/20261110T0300Z/REPORT.md (rows 2, 3).

  • Unpaired: different hardware, network position (Drums' client was local) and guests; vendor list price vs AWS on-demand.
  • The E2B SDK caps fork at 20 copies per call (client-side).

E2B hosted service: valid rollouts per USD on the 55-task suite at N = 32 (one rung, unpaired against Drums from another study) preliminary2026-11-10 · driver + E2B cloud

MetricValueUnitnNote
E2B valid rollouts per USD, pre-registered basis28.2per USD579234 of 579 valid (40.4%); CI over tasks 14.8 to 49.7
E2B valid per USD, S1 + S2 (most favourable to E2B, post hoc)156.1per USD547545 of 547 valid; labelled sensitivity, never the headline
Drums-fixed reference (T1-fixed, N = 128, local client; another study)1,611.7 to 1,616.1per USDhosts B and A; see EXP-032
Ratio E2B / Drums, unpaired, by basis0.017 to 0.097xpre-registered basis 0.017 to most favourable 0.097 (156 vs 1,612); about 10x to 57x below Drums
E2B M5 alive gate: rollouts invalid333 / 579rolloutsthe only missing process was gvfsd-metadata, which also exits when idle; counted invalid as the frozen rule says

Host: driver m7i.2xlarge (us-east-1); E2B osworld-e2b template sandboxes 4 vCPU / 8 GiB, E2B guest from the third-party osworld-e2b template. E2B backend region not disclosed; paid from E2B credits. Workload: 55-task OSWorld suite, T1 scripted policy, N = 32 (E2B's 100-sandbox limit and the budget), one rung. Method: Pre-registered (frozen 2026-11-10T03:03:18Z, sha256 3d074747...). Pre-registered basis: the rung-wall window (the T1 rule). Sensitivities S1 (first 1,200 s only) and S2 (gvfsd-metadata-only alive failures counted valid) were written after the outcome.. Source: research archive: runbooks/sota/e2b/20261110T0300Z/REPORT.md; research archive: runbooks/sota/e2b/20261110T0300Z/raw/m5/out/m5main/m5-analysis.json; research archive: runbooks/sota/e2b/20261110T0300Z/raw/m5/out/m5main/m5-sensitivity.json.

  • Unpaired and cross-study: different hosts, pricing, guest image, network position and N (E2B 32, Drums 128). The Drums side was not measured in this study.
  • Not replicated: the M5 rung ran once. The E2B rung was not sustained by the T1 rule (useful share 0.40 against 0.95).
  • List prices on both sides (E2B list price; AWS on-demand for Drums). Create time is not billed in our ledger, which favours E2B (about 2% of sandbox-seconds).

Failures and surprises

Limitations

Unpaired: different hardware, network position, guests and N; vendor list price against AWS on-demand. Create time is not billed in our ledger, which favours E2B. A young third-party template and two harness-gate interactions. Vendor numbers hold for the named template builds on that date.

Reproduce

REPRODUCE.md in the study folder; code on sota/e2b (119d5e6 frozen driver, c60455a M5 adapter and analysis, bc91d68 sensitivity), T1 harness sota/t1 d2d0acb.

Sources

Pre-registration, report, deviations and M5 analysis files under Evidence. Research archive, internal for now.