DrumsBench

Experiments / EXP-032

Density: Drums sustained 512 desktops on three separate hosts, tuned KVM 128 on the same machine type, tuned Firecracker 16 to 32; never 512 on both hosts of one pre-registered study

Measure the largest number of concurrent desktops each stack sustains on one r8i.16xlarge under the T1 sustained-step rule, and Drums' valid rollouts per dollar against tuned KVM with the pipe-limit fix.

preliminaryA real observation that needs more runs, a positive control, or an independent check before it counts.
Date
2026-11-09 to 2026-11-12
Arms
drums-api, qemu-tuned-shared, firecracker
Hardware
r8i.16xlarge (64 vCPU, 512 GiB, nested virtualization on), on-demand; five hosts across three jobs
OS
Ubuntu 24.04, kernel 7.0.0-1014-aws
Workload
55 OSWorld tasks, scripted mock policy (no model), ladder doubling from 8 to 512 desktops
Code
sota/t1-pipefix at a8d118b (private research archive)
Evidence
runbooks/sota/t1/20261109T2203Z-fixed/PREREGISTRATION.md and FREEZE.txt (frozen 2026-11-09T22:04:28Z, sha256 1958a81c...)
runbooks/sota/t1/20261109T2203Z-fixed/REPORT.md and summary/tables.md
runbooks/sota/integrate/20261111T1725Z/REPORT.md and raw/t1/summary/tables.md
runbooks/sota/round2/firecracker/20261112T0226Z/PLAN.md, FREEZE.txt (frozen 2026-11-12T04:18:13Z, sha256 b7b15c96...), REPORT.md, summary-A/, summary-B/, raw/SHA256SUMS
Limitation
No density claim passes the frozen rule (same direction on both hosts of one study): 512 on T1-fixed host A but 128 on host B, and no Drums result on the Firecracker study's host A.

In one paragraph

With one five-line installer fix (fs.pipe-user-pages-soft = 0), the Drums release sustained 512 desktops on T1-fixed host A (1,201 rollouts, 100% valid, 144 GiB of memory left), on the fresh host of the v0.2.0 integration re-run (1,151 of 1,152 valid) and on host B of the Firecracker study (1,171 of 1,172 valid). On the same machine type tuned KVM sustained 128 on every host it ran on (at 256 only 139 to 143 VMs fit in memory), and tuned Firecracker sustained 16 on one host and 32 on the other. Rungs double, so 512 is the last sustained doubling, not a ceiling. The same pre-registered study gave Drums 1.38x tuned KVM's valid rollouts per dollar on host A and 1.34x on fresh host B, both CIs excluding 1. What does not hold yet: T1-fixed host B stopped at 128 (one quarantined environment produced 10,535 instant failures, counted against Drums), and on the Firecracker study's pre-registered host A one create failed at rung 8 and Drums has no result. So 512 is measured on three hosts, and no single study shows it on both of its hosts.

Setup

All hosts r8i.16xlarge on-demand (spot had no capacity), Ubuntu 24.04, kernel 7.0.0-1014-aws, gp3 600 GiB at 16k IOPS. T1-fixed: drumsd sota/t1-pipefix a8d118b (rc1 bd43483 plus a 5-line installer change), drums-lab cb3145d, drums-criu b4fbfddf, hardened profile on, 4 vCPU and 4 GiB per environment; tuned KVM as in EXP-031. Integration: the v0.2.0 release tarball, the frozen T1-fixed procedure on one fresh host. Firecracker study: Drums v0.2.0 against Firecracker v1.18.0 (one microVM per desktop, OSWorld image, 20 s snapshot, plus a quiescent-snapshot variant), harness bench-r2/firecracker frozen at a4de57c, us-east-1f. Sustained step: at least 95% valid, 0 contamination, MemAvailable at least 3 GiB, step p95 at most twice the first step's.

What ran

T1-fixed: Drums ladder then KVM ladder on host A, the same on fresh host B. Integration (2026-11-11): the same procedure on the v0.2.0 tarball. Firecracker study (2026-11-12): both arms' ladders on hosts A and B, plus a same-host fork and reset workload.

Results

Density and per-dollar, T1-fixed: Drums release with the pipe fix against tuned KVM, hosts A and B preliminary2026-11-10 · hosts A and B

MetricValueUnitnNote
Max sustained desktops, Drums, host A512desktops512 rung: 1,201 rollouts, valid share 1.0000, 6,194 valid/h, min MemAvailable 143.8 GiB
Max sustained desktops, Drums, host B128desktops256 not sustained: one env quarantined, 10,535 instant reset failures on its slot, counted against Drums
Max sustained desktops, tuned KVM, hosts A and B128desktopsat 256 only 143 VMs fit in memory, on both hosts
Drums / tuned KVM, valid rollouts per USD, host A1.38x5595% CI [1.32, 1.44]; Drums better
Drums / tuned KVM, valid rollouts per USD, host B (replication)1.34x5595% CI [1.28, 1.41]; Drums better; inside +-20%
Memory per desktop at N 128, Drums vs tuned KVM, host A0.708 to 2.68GiBDrums 0.708 [0.688, 0.741], tuned KVM 2.676 [2.631, 3.015]
Reset p50 / p99 at N 128, Drums, host A1.79sp99 20.379 s (tuned KVM 2.630 / 7.371 s)
Invalid rollout rate, Drums and tuned KVM0shareDrums 0.0000 [0.0000, 0.0036], KVM 0.0000 [0.0000, 0.0055]

Host: r8i.16xlarge (64 vCPU, 512 GiB, nested virtualization on), on-demand, us-east-1a, gp3 600 GiB 16k IOPS, Ubuntu 24.04, kernel 7.0.0-1014-aws. B fresh. Workload: Same 55 tasks, scripted mock policy (no model), seed 20261108; ladder doubling from 8 to 512. Method: Pre-registered (frozen 2026-11-09T22:04:28Z, sha256 1958a81c...): changes only the Drums build (RC1 plus fs.pipe-user-pages-soft = 0). Sustained step: at least 95% valid, 0 contamination, MemAvailable >= 3 GiB, step p95 <= 2x the first step's. Max desktops claim rule: the same direction on both hosts. valid/USD: paired bootstrap over tasks, 10,000 resamples.. Source: research archive: runbooks/sota/t1/20261109T2203Z-fixed/REPORT.md; research archive: runbooks/sota/t1/20261109T2203Z-fixed/summary/tables.md.

  • Max desktops: 512 on A, 128 on B, so by the frozen rule there is NO density claim from this study alone.
  • Steps double, so '128' means 128 to 255 desktops and '512' is the last sustained doubling, not a ceiling.
  • Drums is hardened while the KVM agent has root in its VM; each is its product's default. No model in the loop.
  • Disclosed before the freeze: an exploratory run had already shown this build reaching 256 desktops.

Density on two more hosts: the v0.2.0 integration re-run (vs tuned KVM) and the Firecracker study (vs tuned Firecracker) preliminary2026-11-12 · integration host; Firecracker host B

MetricValueUnitnNote
Integration host: Drums 512 rung1151 / 1152rolloutsvalid share 0.9991, 6,145 valid/h, min MemAvailable 171.1 GiB: sustained
Integration host: tuned KVM max sustained128desktopsat 256 only 139 slots came up
Integration host: Drums / tuned KVM valid per USD1.4x5595% CI [1.33, 1.47]; agrees with T1-fixed within +-20%
Firecracker host B: Drums v0.2.0 512 rung1171 / 1172rolloutsvalid share 0.9991, 6,152 valid/h: sustained; operating N 256 (7,052 valid/h)
Firecracker host B: tuned Firecracker max sustained (20 s snapshot)16desktops32 failed on B; on host A 32 sustained, 64 failed: 16 to 32 across the two hosts
Firecracker host A: Drums result0rolloutsnone: one of eight creates failed at rung 8 (a slot service-plan read/write race) and the frozen rule stopped the ladder
Firecracker host B: Drums reset at its operating N 25612.56sp99 48.58 s; at equal N 16: Drums 0.46 / 0.79 s vs Firecracker 1.52 / 3.31 s
Firecracker host B: per-task valid/USD, Drums / Firecracker (20 s arm)7.6x4595% CI [7.19, 8.01]; NO CLAIM by the frozen power rule (Firecracker under 200 rollouts at its operating N)

Host: r8i.16xlarge (64 vCPU, 512 GiB, nested virtualization on), on-demand (one fresh host per job), Ubuntu 24.04, kernel 7.0.0-1014-aws. Firecracker study in us-east-1f; Firecracker without jailer, Drums hardened. Workload: Same 55-task scripted suite and ladder as T1; Firecracker v1.18.0 one microVM per desktop, OSWorld image, 20 s snapshot (and a quiescent-snapshot variant). Method: Integration: the frozen T1-fixed procedure re-run on the v0.2.0 tarball on a fresh host, no new freeze. Firecracker: PLAN frozen 2026-11-12T04:18:13Z (sha256 b7b15c96...), harness commit a4de57c, same sustained-step rule.. Source: research archive: runbooks/sota/integrate/20261111T1725Z/REPORT.md; research archive: runbooks/sota/integrate/20261111T1725Z/raw/t1/summary/tables.md; research archive: runbooks/sota/round2/firecracker/20261112T0226Z/REPORT.md; research archive: runbooks/sota/round2/firecracker/20261112T0226Z/summary-B/tables.md.

  • 512 was sustained on three separate hosts across three jobs (T1-fixed host A, the integration host, Firecracker host B), but never on both hosts of one pre-registered study. T1-fixed host B stopped at 128 and Firecracker host A has no Drums result.
  • The integration re-run has no freeze of its own and one host.
  • The Firecracker host uses nested virtualization: the VM arm pays its exit cost and Drums does not; no .metal check. Firecracker ran without jailer.
  • At its own operating point Drums' reset is slow (12.56 s p50 at 256 desktops); report it next to the 512.

Failures and surprises

Limitations

No model in the loop. One instance type. Drums at its own operating point resets slowly (12.56 s p50, 48.58 s p99 at 256 desktops on the Firecracker host). The Firecracker host uses nested virtualization, so the VM arms pay its exit cost and Drums does not; Firecracker ran without jailer while Drums was hardened. The Firecracker per-dollar ratio is "no claim" under the frozen power rule. Desktop capacity is not task throughput and not a cost saving by itself.

Reproduce

T1-fixed: REPRODUCE.md in its folder, drumsd a8d118b. Firecracker: REPRODUCE.md in its folder, harness a4de57c.

Sources

Pre-registrations, freezes, reports, summary tables and raw checksums under Evidence. Research archive, internal for now.