This page is the plain-language summary of our first benchmark. Every number here was measured on one server; the raw data is kept with each run.
What we compared
All setups ran on the same cloud server (64 cores, about 500 GB of memory), on real desktop tasks from OSWorld, with the same AI agent (Claude Opus 5.5).
| Setup | What it is |
|---|---|
| Drums | Many desktops share one running operating system; each keeps its own private state |
| Tuned VMs (QEMU/KVM) | One virtual machine per desktop, restored from a snapshot kept in memory |
| Firecracker | One lightweight microVM per desktop, restored from a snapshot |
| Desktop container | One container per desktop; it cannot copy a running desktop, so it repeats the steps instead |
| OSGym-style | Our own version built from the OSGym paper (its code is not public) |
| OSWorld Docker | OSWorld's own setup, unchanged |
| Cua | Cua's open-source parts only; its fast-restore server has a license we cannot use |
Test 1: copies of the same moment
The agent works part of the way through a task. We then make 8 independent copies of that exact moment and let the agent finish each one. We count correct finishes and divide by everything we paid: the AI model and the server, including failed attempts.
80 paired test cases, with a 95% confidence interval on each comparison.
| Drums compared with | Ratio | 95% interval | Verdict |
|---|---|---|---|
| Tuned VMs | 0.80 | 0.68 to 0.93 | Drums lower |
| Firecracker | 0.83 | 0.69 to 1.00 | Drums lower |
| Desktop container | 1.12 | 0.93 to 1.36 | No clear difference |
| Setup | Correct finishes | Valid copies | Total cost | Cost per correct finish |
|---|---|---|---|---|
| Drums | 294 | 357 of 400 | $84 | $0.29 |
| Tuned VMs | 494 | 544 of 552 | $113 | $0.23 |
| Firecracker | 463 | 504 of 504 | $111 | $0.24 |
| Desktop container | 346 | 414 of 632 | $111 | $0.32 |
Why Drums lost. In 14 of the 80 test cases Drums could not save the running desktop at all:
- Chrome and VS Code run their security sandboxes in a way the save tool (CRIU) cannot handle.
- Some system-settings tasks failed on one settings file; this needs one extra option.
- Thunderbird saved fine, but the memory-sharing restore refused it. A plain restore of the same save worked.
On the 45 cases where every setup completed, Drums tied the VMs (1.00 against tuned VMs, 0.96 against Firecracker). That check was not planned in advance, so treat it as a hint, not a result.
Why the container lost. It repeats the steps instead of copying the moment, and repeated steps do not always land in the same place: saved documents came out different.
The bigger lesson. The AI model was about 95% of the total cost on every setup. Faster or smaller environments barely move the bill while the model is this expensive.
Test 2: the environment alone
No AI agent; just resetting, copying and packing desktops.
| Setup | Reset | Save a running desktop | 8 copies ready | Memory per copy | Most desktops on the server |
|---|---|---|---|---|---|
| Drums | 1.3 s | 3.6 to 4.6 s | 1.0 to 1.1 s | about 270 MB | about 300 by default; 1,024 with one system limit raised |
| Tuned VMs | 1.0 s | 0.8 to 1.0 s | 1.0 to 1.2 s | about 2.5 GB | 128 |
| Firecracker | 0.7 s | 5.0 to 5.4 s | 3.3 to 5.0 s | about 160 to 270 MB | about 128 |
| Desktop container | 1.8 s | cannot copy | 13 s (repeats steps) | about 500 to 740 MB | 512 or more |
| OSGym-style | 16 s | cannot copy | 34 s (repeats steps) | about 5 GB | 32 |
| OSWorld Docker | 20 s | cannot copy | 20 s | about 3.2 GB | 48 or more |
| Cua (open-source parts) | 95 s | cannot copy | 117 s | about 2.6 GB | 48 or more |
"Or more" means we stopped at our test limit before the setup failed. The 1,024 Drums figure comes from a single run.
Wins
- About 9 times less memory per copy than tuned VMs.
- 1,024 desktops on one server with memory to spare; the VM setups stopped near 128.
- 8 copies in about 1 second, 3 to 4 times faster than Firecracker.
- Resets in about 1.3 seconds, against 16 to 95 seconds for the open setups people commonly use.
- Zero leaks between copies in every test, on every setup.
- Fixes to CRIU, the open-source save-and-restore tool:
- file locks that were lost on restore;
- a background service that was silently killed on every restore;
- saving a desktop dropping from about 2 minutes to a few seconds.
Losses
- Fewer correct finishes per dollar than tuned VMs and Firecracker, the pre-registered headline.
- Chrome, VS Code, some settings tasks and Thunderbird cannot be saved yet.
- Tuned VMs save a running desktop faster; Firecracker resets faster.
- Its density only pays off when many desktops share one server, and this test did not reward that.
A first training run
We trained a 9-billion-parameter computer-use model with reinforcement learning (full fine-tuning, not adapters), with Drums supplying every desktop. It ran 17 training steps with no desktop failures. The run was too short to show the model improving, and the tasks we held out for testing were too easy. It shows the pipeline works; it does not show a training benefit yet.
What we are doing next
- Make the missing apps work (Chrome, VS Code, settings, Thunderbird), then rerun the same test against the same results.
- Test where memory and density decide the cost: many desktops per server, and cheaper AI models.
- Run the training comparison against virtual machines.
Fine print
- One server and one run of the plan, with 80 of the 135 planned test cases. The rest were stopped when our access to the AI model was paused; they were dropped from every setup equally.
- The plan and thresholds were written down before the first run. Five documented changes were made before the results they could affect.
- Prices come from the published price lists on the day of the run.