First paid AWS host launch terminated by its own heartbeat backstop

In one paragraph

aws_run is the tooling that is allowed to spend money: one EC2 host per phase, with a budget-derived lifetime, two independent kill switches and a tag-based teardown that is verified afterwards. The first real launch under these gates ended within minutes, not because the host failed but because one of the kill switches fired early. The CloudWatch alarm on the heartbeat metric was created with its terminate action enabled before any heartbeat datapoint existed; the alarm treated its twelve empty periods as breaching, went straight from INSUFFICIENT_DATA to ALARM at its first evaluation, and terminated the instance about a minute before the first heartbeat arrived. The backstop worked, against us. The ordering was changed so that a heartbeat is seeded first, the alarm is created with actions disabled, a healthy OK state is read back, and only then are actions enabled. All of this is author-claimed; no run artifact is on the branch.

Setup

What the gates are

Before any AWS mutation, run --execute requires:

• The reviewed prereg sha256 (--reviewed-prereg-sha256) equal to the digest of the configs, which include volume size, budget-derived hours, idle limit, prices and ceilings; --volume-gib, --hard-max-hours and --idle-limit-s are not run options, so a repeat must equal the preregistered value.
• A valid, single-use provenance record (tool-generated, bound to one phase and prereg hash, with a nonce and a 12 h staleness guard) that "says on its face that it is not an approval and was not written by the owner", plus the operator's typed assertion that they read the original grant. The tool validates bounds and binding, not authenticity.
• An --aws-wrapper executable that selects the authorized profile; the child environment is scrubbed of AWS_* and the caller account must equal the authorized one. No credential is read, stored or logged by the tool.
• Immediately before creating anything: account, instance type, AMI id and default VPC are re-read and must still be the preflighted ones; any tagged instance from any run blocks the launch; a local lock blocks a second orchestrator.
• run-instances --dry-run must fail with exactly DryRunOperation; UnauthorizedOperation, InvalidParameter or rc 0 all refuse.
• The AWSServiceRoleForCloudWatchEvents service-linked role must exist or be created before any billable resource, because EC2 terminate actions on alarms need it (found live: the role was missing in this account before the run, author-reported).

Fixed inputs, operator-supplied and labelled as such on the branch: region us-east-1, instance type c8i.2xlarge (8 vCPU / 16 GiB, nested virtualization via --cpu-options NestedVirtualization=enabled), the official Canonical Ubuntu 24.04 AMI, one 4 vCPU / 4 GiB guest (design value), a 120 GiB root volume ("a guess"), an idle limit default of 3600 s (design value). Network: SSH from exactly one operator /32, port 5000 never opened, IMDSv2 required, no instance profile, encrypted gp3 root deleted on termination, shutdown behaviour terminate.

The two backstop layers

1. AWS-managed: a CloudWatch alarm on drumsbench/Heartbeat (statistic Maximum, unit Count, period 300 s, 12 evaluation periods, missing data = breaching; all design values) whose action is arn:aws:automate:us-east-1:ec2:terminate. The orchestrator publishes a heartbeat only when the host's own progress heartbeat advanced or an operator --keepalive-file is fresh, and only while the estimated hours are inside the run budget. A healthy run is never cut off by elapsed time before the budget boundary.
2. Host-side: a watchdog installed by cloud-init user data at boot with the budget-derived deadline and the idle limit, so there is no unguarded window before bootstrap.

The orchestrator's finally always tries evidence collection, then a bounded-retention snapshot if that fails, then terminates and verifies a tag-based teardown. Exit codes: 0 all passed including human review; 3 automated steps passed, human GUI review pending; 10 phase failed; 20 teardown not clean; 30 evidence not preserved; 2 refused.

What ran

Author-claimed, from tests/test_aws_run.py (class BackstopInitTests), aws_run/provision.py (create_backstop) and commits 8f6ae5a, eb55440, 7e45267 (19:06 to 20:09 UTC): one real EC2 launch happened on 2026-10-25, with the instance id, run id and receipts in the gitignored aws-runs/ directory. A CloudWatch datapoint timestamp captured into a regression test puts the incident at about 19:09 UTC; that timestamp is a captured CLI payload, not a measured benchmark value. The branch's internal label for the launch is undefined and not published here.

Results

No measured result. The author's description of the incident, quoted from the test docstring: "an alarm created with actions enabled and no datapoint went INSUFFICIENT_DATA -> ALARM at its first evaluation (12 empty periods treated as breaching) and terminated the instance a minute before the first heartbeat arrived."

Two smaller findings from the same launch, both from commit messages: the real CLI prints the datapoint Timestamp as a JSON number, not an ISO string, which the readback parser had not accepted (eb55440); and the real payload carried Maximum, not Sum, so the Sum fallback was dropped and the regression test now runs the exact captured payload through the whole init guard (7e45267).

The fix

provision.create_backstop now documents and enforces this order ("ORDER (the first live launch showed why)"):

1. Seed the heartbeat for exactly this instance id, before the alarm exists.
2. Create the alarm with --no-actions-enabled and the terminate action attached but inert.
3. Read back the configuration, the metric (exact dimension, Count, threshold, a current timestamp) and a healthy OK state, polling until they hold or a bounded timeout (600 s, design value) passes; on timeout fail closed and terminate.
4. Only then enable-alarm-actions.
5. Re-verify everything with actions enabled.

The state is never forced (no set-alarm-state) and nothing is slept on without a readback. The commit calls itself "REVIEW CANDIDATE (targeted tests pass; legacy full test_aws_run not re-run)".

Failures and surprises

• The backstop fired. STATUS.md had listed "Backstop action unproven ... whether CloudWatch's terminate action actually fires can only be shown by triggering it. No such trigger has been run." It was triggered, by accident, and it did terminate the instance. That is the one thing this run demonstrated (author-claimed).
• The same branch contradicts itself: STATUS.md line 1 says "nothing has been executed against AWS" and line 20 "no paid launch has happened", while the later test and provision docstrings describe "the first live launch". STATUS.md was not updated. We report the later statements as author-claimed.
• Because aws-runs/ and *.pem are gitignored by design, no AWS artifact can ever appear on this branch; the evidence lives only on the operator's machine.

Limitations

• No run-summary.json, steps.jsonl, aws-calls.jsonl, provision-receipt.json or termination-receipt-*.json on the branch.
• The fixed order has not been exercised on a second live launch per the branch.
• Everything about the host (nested KVM, query-kvm under nesting, savevm/loadvm under nesting, the image download from EC2) remains "never run".
• The pins (CRIU tag and archive hash, instance-type advertisement, prices) are operator-supplied inputs that the host re-verifies but the code does not.

Reproduce

From the README's exact sequence (benchmark/):

python -m aws_run plan --run-budget-usd <usd> --ebs-usd-gb-month <p> --ipv4-usd-hour <p> --transfer-usd-gb <p> \
    --egress-gb-ceiling <gb> --snapshot-usd-gb-month <p> --snapshot-retention-days <days>
python -m aws_run preflight --aws-wrapper ./aws-as-profile --ssh-cidr <your.ip>/32 ...      # read-only AWS calls
python -m aws_run provenance-record --phase baseline --preflight-dir aws-runs/preflight --out provenance-baseline.json
python -m aws_run run --execute --phase baseline --aws-wrapper ./aws-as-profile --preflight-dir aws-runs/preflight \
    --reviewed-prereg-sha256 <hash> --provenance-file provenance-baseline.json --operator-read-original-grant '<id>'
python -m aws_run status --aws-wrapper ./aws-as-profile
python -m aws_run cleanup --aws-wrapper ./aws-as-profile --run-id <id> --confirm-terminate

A second launch with the reordered backstop, with run-summary.json, steps.jsonl and the alarm readback published (identifiers redacted), is what would turn this into a verified record of the backstop behaving.

Sources

• benchmark/aws_run/provision.py (create_backstop, docstring "ORDER (the first live launch showed why)")
• benchmark/tests/test_aws_run.py (BackstopInitTests docstring; the captured-payload regression)
• benchmark/aws_run/STATUS.md (execution gate, spend bounds, open items; stale on the launch)
• benchmark/README.md, section "AWS run source"
• benchmark/aws_run/prereg.py, benchmark/aws_run/packages.lock.json, benchmark/.gitignore
• Commits 1bcb0cd, f9d39a1, 64b143e, 410f855, 8f6ae5a, eb55440, 7e45267 (2026-10-25)