Haltproof

HaltProof

Emergency-shutdown orchestration for Slurm, Kubernetes, and IPMI clusters, with an Ed25519-signed, hash-chained attestation record of exactly what ran.

Signed, dry-run-by-default shutdown orchestration for Slurm, Kubernetes, and IPMI clusters, with a tamper-evident audit trail.

HaltProof is an emergency-shutdown orchestration and cryptographic audit-proof layer for compute clusters. It does not implement a new low-level shutdown mechanism. It coordinates cluster primitives you already trust (Slurm, Kubernetes, IPMI/BMC) and produces a signed, tamper-evident record of exactly what was targeted, what ran, and who authorized it.

pip install haltproof-cli

(Full install options, including the npm wrapper, are in Install below.)

Why

Operators already have the tools to drain a Slurm partition, cordon a Kubernetes node pool, or power-fence a physical host over IPMI. What's usually missing is:

1. One consistent interface across those tools during an incident, instead of three different command sets under pressure.
2. A dry-run-by-default safety rail, so a mistyped target group doesn't take down the wrong nodes.
3. A signed, tamper-evident record of what happened: who ran it, what commands were issued, against which nodes, whether each step succeeded, and whether the record itself has been altered or has a piece missing.

HaltProof is that layer. It calls out to scontrol, kubectl, and ipmitool (or mcp-invokable equivalents) and wraps every invocation in an Ed25519-signed, hash-chained attestation.

Install

pip install haltproof-cli

An npm wrapper is also published for Node.js-based tooling and agent runtimes:

npm install -g haltproof-cli

[!WARNING]

The npm package requires the Python haltproof-cli package to already be

installed and on PATH. It is a thin execFileSync wrapper, not a

reimplementation. If it can't find the Python CLI, it prints an actionable

error and exits non-zero rather than silently doing something else.

Table of Contents

• Features
• Quickstart
• Commands
• Architecture
• Security model
• MCP server (for AI agents)
• Comparison
• What Is HaltProof, and Why Does It Exist
• FAQ
• Contributing
• License

Features

• Dry-run required by default. haltproof halt prints exactly which commands it would run, against which nodes, and still writes a signed attestation of that plan, unless you pass --confirm. The gate lives at a single chokepoint (haltproof.backends.runner.run_step) that all three backends route through, so it can't be quietly bypassed by one backend's code path.
• One interface across three real backends. kubernetes (via kubectl), slurm (via scontrol), and ipmi (via ipmitool) each implement the same four-operation ClusterBackend interface: drain, isolate_network, power_fence, status. A backend that doesn't support an operation, for example Kubernetes has no physical power-fencing primitive, returns a SKIPPED result with an explanation instead of raising, so a halt across a mixed environment still produces one complete attestation.
• Ed25519-signed, hash-chained attestations. Every halt, dry-run or executed, appends one signed record to a local append-only NDJSON log. Each record embeds prev_hash, the content hash of the record before it, so haltproof verify --chain can catch a deleted or reordered entry that a single record's own signature check would miss.
• Structured output on every command. --json (or --format json) is available on halt, verify, status, and keygen, so a script or an agent can parse the result without screen-scraping human-formatted text.
• A real MCP server, built for agents. haltproof mcp-server exposes halt, verify, verify_chain, status, and keygen as MCP tools over stdio, built on the official mcp Python SDK. It calls the same haltproof.core functions the CLI calls, so an agent invoking the MCP tool gets the identical dry-run-by-default behavior a human gets from the CLI.
• Tested. 76 tests, 90% overall line coverage, run and confirmed in this audit with pytest -v --cov=haltproof --cov-report=term-missing. Backend tests mock subprocess calls; none of them require a real cluster.

Quickstart

[!IMPORTANT]

haltproof halt is a no-op without --confirm. It prints the plan and

writes a signed dry-run attestation, but nothing destructive runs until

you pass --confirm explicitly.

# Generate a signing key for attestations (do this once).

haltproof keygen --output ~/.config/haltproof/ed25519_key

See what backend HaltProof auto-detected on this host.

haltproof status

Dry-run a halt against an explicit node list. Nothing runs yet.

haltproof halt gpu-pod-a --nodes node-1,node-2

Same halt, actually executed.

haltproof halt gpu-pod-a --nodes node-1,node-2 --confirm

Verify a past attestation's signature and print its timeline.

haltproof verify 1

Verify the entire attestation log's hash chain instead of a single

record: this catches a deleted or reordered entry that a

single-record signature check alone would miss.

haltproof verify --chain --attestation-log ~/.local/share/haltproof/attestations.jsonl

Every command also supports --json for agent-parseable output:

haltproof status --json

haltproof halt gpu-pod-a --nodes node-1 --json

Config file (optional)

Target groups, the preferred backend, and default paths can live in a haltproof.toml file (checked in the current directory, then ~/.config/haltproof/config.toml, or at a path given by --config / HALTPROOF_CONFIG):

backend = "kubernetes"

operator_id = "ops-team" attestation_log_path = "/var/log/haltproof/attestations.jsonl"

[groups] gpu-pod-a = ["node-1", "node-2", "node-3"]

[kubernetes] namespace = "training"

[ipmi] user = "admin"

Every config value has a matching CLI flag, and an explicit flag always wins over the config file.

Commands

Reference below is regenerated from the CLI's actual --help output.

haltproof halt TARGET_GROUP

Drain, isolate, and power-fence TARGET_GROUP. Dry-run is the default; nothing destructive runs without --confirm.

Options:

--nodes TEXT Comma-separated explicit node list, overrides config group lookup. --backend TEXT Backend to use: kubernetes, slurm, or ipmi. --confirm Actually execute the halt. Without this, dry-run only. --reason TEXT Reason recorded for drain operations. --operator-id TEXT Operator identity to record. Defaults to OS user. --config TEXT Path to a haltproof.toml config file. --attestation-log TEXT Path to the attestation log file. --attestation-key TEXT Path to the Ed25519 private key used to sign the attestation. --no-sign Skip signing (not recommended); still logs the record unsigned. --remote-collector TEXT Optional URL to POST the signed attestation to. --format [human|json] Output format. --json Shorthand for --format=json.

haltproof verify [ATTESTATION_REF]

Verify one attestation record's signature, or an entire log's hash chain. ATTESTATION_REF is a path to a record file, or an attestation id / sequence number to look up in the log. Pass --chain instead of ATTESTATION_REF to verify a whole --attestation-log: every record's signature holds, and none has been deleted or reordered.

haltproof status

Show backend auto-detection results and, optionally, target-group health.

haltproof keygen

Generate an Ed25519 keypair for attestation signing. The private key is written with 0600 permissions and is never printed or logged.

Options:

--output TEXT Path to write the private key to. --force Overwrite an existing key at the output path. --format [human|json] Output format. --json Shorthand for --format=json.

haltproof mcp-server

Starts the HaltProof MCP server on stdio, exposing halt, verify, verify_chain, status, and keygen as MCP tools. No flags beyond --help.

Every subcommand also accepts --help directly for the same reference shown above.

Architecture

HaltProof has three layers:

1. Transport layer: the Click-based CLI (haltproof.cli) and the MCP server (haltproof.mcp_server). Both are thin wrappers over the same core functions in haltproof.core, so a human running the CLI and an agent calling the MCP tool get identical behavior, including the dry-run-by-default gate on halt.

1. Backend layer: a ClusterBackend abstract interface (haltproof.backends.base) with four operations:

   ClusterBackend

├── drain(nodes, dry_run, reason) -> stop new work being scheduled ├── isolate_network(nodes, dry_run) -> cut nodes off from the workload network ├── power_fence(nodes, dry_run) -> hard power off └── status(nodes) -> current reachability/state

Three backends implement it today:

A backend that doesn't support a given operation returns a SKIPPED result with an explanation instead of raising, so a halt sequence against a mixed environment still produces one complete, honest attestation record. Adding support for a new scheduler or fencing tool means adding one backend module and registering it in haltproof.backends.detect, not touching the orchestration or attestation layers.

Backend selection order: an explicit --backend flag, then the config file's `b