Workloadtruth

WorkloadTruth

Install • Quickstart • CLI reference • Comparison • FAQ

Classify a GPU workload as `TRAINING`, `INFERENCE`, or `IDLE` from telemetry alone. No code changes to the workload, no self-reported job labels.

Every GPU scheduler in common use today, including run:ai, Slurm, and Kubernetes GPU operators, asks you to *declare* whether a job is training or inference at submission time. None of them check. WorkloadTruth reads GPU telemetry (utilization, memory pattern, power draw) and answers the question independently, so a mislabeled or misbehaving job doesn't go unnoticed.

Install

# Real NVIDIA GPU telemetry (requires an NVIDIA driver on the host)

pip install "workloadtruth-cli[nvml]"

Try it without a GPU, using the synthetic backend

pip install workloadtruth-cli

npm launcher (thin wrapper around the PyPI package, see "Why two registries")

npx workloadtruth-cli --help

[!NOTE]

The npm package is a launcher, not a standalone install. npx workloadtruth-cli execs the real workloadtruth binary from PATH, so the PyPI package (pip install workloadtruth-cli) must already be installed first.

Quickstart

# No GPU required. Classify a synthetic "training" telemetry trace.

$ workloadtruth classify --backend synthetic --profile training --samples 10 --interval 0 workload_type : TRAINING confidence : 1.00 gpu_index : 0 samples : 10 over 9.0s reasons:

• avg GPU utilization 87.8% >= training threshold 65.0%
• low GPU utilization variance (std=3.4) <= training ceiling 15.0
• memory growing 120.0 MiB/sample >= training threshold 5.0
• low power-draw variance (std=9.4W) <= training ceiling 25.0W

Real hardware

$ workloadtruth classify --backend nvml --samples 10 --interval 1 --json

--json on every command switches to machine-readable output for scripts and agents.

Quick summary

• Use it for: catching cost-misallocated GPU jobs (a job billed as low-priority "inference" that's actually running full training) and unauthorized workload changes (an inference endpoint that starts training on live traffic without sign-off)
• What it's not: a compliance or regulatory-audit tool. No regulation currently requires this kind of monitoring, see What WorkloadTruth is not below
• Prior art: builds on and cites arXiv:2606.19262 (ICML 2026), see Relationship to prior research

How classification works

WorkloadTruth currently ships a rule-based classifier only: a set of documented, inspectable thresholds over four features extracted from a telemetry window (average and variance of GPU utilization, memory-growth slope, average and variance of power draw). Every threshold lives as a named constant in src/workloadtruth/classifier/rules.py with a comment explaining its intuition. Nothing is a black box.

--experimental (an ML-based classifier) is present as a flag but fails loudly with an explanation rather than shipping a fake result. arXiv:2606.19262's trained model and dataset were never published, and this project has no NVIDIA GPU in its build environment to collect real training data. An ML classifier ships here only once it's trained on a real, disclosed dataset and independently shown to beat the rule-based baseline, not before.

Benchmark

workloadtruth benchmark measures the classifier's accuracy against documented synthetic GPU telemetry traces (see synthetic_backend.py for the exact profile parameters), both clean and under a deliberate evasion transform that mimics an operator disguising a training job as inference (capping utilization swings, flattening memory growth, injecting artificial idle gaps).

This is run on synthetic data, not live NVIDIA hardware, so it is not directly comparable to arXiv:2606.19262's real-hardware numbers. Both are reported below, side by side, never blended into one figure.

The overall evasion number hides the real finding. Broken out by workload type, WorkloadTruth's rule-based classifier is currently completely fooled on evasive training workloads (0% accuracy), while staying accurate on evasive inference and idle traces (100% each). Capping GPU utilization alone is enough to make a training job read as inference to this classifier today.

This is the concrete, disclosed gap the benchmark suite exists to surface and track, not a footnote. A rule-based classifier that only looks at utilization variance is trivially evadable. Closing this gap, through multi-signal fusion or a real trained classifier once data exists, is the roadmap, not a solved problem. Reproduce it yourself:

workloadtruth benchmark --trials 300 --window 30 --json

CLI reference

$ workloadtruth --help

Usage: workloadtruth [OPTIONS] COMMAND [ARGS]...

Classify a GPU workload as INFERENCE, TRAINING, or IDLE from telemetry alone.

Options: --version Show the version and exit. --help Show this message and exit.

Commands: benchmark Run the evasion-robustness benchmark against synthetic... classify One-shot classification of the current GPU workload. mcp Start an MCP server exposing classify/benchmark/verify-log... verify-log Verify the hash chain of a local audit log has not been... watch Continuously classify and append hash-chained entries to...

Every command supports --json. Full flag reference: workloadtruth --help.

MCP Server

WorkloadTruth ships a Model Context Protocol server so an AI agent (Claude, Cursor, or any MCP-compatible client) can classify GPU workloads, run the evasion-robustness benchmark, and verify the audit log directly, without a human invoking the CLI by hand.

Install the extra:

pip install "workloadtruth-cli[mcp]"

Add it to your MCP client's config (for Claude Desktop, claude_desktop_config.json). The server is started via the workloadtruth mcp subcommand, not a separate console script:

{

"mcpServers": { "workloadtruth": { "command": "uvx", "args": ["--from", "workloadtruth-cli", "workloadtruth", "mcp"] } } }

The server exposes three tools over stdio:

• `classify_workload(backend="nvml", profile="training", gpu_index=0, samples=10, interval_seconds=1.0, write_to_audit_log=False)`: samples GPU telemetry and classifies it as TRAINING, INFERENCE, or IDLE. backend is "nvml" (real hardware) or "synthetic" (documented synthetic traces, no GPU required). Optionally appends the result to the hash-chained audit log.
• `run_benchmark(trials=50, window=30)`: runs the evasion-robustness benchmark against synthetic telemetry and returns per-profile accuracy under clean and evasion-obfuscated conditions.
• `verify_audit_log(log_file="workloadtruth.log.jsonl")`: re-derives the hash chain of a local audit log and reports whether it has been tampered with.

Example call, classifying a synthetic training trace with no GPU required:

classify_workload(backend="synthetic", profile="training", samples=10, interval_seconds=0)

-> {"workload_type": "TRAINING", "confidence": 1.0, "gpu_index": 0, ...}

Transport is stdio, so there is nothing to host: the MCP client spawns workloadtruth mcp as a local subprocess. A .well-known/agent.json manifest is also shipped at the repo root for A2A-style discovery, listing both the CLI and MCP interfaces and the packages that provide them. Source: src/workloadtruth/mcp_server.py.

Audit log

workloadtruth watch appends a hash-chained entry to workloadtruth.log.jsonl on every classification window. Each entry's hash covers its own content plus the previous entry's hash, so any edit, reorder, or deletion after the fact breaks the chain from that point forward. workloadtruth verify-log re-derives every hash and reports the first broken link, if any.

This proves what was classified, when, and that the local record hasn't been silently altered afterward. It does not prove the classification itself was correct, and it is not evidence of regulatory compliance. See below.

Why two registries

WorkloadTruth's implementation is Python. NVML access (pynvml/nvidia-ml-py) is the mature, official way to read NVIDIA GPU telemetry, and it's also what the closest prior art (arXiv:2606.19262) uses. The npm package (workloadtruth-cli) is a thin launcher, not a reimplementation. It locates and execs the real workloadtruth binary installed from PyPI, so npx workloadtruth-cli works for npm-first agent tooling without duplicating the class