LLM Preflight

LLM Preflight

Last reviewed: 2026-10-29 · As of: v2.23.0

Check LLM responses against the output rules your application depends on—before changing a model, prompt, schema, or provider.

Define the JSON shape, routing labels, or exact output your feature needs. Check responses against those rules, preview provider requests and estimated cost, and keep evidence for the change review.

Built for engineers maintaining an LLM feature. Runs locally as a CLI, GitHub Action, or MCP server. Python 3.10+; no third-party runtime dependencies.

Try it · Observed models · Use your own contract · CI and agents

Watch an invalid response get caught and fixed (23 seconds).

billing passes → change the response to sales → check fails → restore billing → check passes. The allowed labels stay unchanged.

*Real CLI output with synthetic response fixtures. This checks the validator; it makes no provider requests and does not execute your application's parser.* Watch the MP4.

New in 2.23.0

This release adds GPT-6.1 Terra pricing and native Kimi, MiniMax, Mistral and Cohere text checks, plus GLM Flash and restricted Mythos coverage. See native provider checks for bounded examples and request limits. New coverage has deterministic protocol fixtures; live observations below cover the listed 19 routes. Other added providers retain fixture-only coverage.

Try it in 60 seconds

In a new directory, install the tool and run the local mock—no API key or provider request:

python3 -m venv .venv

. .venv/bin/activate python3 -m pip install llm-preflight llm-preflight init llm-preflight benchmark.json --no-save --no-env-file

The activation command above is for POSIX shells; PowerShell uses .\.venv\Scripts\Activate.ps1.

init refuses to overwrite an existing config. Look for local-mock with PASS in the quality gate: the mock replied ok and passed the exact-output test. The overall decision is inconclusive (exit 3) because a local mock provides no live-provider evidence. The demo worked; it cannot approve a model. --no-save skips JSON/Markdown result artifacts; --no-env-file skips credential files.

Watch the interactive default template (30 seconds)

Select the mock and default Reply with ok test, review the plan, and confirm the local run. The session also reads the same decision through CLI JSON:

*Local test: passed. Live provider: not tested. No API keys or provider calls.* Watch the MP4 or start with the text walkthrough and interactive CLI JSON example.

For benchmark runs, read the decision before acting:

• `pass`, exit `0`: the configured checks passed with the required evidence.
• `fail`, exit `1`: a configured check failed; inspect the cause.
• `inconclusive`, exit `3`: evidence is insufficient; read the blocking warnings.

A passing run is evidence for the configured workload, not production approval.

Observed results on a support-routing contract

The latest measurement, 2026-10-29, tested 19 native model routes on eight synthetic support-routing and strict JSON cases each. 144 of 152 checks passed: three outputs failed their contract and five Kimi calls returned HTTP 429. 15 models passed all eight checks. These are workload observations, not a ranking of general intelligence or production approval.

Latency includes successful API responses, including contract failures, and excludes API errors. The known token-cost subtotal is $0.06902127; total cost is unknown because five error responses supplied no usage. Each request used a 1,024-token output allowance and one attempt. No retries were made. Read all 19 rows, the method and failure analysis. Previous dated measurements remain available as history.

First live run

Start with one rule your application already enforces. The project integration guide provides a complete routing contract with accepted/rejected fixtures and a no-spend pull-request check. Match your deployed prompt and response parser; local fixture checks do not execute application code automatically.

To try a minimal provider starter instead:

llm-preflight init live.json --template provider \

--provider openai --model gpt-6.1-sol --api-key-env OPENAI_API_KEY

The starter asks for exactly ok. Add this top-level field to live.json, keeping its existing fields, to check the validator locally:

{

"validation_fixtures": [ {"name": "accepted-label", "response": "ok", "expect": "pass"}, {"name": "rejected-label", "response": "not ok", "expect": "fail"} ] }

Set OPENAI_API_KEY in your shell or in an uncommitted .env.production beside live.json. The config stores the variable name, never the key. Preview the check without making a generation request:

llm-preflight live.json --contract-check

llm-preflight live.json --doctor llm-preflight live.json --pricing-check llm-preflight live.json --dry-run

The starter has a three-request limit and a $0.05 estimated-cost cap. Resolve warnings and review the route, retry-expanded request bound, pricing and retention settings. The starter saves failed responses by default; review artifacts before sharing them.

Paid request: only after reviewing and authorizing that plan, run:

llm-preflight live.json

A direct CLI run has no separate approval prompt. Results are saved as JSON and Markdown under results/ unless you use --no-save. The CLI reads the config-adjacent .env.production without replacing existing shell values; choose another file with --env-file PATH or skip it with --no-env-file.

Choose your path

• Set up a real project. Adapt one application rule with the project integration guide.
• Validate a change. Compare approved and candidate configurations with the model-change guide.
• Automate an established contract. Use CI and JSON output or the GitHub Action.
• Configure a coding agent. Start with llm-preflight-mcp --workspace "$PWD" in a trusted repository and follow the MCP server guide.

See a contract change fail

A review artifact retains the decision, contract validity, latency, usage, estimated cost and blocking warnings. Render saved schema-version-1 JSON as a standalone report without another provider request. Substitute the saved JSON path printed by your run for results/run.json:

llm-preflight report results/run.json --output report.html

The renderer omits prompts, responses, local paths and identifying run metadata. For reproducible examples, run these from a source checkout:

python3 -m llm_preflight report examples/reports/schema-baseline.json --format markdown

python3 -m llm_preflight report examples/reports/schema-break.json --format markdown

The first synthetic result passes; the second reports contract_failure, 67% contract validity and one contract-only failure. The report gallery also demonstrates a gate blocked with no_record (exit 1). These fixtures make no provider calls and cannot supply attested CI evidence.

CLI, CI, and MCP

Works as a CLI, GitHub Action, and local MCP server. The Action defaults to no generation requests; a live MCP run requires explicit paid-run confirmation. Keep the client's tool approval enabled: its confirmation field is not proof that a person authorized spending.

No preflight, no merge. With a required check configured, the offline gate requires a fresh passing CI record matching the current contract, routes, prompts and run selection. Protect the producer and gate workflows; signed records also need gh attestation verify before the gate. Follow proof of preflight to configure it. A local fixture pass or installed package does not enforce a merge policy.

It doesn't certify production behaviour. It proves the check was done.

MCP for coding agents

Configure the installed stdio command in your MCP client:

{

"mcpServers": { "llm-preflight": { "command": "llm-preflight-mcp", "args": ["--workspace", "/absolute/path/to/repository"] } } }

The eight tools (migrate_plan is main branch only, not in 2.23.1) audit literal model IDs, validate a config, preview a plan, run a preflight, compare saved baselines, verify a record, gate CI evidence and plan a retired-model migration. For an existing contract, begin with validate_config and dry_run_plan. Agents must read the structured decision and report missing evidence. See the MCP guide for request/result examples and workspace boundaries.

The agent-smoke suite with --smoke makes six requests per model before configured retries. Use the coding-agent workflow for CLI commands. An agent must not weaken a validator to get a pass or infer permission for paid work from a gate result.

Common jobs

• **Audit existing model