API Shakedown

API Shakedown

Give it an OpenAPI spec. Get back a ready-to-run test harness: WireMock stubs, contract tests, robustness probes, and chaos scenarios, written by a team of LLM agents that review each other's work.

Runs locally on [Ollama](https://ollama.com) by default: no API key, no data leaves your machine. Cloud models (OpenAI, Anthropic) are bring-your-own-key. Available as a CLI, a Docker image, and an MCP server so an AI assistant can drive it.

Status: alpha (0.1.3). Works end to end on the example spec with a local 8B model.

*A real run on the bundled example with qwen3:8b on a laptop, sped up: generation took about 9 minutes. Recording: docs/demo.cast.*

Contents

• Why
• Where it helps
• Quick start
• What you get
• How it works
• Install
• Usage
• Configuration
• MCP server
• Docker
• Architecture
• Limitations
• Roadmap
• Development
• Contributing
• License

Why

Teams that consume an API need three things that are tedious to write by hand and go stale quickly:

1. A realistic mock of the API to develop and test against.
2. Contract tests that tell you when the real API stops matching its spec.
3. Failure drills: what does your client do when the API is slow, flaky, rate limited, returns garbage, or goes down?

API Shakedown generates all three from the spec you already have, and keeps them honest: the LLMs never write code that runs. They propose *data* (response bodies, test cases, failure scenarios, test records), and code validates every item against the spec, feeds problems back to the model, and falls back to deterministic output when a model cannot get it right. Without any model at all (--offline), every part still works.

Where it helps

It is not a load-testing tool or a security scanner, and the virtual service does not keep state between calls; see Limitations.

Quick start

Prerequisites: Python 3.11+, Docker (for WireMock), and Ollama with a model (ollama pull qwen3:8b) unless you use --offline or a cloud model.

pip install api-shakedown

shakedown doctor # is the model ready? shakedown run openapi.yaml --out my-harness # minutes on a local model cd my-harness docker compose up -d # WireMock with your stubs on :8080 pip install -r requirements.txt SHAKEDOWN_SELECT_STATUS=1 pytest tests/test_contract.py python chaos/chaos.py on flaky-start # now point your client at :8080 python chaos/chaos.py off

Try it on the bundled example: shakedown run examples/petstore.yaml --out petstore.

A real run on the example spec with qwen3:8b:

Petstore 1.0.0: 5 operations (ollama / qwen3:8b)

Generating test data... Test data ready (3 Pet). Generating stubs, contract tests, and chaos scenarios... stubs: done contracts: done chaos: done review contracts: approved review stubs: approved review chaos: approved

Wrote 14 files to harness in 436s: 15 stubs, 11 contract cases, 11 chaos scenarios, 30 probes. Next: cd harness && docker compose up -d (then see README.md)

What you get

my-harness/

├── README.md # generated: how to run it, every scenario, review notes ├── docker-compose.yml # WireMock 3 serving the stubs (port: SHAKEDOWN_WIREMOCK_PORT) ├── wiremock/mappings/ # stub mappings ├── tests/ │ ├── test_contract.py # contract tests (cases in contract_cases.json) │ └── test_robustness.py # malformed-request probes (cases in robustness_cases.json) ├── chaos/ │ ├── chaos.py # switch failure scenarios on/off (standard library only) │ └── chaos_scenarios.json ├── data/dataset.json # test records and edge-case values per field ├── requirements.txt # pytest, httpx, openapi-schema-validator └── shakedown.json # manifest: counts, review outcome, generation notes

Baseline chaos scenarios:

LLM-targeted scenarios from a real run include *createPet fails 3 times with 503 then recovers*, *listPets is rate limited*, and *getPet loses required fields*.

How it works

flowchart LR

spec[/"OpenAPI spec"/] --> load["Load, validate, normalize, chunk"] load --> data["Test data agent"] data --> stubs["Stub agent"] data --> contracts["Contract agent"] data --> chaos["Chaos agent"] stubs --> review{"Reviewer agent"} contracts --> review chaos --> review review -- "blocker: revise with feedback" --> stubs review -- "blocker" --> contracts review -- "blocker" --> chaos review -- "approved, or budget spent" --> coord["Coordinator"] coord --> harness[/"Harness directory"/]

1. Spec layer: loads YAML/JSON from a path or URL, validates it (problems become warnings), inlines $refs safely, and splits operations into prompt-sized chunks so large specs fit small models.
2. Test data agent runs first: the stubs and tests share its records.
3. Stub, contract, and chaos agents run in parallel.
4. Reviewer judges each artifact. Blockers send it back to its author with the feedback, for a bounded number of rounds; suggestions are kept as notes.
5. Coordinator assembles the harness and its README, including anything the reviewer still rejected when the revision budget ran out.

Every agent follows the same pattern, which is what keeps small local models reliable:

sequenceDiagram

participant A as Agent participant M as LLM participant C as Code checks A->>M: task + JSON schema (constrained decoding on Ollama) M-->>A: proposal (bodies, cases, scenarios, records) A->>C: validate against the spec alt problems found C-->>A: exact errors, e.g. "listPets 200: 'id' is a required property" A->>M: fix these problems M-->>A: corrected proposal end A->>A: keep valid items, fall back to deterministic ones for the rest

Things code guarantees, so the model never has to:

• Request matching, URL patterns, priorities, and path-parameter types in the stubs.
• Response bodies valid against their schemas.
• Test inputs that route (valid path parameters, required query parameters), success cases that use ids that exist, 404 cases that use ids that do not.
• Authentication from the spec's security schemes.
• Chaos scenarios that are internally consistent (a "flaky" failure must recover, a rate limit is a 429, a partial outage leaves some endpoints up).
• Reviewer findings that cite real items, and blockers only on what the author controls.

Install

pip install api-shakedown              # local models via Ollama; includes the MCP server

pip install "api-shakedown[cloud]" # plus OpenAI and Anthropic

Or use the Docker image, which includes everything: see Docker.

Generated harnesses need Docker (for WireMock) and Python for their tests.

Usage

shakedown doctor                     Check the LLM provider is reachable and ready

shakedown inspect SPEC [--json] Show the operations the agents will see shakedown run SPEC [options] Generate a harness shakedown version

shakedown run options:

| --no-review | review on | Skip the revie