Model Finder

Find the cheapest AI model that passes your own tests, and keep it that way.

Claude · Gemini · OpenAI · Bedrock · Vertex AI · local models — published by Indigloo

Install · How it works · MCP tools · Quick start · Privacy · Security

[!NOTE]

Runs locally. Model Finder is a set of four Agent Skills plus a local MCP server (mcp/server.py, Python 3.9+ standard library, no dependencies). The server makes no network calls, needs no API keys, never writes files and never sends anything to Indigloo. Only the eval script and the router it generates call model providers, with your own keys, when you run them.

How it works

flowchart LR

A["Audit models and SDKs in your repo "] --> B["Draft evals from your prompts, PII dropped "] B --> C["Evaluate your keys · pass rate · latency · cost "] C --> D["Rank cost per passing answer "] D --> E["Route cheapest first, escalate on failure "] F["Size weights + KV cache vs your GPU "] -.-> C classDef step fill:#f3f0e8,stroke:#1c1b19,color:#1c1b19; classDef pick fill:#f3f0e8,stroke:#cd462a,color:#cd462a; class A,B,C,E,F step; class D pick;

Advisory only: nothing is deployed, your active model is never changed, and data goes only to providers you configure and approve. Personal data and secrets are blocked before sending by default; see DATA-PROTECTION.md for how this maps to India's DPDP Act.

Install

Requires Python 3.9+ on your PATH as python3 (see Windows). Nothing to pip install.

Gemini CLI only starts extension MCP servers in folders you have trusted; it asks the first time you open a project.

Manual MCP setup

VS Code — .vscode/mcp.json (top-level key is servers )

{

"servers": { "model-finder": { "type": "stdio", "command": "python3", "args": ["/absolute/path/to/EvalHarness/mcp/server.py"], "env": { "MODEL_FINDER_ROOT": "${workspaceFolder}" } } } }

</details>

Cursor — .cursor/mcp.json in your project, or ~/.cursor/mcp.json

{

"mcpServers": { "model-finder": { "command": "python3", "args": ["/absolute/path/to/EvalHarness/mcp/server.py"], "env": { "MODEL_FINDER_ROOT": "${workspaceFolder}" } } } }

</details>

JetBrains, Windsurf, Claude Desktop and others — mcpServers

If the client has no workspace variable, write the project path:

{

"mcpServers": { "model-finder": { "command": "python3", "args": ["/absolute/path/to/EvalHarness/mcp/server.py"], "env": { "MODEL_FINDER_ROOT": "/absolute/path/to/your/project" } } } }

</details>

Clone the repo once and replace /absolute/path/to/EvalHarness. Repository tools read only inside the workspace: the roots your client shares over MCP, else MODEL_FINDER_ROOT, else the server's working directory. A filesystem root or home directory is refused.

Windows

Codex, Copilot CLI and Cursor start the server as ./mcp/server.py, which relies on its #!/usr/bin/env python3 line; on Windows use the manual setup or uvx model-finder-mcp. If python3 is not on your PATH, the MCP server will not start. Install Python from python.org (tick "Add python.exe to PATH") and either create a python3 alias or use the manual setup with "command": "py" and "args": ["-3", "C:/path/to/EvalHarness/mcp/server.py"].

MCP tools

Ten read-only tools, so the same features work as tools in every supported IDE:

Every tool returns a readable markdown summary by default and the full result as structured JSON (response_format: "json" puts the JSON in the text too). Large results are paginated (limit, offset, has_more, next_offset), modelfinder_audit_models can filter by provider family or hard-coded IDs, and unknown arguments are rejected with the list of valid ones. The server never opens a network connection, never reads API keys and never writes files. Only eval_models.py and the generated router.py call model providers, with your keys, when you run them.

Quick start

In your IDE, ask: *"Audit which AI models this repo uses, then draft evals from our prompts and test whether a cheaper model passes."* The assistant uses the MCP tools to audit and draft, agrees checks with you, and asks before the paid run.

From a terminal:

cp skills/eval-models/assets/example-config.json my-eval.json   # fill in verified model IDs and prices

export ANTHROPIC_API_KEY=... GEMINI_API_KEY=... OPENAI_API_KEY=... python3 skills/eval-models/scripts/repo_scan.py audit . # what the repo uses python3 skills/eval-models/scripts/repo_scan.py examples . # draft examples from prompt files python3 skills/eval-models/scripts/eval_models.py my-eval.json --dry-run python3 skills/eval-models/scripts/eval_models.py my-eval.json --out results python3 skills/route-models/scripts/make_routing.py --results results/results.json --guards guards.json --out routing python3 routing/router.py routing/routing.json "your prompt"

Size a local model:

python3 skills/size-local-model/scripts/estimate_memory.py --params-b 8.03 --bits 16 \

--layers 32 --kv-heads 8 --head-dim 128 --context 8192 --available-gib 24

Tests

python3 -m unittest discover -s tests -v

End-to-end tests run against a local mock of the OpenAI-compatible, Anthropic, and Gemini APIs, and drive the MCP server over stdio; no key or network access is needed. evaluations/ holds a 10-question agent evaluation of the MCP tools.

Listed on

Also on Glama, MCPNav, Socket, PyPI and the MCP Registry.

Sponsor

Model Finder i