ModelLens

ModelLens

See which of your models actually works.

An MCP server that turns any OpenAI-compatible gateway into an observable, rankable set of models. It discovers what a provider advertises, probes what works *right now* — latency, success, cancellation, streaming, tool calling — keeps the history, and answers "which model should I use?"

Zero third-party dependencies. Python 3.9+ standard library only.

Glama PyPI

---

Why

If you run more than one model provider, you already have this problem:

• The gateway says it serves 89 models. How many actually answer?
• Which one is fast *today*? Yesterday's answer is already stale.
• Your router picks the first entry in a list — and the first entry is slow.
• Failures are invisible until a user complains.

ModelLens makes that visible. It is the difference between trusting a model list and measuring it.

Real example from a live gateway (89 models, 16 probed):

magpie/kilo/nvidia/nemotron-3.5-super-vl:free     97.6  100.0%    0.0%      628ms   healthy

magpie/auto/beam-free 96.6 100.0% 0.0% 909ms healthy magpie/longcat/LongCat-2.5 88.4 100.0% 0.0% 3104ms mixed magpie/group/daily 70.0 100.0% 0.0% 77345ms mixed magpie/radeon-cloud/DeepSeek-V4.1-Flash 70.0 100.0% 0.0% 87388ms mixed magpie/nvidia/deepseek-ai/deepseek-v4.1-flash 0.0 0.0% 100.0% - unreliable magpie/modelscope/Qwen/Qwen3.8-27B 0.0 0.0% 0.0% - quota_blocked

The default routing group was putting an 87-second model first and a 0.9-second model third. ModelLens found that in one pass.

---

Install

# from PyPI

pip install modellens-mcp modellens --config config.json --data history.json --once ranking

or straight from source — no dependencies, no install step:

curl -fsSL https://raw.githubusercontent.com/Greatbeing/modellens/main/modellens.py -o modellens.py

python3 modellens.py --config config.json --data history.json --once ranking

Configure

{

"endpoints": [ {"name": "magpie", "base_url": "http://127.0.0.1:3425/v1", "api_key": ""}, {"name": "openai", "base_url": "https://api.openai.com/v1", "api_key": "sk-..."}, {"name": "ollama", "base_url": "http://127.0.0.1:11434/v1"} ] }

api_key may be empty for a localhost gateway. models optionally filters the advertised list; enabled: false disables an endpoint without deleting it.

Run as an MCP server

Point any MCP client at it:

{

"mcpServers": { "modellens": { "command": "python3", "args": ["/path/to/modellens.py", "--config", "/path/to/config.json", "--data", "/path/to/history.json"] } } }

Works with Claude Desktop, Cursor, Windsurf, VS Code, and any other MCP client.

DeepSeek Harness

Register it in the profile's cordis.patch.yml (note: a new entry needs - insert:, not a bare - id: — a bare one means "modify an existing entry"):

- insert:

• id: modellens name: "@deepseek-ai/dsh-mcp-client" config: serverName: "modellens" transport: "stdio" command: "python3" args:
• "/data/workspace/modellens/modellens.py"
• "--config"
• "/data/workspace/modellens/config.json"
• "--data"
• "/data/workspace/modellens/history.json"

Then restart dsh web. Verified: the server runs as a child process of dsh-web (pgrep -P $(systemctl --user show dsh-web -p MainPID)).

Tools

Example

probe(limit=20, check_tools=true)

ranking(limit=10) recommend(profile="fast")

---

How it scores

Every probe is a real request, so the numbers are evidence, not estimates.

Verdicts: healthy (≥95% success, <3 s median) · mixed · unreliable (≥30% cancelled) · quota_blocked (all 429) · dead (all 404) · untested.

History is kept per model (newest 200 probes) in history.json, so rankings improve as you use it.

---

CLI

The server doubles as a command-line tool:

python3 modellens.py --config config.json --data history.json --once probe --limit 20

python3 modellens.py --config config.json --data history.json --once ranking

---

Use cases

• Before you pick a model — recommend(profile="fast")
• When a provider goes quiet — ranking shows it dropping before your users do
• When your router misbehaves — compare group members, find the slow one
• Cost control — spot the expensive model that is never chosen
• Onboarding a new provider — probe once, see what actually works

---

Status and roadmap

Working today: discovery, probing, streaming/tool-call checks, scoring, ranking, recommendation, markdown report, MCP stdio transport, CLI.

Roadmap: OpenAI-compatible /v1/models only for now; Anthropic and Gemini transports, per-endpoint cost accounting, alerting webhooks, and a hosted multi-team version.

---

License

MIT. Use it, ship it, charge for your own hosted version.

---

Honest notes

• Probing costs tokens. Each probe is a real completion. Keep limit sensible and don't probe a hundred models every minute.
• A single probe is weak evidence. The score is meaningful after several probes; the report says n=1 so you can see the sample size.
• Latency depends on you. Your network, not just the provider, is in the measurement. Run it from where your agents run.