Llmtest MCP

Name: Llmtest MCP
Author: tjacquesson

by tjacquessonversion 0.6.1i★ –iRuns on your computer

Registry nameio.github.tjacquesson/llmtest-mcp

Install

Overview

Description from the maker (original English)

Benchmark AI models on real prompts. Find cheaper, faster alternatives across 340+ models.

PopNx MCP’s three pillars

These three are never merged into one score: opinions, measurements and popularity are different things.

User voiceiNo reviews yetReviews open together with accounts in a later step
Our basic checksi1 of 1 passedChecked from public information; code not scanned yet
UsageiNot measured yet★ –

What you can do with it

We have not written a plain-language summary for this one yet. Below are the facts we read from the registry.

Connection: Runs on your computer
Transport: stdio
Category: Other
You need to fill in: LLMTEST_API_KEY

Install

Pick the AI app you use, then copy the settings.

Before you install

Basic checks: 1 of 1 passed
Runs on your computer with your user’s permissions (files, network, apps).
You fill in yourself: LLMTEST_API_KEY

Values marked secret are keys to your account. Never share them or paste them into a chat.

You need

npmNode.js installed (for the npx command)

Which app do you use?

ClaudeClaude CodeCursorVS CodeChatGPT

Pick your app above. We remember it, so next time the button says “Install in …”.

Open Claude Desktop → Settings → Developer → Edit Config, and paste this into claude_desktop_config.json

{
  "mcpServers": {
    "llmtest": {
      "command": "npx",
      "args": [
        "-y",
        "llmtest-mcp"
      ],
      "env": {
        "LLMTEST_API_KEY": "your-value"
      }
    }
  }
}

Run this in your terminal

claude mcp add llmtest --env LLMTEST_API_KEY=your-value -- npx -y llmtest-mcp

Fill in the empty values before running it.

Open in Cursor

Or copy the settings yourself

Paste into .cursor/mcp.json in your project, or ~/.cursor/mcp.json

{
  "mcpServers": {
    "llmtest": {
      "command": "npx",
      "args": [
        "-y",
        "llmtest-mcp"
      ],
      "env": {
        "LLMTEST_API_KEY": "your-value"
      }
    }
  }
}

Your browser asks to open the app, then the app shows the settings and asks you to confirm. Key values are left empty for you to fill in.

Open in VS Code

Or copy the settings yourself

Paste into .vscode/mcp.json in your project

{
  "servers": {
    "llmtest": {
      "type": "stdio",
      "command": "npx",
      "args": [
        "-y",
        "llmtest-mcp"
      ],
      "env": {
        "LLMTEST_API_KEY": "your-value"
      }
    }
  }
}

Your browser asks to open the app, then the app shows the settings and asks you to confirm. Key values are left empty for you to fill in.

This one runs on your computer, and ChatGPT can only connect to servers on the internet. Pick another app.

Your choice stays on this device in a cookie; no account needed.

Values you fill in

LLMTEST_API_KEY secret required env
Your LLMTest API key from https://llmtest.io/dashboard

These settings are generated from the maker’s registry entry. If the README says otherwise, follow the README.

What it can do

Runs on your computer; the tool list appears in your AI app after you install it.

Basic checksi

We check information anyone can see; we have not read the code. Checks that do not apply are left out of the count. · Checked Nov 7, 2026

✓ Pass
Maker matches the repo
The registry name was verified with the GitHub account tjacquesson, which owns the repo
… Not checked yet
Still maintained
We have not read this repo from GitHub yet
… Not checked yet
Has a licence
We have not read this repo from GitHub yet
– Not applicable
Endpoint responds
This one runs on your computer; there is no internet endpoint
– Not applicable
Sign-in with OAuth
Runs on your computer and uses local settings instead
… Not checked yet
No hidden instructions in tool descriptions
Runs on your computer; we can only read tool lists from internet endpoints

Compared with others in Otheri

More stars than 0% of the other 32680 servers in this category
Published to the registry more recently than 10% of the other 32680 servers in this category
Basic checks equal to or better than 100% of the other 32680 servers in this category

Compared with every active server in this category in the registry. Stars count only when the publisher owns the repo.

Common questions

Is Llmtest MCP safe?

It passed 1 of 1 applicable basic checks. We have not read the code. Before adding a key or token, see what access the server asks for and use a key with only the permissions you need.

What keys or codes does it need?

You need to fill in: LLMTEST_API_KEY

Does it run on my computer or in the cloud?

It runs on your computer. Your AI app starts it when needed, and data does not pass through the maker’s servers (unless it calls an online service itself).

Is it still maintained?

The GitHub repo last changed –, and the latest version (0.6.1) reached the registry 6 months ago.

Which apps can use it?

Apps that support MCP, such as Claude Desktop, Claude Code, Cursor and VS Code. You also need Node.js on your computer.

What can I use instead?

In the same category: Spec Driven Development, Remote Desktop Commander, EmbeddedCI BenchPod

Alternatives

Servers in the same category from other makers, sorted by stars.

Reading a card:★ GitHub starsiThree pillars: reviews · checks · usageiUpdated = latest version published to the registryi

Spec Driven Development

by formulahendry

MCP Server that facilitates spec-driven development workflows, not just Vibe Coding.

Runs on your computer★ 440MITUpdated 14 months ago

Install

Remote Desktop Commander

by desktop-commander

Remote Desktop Commander: use files, terminal and processes on your own computers from any AI app.

Runs in the cloud★ 211Updated 2 days ago

Install

EmbeddedCI BenchPod

by embeddedci-com

Drive a BenchPod hardware-in-the-loop bench: power, flash, UART, I2C, CAN, analog and logic.

Runs on your computerUpdated today

Install

README from the maker

Open the README

Copied from the maker’s repo as written. PopNx MCP has not reviewed or endorsed it. HTML, images and scripts are removed. Links lead off this site.

LLMTest MCP Server

MCP server that benchmarks AI models on your actual prompts and finds cheaper, faster alternatives. Works with Claude Code, Cursor, Windsurf, and any MCP-compatible tool.

Quick Start

1. Get your API key

2. Add to your tool

Claude Code:

claude mcp add llmtest -- npx llmtest-mcp

Then set your key:

export LLMTEST_API_KEY=llmt_your_key_here

Cursor / Windsurf / Other MCP clients:

Add to your MCP config file:

"mcpServers": { "llmtest": { "command": "npx", "args": ["llmtest-mcp"], "env": { "LLMTEST_API_KEY": "llmt_your_key_here" } } } }

3. Talk to your AI

Just ask in natural language:

"Check my LLMTest status"
"Find cheaper models for my AI calls"
"Run a benchmark on my blog-writer flow"
"What models are trending?"

How It Works

LLMTest is a proxy that sits between your app and AI providers. Point your app at https://llmtest.io/v1 instead of calling OpenAI/Anthropic directly, and LLMTest tracks your usage, benchmarks alternatives, and suggests cost savings.

This MCP server gives your AI assistant access to LLMTest's tools so it can manage everything for you.

Available Tools

Tool	Description
`status`	Show proxy status and activity summary
`list_flows`	List all AI flows with cost and latency stats
`get_suggestions`	Get pending model-switch recommendations
`update_suggestion`	Accept or dismiss a suggestion
`run_benchmark`	Benchmark a flow against challenger models
`optimize_prompt`	Rewrite a flow's prompt and find a cheaper model that still works
`seed_samples`	Add test prompts for pre-launch benchmarking
`list_samples`	Show stored test samples per flow
`list_new_models`	Show new and trending models
`get_account`	Check credit balance and usage
`get_autopilot_status`	Check whether autopilot is on and whether the account is eligible
`enable_autopilot`	Turn on weekly auto-optimization with safety gates + drift-based auto-revert
`disable_autopilot`	Turn off autopilot (existing optimizations stay active)
`list_active_optimizations`	List auto-accepted optimizations still inside their 24h revert window
`revert_optimization`	Roll an auto-accepted optimization back to the previous prompt

Autopilot

Autopilot automatically optimizes your flows on a weekly cadence. Changes that pass every safety gate go live with a 24-hour revert window. Drift detection keeps checking after that and rolls back if quality slips.

To enable from your IDE: ask your AI assistant something like "enable LLMTest autopilot". It will call enable_autopilot. Use get_autopilot_status to confirm prerequisites.

Prerequisites (checked per flow each cycle):

Autopilot enabled on the account
Email verified
Account age ≥ 14 days (trust ramp)
Flow has ≥ 20 real calls in the last 7 days
Flow not optimized by autopilot in the last 14 days (cooldown)
Positive credit balance (~$1–2 per run)

Safety gates (all must pass for auto-accept): 95% CI lower bound > 50% win rate, multi-judge agreement ≥ 80%, ≥ 20% total savings, no length-bias warning, golden-set regression check.

Revert: 24h window after auto-accept. After that, only drift detection can roll back.

Typical Workflow

Pre-launch (no traffic yet):

Tell your AI: "I'm building a support chatbot using gpt-4o"
It seeds realistic test samples with seed_samples
It runs run_benchmark to compare models
It shows you get_suggestions with cheaper alternatives

Post-launch (with real traffic):

Route your AI calls through https://llmtest.io/v1
LLMTest monitors usage and auto-benchmarks when flows hit 50+ calls
Ask "any cost-saving suggestions?" to see recommendations
Accept a suggestion and update your code

Environment Variables

Variable	Required	Description
`LLMTEST_API_KEY`	Yes	Your API key from llmtest.io/dashboard
`LLMTEST_BASE_URL`	No	Custom API URL (defaults to `https://llmtest.io`)

License

MIT

Llmtest MCP

Overview

PopNx MCP’s three pillars

What you can do with it

Install

Before you install

Values you fill in

What it can do

Basic checksi

Maker matches the repo

Still maintained

Has a licence

Endpoint responds

Sign-in with OAuth

No hidden instructions in tool descriptions

Compared with others in Otheri

Common questions

Alternatives

Spec Driven Development

Remote Desktop Commander

EmbeddedCI BenchPod

README from the maker

LLMTest MCP Server

Quick Start

1. Get your API key

2. Add to your tool

3. Talk to your AI

How It Works

Available Tools

Autopilot

Typical Workflow

Environment Variables

Links

License