EvalForge Lite

Name: EvalForge Lite
Author: thejaredchapman

by thejaredchapmanversion 1.0.1i★ –iRuns on your computer

Registry nameio.github.thejaredchapman/evalforge-lite

Install

Overview

Description from the maker (original English)

Compare text LLMs across OpenRouter, Bedrock, Vertex AI and Foundry with automated grading.

PopNx MCP’s three pillars

These three are never merged into one score: opinions, measurements and popularity are different things.

User voiceiNo reviews yetReviews open together with accounts in a later step
Our basic checksi1 of 1 passedChecked from public information; code not scanned yet
UsageiNot measured yet★ –

What you can do with it

We have not written a plain-language summary for this one yet. Below are the facts we read from the registry.

Connection: Runs on your computer
Transport: stdio
Category: Other
You need to fill in: OPENROUTER_API_KEY

Install

Pick the AI app you use, then copy the settings.

Before you install

Basic checks: 1 of 1 passed
Runs on your computer with your user’s permissions (files, network, apps).
You fill in yourself: OPENROUTER_API_KEY

Values marked secret are keys to your account. Never share them or paste them into a chat.

You need

PyPIPython’s uv installed (for the uvx command)

Which app do you use?

ClaudeClaude CodeCursorVS CodeChatGPT

Pick your app above. We remember it, so next time the button says “Install in …”.

Open Claude Desktop → Settings → Developer → Edit Config, and paste this into claude_desktop_config.json

{
  "mcpServers": {
    "evalforge-lite": {
      "command": "uvx",
      "args": [
        "evalforge-lite"
      ],
      "env": {
        "OPENROUTER_API_KEY": "your-value"
      }
    }
  }
}

Run this in your terminal

claude mcp add evalforge-lite --env OPENROUTER_API_KEY=your-value -- uvx evalforge-lite

Fill in the empty values before running it.

Open in Cursor

Or copy the settings yourself

Paste into .cursor/mcp.json in your project, or ~/.cursor/mcp.json

{
  "mcpServers": {
    "evalforge-lite": {
      "command": "uvx",
      "args": [
        "evalforge-lite"
      ],
      "env": {
        "OPENROUTER_API_KEY": "your-value"
      }
    }
  }
}

Your browser asks to open the app, then the app shows the settings and asks you to confirm. Key values are left empty for you to fill in.

Open in VS Code

Or copy the settings yourself

Paste into .vscode/mcp.json in your project

{
  "servers": {
    "evalforge-lite": {
      "type": "stdio",
      "command": "uvx",
      "args": [
        "evalforge-lite"
      ],
      "env": {
        "OPENROUTER_API_KEY": "your-value"
      }
    }
  }
}

Your browser asks to open the app, then the app shows the settings and asks you to confirm. Key values are left empty for you to fill in.

This one runs on your computer, and ChatGPT can only connect to servers on the internet. Pick another app.

Your choice stays on this device in a cookie; no account needed.

Values you fill in

OPENROUTER_API_KEY secret env
Optional. OpenRouter API key held by the server so tool calls need no credentials for OpenRouter. If unset, pass creds in each tool call.

These settings are generated from the maker’s registry entry. If the README says otherwise, follow the README.

What it can do

Runs on your computer; the tool list appears in your AI app after you install it.

Basic checksi

We check information anyone can see; we have not read the code. Checks that do not apply are left out of the count. · Checked Nov 7, 2026

✓ Pass
Maker matches the repo
The registry name was verified with the GitHub account thejaredchapman, which owns the repo
… Not checked yet
Still maintained
We have not read this repo from GitHub yet
… Not checked yet
Has a licence
We have not read this repo from GitHub yet
– Not applicable
Endpoint responds
This one runs on your computer; there is no internet endpoint
– Not applicable
Sign-in with OAuth
Runs on your computer and uses local settings instead
… Not checked yet
No hidden instructions in tool descriptions
Runs on your computer; we can only read tool lists from internet endpoints

Compared with others in Otheri

More stars than 0% of the other 32725 servers in this category
Published to the registry more recently than 52% of the other 32725 servers in this category
Basic checks equal to or better than 100% of the other 32725 servers in this category

Compared with every active server in this category in the registry. Stars count only when the publisher owns the repo.

Common questions

Is EvalForge Lite safe?

It passed 1 of 1 applicable basic checks. We have not read the code. Before adding a key or token, see what access the server asks for and use a key with only the permissions you need.

What keys or codes does it need?

You need to fill in: OPENROUTER_API_KEY

Does it run on my computer or in the cloud?

It runs on your computer. Your AI app starts it when needed, and data does not pass through the maker’s servers (unless it calls an online service itself).

Is it still maintained?

The GitHub repo last changed –, and the latest version (1.0.1) reached the registry 37 days ago.

Which apps can use it?

Apps that support MCP, such as Claude Desktop, Claude Code, Cursor and VS Code. You also need Python (uv) on your computer.

What can I use instead?

In the same category: Spec Driven Development, Remote Desktop Commander, Signoff

Alternatives

Servers in the same category from other makers, sorted by stars.

Reading a card:★ GitHub starsiThree pillars: reviews · checks · usageiUpdated = latest version published to the registryi

Spec Driven Development

by formulahendry

MCP Server that facilitates spec-driven development workflows, not just Vibe Coding.

Runs on your computer★ 440MITUpdated 14 months ago

Install

Remote Desktop Commander

by desktop-commander

Remote Desktop Commander: use files, terminal and processes on your own computers from any AI app.

Runs in the cloud★ 211Updated 2 days ago

Install

Signoff

by ayuanxx-max

Five gates that make an agent's 'done' provable: run them, read the verdict, audit the record.

Runs on your computerUpdated today

Install

README from the maker

Open the README

Copied from the maker’s repo as written. PopNx MCP has not reviewed or endorsed it. HTML, images and scripts are removed. Links lead off this site.

EvalForge Lite

Compare text LLMs side by side. Write a few test prompts, pick up to four models across OpenRouter, Amazon Bedrock, Google Vertex AI and Microsoft Foundry, and EvalForge Lite sends the same prompts to all of them, scores the answers automatically, and shows a leaderboard with letter grades, response time, speed and estimated cost. Use it as a web app or as an MCP server for Claude and other assistants. You bring your own credentials, or the person hosting it keeps them on the server for you.

Documentation site: https://thejaredchapman.github.io/evalforge-lite/

Why use it

Test on your own prompts. Choose models from evidence about your tasks, not a generic benchmark.
Up to 4 models per run, across 4 backends. X (OpenRouter) and X@bedrock are separate targets, so you can check one model on two platforms in a single run.
Automatic grading. A judge model scores each answer against your rubric, and rule checks (contains, regex, json_valid, max_length, available through the API and MCP) add a pass or fail. You get a 0-100 score and a letter grade.
A second opinion on every response. Each answer is also evaluated on six criteria (answered, quality, instruction following, completeness, helpfulness, safety) with strengths and weaknesses written out.
Speed and cost beside quality. Latency, tokens per second and estimated cost for every model, and a "What matters most?" selector that moves the "Best for ..." badge without a new run.
Policy gate. Upload a company policy and prompts that violate it are blocked before any model is called. If the check itself fails, the prompt is blocked.
Reports. Download a PDF or a CSV for any of your last five runs.
No accounts, no database. Credentials are used for one request and not stored. Nothing is written to disk.

Quick start

1. Run the web app on your computer

Requires Python 3.10 or newer (3.12 recommended; download from https://www.python.org/downloads/) and git.

git clone https://github.com/thejaredchapman/evalforge-lite.git cd evalforge-lite python3.12 -m venv venv source venv/bin/activate pip install -r requirements.txt python app.py

Open http://localhost:8000, paste a key for at least one backend (an OpenRouter API key is the quickest: https://openrouter.ai/workspaces/default/keys), add a test case, pick two to four models, and click Run comparison. Runs are limited to 3 per 8 hours per browser session. Full walkthrough: Getting started.

2. Use it from Claude (MCP server)

With uv installed:

uvx evalforge-lite

Add it to Claude Code in one line:

claude mcp add evalforge-lite -- uvx evalforge-lite

Or install the Claude Code plugin, which bundles the same server:

claude plugin marketplace add thejaredchapman/evalforge-lite claude plugin install evalforge-lite@evalforge

Then ask your assistant to compare models. It gets 9 tools: list_models, suggest_models, list_availability, set_policy, evaluate_prompt, run_comparison, list_runs, get_report, get_report_csv. Details, Claude Desktop config and credential shapes: MCP server.

3. Host it for other people

Deploy with the included render.yaml (gunicorn, one worker) or any host that can run gunicorn --workers 1 --threads 4 --bind 0.0.0.0:$PORT app:app. By default every visitor supplies their own key. Optionally keep provider keys on the server with environment variables and a shared daily cap (50 per 24 hours by default). Keep it at one worker: all state is in memory per process. Full guide: Hosting and server-side keys.

Good to know

The app does not read .env by itself. To use values from it, run set -a; source .env; set +a before python app.py.
Each run is limited to 4 models, and each browser session gets 3 runs per 8 hours.
All state lives in memory and is cleared when the server restarts. See Privacy and limits.
Upgrading from an older version and reading the CSV or API fields? See the notes in Troubleshooting and FAQ.

Backends and credentials

Backend	What you provide
OpenRouter	One API key
Amazon Bedrock	A region, plus a Bedrock API key or AWS access keys (optional session token)
Google Vertex AI	A project id and region, plus an access token or service-account JSON
Microsoft Foundry	A resource name and region, plus an API key or Entra ID access token

A separate judge backend setting chooses where the judge and policy gate run. Bedrock, Vertex and Foundry costs are estimates from catalog prices, not your cloud bill. See Backends and credentials.

Documentation

Page	What is in it
Overview	What it is, who it is for, the three ways to use it
Getting started	Install, run, and your first comparison
Web app guide	Every part of the screen, in order
Comparing models	Reading metrics, grades, evaluation, cost and their limits
Backends and credentials	Keys, regions, `X@backend` targets
MCP server	Install paths, all 9 tools, example prompts
Hosting and server-side keys	Deploying for others, operator-held keys, daily cap
Troubleshooting and FAQ	Common messages, fixes, and notes on CSV/API field changes
Privacy and limits	What data goes where, what is stored, every limit

Test

pytest tests/ -v

Every model and HTTP call is mocked, so the suite needs no API key and makes no network calls.

Contributing

Contributions are welcome: bug reports, model-catalog updates, new checks, docs, and new backends. See CONTRIBUTING.md for setup, tests and the pull request process. When the app shows an error, the popup's Report an issue on GitHub button opens a pre-filled bug report.

License

MIT

EvalForge Lite

Overview

PopNx MCP’s three pillars

What you can do with it

Install

Before you install

Values you fill in

What it can do

Basic checksi

Maker matches the repo

Still maintained

Has a licence

Endpoint responds

Sign-in with OAuth

No hidden instructions in tool descriptions

Compared with others in Otheri

Common questions

Alternatives

Spec Driven Development

Remote Desktop Commander

Signoff

README from the maker

EvalForge Lite

Why use it

Quick start

1. Run the web app on your computer

2. Use it from Claude (MCP server)

3. Host it for other people

Good to know

Backends and credentials

Documentation

Test

Contributing

License