EvalForge Lite

Name: EvalForge Lite
Author: thejaredchapman

作者：thejaredchapman版本 1.0.1i★ –i在你的电脑上运行

注册名称io.github.thejaredchapman/evalforge-lite

安装

概览

作者提供的说明（英文原文）

Compare text LLMs across OpenRouter, Bedrock, Vertex AI and Foundry with automated grading.

PopNx MCP 的三大支柱

这三项永远不会合并成一个分数：意见、测量和人气是不同的东西。

用户评价i暂无评价评价功能将在后续阶段与账户功能一起开放
我们的基础检查i1 项中通过 1 项依据公开信息检查；尚未扫描代码
使用情况i尚未统计★ –

它能帮你做什么

我们还没有为它写通俗的介绍。下面是我们从注册中心读取到的事实。

连接方式: 在你的电脑上运行
传输方式: stdio
分类: 其他
你需要填写：OPENROUTER_API_KEY

安装

选择你使用的 AI 应用，然后复制设置。

安装之前

基础检查：1 项中通过 1 项
在你的电脑上运行，拥有你的用户账户的权限（文件、网络、应用）。
需要你自己填写： OPENROUTER_API_KEY

标为机密的值是你账户的钥匙。千万不要分享给别人，也不要粘贴到聊天里。

你需要

PyPI已安装 Python 的 uv（用于 uvx 命令）

你用的是哪个应用？

ClaudeClaude CodeCursorVS CodeChatGPT

在上方选择你的应用。我们会记住它，下次按钮会显示“安装到 …”。

打开 Claude Desktop → Settings → Developer → Edit Config，把下面的内容粘贴到 claude_desktop_config.json

{
  "mcpServers": {
    "evalforge-lite": {
      "command": "uvx",
      "args": [
        "evalforge-lite"
      ],
      "env": {
        "OPENROUTER_API_KEY": "your-value"
      }
    }
  }
}

在终端中运行这条命令

claude mcp add evalforge-lite --env OPENROUTER_API_KEY=your-value -- uvx evalforge-lite

运行前请先填写空着的值。

在 Cursor 中打开

或者自己复制设置

粘贴到项目中的 .cursor/mcp.json，或 ~/.cursor/mcp.json

{
  "mcpServers": {
    "evalforge-lite": {
      "command": "uvx",
      "args": [
        "evalforge-lite"
      ],
      "env": {
        "OPENROUTER_API_KEY": "your-value"
      }
    }
  }
}

浏览器会先询问是否打开应用，然后应用会显示设置并请你确认。密钥的值会留空，由你自己填写。

在 VS Code 中打开

或者自己复制设置

粘贴到项目中的 .vscode/mcp.json

{
  "servers": {
    "evalforge-lite": {
      "type": "stdio",
      "command": "uvx",
      "args": [
        "evalforge-lite"
      ],
      "env": {
        "OPENROUTER_API_KEY": "your-value"
      }
    }
  }
}

浏览器会先询问是否打开应用，然后应用会显示设置并请你确认。密钥的值会留空，由你自己填写。

这个服务器在你的电脑上运行，而 ChatGPT 只能连接互联网上的服务器。请选择其他应用。

你的选择通过 Cookie 保存在这台设备上，无需账户。

需要你填写的值

OPENROUTER_API_KEY 机密 env
Optional. OpenRouter API key held by the server so tool calls need no credentials for OpenRouter. If unset, pass creds in each tool call.

这些设置是根据作者在注册中心的条目生成的。如果 README 的写法不同，请以 README 为准。

它能做什么

它在你的电脑上运行；安装后，工具列表会显示在你的 AI 应用中。

基础检查i

我们检查的是任何人都能看到的信息；我们没有阅读代码。不适用的检查项不计入总数。 · 检查于 2026年11月7日

✓ 通过
作者与仓库一致
注册名称已通过 GitHub 账户 thejaredchapman 验证，该账户拥有此仓库
… 尚未检查
仍在维护
我们还没有从 GitHub 读取这个仓库
… 尚未检查
有许可证
我们还没有从 GitHub 读取这个仓库
– 不适用
端点有响应
它在你的电脑上运行，没有互联网端点
– 不适用
使用 OAuth 登录
在你的电脑上运行，改用本地设置
… 尚未检查
工具描述中没有隐藏指令
它在你的电脑上运行；我们只能读取互联网端点的工具列表

与“其他”中的其他服务器比较i

星标数多于此分类中其他 32727 个服务器的 0%
发布到注册中心的时间比此分类中其他 32727 个服务器的 52% 更新
基础检查结果等于或优于此分类中其他 32727 个服务器的 100%

与注册中心里这个分类中所有活跃的服务器比较。只有当发布者拥有该仓库时，星标数才计入。

常见问题

EvalForge Lite 安全吗？

它在 1 项适用的基础检查中通过了 1 项。我们没有阅读代码。在添加密钥或令牌之前，请先看看这个服务器要求哪些权限，并使用只具备所需权限的密钥。

它需要哪些密钥或代码？

你需要填写：OPENROUTER_API_KEY

它在我的电脑上运行还是在云端？

它在你的电脑上运行。你的 AI 应用会在需要时启动它，数据不会经过作者的服务器（除非它自己调用在线服务）。

它还在维护吗？

GitHub 仓库最后更新于–，最新版本（1.0.1）于37天前进入注册中心。

哪些应用可以使用它？

支持 MCP 的应用，例如 Claude Desktop、Claude Code、Cursor 和 VS Code。你的电脑上还需要有 Python (uv)。

有什么可以替代它？

同一分类中有：Spec Driven Development, Remote Desktop Commander, Foliosio

替代选择

同一分类中其他作者的服务器，按星标数排序。

如何看卡片：★ GitHub 星标i三大支柱：评价 · 检查 · 使用情况i更新 = 最新版本发布到注册中心的时间i

Spec Driven Development

作者：formulahendry

MCP Server that facilitates spec-driven development workflows, not just Vibe Coding.

在你的电脑上运行★ 440MIT14个月前更新

安装

Remote Desktop Commander

作者：desktop-commander

Remote Desktop Commander: use files, terminal and processes on your own computers from any AI app.

在云端运行★ 211前天更新

安装

Foliosio

作者：foliosio.com

Read your support inbox, tag conversations and save draft replies from your AI agent. Never sends.

在云端运行今天更新

安装

作者的 README

打开 README

按原文从作者的仓库复制。PopNx MCP 没有审查或背书其内容。HTML、图片和脚本已被移除。链接会离开本站。

EvalForge Lite

Compare text LLMs side by side. Write a few test prompts, pick up to four models across OpenRouter, Amazon Bedrock, Google Vertex AI and Microsoft Foundry, and EvalForge Lite sends the same prompts to all of them, scores the answers automatically, and shows a leaderboard with letter grades, response time, speed and estimated cost. Use it as a web app or as an MCP server for Claude and other assistants. You bring your own credentials, or the person hosting it keeps them on the server for you.

Documentation site: https://thejaredchapman.github.io/evalforge-lite/

Why use it

Test on your own prompts. Choose models from evidence about your tasks, not a generic benchmark.
Up to 4 models per run, across 4 backends. X (OpenRouter) and X@bedrock are separate targets, so you can check one model on two platforms in a single run.
Automatic grading. A judge model scores each answer against your rubric, and rule checks (contains, regex, json_valid, max_length, available through the API and MCP) add a pass or fail. You get a 0-100 score and a letter grade.
A second opinion on every response. Each answer is also evaluated on six criteria (answered, quality, instruction following, completeness, helpfulness, safety) with strengths and weaknesses written out.
Speed and cost beside quality. Latency, tokens per second and estimated cost for every model, and a "What matters most?" selector that moves the "Best for ..." badge without a new run.
Policy gate. Upload a company policy and prompts that violate it are blocked before any model is called. If the check itself fails, the prompt is blocked.
Reports. Download a PDF or a CSV for any of your last five runs.
No accounts, no database. Credentials are used for one request and not stored. Nothing is written to disk.

Quick start

1. Run the web app on your computer

Requires Python 3.10 or newer (3.12 recommended; download from https://www.python.org/downloads/) and git.

git clone https://github.com/thejaredchapman/evalforge-lite.git cd evalforge-lite python3.12 -m venv venv source venv/bin/activate pip install -r requirements.txt python app.py

Open http://localhost:8000, paste a key for at least one backend (an OpenRouter API key is the quickest: https://openrouter.ai/workspaces/default/keys), add a test case, pick two to four models, and click Run comparison. Runs are limited to 3 per 8 hours per browser session. Full walkthrough: Getting started.

2. Use it from Claude (MCP server)

With uv installed:

uvx evalforge-lite

Add it to Claude Code in one line:

claude mcp add evalforge-lite -- uvx evalforge-lite

Or install the Claude Code plugin, which bundles the same server:

claude plugin marketplace add thejaredchapman/evalforge-lite claude plugin install evalforge-lite@evalforge

Then ask your assistant to compare models. It gets 9 tools: list_models, suggest_models, list_availability, set_policy, evaluate_prompt, run_comparison, list_runs, get_report, get_report_csv. Details, Claude Desktop config and credential shapes: MCP server.

3. Host it for other people

Deploy with the included render.yaml (gunicorn, one worker) or any host that can run gunicorn --workers 1 --threads 4 --bind 0.0.0.0:$PORT app:app. By default every visitor supplies their own key. Optionally keep provider keys on the server with environment variables and a shared daily cap (50 per 24 hours by default). Keep it at one worker: all state is in memory per process. Full guide: Hosting and server-side keys.

Good to know

The app does not read .env by itself. To use values from it, run set -a; source .env; set +a before python app.py.
Each run is limited to 4 models, and each browser session gets 3 runs per 8 hours.
All state lives in memory and is cleared when the server restarts. See Privacy and limits.
Upgrading from an older version and reading the CSV or API fields? See the notes in Troubleshooting and FAQ.

Backends and credentials

Backend	What you provide
OpenRouter	One API key
Amazon Bedrock	A region, plus a Bedrock API key or AWS access keys (optional session token)
Google Vertex AI	A project id and region, plus an access token or service-account JSON
Microsoft Foundry	A resource name and region, plus an API key or Entra ID access token

A separate judge backend setting chooses where the judge and policy gate run. Bedrock, Vertex and Foundry costs are estimates from catalog prices, not your cloud bill. See Backends and credentials.

Documentation

Page	What is in it
Overview	What it is, who it is for, the three ways to use it
Getting started	Install, run, and your first comparison
Web app guide	Every part of the screen, in order
Comparing models	Reading metrics, grades, evaluation, cost and their limits
Backends and credentials	Keys, regions, `X@backend` targets
MCP server	Install paths, all 9 tools, example prompts
Hosting and server-side keys	Deploying for others, operator-held keys, daily cap
Troubleshooting and FAQ	Common messages, fixes, and notes on CSV/API field changes
Privacy and limits	What data goes where, what is stored, every limit

Test

pytest tests/ -v

Every model and HTTP call is mocked, so the suite needs no API key and makes no network calls.

Contributing

Contributions are welcome: bug reports, model-catalog updates, new checks, docs, and new backends. See CONTRIBUTING.md for setup, tests and the pull request process. When the app shows an error, the popup's Report an issue on GitHub button opens a pre-filled bug report.

License

MIT

EvalForge Lite

概览

PopNx MCP 的三大支柱

它能帮你做什么

安装

安装之前

需要你填写的值

它能做什么

基础检查i

作者与仓库一致

仍在维护

有许可证

端点有响应

使用 OAuth 登录

工具描述中没有隐藏指令

与“其他”中的其他服务器比较i

常见问题

替代选择

Spec Driven Development

Remote Desktop Commander

Foliosio

作者的 README

EvalForge Lite

Why use it

Quick start

1. Run the web app on your computer

2. Use it from Claude (MCP server)

3. Host it for other people

Good to know

Backends and credentials

Documentation

Test

Contributing

License