Judges Panel

Judges Panel

An MCP (Model Context Protocol) server that provides a panel of 45 specialized judges to evaluate AI-generated code — acting as an independent quality gate regardless of which project is being reviewed. Combines deterministic pattern matching & AST analysis (instant, offline, zero LLM calls) with LLM-powered deep-review prompts that let your AI assistant perform expert-persona analysis across all 45 domains.

Highlights:

• Includes an App Builder Workflow (3-step) demo for release decisions, plain-language risk summaries, and prioritized fixes — see Try the Demo.
• Includes V2 context-aware evaluation with policy profiles, evidence calibration, specialty feedback, confidence scoring, and uncertainty reporting.
• Includes public repository URL reporting to clone a repo, run the full tribunal, and output a consolidated markdown report.
• 200+ deterministic auto-fix patches (see src/patches/index.ts) plus LLM-powered deep review.

🧪 Many commands in printHelp are experimental/roadmap. By default, we show GA commands only. Set JUDGES_SHOW_EXPERIMENTAL=1 to reveal stubs; these may not be wired yet.

🔰 Packages

- CLI: @kevinrabun/judges-cli → binary judges (use npx @kevinrabun/judges-cli eval --file app.ts).

- MCP/API: @kevinrabun/judges → programmatic API + MCP server (npm install @kevinrabun/judges).

- VS Code extension: see vscode-extension/.

- GitHub Action: uses: KevinRabun/judges@main (see CI quickstart).

---

Quickstart

CLI (one-off)

# Using the CLI package (recommended)

npx @kevinrabun/judges-cli eval --file src/app.ts

Show GA commands only (default)

npx @kevinrabun/judges-cli --help

Show experimental/roadmap commands

echo "JUDGES_SHOW_EXPERIMENTAL=1" >> $GITHUB_ENV npx @kevinrabun/judges-cli --help

License scan (supply-chain & license compliance)

npx @kevinrabun/judges-cli license-scan --dir .

CLI vs API: If you want to embed Judges in your app (MCP/API), install @kevinrabun/judges. For the command-line, use @kevinrabun/judges-cli (binary judges).

GitHub Action

name: Judges

on: [pull_request, push] jobs: judges: runs-on: ubuntu-latest steps:

• uses: actions/checkout@v4
• uses: KevinRabun/judges@main with: path: . diff-only: true # evaluate only changed lines in PRs (default true) fail-on-findings: true # fail on critical/high findings upload-sarif: true # upload SARIF to GitHub Code Scanning

Programmatic API (MCP server included)

npm install @kevinrabun/judges

import { evaluateCode } from "@kevinrabun/judges/api";

const verdict = evaluateCode("const password = 'ProdSecret';", "typescript"); console.log(verdict.overallVerdict, verdict.overallScore);

MCP server

The MCP server runs on stdio and is started by your MCP client (VS Code, Claude Desktop, etc.). Configure it in your MCP settings (e.g. mcp.json):

{

"servers": { "judges": { "type": "stdio", "command": "npx", "args": ["-y", "@kevinrabun/judges"] } } }

Or run the server directly:

npx @kevinrabun/judges

Starts the MCP server on stdio

Config file: .judgesrc.json (supports ${ENV_VAR} substitution via expandEnvPlaceholders). See Configuration.

---

Why Judges?

AI code generators (Copilot, Cursor, Claude, ChatGPT, etc.) write code fast — but they routinely produce insecure defaults, missing auth, hardcoded secrets, and poor error handling. Human reviewers catch some of this, but nobody reviews 45 dimensions consistently.

Judges doesn't replace linters — it covers the dimensions linters don't: authentication strategy, data sovereignty, cost patterns, accessibility, framework-specific anti-patterns, and architectural issues across multiple files.

---

Quick Start

Prereqs: Node.js >=18 (>=20 recommended), npx available. The judges CLI binary ships with @kevinrabun/judges-cli (preferred) and also works via npx @kevinrabun/judges.

Packages:

- CLI: npm install -g @kevinrabun/judges-cli (or npx @kevinrabun/judges-cli ...)

- MCP/API: npm install @kevinrabun/judges

Use @kevinrabun/judges for the MCP server and programmatic API. Use @kevinrabun/judges-cli when you want the judges terminal command.

Try it now (no clone needed)

# Install the CLI globally

npm install -g @kevinrabun/judges-cli

Evaluate any file

judges eval src/app.ts

Pipe from stdin

cat api.py | judges eval --language python

Single judge

judges eval --judge cybersecurity server.ts

SARIF output for CI

judges eval --file app.ts --format sarif > results.sarif

HTML report with severity filters and dark/light theme

judges eval --file app.ts --format html > report.html

Fail CI on findings (exit code 1)

judges eval --fail-on-findings src/api.ts

Suppress known findings via baseline

judges eval --baseline baseline.json src/api.ts

Use a named preset

judges eval --preset security-only src/api.ts

Use a config file

judges eval --config .judgesrc.json src/api.ts

Set a minimum score threshold (exit 1 if below)

judges eval --min-score 80 src/api.ts

One-line summary for scripts

judges eval --summary src/api.ts

Agentic skills (orchestrated judge sets)

judges skill ai-code-review --file src/app.ts judges skill security-review --file src/api.ts --format json judges skill release-gate --file src/app.ts judges skills # list available skills

Full catalog: docs/skills.md

List all 45 judges

judges list

Additional CLI Commands

# Interactive project setup wizard

judges init

Preview auto-fix patches (dry run)

judges fix src/app.ts

Apply patches directly

judges fix src/app.ts --apply

License compliance scan (copyleft/unknown detection)

judges license-scan --format json --risk high

Watch mode — re-evaluate on file save

judges watch src/

Project-level report (local directory)

judges report . --format html --output report.html

Evaluate a unified diff (pipe from git diff)

git diff HEAD~1 | judges diff

Analyze dependencies for supply-chain risks

judges deps --path . --format json

Run GitHub App server (zero-config PR reviews)

judges app serve --port 4567

Run GitHub PR review (gh CLI required)

judges review --pr 123 --repo owner/name --diff-only

Auto-tune presets and configs

judges tune --dir . --apply

Create a baseline file to suppress known findings

judges baseline create --file src/api.ts -o baseline.json

Generate CI template files

judges ci-templates --provider github judges ci-templates --provider gitlab judges ci-templates --provider azure judges ci-templates --provider bitbucket

Generate per-judge rule documentation

judges docs judges docs --judge cybersecurity judges docs --output docs/

Install shell completions

judges completions bash # eval "$(judges completions bash)" judges completions zsh judges completions fish judges completions powershell

Install pre-commit hook

judges hook install

Uninstall pre-commit hook

judges hook uninstall

🔎 Tip: The CLI help now defaults to GA commands only. To see experimental/roadmap commands, run:

`bash

JUDGES_SHOW_EXPERIMENTAL=1 judges --help

`

GitHub App (self-hosted webhook)

Run a zero-config PR reviewer as a GitHub App:

# Run the webhook server locally

judges app serve --port 4567

Required env vars:

• JUDGES_APP_ID – GitHub App ID
• JUDGES_PRIVATE_KEY or JUDGES_PRIVATE_KEY_PATH – PEM private key
• JUDGES_WEBHOOK_SECRET – signature verification secret

Optional:

• JUDGES_MIN_SEVERITY (default: medium)
• JUDGES_MAX_COMMENTS (default: 25)
• JUDGES_TEST_DRY_RUN=1 to avoid live network calls during tests

For local testing, you can expose http://localhost:4567/webhook via ngrok http 4567 and configure the GitHub App webhook URL accordingly.

Use in GitHub Actions

Add Judges to your CI pipeline with zero configuration:

# .github/workflows/judges.yml

name: Judges Code Review on: [pull_request]

jobs: judges: runs-on: ubuntu-latest permissions: contents: read security-events: write # only if using upload-sarif steps:

• uses: actions/checkout@v4
• uses: KevinRabun/judges@main with: path: src/api.ts # file or directory format: text # text | json | sarif | markdown upload-sarif: true # upload to GitHub Code Scanning fail-on-findings: true # fail CI on critical/high findings

Outputs available for downstream steps: verdict, score, findings, critical, high, sarif-file.

Use with Docker (no Node.js required)

# Build the image

docker build -t judges .

Evaluate a local file

docker run --rm -v $(pwd):/code judges eval --file /code/app.ts

Pipe from stdin

cat api.py | docker run --rm -i judges eval --language python

List judges

docker run --rm judges list

Or use as an MCP server

1. Install and Build

git clone https://github.com/KevinRabun/judges.git

cd judges npm install npm run build

2. Try the Demo

Run the included demo to see all 45 judges evaluate a purposely flawed API server:

npm run demo

This evaluates examples/sample-vulnerable-api.ts — a file intentionally packed with security holes, performance anti-patterns, and code quality issues — and prints a full verdict with per-judge scores and findings.

The demo now also includes an App Builder Workflow (3-step) section. In a single run, you get both tribunal output and workflow output:

• Release decision (Ship now / Ship with caution / Do not ship)
• Plain-language summaries of top risks
• Prioritized remediation tasks and AI-fixable P0/P1 items

Sample workflow output (truncated):

╔══════════════════════════════════════════════════════════════╗

║ App Builder Workflow Demo (3-Step) ║ ╚══════════════════════════════════════════════════════════════╝

Decision : Do not ship Verdict : FAIL (47/100) Risk Counts : Critical