Prototype · Preview: every server in the official registry, synced every 30 minutes. Sign in with Google to write reviews.Prototype · updated every 30 min
PopNx MCPMCP servers for people who use AI · explained in plain words

evals-skills Skill

Repository ↗

by ai-evals-course★ 1,823 (whole repo)9 skills

★ Stars are for the whole repo, not each skill

Pick the skill to install

What it does

This repo is a collection of 9 skills; it is a guide with example skills

  • Connects Instruction files the AI reads
  • Account or key No key found in README or SKILL.md
  • Category Coding
Maker’s original description
Skills that guide AI coding agents to help you build product-specific AI evals.
For makers: add a badge to your README

If you made this, paste this code into your README. The badge links back to this page.

[![Featured on PopNx MCP](https://mcp.popnx.com/badge/skill/ai-evals-course-evals-skills.svg)](https://mcp.popnx.com/en/skills/ai-evals-course-evals-skills)

Use together with

MCP servers that connect the AI to services for the same work

REA

Reverse engineer anything from your terminal or agent with one CLI and MCP server.

by morlutoWidely used↓ 388.7K/wk★ 468.8KUpdated 7 days ago

MCP · Connects a serviceRuns on your computer

Fallow

Codebase intelligence for TypeScript and JavaScript, exposed as structured MCP analysis tools.

by fallow-rsWidely used↓ 2.4M/wk★ 5.4KUpdated today

MCP · Connects a serviceRuns on your computer

Desktop Commander

MCP server for terminal commands, file operations, and process management

by wonderwhy-erWidely used↓ 740.3K/wk★ 10.4KUpdated yesterday

MCP · Connects a serviceRuns on your computer

Similar Skills

Same kind of work, ordered like the Skills list

Compared with similar ones
This one
evals-skills
reaandrej-karpathy-skillsgraphifyMoneyPrinterTurbo
Makerai-evals-coursemorlutomultica-aiGraphify-Labsharry0703
Skills91111
GitHub stars1,823468,733234,332150,287144,566
Last code changeOct 25, 2026Nov 19, 2026Apr 20, 2026Nov 18, 2026Nov 18, 2026
LicenseApache-2.0MIT–Apache-2.0MIT

9 skills (instruction sets) by ai-evals-course that an AI reads and follows · not an MCP server

Pick the skill to install

Install only the ones you use. Open a skill to see how to install it

Install the whole set (9)

If you want every one, let the AI install them in one go. Know 2 things first

  • Every installed skill’s description is loaded into every chat. The more you install, the easier it is for the AI to pick the wrong one
  • Some skills duplicate tools the AI already has built in. Installing only the ones you use works better
Where to install
Please install all 9 skills from https://github.com/ai-evals-course/evals-skills: build-review-interface, error-discovery, eval-audit, evals-start, evaluate-rag, generate-synthetic-data, validate-evaluator, write-code-eval, write-judge-prompt, for all my projects, putting each at ~/.claude/skills/<skill name> (in Codex use ~/.agents/skills/<skill name>, in Cursor ~/.cursor/skills/<skill name>). Folders in the repo: build-review-interface = skills/build-review-interface · error-discovery = skills/error-discovery · eval-audit = skills/eval-audit · evals-start = skills/evals-start · evaluate-rag = skills/evaluate-rag · generate-synthetic-data = skills/generate-synthetic-data · validate-evaluator = skills/validate-evaluator · write-code-eval = skills/write-code-eval · write-judge-prompt = skills/write-judge-prompt. First show me every file and tell me whether any has scripts or changes machine settings, then tell me the full path you will install each one to and wait for me to agree.
Please install all 9 skills from https://github.com/ai-evals-course/evals-skills: build-review-interface, error-discovery, eval-audit, evals-start, evaluate-rag, generate-synthetic-data, validate-evaluator, write-code-eval, write-judge-prompt, into the project this chat is open in (at .claude/skills/<skill name>; in Codex use .agents/skills/<skill name>, in Cursor .cursor/skills/<skill name>). Folders in the repo: build-review-interface = skills/build-review-interface · error-discovery = skills/error-discovery · eval-audit = skills/eval-audit · evals-start = skills/evals-start · evaluate-rag = skills/evaluate-rag · generate-synthetic-data = skills/generate-synthetic-data · validate-evaluator = skills/validate-evaluator · write-code-eval = skills/write-code-eval · write-judge-prompt = skills/write-judge-prompt. First show me every file and tell me whether any has scripts or changes machine settings, then tell me the project name and the full path you will install each one to and wait for me to agree.

Uploading to Claude on the web and apps: you need 9 uploads, one ZIP per skill. Open each skill in the list below to download it

Checked against official docs Nov 12, 2026 · Source: support.claude.com, code.claude.com, learn.chatgpt.com, cursor.com

  • build-review-interface

    Build a custom browser-based annotation interface tailored to your data for reviewing LLM traces and collecting structured feedback. Use when you need to build an annotation tool, review traces, or collect human labels.

  • error-discovery

    Run error analysis on a dataset. Build a review UI, select diverse samples, monitor annotations, and organize failure modes.

  • eval-audit

    Audit an LLM eval pipeline and surface problems: missing error analysis, unvalidated judges, vanity metrics, etc. Use when inheriting an eval system, when unsure whether evals are trustworthy, or as a starting point when no eval infrastructure exists. Do NOT use when the goal is to build a new eval…

  • evals-start

    Entry point for evals. Use when the user asks for help with evals, does not know where to begin, or asks for something no other skill in this plugin matches. Do NOT use when a more specific skill in this plugin already matches; load that skill directly.

  • evaluate-rag

    Guides evaluation of RAG pipeline retrieval and generation quality. Use when evaluating a retrieval-augmented generation system, measuring retrieval quality, assessing generation faithfulness or relevance, generating synthetic QA pairs for retrieval testing, or optimizing chunking strategies.

  • generate-synthetic-data

    Create diverse synthetic test inputs for LLM pipeline evaluation using dimension-based tuple generation. Use when bootstrapping an eval dataset, when real user data is sparse, or when stress-testing specific failure hypotheses. Do NOT use when you already have 100+ representative real traces (use s…

  • validate-evaluator

    Calibrate an LLM judge against human labels using data splits, TPR/TNR, and bias correction. Use after writing a judge prompt (write-judge-prompt) when you need to verify alignment before trusting its outputs. Do NOT use for code-based evaluators (those are deterministic; test with unit tests per `…

  • write-code-eval

    Write code evaluators for known failure modes with objective rules. Use when code can check the rule from a trace, with or without a reference answer. Use `write-judge-prompt` when the rule requires interpretation.

  • write-judge-prompt

    Design LLM-as-Judge evaluators for subjective criteria that code-based checks cannot handle. Use when a failure mode requires interpretation (tone, faithfulness, relevance, completeness). Do NOT use when the failure mode can be checked with code (regex, schema validation, execution tests); use `wri…

How a Skill differs from MCP

MCP = the hands

Connects the AI to an outside service or data, such as reading Notion or creating an invoice. A program or server has to be running.

Skill = the manual

A text file that teaches the AI how to do a task. It connects to nothing.

Some overlap: a few Skills run scripts that call tools, and a few MCP servers only send knowledge. Start from the task you want done.

Works with

Not stated by the maker · SKILL.md skills work in apps that support Agent Skills, such as Claude Code, Codex and Cursor

Install

Pick a skill first

This repo has 9 skills. Pick one from the list above; its page shows every way to install it, including letting the AI install it and a ZIP

Let the AI install it
Upload to claude.ai

Go to the skill list ↑

The maker’s command installs the whole set of 9
Install Node.js (if you don’t have it)

Already installed? Check first · Open Terminal, type this command and press Enter

node -v
  • A version number such as v26.13.1 = installed, skip this step
  • "command not found" (on Windows: "is not recognized") = not installed yet, or installed but Terminal needs to be closed and reopened
How to open Terminal

Mac: press ⌘ + Space, type Terminal, press Enter

Windows: open Start, type Windows PowerShell, then Open

Not installed: Open the official download page, pick the LTS version and the click-through installer: .pkg on Mac · .msi on Windows. Double-click and follow it to the end. No curl command needed

Open the Node.js download page ↗

When it’s done, close Terminal, open it again and run the same command. You should see a version number: node -v

Install once; it works for everything that needs Node.js

npx skills add https://github.com/ai-evals-course/evals-skills

Copied from the maker’s README; it downloads and runs the maker’s code

Discussion

Ask, share a tip or report that it does not work. Sorted by what people marked helpful.

No posts yet. Be the first to ask or share.

Posts here are written by users, not the PopNx team.

Sign in to post

From GitHub

GitHub stars
1,823
Stars are for the whole repo, not each skill
Maker
ai-evals-course
License
Apache-2.0
Last code change
Oct 25, 2026
SKILL.md files in the repo (including translated copies and templates)
9