Nonobench

Name: Nonobench
Author: mauricekleine

by mauricekleineversion 1.1.0i★ 10iRuns in the cloud

Registry nameio.github.mauricekleine/nonobench

Install

Overview

Description from the maker (original English)

An open-source benchmark of how well LLMs solve nonogram puzzles, from 5x5 to 20x20.

PopNx MCP’s three pillars

These three are never merged into one score: opinions, measurements and popularity are different things.

User voiceiNo reviews yetReviews open together with accounts in a later step
Our basic checksi4 of 4 passedChecked from public information; code not scanned yet
UsageiEarly★ 10

What you can do with it

We have not written a plain-language summary for this one yet. Below are the facts we read from the registry.

Connection: Runs in the cloud
Transport: Streamable HTTP
Category: Search & AI memory

Install

Pick the AI app you use, then copy the settings.

Before you install

Basic checks: 4 of 4 passed
Runs on the maker’s server: what you ask it goes through their server.
Can do 11 things (tools) get_leaderboard, list_providers, list_families, compare_models …
The registry lists no key to fill in.

You need

Runs in the cloudNothing to install; it connects over the internet

Which app do you use?

ClaudeClaude CodeCursorVS CodeChatGPT

Pick your app above. We remember it, so next time the button says “Install in …”.

Works in Claude on the web, desktop and mobile:

Open Settings → Connectors
Press Add custom connector
Paste the address below and press Add
If it asks you to sign in, sign in with the maker

https://www.nonobench.com/mcp

Run this in your terminal

claude mcp add --transport http nonobench https://www.nonobench.com/mcp

Open in Cursor

Or copy the settings yourself

Paste into .cursor/mcp.json in your project, or ~/.cursor/mcp.json

{
  "mcpServers": {
    "nonobench": {
      "url": "https://www.nonobench.com/mcp"
    }
  }
}

Your browser asks to open the app, then the app shows the settings and asks you to confirm. Key values are left empty for you to fill in.

Open in VS Code

Or copy the settings yourself

Paste into .vscode/mcp.json in your project

{
  "servers": {
    "nonobench": {
      "type": "http",
      "url": "https://www.nonobench.com/mcp"
    }
  }
}

Your browser asks to open the app, then the app shows the settings and asks you to confirm. Key values are left empty for you to fill in.

ChatGPT connects only to servers on the internet:

Open ChatGPT on the web → Settings → Connectors (Apps)
Press Create or Add (some plans need Developer mode under Advanced first)
Paste the address below, give it a name and save
If it asks you to sign in, sign in with the maker

https://www.nonobench.com/mcp

Menu names can differ slightly by plan and over time.

Your choice stays on this device in a cookie; no account needed.

These settings are generated from the maker’s registry entry. If the README says otherwise, follow the README.

Which apps can use it

Based on how this server connects and on each app’s own docs, checked Nov 10, 2026.

Connect over the internet in

Claude ChatGPT Goose LM Studio Msty Studio Cherry Studio Cursor VS Code + GitHub Copilot Visual Studio Windsurf (Devin Desktop)Zed Kiro Cline Continue JetBrains AI Assistant Claude Code Codex CLI Gemini CLI OpenCode nodecode Warp LibreChat Open WebUI Raycast

All apps, with pros and cons →

What it can do

Tool names and descriptions exactly as the server reported them on Nov 10, 2026. The maker wrote them, not us; we show them as plain text.

get_leaderboard
Models ranked by accuracy; defaults to all effort levels for compatibility.
list_providers
Provider ids, names, families and variant counts.
list_families
Model families, available efforts and best variants.
compare_models
Side-by-side core overall and per-size accuracy, cost, latency and token results for model or family names.
get_model_results
Accuracy, cost, latency and token use for one model, broken down by grid size.
list_puzzles
The benchmark puzzles with their ids and row/column clues.
get_puzzle
One puzzle, including the clue text models were prompted with. The reference solution is only included on request; some puzzles have several valid solutions.
check_solution
Check a grid against a puzzle's clues, using the same rule as the benchmark grader. Reports which rows and columns do not match.
get_puzzle_results
Per-model outcomes for one puzzle. Answers are omitted unless requested.
get_model_puzzles
Which puzzles one model solved, missed, timed out on, or has not run.
list_runs
Individual benchmark runs (one model on one puzzle), optionally with the raw prompt and model output.

Basic checksi

We check information anyone can see; we have not read the code. Checks that do not apply are left out of the count. · Checked Nov 11, 2026

✓ Pass
Maker matches the repo
The registry name was verified with the GitHub account mauricekleine, which owns the repo
✓ Pass
Still maintained
Code last changed 7 days ago
… Not checked yet
Has a licence
We have not read this repo from GitHub yet
✓ Pass
Endpoint responds
Responded in 248 ms (HTTP 200)
– Not applicable
Sign-in with OAuth
No sign-in needed
✓ Pass
No hidden instructions in tool descriptions
Read 11 tool descriptions and found no hidden instructions to the AI

Compared with others in Search & AI memoryi

More stars than 91% of the other 5868 servers in this category
Published to the registry more recently than 49% of the other 5868 servers in this category
Basic checks equal to or better than 100% of the other 5868 servers in this category

Compared with every active server in this category in the registry. Stars count only when the publisher owns the repo.

Common questions

Is Nonobench safe?

It passed 4 of 4 applicable basic checks. We have not read the code. Before adding a key or token, see what access the server asks for and use a key with only the permissions you need.

What keys or codes does it need?

The registry does not list a required key, though some servers ask you to sign in on first use.

Does it run on my computer or in the cloud?

It runs in the maker’s cloud. Connect over the internet with nothing to install, but what you send passes through the maker’s servers.

Is it still maintained?

The GitHub repo last changed 7 days ago, and the latest version (1.1.0) reached the registry 42 days ago.

Which apps can use it?

Apps that support MCP, such as Claude Desktop, Claude Code, Cursor and VS Code. It connects by URL, so apps that accept remote MCP servers work too, such as ChatGPT in developer mode.

What can I use instead?

In the same category: Ruflo, Context7, Metabase

Alternatives

Servers in the same category from other makers, sorted by stars.

Reading a card:↓ Downloads per weeki★ GitHub starsiThree pillars: reviews · checks · usageiUpdated = latest version published to the registryi

Ruflo

Agent teams, swarms, memory and hooks for Claude Code and Codex, served over MCP.

by ruvnetWidely used↓ 74.3K/wk★ 81.4KUpdated today

Runs on your computer

Install

Context7

Feeds AI the latest docs for libraries and frameworks before it answers, so code it writes does not use outdated, removed functions.

by upstashWidely used↓ 1.2M/wk★ 64.4KUpdated yesterday

Computer or cloud

Install

Metabase

Lets AI clients search, explore, query, and visualize data in a Metabase instance.

by metabaseWidely used★ 50.3KUpdated 6 months ago

Runs in the cloud

Install

README from the maker

Open the README

Copied from the maker’s repo as written. PopNx MCP has not reviewed or endorsed it. HTML, images and scripts are removed. Links lead off this site.

Nonobench

A benchmark suite for evaluating LLM reasoning capabilities on Nonogram (Picross) puzzle solving across different grid sizes. Results are published at nonobench.com.

Built by Maurice Kleine.

What is a Nonogram?

Nonograms (also known as Picross, Griddlers, or Paint by Numbers) are logic puzzles where you fill in cells on a grid based on numeric clues for each row and column. The clues indicate consecutive groups of filled cells, separated by at least one empty cell. Solving these puzzles requires logical deduction and constraint satisfaction - making them an excellent test of LLM reasoning abilities.

Project Structure

nonobench/

├── bench/ # Benchmark runner, results database and exporter └── visualizer/ # TanStack Start dashboard on Cloudflare Workers (nonobench.com), also home of the puzzle set

Prerequisites

Bun runtime (v1.4+)
Node.js v24+ (for the visualizer)
OpenRouter API key

Quick Start

1. Clone the Repository

git clone https://github.com/mauricekleine/nonobench.git

cd nonobench

2. Set Up Environment Variables

Create a .env file in the bench/ directory (see bench/.env.example):

OPENROUTER_API_KEY=your_openrouter_api_key_here

This is the only variable needed. The visualizer builds without any.

3. Running Benchmarks

cd bench

bun install bun run bench # prints the plan and exits, no API calls bun run bench --model # run one model (repeat --model for more) bun run bench --all-missing # run every configured model with missing work bun run bench --model --sizes 20x20 # opt in to the extended tier

Runs are incremental and append-only: a model/puzzle pair that already has a successful result is never run again, and the database refuses to overwrite it. Failed runs are retried on the next invocation. Results are stored in bench/results.db (SQLite).

Useful flags:

--max-cost stops launching new puzzles once this session's spend reaches the amount. Requests already in flight still finish, so a session can overshoot by up to --parallel requests per selected model. Use the OpenRouter key's own limit as the hard ceiling.
--parallel sets concurrent requests per model (default 10); lower it for rate-limited providers.
--limit runs only the first n puzzles of each size, for pilots against a scratch database (NONOBENCH_DB=/tmp/copy.db).

Output modes

New runs ask for the answer as strict structured output (a JSON schema, only routed to endpoints that enforce it), so models cannot wrap the grid in prose. For a few models the schema-enforcing endpoints measurably hurt answers; those run in text mode instead (outputMode: "text" in bench/constants.ts, chosen by a 5x5 A/B with the benchmark prompt). Every run records its mode, and grading is identical for both: the answer must satisfy every clue. The runner also stops a model that solves none of the 5x5 puzzles with structured output, and one whose early runs mostly report zero reasoning tokens, since both point at the harness rather than the model.

After benchmarking, export results for the visualizer:

bun run export

This writes visualizer/app/results.json (aggregates) and visualizer/public/results-raw.json (every run, including prompts and outputs).

Other scripts:

bun test - parser, grader, database-policy and puzzle checks
bun run typecheck - TypeScript check
bun run regrade - read-only comparison of stored grades against the current grader

Running Against a Local Model

The runner can benchmark a model that you serve yourself. Any server with an OpenAI-compatible /v1 endpoint works: vLLM, SGLang, llama.cpp, LM Studio, and Ollama.

This setup uses two machines. The model server is the workstation that serves the model. The bench machine is the laptop that runs the benchmark.

1. Serve the model on the workstation

Bind the server to all interfaces. The default bind address, 127.0.0.1, accepts connections only from the model server itself.

vLLM:

vllm serve Qwen/Qwen3-32B --host 0.0.0.0 --port 8000

Ollama:

ollama pull qwen3:32b

OLLAMA_HOST=0.0.0.0:11434 ollama serve

llama.cpp:

llama-server -m ./qwen3-32b-Q4_K_M.gguf --host 0.0.0.0 --port 8080

SGLang:

python -m sglang.launch_server --model-path Qwen/Qwen3-32B --host 0.0.0.0 --port 30000

LM Studio: open the Developer tab, then select Start Server. Turn on the Network Server switch, so the server listens on all interfaces.

CAUTION: Do not expose the port to the internet. Most local servers have no authentication, so anyone who reaches the port can run the model.

2. Open the port on the model server

Restrict the rule to your own subnet. Allow TCP traffic from your local network to the port:

sudo ufw allow from 192.168.1.0/24 to any port 8000 proto tcp

On macOS, allow incoming connections for the server app in System Settings.

3. Verify the connection from the laptop

Find the LAN address of the model server. On Linux, run ip -4 addr show. On macOS, run ipconfig getifaddr en0.

From the laptop, request the model list:

curl http://192.168.1.20:8000/v1/models

Make sure that the answer lists your model. Copy the exact id value. You need that value for NONOBENCH_LOCAL_MODEL.

If the request fails, check the bind address from step 1. Then check the firewall rule, the address, and the port.

If the model server is not reachable on your network, create an SSH tunnel to it instead. Then use the address http://127.0.0.1:8000/v1:

ssh -N -L 8000:127.0.0.1:8000 user@192.168.1.20

4. Run the benchmark

cd bench

bun install NONOBENCH_LOCAL_BASE_URL=http://192.168.1.20:8000/v1 \ NONOBENCH_LOCAL_MODEL=Qwen3-32B \ bun run bench:local --model Qwen3-32B

Do a pilot first. The next command runs two 5x5 puzzles into a scratch database:

NONOBENCH_LOCAL_BASE_URL=http://192.168.1.20:8000/v1 \

NONOBENCH_LOCAL_MODEL=Qwen3-32B \ NONOBENCH_DB=local-pilot.db \ bun run bench --model Qwen3-32B --sizes 5x5 --limit 2

Make sure that the plan lists your model as [local, text].

Environment variables for a local model

Variable	Meaning
`NONOBENCH_LOCAL_BASE_URL`	The `/v1` endpoint of the server. Ollama uses port `11434`.
`NONOBENCH_LOCAL_MODEL`	The model `id` that the server returns from `GET /v1/models`.
`NONOBENCH_LOCAL_NAME`	The display name and the value for `--model`. Defaults to the model `id`. Must not match a cloud model name.
`NONOBENCH_LOCAL_EFFORT`	The reasoning effort your server is configured with (`none`, `low`, `xhigh`, …). Label only — the runner never sends reasoning settings to a local server. Defaults to `none`.
`NONOBENCH_LOCAL_API_KEY`	The bearer token, when the server requires one. Defaults to `local`.

The runner adds the local model to the plan only when you set both NONOBENCH_LOCAL_BASE_URL and NONOBENCH_LOCAL_MODEL.

Notes on local runs

Local runs cost $0. Keep them out of the published dataset: run them with bun run bench:local, which is bun run bench with NONOBENCH_DB=local-results.db. bench/results.db is the shared dataset, and the export-contract test checks it against the committed exports. The runner refuses to write a local model to the shared database.
Before starting, the runner checks the model id against the server's GET /v1/models list. A typo fails with the models the server does serve, and an unreachable server fails before any run is recorded.
Text mode is the default for a local model. Many local servers accept a JSON schema request and then ignore it. If your server enforces the schema, set NONOBENCH_OUTPUT_MODE=json_schema.
Start with --parallel 1. One GPU serves fewer requests at the same time than a cloud provider does.
The 20x20 tier asks for 128,000 output tokens. If your server has a smaller context, run only the core sizes.
The runner does not apply the reasoning-token circuit breaker to a local model. Some local servers report zero reasoning tokens for every run.
Set the same variables in bench/.env to avoid the prefix on every command.
A bench run registers the local model in bench/local-models.json (name, server model id, family, server URL), so a later bun run export and the visualizer pick it up without any env vars. To register a model you already benched, add one entry there: " ": { "baseURL": "http://host:port/v1", "modelId": " ", "family": " " }. Omit modelId when the name is the server's id.

4. Viewing Results

cd visualizer

bun install bun run dev

Then open http://localhost:3000 to view the interactive dashboard.

Your own runs live in a separate database. Serve them with:

cd visualizer

bun run dev:local

dev:local exports bench/local-results.db into the dashboard's data files, starts the dev server, and restores the committed files when you stop it. While it runs, the dashboard shows only the models in that database — your local runs, not the published leaderboard. Set NONOBENCH_LOCAL_DB to serve another database.

While that server runs, bun test in bench fails one test: the export-contract test compares the dashboard's data files against bench/results.db, and dev:local has put your runs in them. Stop the server, let the script restore the committed files, and the suite passes again.

5. Deploying

The visualizer is a Cloudflare Worker (visualizer/wrangler.jsonc). Pages are prerendered at build time and served as static assets; the Worker serves the API, the MCP server and the agent routes, reading the exported JSON through its assets binding. Workers Builds owns production deploys on merge to main: it runs bun install --frozen-lockfile && bun run build, then bunx wrangler deploy, from visualizer/ with Cloudflare credentials. Local deploy scripts and Wrangler logins are retired; agents use cf for Cloudflare access. From visualizer/:

CLOUDFLARE_ENV=beta bun run build builds the beta Worker for local preview. Nothing deploys it.
bun run deploy:verify waits until production runs a build containing that commit on main. Branch commits fail immediately unless DEPLOY_VERIFY_URL names a separately deployed target. The repository root runs the same script, so bun run deploy:verify works from either directory.

Agent Access

nonobench.com exposes the benchmark data to agents, with no authentication:

REST API under /api/v1 (leaderboard, models, puzzles, a solution checker, individual runs). The spec is at /api/openapi.json, and /.well-known/api-catalog (RFC 9727) points to it.
MCP server at /mcp (stateless Streamable HTTP, MCP 2026-07-28 with 2025 client compatibility), described by /.well-known/mcp/server-card.json. Add it to a client with `claude mcp add --transport http nonobench https://www.nonob

Only the beginning is shown; read the rest in the repo

Nonobench

Overview

PopNx MCP’s three pillars

What you can do with it

Install

Before you install

Which apps can use it

Related guides

What it can do

Basic checksi

Maker matches the repo

Still maintained

Has a licence

Endpoint responds

Sign-in with OAuth

No hidden instructions in tool descriptions

Compared with others in Search & AI memoryi

Common questions

Alternatives

Ruflo

Context7

Metabase

README from the maker

Nonobench

What is a Nonogram?

Project Structure

Prerequisites

Quick Start

1. Clone the Repository

2. Set Up Environment Variables

3. Running Benchmarks

Output modes

Running Against a Local Model

1. Serve the model on the workstation

2. Open the port on the model server

3. Verify the connection from the laptop

4. Run the benchmark

Environment variables for a local model

Notes on local runs

4. Viewing Results

5. Deploying

Agent Access