Poordjaevin

Part of the [DEVIN ecosystem](https://github.com/Icaro0310/awesome-devin)

Track: Control · Nature: product

For: AI engineers, QA engineers

Interface: CLI / Python library / MCP server / Bridge

Path: QA engineers · step 3/3 — after devin-evals

Where this fits

• Job: Control
• Product: devin-judge
• Packages: poordjaevin
• Mode: read-only
• Ecosystem: awesome-devin · registry: devin-powerups

poordjaevin

Renamed (Oct 2026): this repository moved from Icaro0310/poordjaevin to Icaro0310/devin-judge . The PyPI package, console script and MCP registry name stay poordjaevin ; stars, issues and history are preserved by the redirect.

The poor man's Jev. An open source, local-first "System One" decision layer for LLM apps: typed decisions with provably calibrated confidence . No API key. No waitlist.

Community project. Not affiliated with, endorsed by, or sponsored by Cognition AI. "Devin" is a trademark of Cognition AI.

[Linux](README.linux.md) · [Personal Windows](README.windows.md) · [Corporate Windows](README.corporate-windows.md)

Part of the awesome-devin ecosystem: the curated hub for the devin-* tools.

---

Your model's `0.9` is a vibe. poordjaevin's `0.9` is a measurement.

Every LLM-in-JSON-mode hands you a confidence score and hopes you don't check it. poordjaevin checks it. On the shipped eval set it cuts calibration error (ECE) from 0.156 to 0.072 with zero loss of accuracy, and it runs on your laptop with no API key.

Left: raw confidences, overconfident. Right: calibrated, a stated 0.8 really is right about 80% of the time.

Quickstart

# Linux / macOS

pip install "poordjaevin[local]"

# Windows (PowerShell)

py -m pip install "poordjaevin[local]"

[local] pulls torch + transformers for the offline NLI backend. If you only plan to use the Devin ACP backend (poordjaevin serve default), a plain pip install poordjaevin is enough. To install the development version straight from GitHub, append @ git+https://github.com/Icaro0310/devin-judge.git to the package spec.

devin-judge is the new PyPI distribution name: pip install devin-judge and pip install poordjaevin install the same tool. The binary remains poordjaevin for historical compatibility, and a devin-judge command is provided as well.

from poordjaevin import Client, Choice, Score, Noul

client = Client() # local model, no key, offline after one download

result = client.ask( state="I've emailed three times and I'm STILL being double-charged. Cancel my account today.", questions={ "topic": Choice(["billing", "technical", "account", "shipping", "other"]), "frustration": Score(levels=["low", "medium", "high"]), "is_urgent": Noul("The customer needs a response today."), "wants_cancel": Noul("The customer wants to cancel their account."), }, )

result["topic"].value # "billing" always one of your options, by construction result["topic"].confidence # 0.86 calibrated, not a vibe result["frustration"].value # "high" result["is_urgent"].value # True result["wants_cancel"].value # True

One call, one model pass, four typed answers. No prompt engineering, no JSON parsing, no "the model returned prose."

Use it with Devin (MCP server, Devin-only)

poordjaevin ships an MCP server, so Devin (or any MCP client) can make fast, calibrated decisions as tools. The obvious use: gate a risky tool call before the agent runs it.

This mode needs nothing but Devin. The default backend (acp) talks to devin acp through a small packaged Node bridge, so every decision is scored by the model your Devin plan already provides, with automatic model rotation. No Ollama, no VM, no tunnel, no second API key: the bridge reads the same credentials.toml the Devin CLI uses.

# Devin users: no local model needed

uv tool install "poordjaevin[mcp]"

or with pipx (alternative):

pipx install "poordjaevin[mcp]"

Add it to your Devin MCP configuration:

{

"mcpServers": { "poordjaevin": { "command": "poordjaevin", "args": ["serve"] } } }

Requirements for the ACP backend: the devin CLI on PATH (or DEVIN_CLI_PATH), Node.js >= 18 on PATH, and valid Devin credentials at %APPDATA%\devin\credentials.toml (Windows) or ~/.local/share/devin/credentials.toml (Linux, override with DEVIN_CREDENTIALS_PATH).

Backend selection and tuning, all optional:

Honesty note: with acp and vscode-lm the confidence is self_report (the model's own stated probability, temperature-adjusted), not NLI logprobs. Every tool response carries a confidence_source field so callers never mistake one for the other, and model/cost are logged per call for quota monitoring.

The vscode-lm backend scores through whichever chat model VS Code is serving (Copilot, Codex, BYOK). vscode.lm only exists inside the extension host, so the icaro0310.judge extension serves a small localhost endpoint and advertises it via vscode-lm-endpoint.json in the platform data dir (next to djaevin_log.db). If the extension is not running the backend fails closed, or falls back to nli when POORDJAEVIN_VSCODE_LM_FALLBACK=nli.

No Devin on the machine? Use the offline path:

uv tool install "poordjaevin[local,mcp]"

or with pipx (alternative):

pipx install "poordjaevin[local,mcp]"

POORDJAEVIN_BACKEND=nli poordjaevin serve   # ~400MB one-time model download, then offline

The server also works with Claude Code/Desktop (claude mcp add poordjaevin -- poordjaevin serve), same JSON config shape.

The agent then has these local tools:

Why this beats asking an LLM to judge: on the default NLI backend it is local (private), free (no tokens), fast, and the confidence is calibrated instead of made up. On acp or vscode-lm your scoring text goes to whichever model provider you picked; the confidence stays honestly labeled via confidence_source, calibrated when a matching calibrator is loaded and raw otherwise.

Gate your commits (pre-commit hook)

The repo ships a pre-commit hook that runs poordjaevin gate on an action file before every commit:

# .pre-commit-config.yaml (in the repo you want to protect)

• repo: https://github.com/Icaro0310/devin-judge

the hook manifest ships in v0.1.2+; use a tag once cut

rev: main hooks:

• id: devin-judge-gate

optional: point the gate at your own action file

args: [--action-file, path/to/action.txt]

Setup is required: the hook fails closed. Without the action file every commit exits 2, so create it once:

mkdir -p .devin

echo "The risky action this commit performs." > .devin/judge-action.txt

Two details worth knowing: pre-commit appends args after the flag already embedded in entry, so repeat --action-file to override the default path. And the hook runs on every commit (always_run, pass_filenames: false), so it needs a working backend: the acp default, or POORDJAEVIN_BACKEND=nli with the local extra installed.

Why poordjaevin exists

Most production AI work is not chat. It is fast structured decisions: route a ticket, classify an intent, score a sentiment, extract a field, gate a tool call. TypeSafe's Jev named this category ("System One" models) and nailed the thesis — and, per the independent cross-system benchmark below, it currently backs its calibration claims up: it's the strongest model measured here. It's also closed, hosted, and behind a waitlist.

poordjaevin exists for the deployments whe