SQL Benchmarks Lab

Name: SQL Benchmarks Lab
Author: rctruta

by rctrutaversion 0.1.2i★ –iRuns on your computer

Registry nameio.github.rctruta/sqlbenchdag

Install

Overview

Description from the maker (original English)

Query pre-computed SQL engine benchmarks. Runs standalone, no server setup required.

PopNx MCP’s three pillars

These three are never merged into one score: opinions, measurements and popularity are different things.

User voiceiNo reviews yetReviews open together with accounts in a later step
Our basic checksi1 of 1 passedChecked from public information; code not scanned yet
UsageiNot measured yet★ –

What you can do with it

We have not written a plain-language summary for this one yet. Below are the facts we read from the registry.

Connection: Runs on your computer
Transport: stdio
Category: Databases & datasets

Install

Pick the AI app you use, then copy the settings.

Before you install

Basic checks: 1 of 1 passed
Runs on your computer with your user’s permissions (files, network, apps).
You fill in yourself: SB_RESULTS_DIR, SB_API_BASE

You need

PyPIPython’s uv installed (for the uvx command)

Which app do you use?

ClaudeClaude CodeCursorVS CodeChatGPT

Pick your app above. We remember it, so next time the button says “Install in …”.

Open Claude Desktop → Settings → Developer → Edit Config, and paste this into claude_desktop_config.json

{
  "mcpServers": {
    "sqlbenchdag": {
      "command": "uvx",
      "args": [
        "sqlbenchdag"
      ],
      "env": {
        "SB_RESULTS_DIR": "your-value",
        "SB_API_BASE": "your-value"
      }
    }
  }
}

Run this in your terminal

claude mcp add sqlbenchdag --env SB_RESULTS_DIR=your-value --env SB_API_BASE=your-value -- uvx sqlbenchdag

Fill in the empty values before running it.

Open in Cursor

Or copy the settings yourself

Paste into .cursor/mcp.json in your project, or ~/.cursor/mcp.json

{
  "mcpServers": {
    "sqlbenchdag": {
      "command": "uvx",
      "args": [
        "sqlbenchdag"
      ],
      "env": {
        "SB_RESULTS_DIR": "your-value",
        "SB_API_BASE": "your-value"
      }
    }
  }
}

Your browser asks to open the app, then the app shows the settings and asks you to confirm. Key values are left empty for you to fill in.

Open in VS Code

Or copy the settings yourself

Paste into .vscode/mcp.json in your project

{
  "servers": {
    "sqlbenchdag": {
      "type": "stdio",
      "command": "uvx",
      "args": [
        "sqlbenchdag"
      ],
      "env": {
        "SB_RESULTS_DIR": "your-value",
        "SB_API_BASE": "your-value"
      }
    }
  }
}

Your browser asks to open the app, then the app shows the settings and asks you to confirm. Key values are left empty for you to fill in.

This one runs on your computer, and ChatGPT can only connect to servers on the internet. Pick another app.

Your choice stays on this device in a cookie; no account needed.

Values you fill in

SB_RESULTS_DIR env
Directory of experiment result capsules. The package ships the SQL suites, taxonomy and templates but NOT capsules, which are run output — so on a fresh install the catalog tools work and list_result…
SB_API_BASE env
Optional. Base URL of a running SQL Benchmarks REST API. Leave unset and the read tools mount the API in-process — no server needed. Set it only when the API runs elsewhere, or to use submit_experime…

These settings are generated from the maker’s registry entry. If the README says otherwise, follow the README.

What it can do

Runs on your computer; the tool list appears in your AI app after you install it.

Basic checksi

We check information anyone can see; we have not read the code. Checks that do not apply are left out of the count. · Checked Nov 6, 2026

✓ Pass
Maker matches the repo
The registry name was verified with the GitHub account rctruta, which owns the repo
… Not checked yet
Still maintained
We have not read this repo from GitHub yet
… Not checked yet
Has a licence
We have not read this repo from GitHub yet
– Not applicable
Endpoint responds
This one runs on your computer; there is no internet endpoint
– Not applicable
Sign-in with OAuth
Runs on your computer and uses local settings instead
… Not checked yet
No hidden instructions in tool descriptions
Runs on your computer; we can only read tool lists from internet endpoints

Compared with others in Databases & datasetsi

More stars than 0% of the other 953 servers in this category
Published to the registry more recently than 24% of the other 953 servers in this category
Basic checks equal to or better than 100% of the other 953 servers in this category

Compared with every active server in this category in the registry. Stars count only when the publisher owns the repo.

Common questions

Is SQL Benchmarks Lab safe?

It passed 1 of 1 applicable basic checks. We have not read the code. Before adding a key or token, see what access the server asks for and use a key with only the permissions you need.

What keys or codes does it need?

The registry does not list a required key, though some servers ask you to sign in on first use.

Does it run on my computer or in the cloud?

It runs on your computer. Your AI app starts it when needed, and data does not pass through the maker’s servers (unless it calls an online service itself).

Is it still maintained?

The GitHub repo last changed –, and the latest version (0.1.2) reached the registry 3 months ago.

Which apps can use it?

Apps that support MCP, such as Claude Desktop, Claude Code, Cursor and VS Code. You also need Python (uv) on your computer.

What can I use instead?

In the same category: Supabase, MongoDB MCP Server, Elasticsearch MCP Server

Alternatives

Servers in the same category from other makers, sorted by stars.

Reading a card:★ GitHub starsiThree pillars: reviews · checks · usageiUpdated = latest version published to the registryi

Supabase

by supabase

Supabase’s official server lets AI create tables, query data and review project settings in your Supabase database.

Computer or cloud★ 2,993Apache-2.0Updated yesterday

Install

MongoDB MCP Server

by mongodb-js

MongoDB Model Context Protocol Server

Runs on your computer★ 1,157Apache-2.0Updated 2 days ago

Install

Elasticsearch MCP Server

by cr7258

MCP server for interacting with Elasticsearch

Runs on your computer★ 315Apache-2.0Updated 14 months ago

Install

README from the maker

Open the README

Copied from the maker’s repo as written. PopNx MCP has not reviewed or endorsed it. HTML, images and scripts are removed. Links lead off this site.

SQL Benchmarking Laboratory

A deterministic, orchestrated harness for verifying database performance at scale.

Developed by Ramona C. Truta

The Mission: "Ground Truth" as Code

This platform is a specialized laboratory for testing SQL performance hypotheses. It transforms query tuning from intuition into a reproducible science.

The core of the system is a Deterministic Orchestration Harness that guarantees that if the logic or the environment changes, the benchmark result changes. If they do not, the result is addressable and cached.

Scope: the focus to date is synthetic and canonical (TPC-H) data, which is the right instrument for *mechanism* experiments — where controlled, reproducible data isolates the variable under test. Real-data support exists but is experimental; see the FAQ for the synthetic-vs-real rationale, the container model, and the roadmap (AI-security testbed, real-data trust chain).

Key Features & Innovations

1. Context-Aware Semantic Hashing (The Experiment ID)

The Heart of the system is the Experiment ID, an 8-character hash that governs the entire lifecycle. This hash is a SHA-256 fingerprint generated from:

The Config: Every dimension in your YAML (rows, skew, parameters).
The SQL Logic: The actual content of the benchmarked scripts.
The Code: All measurement-relevant Python — orchestration (assets/), engine clients (resources/), and data generators (plugins/).

Semantic Normalization: The hashing engine distinguishes between a logic change and a formatting change.

SQL: Comments, whitespace, and case are normalized before hashing.
Python: Orchestration scripts are parsed into an Abstract Syntax Tree (AST) to strip docstrings and formatting variations, ensuring the Experiment ID only changes when execution logic changes.

2. Multi-Layer Cold-Cache Isolation

To ensure IO-bound performance is not masked by memory buffers, engine-level buffers are cleared before every query:

Postgres: Mandatory Docker Container Restarts clear shared buffers.
Versioned DuckDB (duckdb_1_5, duckdb_2_0): every query runs in a fresh worker process, so DuckDB's buffer pool starts empty.

OS page cache: not evicted. Engines call thrash_os_cache, which dirties an anonymous memory region (half of available RAM, capped at 4 GB). On macOS this was measured to leave a cached 1 GB file fully resident, so measurements should be read as warm at the OS level. DuckDB clients take a required cold argument, and cold=True raises until a verified eviction step exists.

3. Agentic AI Integration

The platform is built for the future of Autonomous Engineering. The Experiment ID allows AI agents to treat the laboratory as a Deterministic Performance API.

See AGENTS.md for the full Agentic Benchmarking Protocol.

4. Declarative Matrix Orchestration

Benchmarks are defined as N-dimensional matrices in YAML. The platform expands these into a Cartesian product of Independent Dagster Partitions. This allows for parallel dispatch and granular retries.

Usage & Technical Setup

Prerequisites

[uv](https://astral.sh/uv): the project's Python environment & dependency manager (fast, modern). setup.sh uses it, and it can install Python 3.11 for you.
Python 3.11+: core runtime (uv provisions it if missing).
Docker: for the containerized engines (Postgres, TypeDB), which the harness manages itself — *not* required for the DuckDB-only quickstart.

Installation & Setup

From PyPI — the lab is installable, which gives you the tooling and the sqlbench CLI:

pip install sqlbenchdag

From source — clone the repo to also get the published capsules (sql_benchmarks/experiments/results/) and the full harness. The laboratory includes a comprehensive setup script that manages virtual environments, dependencies, and directory initialization.

# 1. Automate Setup

chmod +x setup.sh && ./setup.sh

2. Activate Laboratory

source venv/bin/activate

The Execution Workflow (CLI)

While the system is powered by Dagster, the primary interface is the CLI for automated workflows.

# QUICKSTART: DuckDB only, no Docker required — runs in seconds

./run.sh sql_benchmarks/experiments/queue/quickstart.yaml --auto

FULL BENCHMARK: Run all queued experiments (requires Docker for Postgres)

./run.sh queue --auto

SINGLE EXPERIMENT: Pass a specific config path

./run.sh sql_benchmarks/experiments/queue/quack_execution_modes.yaml --auto

To write your own experiment, start from the annotated template and see the catalog of published runs in [docs/experiments.md](docs/experiments.md).

Project Structure

Managed with a strict separation between Harness and Scenario:

/sql_benchmarks

├── /assets # THE HARNESS: Dagster factories & pipeline logic ├── /resources # THE INFRASTRUCTURE: DB drivers & Docker management ├── /scripts/sql # THE SCENARIOS: Raw SQL partitioned by scenario ├── /plugins # THE DATA: Declarative generators & scenario providers ├── /utils # THE BRAIN: AST-hashing, common logic, & system primitives └── /experiments # THE LABORATORY ├── active.yaml # LOCAL runtime staging — the coordinator overwrites │ # this on every run (gitignored; not shared state) ├── /queue # Staging area for new experiment configs ├── /archive # Library of previously defined experiment templates ├── /configs # Registry of immutable, hash-addressed experiment capsules └── /results # Data capsules: fragments, CSVs, and Dashboards

Published Experiments — the Quack investigation

The first published study with this lab: an independent measurement of DuckDB's Quack client-server protocol (beta, v1.5.3). Each row links the runnable config to its committed, verifiable capsule. Full numbers, scaling exponents, and verification steps in docs/published_capsules.md.

Act	Question	Config	Capsule	Finding
I	What does the protocol cost?	quack_execution_modes.yaml	`b8e2bfaf`	Attach-mode overhead grows with scan size (2.6× → 9.5× at 10M rows); pushdown stays flat at ~2×.
II	Why is pushdown ~2× and not 1×?	quack_residual_threads.yaml	`25b0e134`	The residual tracks reduced server-side parallelism (~2–4 of 8 threads), not transport.
III	Does it generalize to joins?	tpch_quack_validation.yaml	`b198363e`	On canonical TPC-H Q3, pushdown holds (~1.9×); attach mode cannot run the join at all (DNF).
IV	Does it beat the incumbent?	quack_vs_postgres.yaml	`902d1277`	DuckDB-over-Quack (pushdown) beats PostgreSQL by up to 13.2×, and the gap widens with scale.

*Act 0 (b82b4eae) is the exploratory scout that started it — see docs/published_capsules.md.*

The lab also ships scenario suites for other studies — selectivity/, null_logic/, null_sentinel/, recursion/, sort_spill/, tpch/, and more under sql_benchmarks/scripts/sql/ — each runnable across the engine matrix.

Naming & Provenance

The lab works at two layers, named two different ways. Keeping them distinct is the difference between a number you can trace and a number you have to trust.

Layer	What it is	Named by	Example
Capsule	one experiment's complete results	its Experiment ID — an 8-char SHA-256 of the question (config + SQL + measurement code), assigned by the machine	`48c92f31`
Release	a signed set of capsules backing a publication	a human name: `sqlbenchdag- -v - `	`sqlbenchdag-quack-v1-20260614`

A release *contains* capsules; a capsule is one experiment. The machine names capsules (hashes you can't forge); the author names releases (words that mean something). sqlbenchdag is the lab's maker's mark — sql + bench + dag (Dagster, the orchestration that distinguishes this lab from a bare query timer). It is carried on every release this lab produces.

Two hashes, two jobs (a capsule contains both — they are not the same thing):

the Experiment ID (8 chars) is an *address* — it *names* the capsule, and the same question always produces the same address;
the integrity seal (64 chars, in integrity.seal) is a *checksum* — it *verifies* the capsule's contents are untouched. You file by the address; the seal is the tamper-evident lid.

Reproducibility is pinned to the release. Because the Experiment ID hashes the measurement code, it is stable only at a fixed code revision — which is exactly what the signed release tag freezes. To reproduce a published capsule, check out its release tag and re-run its config; the same question yields the same ID, or the comparison is refused by construction.

Provenance roadmap: a per-capsule generator stamp (sqlbenchdag@ )

will record the exact build that produced each capsule. It writes through the

finalization code, which lives inside the Experiment-ID hash, so it ships with

a future release rather than re-numbering capsules already published.

See docs/published_capsules.md for the full trust chain (reproducibility, integrity, timestamp, authorship) and how to verify each. Reproducibility and integrity are automatic on every run; timestamp and signature are optional steps you add only when publishing — the producer workflow is docs/PUBLISHING.md.

License

Built with Dagster, Polars, Docker, DuckDB, and Postgres.

SQL Benchmarks Lab

Overview

PopNx MCP’s three pillars

What you can do with it

Install

Before you install

Values you fill in

What it can do

Basic checksi

Maker matches the repo

Still maintained

Has a licence

Endpoint responds

Sign-in with OAuth

No hidden instructions in tool descriptions

Compared with others in Databases & datasetsi

Common questions

Alternatives

Supabase

MongoDB MCP Server

Elasticsearch MCP Server

README from the maker

SQL Benchmarking Laboratory

The Mission: "Ground Truth" as Code

Key Features & Innovations

1. Context-Aware Semantic Hashing (The Experiment ID)

2. Multi-Layer Cold-Cache Isolation

3. Agentic AI Integration

4. Declarative Matrix Orchestration

Usage & Technical Setup

Prerequisites

Installation & Setup

2. Activate Laboratory

The Execution Workflow (CLI)

FULL BENCHMARK: Run all queued experiments (requires Docker for Postgres)

SINGLE EXPERIMENT: Pass a specific config path

Project Structure

Published Experiments — the Quack investigation

Naming & Provenance

License