Spec Driven Development
MCP Server that facilitates spec-driven development workflows, not just Vibe Coding.
Runs on your computer
Sparkforensics MCP
Overview
Description from the maker (original English)
MCP server for diagnosing Apache Spark event-log performance issues and comparing runs.
PopNx MCP’s three pillars
These three are never merged into one score: opinions, measurements and popularity are different things.
- User voiceiNo reviews yetReviews open together with accounts in a later step
- Our basic checksi1 of 1 passedChecked from public information; code not scanned yet
- UsageiEarly↓ 510/wk · ★ –
What you can do with it
We have not written a plain-language summary for this one yet. Below are the facts we read from the registry.
- Connection: Runs on your computer
- Transport: stdio
- Category: Other
Install
Pick the AI app you use, then copy the settings.
Before you install
- Basic checks: 1 of 1 passed
- Runs on your computer with your user’s permissions (files, network, apps).
- The registry lists no key to fill in.
You need
- npmNode.js installed (for the npx command)
Which app do you use?
Pick your app above. We remember it, so next time the button says “Install in …”.
Open Claude Desktop → Settings → Developer → Edit Config, and paste this into claude_desktop_config.json
{
"mcpServers": {
"sparkforensics": {
"command": "npx",
"args": [
"-y",
"sparkforensics-mcp"
]
}
}
}Run this in your terminal
claude mcp add sparkforensics -- npx -y sparkforensics-mcpOr copy the settings yourself
Paste into .cursor/mcp.json in your project, or ~/.cursor/mcp.json
{
"mcpServers": {
"sparkforensics": {
"command": "npx",
"args": [
"-y",
"sparkforensics-mcp"
]
}
}
}Your browser asks to open the app, then the app shows the settings and asks you to confirm. Key values are left empty for you to fill in.
Or copy the settings yourself
Paste into .vscode/mcp.json in your project
{
"servers": {
"sparkforensics": {
"type": "stdio",
"command": "npx",
"args": [
"-y",
"sparkforensics-mcp"
]
}
}
}Your browser asks to open the app, then the app shows the settings and asks you to confirm. Key values are left empty for you to fill in.
This one runs on your computer, and ChatGPT can only connect to servers on the internet. Pick another app.
Your choice stays on this device in a cookie; no account needed.
These settings are generated from the maker’s registry entry. If the README says otherwise, follow the README.
What it can do
Runs on your computer; the tool list appears in your AI app after you install it.
Basic checksi
We check information anyone can see; we have not read the code. Checks that do not apply are left out of the count. · Checked Nov 8, 2026
- ✓ Pass
Maker matches the repo
The registry name was verified with the GitHub account shuffle-works, which owns the repo
- … Not checked yet
Still maintained
We have not read this repo from GitHub yet
- … Not checked yet
Has a licence
We have not read this repo from GitHub yet
- – Not applicable
Endpoint responds
This one runs on your computer; there is no internet endpoint
- – Not applicable
Sign-in with OAuth
Runs on your computer and uses local settings instead
- … Not checked yet
No hidden instructions in tool descriptions
Runs on your computer; we can only read tool lists from internet endpoints
Compared with others in Otheri
- More stars than 0% of the other 18474 servers in this category
- Published to the registry more recently than 33% of the other 18474 servers in this category
- Basic checks equal to or better than 100% of the other 18474 servers in this category
Compared with every active server in this category in the registry. Stars count only when the publisher owns the repo.
Common questions
Is Sparkforensics MCP safe?
It passed 1 of 1 applicable basic checks. We have not read the code. Before adding a key or token, see what access the server asks for and use a key with only the permissions you need.
What keys or codes does it need?
The registry does not list a required key, though some servers ask you to sign in on first use.
Does it run on my computer or in the cloud?
It runs on your computer. Your AI app starts it when needed, and data does not pass through the maker’s servers (unless it calls an online service itself).
Is it still maintained?
The GitHub repo last changed –, and the latest version (0.1.0) reached the registry 2 months ago.
Which apps can use it?
Apps that support MCP, such as Claude Desktop, Claude Code, Cursor and VS Code. You also need Node.js on your computer.
What can I use instead?
In the same category: Spec Driven Development, Remote Desktop Commander, SwarmMemo
Alternatives
Servers in the same category from other makers, sorted by stars.
Reading a card:↓ Downloads per weeki★ GitHub starsiThree pillars: reviews · checks · usageiUpdated = latest version published to the registryi
Remote Desktop Commander
Remote Desktop Commander: use files, terminal and processes on your own computers from any AI app.
Runs in the cloud✓ desktopcommander.app
SwarmMemo
Talk with agent swarms: rooms, private messages and paid tasks for AI agents
Runs in the cloud✓ swarmmemo.com
README from the maker
Open the README
Copied from the maker’s repo as written. PopNx MCP has not reviewed or endorsed it. HTML, images and scripts are removed. Links lead off this site.
SparkForensics
Your Spark job is slow. SparkForensics tells you why, and what to do about it.
Drop a Spark event log in and get an evidence-backed breakdown of which stages are slow, why, and how to fix them. Nothing installs on your cluster and nothing leaves your machine.
[Try it now →](https://shuffle-works.github.io/sparkforensics/) with your own event log. No install, no account, no upload. Read the documentation for a guided tour.
Spark's own History Server gives you raw metrics. It won't tell you that stage 14 spilled 40GB because of a skewed join key, or that cold start is eating a third of the job. SparkForensics reads the same event logs and turns them into findings.
Why SparkForensics
- Your data stays on your machine. Drop a log file and read the dashboard; nothing is uploaded anywhere. Local-server mode adds a small Node companion that can also pull runs from a Spark History Server.
- Event logs are streamed and parsed off the main thread in a Web Worker, so a 240MB+ (or multi-GB) NDJSON log doesn't freeze the tab.
- The finding types cover skew, shuffle, spill, GC pressure, stragglers, cold start, cache/memory utilization, autoscaling churn, job/stage failures and partition sizing, plus a SQL Plan Advisor (duplicate subtrees, small files, broadcast sizing) and a config audit. Each has a page in the detector catalog, and its thresholds are in the detector table.
- Fix These First orders every finding by its estimated wall-clock impact, not severity alone, so the top row is the fix worth doing first.
- Run comparison (A/B) mode structurally matches stages across two runs of the same job and shows which stages regressed, rather than every difference.
- Every finding tracks what it could and couldn't confirm from the log, so you know when to trust a flag and when it's a hint worth checking manually.
- A headless CLI (
sparkforensics-analyze) evaluates a run against budget thresholds (max runtime, skew, spill, failure rate...) and exits non-zero on violation, so a regression can fail a build. The run can come from a local event-log file/directory or, with--shs-base-url, fetched directly from a Spark History Server. - An MCP server (
sparkforensics-mcp) exposes tools for diagnosing, summarizing and comparing runs, so an LLM agent can investigate a run directly. The full tool list is in MCP tools. - For running the analyze CLI automatically after every Spark job instead of by hand, see sparkforensics-operator, an Airflow operator that runs SparkForensics on a job's event log and acts on the result.
Unlike tools that need a cluster-side plugin or a running server plus an LLM agent (Spark History Server MCP tools), SparkForensics needs nothing installed anywhere else: the event log and a browser tab are enough.
Get started
Run the dashboard locally:
npx sparkforensics-serverThe Getting started guide covers what it serves, the live demo alternative, and loading a run. To work on SparkForensics itself, see Development.
For a one-shot check with no browser, the kind a CI pipeline can gate on:
npx -p sparkforensics-cli sparkforensics-analyze path/to/eventlog --max-runtime 3600000 --max-skew 3The command ships in the sparkforensics-cli package. To install it once instead: npm i -g sparkforensics-cli, then run sparkforensics-analyze directly. The sparkforensics-analyze alias package also works: npx sparkforensics-analyze.
That's a headless run against budget thresholds. See Command-line reference below for HTML export, Spark History Server fetching, baseline comparisons, and redaction. To run a detector check with your own threshold, see Tuning detector thresholds.
Development
Install and run
From a checkout:
git clone git@github.com:shuffle-works/sparkforensics.gitcd sparkforensics npm install npm run dev
Open the URL Vite prints. For a production bundle: npm run build, then serve dist/ with any static file server. Asset URLs are relative, so the bundle also works under a URL subpath.
Note: The app must be served over HTTP:
file://URLs block Web
Worker module imports in Chrome.
Deploy modes
This app runs in two modes:
- Static mode (e.g. GitHub Pages, or any plain file server): Choose file is the normal browser-local path for a single event log. Choose rolling-log folder is available only for an
eventlog_v2_*directory. Deploy the output ofnpm run buildfromdist/. Fetching directly from a Spark History Server is not available: the browser's CORS policy blocks it and a static host can't proxy the request. If a History Server load cannot be used, choose a local event-log file instead. - Local-server mode is a small Node companion that serves this app *and* fetches from a Spark History Server on your behalf (server-to-server, so no browser CORS restriction). Useful when the SHS is behind a VPN or isn't CORS-configured.
Run local-server mode with:
npx sparkforensics-serverIt listens on http://127.0.0.1:4173 by default (override with --port or the PORT env var) and binds to localhost only. Open the URL in Chrome and use the Fetch from Spark History Server disclosure on the intake screen. The same server serves the MCP tools over streamable HTTP at /mcp, for clients on the same machine; see Connecting over HTTP.
History Server fetching requires local-server mode, a History Server reachable from the machine running that server, and a supported base application ID (see Getting started for the full list of accepted forms). Enter an optional attempt separately from the base application ID. The local server sends no credentials and follows no upstream redirects. Its recoverable errors intentionally omit upstream response details; edit the fields or choose a local event-log file to recover.
During development from a checkout, npm run local-server does the same thing without publishing or installing the package.
Server configuration
All settings have working defaults. The port (PORT or --port) and the SPARKFORENSICS_* environment variables are listed in Server configuration.
SHS-proxy route test fixture
The /shs-proxy route (packages/core/src/proxy.js) is covered by packages/server/test/shs-proxy-fixture.test.js, part of the normal npm run test:server suite. It loads a real event log from the spark-event-corpus-data git submodule at dev/log-corpus/ and asserts the route streams it back as a valid ZIP. No Docker, no live Spark process, and no live History Server are needed for this.
To verify the full Fetch-from-SHS UI flow by hand, including its unreachable-upstream and application-not-found recovery states, point npm run local-server at a real Spark History Server you have access to and use its Fetch from Spark History Server disclosure. Automated coverage of /shs-proxy uses the spark-event-corpus-data logs, so no Docker image or live Spark process is needed.
Running tests
npm test # single runnpm run test:watch # watch mode npm test -- tests/finding-filter.test.js # single file npm run test:core # packages/core: detectors, analyzer, parser, MCP tools npm run test:cli # packages/cli npm run test:mcp # packages/mcp npm run test:server # packages/server
Command-line reference
Beyond the plain budget check in Get started, the same sparkforensics-analyze CLI covers a few other jobs.
Write a self-contained HTML dashboard for the run instead of a JSON/Markdown report, viewable offline over file:// with no server:
npx -p sparkforensics-cli sparkforensics-analyze path/to/eventlog --export-html path/to/output-dirFetch the run straight from a Spark History Server instead of a local file (mutually exclusive with the positional path):
npx -p sparkforensics-cli sparkforensics-analyze --shs-base-url http://history-server:18080 --app-id application_1234_0001 --max-skew 3Compare a candidate run against a baseline (A/B) and fail the build on a regression or a newly introduced critical finding:
npx -p sparkforensics-cli sparkforensics-analyze path/to/candidate --baseline path/to/baseline --max-regression-pct 10 --fail-on-introduced criticalSeveral regression budgets in one call, and several candidates against one baseline (NDJSON, one line per candidate; the baseline is parsed once):
npx -p sparkforensics-cli sparkforensics-analyze cand-1 cand-2 cand-3 --baseline base --regression-budget wallClock:10 --regression-budget gcTime:25Compare against a baseline that lives on a Spark History Server, from a local candidate or a History Server one, with --baseline-shs-base-url, --baseline-app-id and --baseline-attempt-id (the baseline can be on a different server):
npx -p sparkforensics-cli sparkforensics-analyze --shs-base-url http://history-server:18080 --app-id application_1234_0002 --baseline-shs-base-url http://history-server:18080 --baseline-app-id application_1234_0001 --max-regression-pct 10Several candidates from one History Server take repeated --app-id values (--app-id : names an attempt) and print NDJSON, one line each.
See Several budgets and several candidates for the --budgets file, the NDJSON line fields and the exit code rules.
List every finding type the detectors can report, without reading a log (the MCP list_detectors tool returns the same records):
npx -p sparkforensics-cli sparkforensics-analyze detectors --format mdRedact the app id, the app name, host names, IP addresses and user names in URLs and paths before sharing the output outside the environment that produced it. Other paths, table names and SQL text stay as written; the CI guide lists what --redact covers and what it does not:
npx -p sparkforensics-cli sparkforensics-analyze path/to/eventlog --redactNarrow the output's findings array to cert
Only the beginning is shown; read the rest in the repo