dashAI (unofficial)

dashai-mcp

An MCP server for [dashAI](https://github.com/DashAISoftware/DashAI), the open source Machine Learning workbench led by the University of Chile (FCFM), built by students of DCC UChile and UTFSM, with CENIA and IMFD.

Unofficial and independent. This is a third-party project. It is not

affiliated with, endorsed by, or maintained by the dashAI project or the

institutions that develop it.

It gives an agent the same surface dashAI gives a person through its GUI: look at datasets, see which models are available, train, follow queued work, and read the metrics.

"Train a random forest on dataset 3 predicting 'species' and tell me the F1"

Status

v0.2.2 — verified against a running dashAI 0.9.7.post1. Deterministic tests (including the predict two-step path) plus live runs: dashai_train_model → dashai_job_status → dashai_get_run with metrics, and dashai_predict → dashai_job_status finished.

Live tabular run (seed students)

Public dashAI seed only. Target placement_status (~83% majority). exam_score was left out of the inputs (it leaks the label). Split 70/15/15. Goal metric: BalancedAccuracy — Accuracy and F1 on the majority class are traps.

The dummy *wins F1* by always answering the majority class. The linear model beats the forest on the metrics that actually measure separation. Each row is its own 70/15/15 draw (not the same test rows) — still enough to stop treating the forest as the default. If a client reports only F1 here, it is lying.

dashai_predict on the finished forest run returned prediction_id and the job finished. That scores the same seed dataset the model was trained on, not a held-out file — do not read it as a generalization check. dashai_get_prediction returns {n, n_classes, class_counts} only.

Live image run (seed cifar10-subset)

Public seed: 200 images, frog vs truck (100/100). LeNet5ImageClassifier, CPU, 32×32. Split 70/15/15 → test n=30. Chance is 0.5.

\*Accuracy, not BalancedAccuracy — on the one-class val/test they collapse. Chance on this seed is 0.5. Dummy tabular does not apply (image task).

The image *path* works. The *0.867 is not a result*: dashAI defaulted shuffle=False, so val/test were 30 trucks and zero frogs. MCC 0 is sklearn on a one-class split. After this server sent shuffle=true + stratify=true, both classes are in val/test and MCC is no longer 0. Train hits 1.0 (140 images memorized). Val 0.767 is the honest-ish number; test 0.900 is a 30-row lottery. Do not publish a leaderboard line.

Verifying against a live instance surfaced gaps between dashAI's documentation and its actual behaviour. Each one has its own regression test. A sixth — sequential splits with shuffle=False — only showed up live because MCC came back 0 next to a moving Accuracy. dashai_predict sending run_id to PredictJob — only showed up live (KeyError: 'prediction_id') because there was no predict test.

Install

pip install dashai-mcp

or: uv pip install dashai-mcp

In your MCP client configuration:

{

"mcpServers": { "dashai": { "command": "dashai-mcp" } } }

dashAI has to be running separately (dashai, or the desktop app). It is looked up at http://localhost:8000 by default.

Tools

Six things dashAI's documentation (or defaults) get wrong

Found by running against a real instance. If you are writing a client for this API, these will bite you:

The component registry also has 13 types, not the four the documentation suggests: Task, GenerativeTask, Model, GenerativeModel, DataLoader, DatasetSource, Metric, Optimizer, Job, LocalExplainer, GlobalExplainer, Explorer, Converter.

And GET /run/{id} returns split_indexes with the full list of indices: on a 10,000-row dataset that is 59 KB, 99% of the response. This server replaces it with the per-split counts, bringing the response down to ~1 KB.

Three design decisions

1. Ten tools, not 142

dashAI exposes 142 REST endpoints. Generating one tool per endpoint is mechanical and it is a mistake: a model with 140 tools burns context reading the catalogue and chooses worse. These ten cover the actual working path.

2. dashai_train_model collapses three calls

In the raw API, training is a chained sequence:

POST /model-session/   → creates the experiment

POST /run/ → creates the run POST /job/ → enqueues the ModelJob

With required fields the GUI fills in on its own and that are undocumented — plot_history_path, plot_slice_path, plot_contour_path, plot_importance_path. On top of that, `splits` travels as a JSON string, not an object, even though dashAI's documentation shows it as an object: the backend's Pydantic schema declares it str. That kind of detail is exactly what makes an agent fail against the raw API.

Here it is a single call, and it does not block: training can take hours, so it returns the job_id immediately and progress is polled with dashai_job_status.

dashai_predict does the same for the two-step GUI path: POST /predict/ then POST /job/ with prediction_id.

3. No tool deletes anything

dashAI's API has no authentication — checked endpoint by endpoint. That is coherent for something local-first, but it means there is no barrier between a misread sentence and an irreversible DELETE /dataset/{id}. Deleting is done from the GUI, looking at what is being deleted.

For the same reason, the server refuses to point at a non-local host:

DASHAI_BASE_URL points to 'ml.example.com', which is not local, and dashAI's API

has no authentication: exposing it to the network leaves the backend open to anyone who can reach it.

This can be disabled on purpose with DASHAI_ALLOW_REMOTE=1, if the target is protected some other way.

Configuration

dashai_get_prediction needs pyarrow in the MCP process to turn the Arrow file into class counts (pip install 'dashai-mcp[counts]', or install the MCP into the same env as dashAI). Without it the tool still returns status and refuses to dump rows.

Development

python -m venv .venv && .venv/bin/pip install -e ".[dev]"

.venv/bin/python -m pytest tests/ -q

The tests stub the HTTP responses with respx: they need neither a dashAI instance nor credentials. They test the contract — which calls are made, in what order, with what body, and what the agent is told when something fails.

Verifying against a live instance

A client tested only against stubs is a hypothesis. scripts/smoke_live.py exercises every tool against a real running dashAI, through the same code paths an agent uses:

python scripts/smoke_live.py            # read-only tools

python scripts/smoke_live.py --train # + a real train -> predict loop

The --train loop creates a model session, a run and a prediction on the target instance — point it at a scratch instance, not a production one. Exit code 0 only if every exercised tool worked. This is how each release gets verified; the version it was last run against is what dashai_server_info reports under compatibility.verified_against.

API compatibility check

dashAI exposes no version endpoint, so dashai_server_info reads the instance's openapi.json and compares the API surface against what this server actually calls: are the endpoints still there, and does POST /model-session/ require fields this server does not send? The verdict comes back as compatibility.status — ok, mismatch (with the exact differences named) or unknown (schema unreadable; everything else may still work). The case it exists for is real: dashAI's development branch already adds an evaluation_strategy field to model sessions.

A note on the SDK

Requires the MCP Python SDK 2.x. Version 2.0 removed mcp.server.fastmcp; it is now mcp.server.mcpserver.MCPServer, and annotations are ToolAnnotations objects instead of dictionaries. Most tutorials still show the 1.x API.

License

MIT, same as dashAI. See the note at the top on affiliation.