Fittest

Fittest

The fittest survive, not the smartest. An evolving agent council that decides what gets compute and lab time.

Intelligence is abundant; ground truth is not. ML models screen candidate molecules and materials by the million, but only a few get a high-fidelity calculation (DFT) or a lab experiment. Fittest is a council of three independent foundation machine-learned interatomic potentials (MLIPs: MACE-MPA-0, AIMNet2, Orb-v3) and four role agents (Skeptic, Calibrator, Oracle, Curator) that share one log. For every structure it decides which answer to serve, how far to trust it, and which structures get ground truth. In this repo ground truth is DFT read from Transition1x (lookups, not live calculations); the decision is the same for lab time.

The project asks two questions, each tested against baselines or a pre-registered rule:

1. What if the models decide as a council? Their disagreement tells the council when to be skeptical (§2.3, §3.3).
2. Can the council evolve to give better predictions? Yes, once the test for new members was fixed: fine-tuned children replace their parents (§2.5).

Built at ScienceClaw 2026 (MIT Media Lab) for the NVIDIA ALCHEMI challenge, on NVIDIA ALCHEMI Toolkit 0.3.0-rc. Formerly TrustSwarm; the old GitHub URL redirects here.

• Code: this repo (fittest/ package, MIT licence)
• Hosted MCP server: https://fittest-mcp.westus2.cloudapp.azure.com/mcp
• Authoritative numbers, all generated from the result files: results/RESULTS.md (Idea 1), results/evolution/SUMMARY.md (Idea 2) and results/story_numbers.json (scripts/story_numbers.py: the critic values, run F's lineages and its absolute errors)

---

TL;DR

• Problem: the best model fails silently where it matters. On 287 held-out Transition1x reactions every model's force error peaks at the transition state, the top of the energy hill that sets the reaction rate. AIMNet2 leads at the reactant (median 0.22 vs 0.35 eV/Å for Orb) and loses its lead at the transition state (0.46 vs 0.47), with no warning. Its own seed ensemble does not warn either: its spread is 2.4× smaller than its error.
• The council: each decision has an owner, no central planner. Three models answer independently and never see each other's answers. The Calibrator keeps the track record and chooses what to serve, the Skeptic audits trusted answers at random and vetoes, the Oracle spends the ground-truth budget, and the Curator breeds and retires members. They coordinate only through an append-only log, so every decision replays. The agents are deterministic programs, not LLMs; an LLM agent uses the whole council as one tool (MCP).
• Idea 1: decide as a council (leak-free split, 200 DFT label lookups per arm).
• Disagreement between independent models flags when the best model is wrong, with no labels: AUROC 0.917 against "AIMNet2 wrong" (pool split, 1,672 frames). Copies of one model are overconfident: their spread is 2.4–36× smaller than their error.
• Served force RMSE 0.206 eV/Å vs 0.286 for the best baseline (−28%). The council discovered from 200 labels that averaging AIMNet2 and Orb is the best answer; that average, picked in hindsight with eval labels, is as good (0.202).
• Adding one role, the Skeptic's random audits of trusted answers, cuts calibration error (ECE) from 0.370 to 0.079 (4.6×).
• Negative: the council's own risk score ranks its remaining errors worse than the simple baselines (AUROC 0.73 vs 0.80–0.87).
• The worst model is the best critic. As an answerer MACE-MPA-0 is wrong on 99.9% of eval frames (Orb 91.3%, AIMNet2 57.2%). As a critic it is the only foundation model whose removal makes the council worse at spotting mistakes: mean ΔAUROC +0.040 over 491 fitness checks (AIMNet2 −0.067, Orb −0.037).
• Idea 2: let the council evolve (pre-registered, failures included). The Curator breeds fine-tuned children on the labels the council already paid for; a child must beat its parent on labels it never trained on, and members that add nothing are retired. The first four designs (runs A–D) failed their rule or were not confirmed. A diagnosis showed why: the test's energy check on a few labels was noise and threw out the best children. With a force-only test (run F, 12 new seeds, 800 lookups for both arms) the evolving council beats the static council in 12 of 12 seeds: −0.026 eV/Å [−0.030, −0.021] (−12%), and −14% on 2,475 structures from new chemical formulas that no run had read. In total that is 0.181 vs 0.286 eV/Å against the best Idea 1 baseline on the same eval frames (−37%), and below the hindsight average. 108 children were bred, 43 sit in the final councils, and 32 of the 36 foundation models were retired. Caveat: growing the council without selection (run E) already gives −0.018 of the −0.026; selection's own share is not isolated yet.
• Use it now. Add the hosted MCP server to Claude Code: claude mcp add --transport http fittest https://fittest-mcp.westus2.cloudapp.azure.com/mcp, then ask *"Use fittest to check whether I can trust an ML potential on this water molecule: O 0 0 0, H 0.96 0 0, H -0.24 0.93 0."* Structures are never stored (logging policy).

---

Contents

1. Use it now — MCP server, local install, Python API, ALCHEMI hook
2. How it works — the problem, the council, Idea 1, the worst model as critic, Idea 2 (evolution), ALCHEMI usage
3. Validation — data, split, metrics, the Idea 1 ablation, self-repair
4. What failed and limitations — and next steps
5. Reproduce
6. Repo map, privacy, licence, acknowledgements

---

1. Use it now

1.1 Hosted MCP server (no install)

fittest-mcp exposes the council to any MCP client over streamable HTTP. The three model agents (MACE-MPA-0, AIMNet2, Orb-v3) run as warm CPU processes on an Azure VM, and their disagreement is turned into a verdict with the Calibrator state learned in the final run (fittest/snapshots/trust_snapshot.json, identical to results/trust_snapshot.json). Landing page: https://fittest-mcp.westus2.cloudapp.azure.com/, public counts at /stats, health at /health. Full guide: docs/MCP.md.

Limits: ≤ 20 structures per call, ≤ 100 atoms each, non-periodic, coordinates within ±1000 Å, no atoms closer than 0.3 Å; element symbols must be exact (Cl or cl; PDB-style C1 is fine, ambiguous CA or HE1 is an error); per-client rate limit (burst 20, then 1 call per 2 s) plus a per-IP limit; 120 s per request; model evaluations are serialized.

The verdict rule. One rule feeds check_structures, explain and the usage counts. A structure gets trust only if all four hold:

1. every model agent supports its elements (AIMNet2 sets the limit; the full committee covers H B C N O F Si P S Cl As Se Br I), otherwise send_to_dft with reason unsupported_elements and no trust score;
2. the forces do not all vanish (max |F| < 0.05 eV/Å for every model means symmetry or exact equilibrium, where disagreement carries no information);
3. σ_F < τ;
4. trust_score ≥ 0.5.

Otherwise the verdict is send_to_dft, with every failing condition listed in verdict_reasons. τ = 0.620 eV/Å is fixed: it is the 70th percentile of σ_F on the round-0 pool, which the Calibrator's threshold rule never moved. trust_score is the Calibrator's Beta posterior mean of P(served answer correct) for the served candidate in the structure's (σ_F, mean |F|) bucket, built from DFT oracle labels on Transition1x with prior Beta(2, 1), so it is never exactly 1. It comes with trust_score_90ci (the 90% credible interval) and labels_behind_score (the DFT labels in that bucket); a bucket with few labels gives a wide interval, so read the score with it. "Correct" means force RMSE ≤ 0.25 eV/Å and relative energy error ≤ 0.25 eV (relative to the reaction's reactant). The ECE 0.080 in §3.3 is for arm D's risk score (max of the isotonic P(served wrong) and the Skeptic's p_S); the calibration of this per-bucket trust_score has not been measured beyond the 12-frame spot check below.

Claude Code

claude mcp add --transport http fittest https://fittest-mcp.westus2.cloudapp.azure.com/mcp

Cursor (~/.cursor/mcp.json)

{ "mcpServers": { "fittest": { "url": "https://fittest-mcp.westus2.cloudapp.azure.com/mcp" } } }

Claude Desktop / claude.ai: Settings → Connectors → Add custom connector → https://fittest-mcp.westus2.cloudapp.azure.com/mcp.

Example prompt: *"Use fittest to check whether I can trust an ML potential on this water molecule: O 0 0 0, H 0.96 0 0, H -0.24 0.93 0."* Atoms may be separated by newlines, commas or semicolons. In the Round-2 verification run this prompt, used verbatim, gave trust, trust score 0.857 (90% CI 0.61–0.99, 4 labels behind it), recommended AIMNet2, σ_F = 0.378 eV/Å below τ = 0.620 eV/Å (docs/MCP.md, verification log).

Hosted CPU vs the GPU run. On the 12 held-out Transition1x frames of results/service_demo.xyz the hosted server's σ_F ≥ τ flags agree 12/12 with the earlier GPU run (results/service_demo.json), with max |Δσ_F| = 0.0005 eV/Å. Under the verdict rule 7 of the 12 are trusted and none of the trusted frames is DFT-wrong; the earlier τ-only rule trusted 2 DFT-wrong frames.

Domain. The Calibrator learned from gas-phase C/H/N/O Transition1x reactions with at most 7 heavy atoms, including transition states and NEB paths. A domain_warning is added for elements outside C/H/N/O that the committee still supports, large