Canli Validation

canli-validation-mcp

Is your best backtest real, or just the luckiest of the variants you tried? This MCP server lets Claude, Cursor or any MCP client answer that with the standard corrections: the deflated Sharpe ratio, the CSCV probability of backtest overfitting, data-snooping tests on every variant tried (Hansen's SPA, White's Reality Check, Romano-Wolf StepM), the minimum track record length, the haircut Sharpe ratio, and luck-equivalent trials. Free and MIT-licensed.

Quick start

# Claude Code, computed on your machine: no key, nothing sent anywhere

claude mcp add canli-local --env CANLI_LOCAL=1 -- npx -y canli-validation-mcp

the same tools with signed, stored receipts from the free API (a free key is issued on first use)

claude mcp add canli -- npx -y canli-validation-mcp

nothing to install: the hosted endpoint

claude mcp add --transport http canli https://canlicapital.com/mcp

Then ask, for example: *"I tried 229 variants and kept the best: an annualised Sharpe of 1.5 over 730 daily returns (365 a year), skew -0.5, kurtosis 5, and the variants' Sharpe ratios spread by 0.57. Is it real?"* The assistant calls validate_deflated_sharpe: the best of 229 skill-less variants would reach 1.60 by luck alone, so the probability that the Sharpe is above zero falls from 98.1% to 44.4% once the search is counted. Claude Desktop and any other stdio client (Cursor, VS Code) run the same npx command; see "Claude Desktop" and "Generic stdio client" below.

What it does

An MCP (Model Context Protocol) server over canlicapital.com's free, keyed validation API, with a lab that computes on your machine. It gives a coding agent twenty tools: issue a free key, run the nine validators (deflated Sharpe, CSCV overfitting, data-snooping tests (Hansen's SPA, White's Reality Check and Romano-Wolf StepM), paper-evidence conformance, breadth ceiling, minimum track record length, minimum backtest length, haircut Sharpe ratio, luck-equivalent trials), audit one backtest with several of them in a single call, verify a receipt's signature offline, read service status, read a company's reported financial history from SEC filings, and six lab tools: backtest a rule over a parameter grid, summarize a long series, stress-test a strategy's returns, check a plan against a broker's limits, check a signal for lookahead, and test a whole research pipeline on placebo data. Every validation result carries, beside the number, the sentences that say what it does not establish and the receipt that records it, success or error, so the agent cannot see a number without its limits.

This package is published to npm as canli-validation-mcp. Also listed on the official MCP Registry (io.github.arhancanli/canli-validation-mcp) and cursor.directory. Run it with npx, no install step, as shown below. A local checkout is only needed to develop or test this package itself; see "Local checkout" near the bottom.

This README describes the version in package.json. Unversioned npx runs npm's latest release; npx -y canli-validation-mcp@ pins one.

How well agents use it

A fixed benchmark gives models these tools and scores whether they pick the right one and return the right answer, against ground truth computed from the same checked code. Results for three models, with every run recorded, are in bench/agent/README.md.

What the API is (and is not)

The engine is the product. The service runs your submitted numbers through the same honesty arithmetic canlicapital.com's own paper record runs on itself and hands back a verdict anyone can recompute from the receipt. It signs every receipt, does not accept market data, does not grade a strategy, and never saw your data source, its costs, or any lookahead in how a series was built. See docs/superpowers/specs/2026-09-05-developer-key-validation-api-design.md in the main repository for the full design.

Tools

Unreleased source adds local journal binding to validate_paper_evidence: run CANLI_LOCAL=1 node mcp/src/server.mjs from the repository, and send exactly one of record or record_file, plus journal_file to check source hashes, signatures and recomputed claims. record_file accepts a standard record, a {record,signature} envelope, or the full execution export bundle; companion metrics/observations are checked for full bundles. Signed records need the detached signature. bindings.all_match must be true for source validation to pass. Without a journal, the result explicitly reports structure-only conformance. Files and signatures are refused in hosted/remote mode and never uploaded. Local results have no Canli receipt. The current npm/hosted release retains the existing record-only API.

validate_deflated_sharpe accepts exactly one of two input shapes, never a mix of both:

• the seven contract fields: observed_sharpe_annualized, observations, periods_per_year, skew, non_excess_kurtosis, effective_independent_trials, cross_trial_sharpe_sd_annualized;
• or a return series plus the trials behind it: returns, periods_per_year, effective_independent_trials, cross_trial_sharpe_sd_annualized.

Sending fields from both shapes, or from neither, is rejected before any request leaves the process; see src/schemas.mjs.

validate_reality_check takes the returns of every variant the search tried, one row per period and one column per variant, as matrix or as matrix_file (a CSV or JSON on your machine), and an optional benchmark series; with none, variants are tested against zero. It runs three tests on one seeded stationary bootstrap:

• Hansen's SPA: the chance that the best variant's studentized excess return is at least this good if no variant has an edge, with lower and upper bounds (spa.p_value, the consistent p-value, is the headline, and monte_carlo_se its sampling error);
• White's Reality Check: the same question without studentizing, so one volatile variant can dominate it;
• Romano and Wolf's StepM: which variants beat the benchmark, with the familywise error held at alpha.

Studentizing divides a variant's mean excess return by its standard deviation. Each bootstrap resample is divided by its own standard deviation, worked out again from that resample, as Romano and Wolf (2005) recommend, and SPA and StepM both use these figures. Earlier versions divided every resample by one figure estimated once from your data. That ignores how noisy the estimate is, and the test rejected too often: on Null Zoo v1's searches with no edge (20 variants, 504 days, 1,000 resamples) it rejected 7.5% at a nominal 5% with normal returns and 12.5% with negatively skewed returns. On 4,000 new searches of each kind the new version rejected 5.1% and 5.4% (6.9% with autocorrelated returns, where the Reality Check also rejected 6.6%). Studentizing helps find a skilled variant that is less volatile than the rest and costs power when the skilled variant is the most volatile; the Reality Check, which does not studentize, is reported beside it for that case.

A p-value is the share of resamples at least as extreme as your result, ties included. When no variant beats the benchmark, the SPA statistic is 0, so its p-values are 1 and StepM names no variant. Earlier versions counted only resamples strictly above 0 and could report a p-value near 0, a rejection, in exactly that case.

A result reproduces exactly from its seed. On four fixed-seed cases the p-values agree within Monte Carlo error with Python's arch 8.0 (run without studentizing) and with a separate numpy implementation of the per-resample studentization, and the StepM sets match both (js/snooping-core.test.js). Only the variants sent are counted: a search that tried more than it sends makes luck look smaller than it was.

company_financial_history is different from the other tools: it reads the public company reference at canlicapital.com, not the validation API. Give it a CIK (1 to 10 digits) to list a company's available financial histories, or a CIK and a us-gaap concept such as Revenues to get the observations, newest first (limit defaults to 40, maximum 200). Every observation keeps its filing accession, form, filed date and unit, and the result carries the SHA-256 of the original SEC response and the record's own boundary sentence: these are accounting values as reported to the SEC, not market prices, returns or a recommendation. Companies and concepts outside the current release return an error with the available concepts listed.

The lab: backtest, summarize, stress, feasibility, lookahead, placebos

Six tools compute in the server process itself, on your machine or on the hosted endpoint, with nothing sent to the API and no receipt stored. Each result names the SHA-256 of exactly the numbers it was computed from, and states what it does not establish.

• `backtest_strategy` runs a rule over every parameter set in a grid, on your prices: sma_cross (fast, slow), momentum (lookback, skip), mean_reversion (window, entry_z), breakout (lookback) or buy_and_hold, long-only or with allow_short, after a cost per unit of turnover. The position held over each period is decided from closes up to the one before it, so no rule can see the future (a test changes later prices and checks that no earlier position moves). It then validates the best variant with the number of variants this call ran, not a number someone declared: the deflated Sharpe ratio across the grid's measured Sharpe dispersion, the CSCV probability of backtest overfitting on every variant's returns, the probabilistic Sharpe and the minimum track record, next to buy and hold over the same window. At most 200 variants and 20,000 prices per call. No code is ever run: a strategy i