Studydiff

StudyDiff

Understand why two scientific studies reach different conclusions — with every claim verified against the source.

Live demo · Quick start · How it works · Examples · MCP server · Roadmap · Contributing

Two well-run papers often reach opposite conclusions. Usually the reason isn't that one is wrong — it's a methodological difference (a cell type, a dose, a follow-up window, an analysis choice) that a reader has to dig out of the methods sections by hand. StudyDiff does that digging. Give it two studies and it extracts each one's design, surfaces the differences that could explain the disagreement, and — critically — grounds every statement in the source text, so it never invents a finding.

It is built for a bench scientist deciding which of two conflicting papers to trust before planning an experiment.

Why it's different

Most "AI literature" tools generate a fluent answer and ask you to trust it. StudyDiff inverts that:

• It shows the evidence, then gets out of the way. The design differences up top; every value with the verbatim sentence that supports it underneath — and click any verified value to see that exact sentence highlighted in the paper's own text.
• It refuses to guess. Any field the source doesn't state is shown as *not reported*, never inferred.
• It verifies itself. A deterministic grounding check (no second LLM acting as judge) confirms every extracted value and every explanation is backed by a verbatim quote and traceable numbers. Anything that fails is downgraded *before* it can be used as a reason.
• It knows what it can't do, because it was measured — on two independent benchmark sets, the second one held out and scored once. See below.

Does it work? A measured answer

StudyDiff used to rank the divergent design dimensions and present the top one as the likely driver of a disagreement. We built a benchmark to test that, and it doesn't work.

15 documented contradictions where the literature has since established *why* the papers disagreed — each label carrying its own citation, the set built blind before any accuracy number existed. Scored against StudyDiff's top-ranked driver:

Top-1 accuracy (strict)      13.3%  (95% CI 3.7-37.9%)   [2/15]

Baseline "always say assay" 13.3% (95% CI 3.7-37.9%) [2/15] → discordant on 0 of 15 cases Oracle ceiling (reachable) 66.7% (95% CI 41.7-84.8%) [10/15] Non-assay-labelled cases 0.0% (95% CI 0.0-22.8%) [0/13]

The ranking was a fixed prior (DRIVER_RANK in src/compare.mjs) in which assay outranks everything. Two papers almost always use somewhat different methods, so assay almost always diverges, so it was picked 13 times out of 15 — and the two hits are exactly the two assay-labelled cases. It is not merely as good as guessing a constant; it is behaviourally identical to it on every case in the set.

Fixing grounding first (Phase 2) removed that excuse. Recovering 20 of 26 false-positive rejections doubled the oracle ceiling from 33% to 67% — the established cause is now an available candidate in 10 of 15 cases instead of 5 — and top-1 accuracy did not move at all. The ranker was handed the right answer five more times and took none of them.

What changed as a result: the app no longer nominates a primary driver. It presents the divergent dimensions as an unranked list, because that list is informative (it contains the established cause 10 times in 15) while the ordering is not. Choosing among them needs domain knowledge the tool doesn't have.

What still holds: which dimensions differ, which are identical (*ruled out*), and the verbatim sentence behind every value. None of that depends on the ranking.

Confirmed blind, on a second set

Those 15 cases had by then been read across two phases — failures analysed, spans re-audited — so every post-fix figure from them is *development-set* accuracy, not a blind measurement. So we built a second set and measured it once.

eval/cases-heldout.json is 15 further documented contradictions, curated to a protocol written and committed *before any case was selected* (eval/HELDOUT-PROTOCOL.md), by a curator kept blind to the dev set's per-case failures, in deliberately different fields: microbiome, marine ecology, toxicology, psychology, critical care, oncology, infectious disease. No paper and no contradiction is shared with the dev set — selftest enforces that mechanically.

Top-1 accuracy (strict)      13.3%  (95% CI 3.7-37.9%)   [2/15]

Baseline "always say assay" 20.0% (95% CI 7.0-45.2%) [3/15] → discordant on 1 of 15 cases Oracle ceiling (reachable) 73.3% (95% CI 48.0-89.1%) [11/15] Non-assay-labelled cases 0.0% (95% CI 0.0-24.3%) [0/12]

On unseen data the prior scores below the constant guess — by exactly one case. The intervals overlap almost entirely and the two strategies disagree on 1 of 15, so the honest statement is that it remains indistinguishable from guessing assay every time, not that it is worse.

The line that does not move is the last one. Across both sets, 25 cases where the established cause was something other than `assay`, the prior identified none of them. And the ceiling here is *higher* than on the dev set — 73.3% against 66.7% — so extraction put the right answer in front of the ranker more often, and it was taken no more often. That is the Phase 1–2 conclusion reproduced on data the development loop never saw, which is the only way it could have been strengthened.

The number is reported as-is, and is never pooled with the dev-set number: summing them into an "n=30" figure would relaunder read data as blind data. Nothing in src/ was changed on the basis of it. The set was fetched once on a single pre-registered arm, but the scoring was not one clean pass — the first run reported n=14 after one paper failed to fetch, and the full set was scored after retrieving it. Both figures, the sixteen defects an adversarial verification pass found and corrected *before* any scoring, and the reasoning behind each label are recorded in the file's own provenance block rather than summarised away.

npm run eval:heldout            # the blind number, offline and free

npm run eval:selftest:heldout # set integrity, incl. zero overlap with the dev set

Full method, the pre-registered decisions, and every prediction that turned out wrong: eval/README.md and eval/PHASE2.md. The benchmark sets are eval/cases.json (development) and eval/cases-heldout.json (held-out).

npm run eval            # offline, free, no API key — regenerates the numbers above

npm run eval:selftest # validates the harness maths and set integrity

eval/cache/ and eval/cache-heldout/ are committed on purpose. They aren't build output, they're evidence: the published numbers are reproducible from artefacts in the repo rather than taken on faith.

Quick start

Under 60 seconds, no API key, no network:

git clone https://github.com/nickjlamb/studydiff && cd studydiff

npm install npm run demo # explains a real, famous contradiction npm run demo -- resveratrol-sirt1 # a second worked example npm run demo -- treg-stability # a third: Treg lineage stability

Run the web app:

cp .env.example .env    # add ANTHROPIC_API_KEY for live comparisons

npm run serve # http://localhost:4173

The built-in examples run on cached data and need no key. To compare live, add your key and use the PMID / DOI, Upload PDF, or Paste inputs.

How it works

flowchart LR

IN["Two papers PMID · DOI · PDF · text"] --> R["Retrieve PubMed / PMC / PDF"] R --> E["Extract Claude → structured study cards"] E --> V["Verify deterministic grounding"] V -- "ungrounded → not reported" --> E V --> C["Compare divergent vs. shared design"] C --> X["Present divergent dimensions + ruled out"]

1. Retrieve — a PubMed/PMC client with a full-text-to-abstract fallback that tags how deep it read (fulltext / abstract / pasted); uploaded PDFs are text-extracted server-side.
2. Extract — Claude turns each paper into a fixed study card (species, model, assay, dose, timing, endpoint, sample size, statistic, finding, limitations). Every field carries a verbatim supporting quote; absent fields default to *not reported*.
3. Verify — grounding runs first: any value whose quote isn't in the source, or whose numbers don't trace, is downgraded to *not reported*. StudyDiff can't cite a fact it hasn't verified.
4. Compare — deterministic: which design dimensions are reported by both papers, which of those diverge, and which are identical. Divergence is a token-overlap test on design dimensions and strict inequality on the conclusion itself, because a conclusion's *polarity* matters where a method list's *content* does.
5. Present — the divergent dimensions unranked, each with both papers' values and the verbatim sentence behind them, and the identical ones explicitly *ruled out*. StudyDiff does not nominate one as the cause: it was measured and it doesn't work.

Two things hold the guarantee up: the API key never leaves the server, and verification is deterministic — grounding and comparison involve no model judgment at all, so the same verified evidence always yields the same divergent dimensions.

Why Claude?

StudyDiff's reliability comes from *how* it uses Claude, not just that it does:

• Structured extraction via tool-use. Each paper becomes a study card through a forced Claude (Sonnet) tool schema, so every field returns validated and carries a *verbatim supporting quote* — no free-te