Research Repo Doctor

Research Repo Doctor

Get your research artifact ready for Artifact Evaluation before the deadline: scan the repo, scaffold the easy fixes, verify the run path, and generate the appendix.

Web demo for public repos: https://research-repo-doctor-bckncrcwwmg6jrbsrd6btj.streamlit.app/ If the hosted demo is waking up or reinstalling dependencies, the local zero-install command below is the reliable fallback:

uvx rrdoctor scan .

Maintainers can check anonymous demo access with python scripts/check_live_demo.py https://research-repo-doctor-bckncrcwwmg6jrbsrd6btj.streamlit.app/.

rrdoctor is a local CLI and GitHub Action for research artifact preparation. It audits whether a repo is reviewable, citable, and close to runnable; scaffolds safe mechanical fixes; maps findings to an AE-style readiness level; and turns the rest into a checklist any coding agent or human can finish.

Make it a pull-request gate

The GitHub Action is the main adoption path: run the CLI while preparing the artifact, then keep the same deterministic preflight on every pull request.

name: Reproducibility preflight

on: pull_request: push: branches: [main]

permissions: contents: read

jobs: rrdoctor: runs-on: ubuntu-latest steps:

• uses: actions/checkout@v7
• uses: Tom409114/research-repo-doctor@v0.2.24 with: profile: standard fail-on: error

The full GitHub Action guide covers sticky PR comments, new-finding baselines, agent plans, appendices, and complete AE prep packets.

AE deadline loop

uvx rrdoctor prepare . --profile acm --out-dir rrdoctor-prep

Or run the pieces explicitly:

uvx rrdoctor scan . --profile acm uvx rrdoctor fix . --write uvx rrdoctor appendix . --profile acm --output ARTIFACT_APPENDIX.md uvx rrdoctor verify . --profile acm uvx rrdoctor verify . --profile acm --run --timeout 600 --fail-on error # trusted repos only

Or pin the official quickstart command as the dynamic gate:

uvx rrdoctor verify . --profile acm --command "python train.py config/default.py" --run --timeout 600 --fail-on error

For trusted repositories, rrdoctor verify --run can go beyond static checks. For supported Python repositories it creates a temporary isolated environment, installs declared dependencies, and executes the declared entrypoint there under a timeout. Other ecosystems retain an explicit resolver preflight. With the default gate (--fail-on error), failed or blocked dynamic L2/L3 steps return a nonzero exit code. Use --command when the artifact has a specific smoke-test or quickstart command that reviewers should run. Dynamic mode may run dependency build/install hooks as well as the entrypoint, so never use it on untrusted code. rrdoctor prepare writes the report, agent plan, artifact appendix, and verification ladder into one local evidence directory.

Artifact Evaluation chairs and lab maintainers can use the AE chair guide for optional pre-submission wording and CI examples.

For public calibration evidence, see the 80-repository data brief and the nanoGPT first-run regression case study.

What it catches

• "Your `--seed` flag does nothing." RRD052 spots code that declares a seed option but never calls random.seed, np.random.seed, torch.manual_seed, tf.random.set_seed, or random_state=seed.
• "This worked on my laptop." Local-only data paths, missing data provenance, and undocumented retrieval steps.
• "The environment silently changed." Unpinned dependencies, missing runtime versions, undeclared imports, and absent dependency manifests.
• "The notebook lies." Stale outputs, out-of-order execution, checkpoint artifacts, and secret-like notebook output.
• "Reviewers cannot tell how to cite or rerun this." Missing license, citation, CI, tests, changelog, results provenance, or experiment entrypoint.

Install

Run once, without installing:

uvx rrdoctor scan .

Alternatives:

pipx run rrdoctor scan .

pip install rrdoctor rrdoctor scan .

Developer install from source:

git clone https://github.com/Tom409114/research-repo-doctor.git

cd research-repo-doctor python -m pip install -e ".[dev]" rrdoctor scan .

Fix the easy gaps

Let rrdoctor create the safe scaffolding for you. It is deterministic, idempotent, and never overwrites existing files.

rrdoctor fix . --write

It can scaffold missing governance docs, citation metadata, data/results provenance notes, a reproducible-seed helper, changelog entries, and common research .gitignore entries. The hard parts become a reviewable plan:

rrdoctor plan . --output plan.md

Use with your coding agent

Paste this into Claude Code, Cursor, GitHub Copilot, or any other coding agent:

Use rrdoctor as the deterministic, offline, no-API-key grader for this research repo.

Run: rrdoctor scan . --format json --output baseline.json rrdoctor plan . --output plan.md

Work through plan.md without weakening rrdoctor checks.

Definition of done: rrdoctor scan . --baseline baseline.json --fail-on-new error

The final command is the objective gate: it verifies the agent's work against the starting baseline and fails only on newly introduced errors.

Copyable agent templates are available for Agent Skills / Claude Code-style workflows and Cursor project rules under integrations/.

Keywords: research software, reproducibility, artifact evaluation, repository audit, auto-fix, coding agents, AGENTS.md, GitHub Action, notebooks, data availability, citation metadata.

Help calibrate the rules

The fastest way to improve rrdoctor is real scan feedback from real research repositories. After one scan, the low-friction 10-minute trial report asks only what was useful, wrong or missing, and whether the tool fits a real deadline. For a focused rule report, please open a false-positive, false-negative, scan-case, or new-rule issue. Include the rule ID, command, rrdoctor version, and a sanitized minimal repo shape. See feedback and calibration for the short checklist.

Why this matters

Research code often lands on GitHub under deadline pressure. A reviewer or future lab member finds a promising repository and then loses hours because the environment is underspecified, data paths are local, notebooks contain stale outputs, dependencies are unpinned, or the citation is unclear.

Research Repo Doctor turns those recurring release blockers into deterministic checks with concrete remediation - and, where it is safe to do so, scaffolds the mechanical starting points. It is built to sit in the ordinary maintenance path: run locally while preparing a release, then run automatically on pull requests through GitHub Actions.

The audit runs without an AI API key, network access, or hosted service. That same determinism makes it an honest grader: it can verify fixes made by a person or a coding agent.

audit -> fix -> plan -> (your coding agent / you) -> verify -> PR

| | | | | | rrdoctor plan rrdoctor scan --baseline | rrdoctor fix --write --fail-on-new error rrdoctor scan

What's new in 0.2.24

• Better systems-artifact coverage: existing checks now recognize Cargo, CMake, Meson, container, and Nix evidence plus documented artifact scripts and Cargo commands across entrypoint, test, CI, and release workflows.
• Quieter static scans: target-source SyntaxWarning noise is suppressed, and escaped Python exception text no longer looks like a local Windows path.
• A rechecked first-run trust case: public PyPI rrdoctor==0.2.23 still scores nanoGPT as Functional at 76/100 with 0 errors, while the original noisy RRD050 and RRD063 findings remain absent.
• An explicit JOSS boundary: the draft-paper notes now separate the more-than-six-month public-history gate from research-use, authorship, conflict-of-interest, and AI-disclosure evidence that maintainers must supply.
• A configuration file that actually governs the scan: CLI and MCP calls load the target repository's .rrdoctor.yml from any working directory; profile, report, threshold, failure, include/exclude, and rule settings now work beneath explicit CLI overrides.

What's new in 0.2.23

• A safer release path: the hosted demo can stay pinned to the latest published PyPI package while a new release propagates, avoiding broken Streamlit deployments during Trusted Publishing.
• A first-class GitHub Action path: a compact pull-request workflow now appears near the top of the README and is guarded by public-readiness tests.
• A shorter feedback loop: completed web-demo scans link to a structured 10-minute trial report for useful findings, false positives, false negatives, and real deadline fit.
• Official MCP Registry distribution: v0.2.24 is published as io.github.Tom409114/rrdoctor, with MCP support remaining an optional extra rather than a core scanner dependency. Registry inclusion is distribution metadata, not endorsement or evidence of adoption.
• Stronger dynamic verification coverage: an offline integration test proves L3 can install and import a declared local wheel inside its temporary Python environment.

What's new in 0.2.22

• Real Python environment verification: rrdoctor verify --run now creates a temporary isolated venv, installs the repository's declared dependencies, and runs L3 with that interpreter and PATH instead of stopping at a resolver dry run.
• Honest execution boundaries: verification reports disclose dependency build-hook risk, redact machine-specific temporary paths, and remove the temporary environment after the run. Other ecosystems retain an explicit resolver preflight.
• Stronger isolation checks: inherited PYTHONHOME and PYTHONPATH values are removed, and versioned launchers such as python3.10 are redirected to the temporary environment.

What's new in 0.2.21

• More honest corpus evidence: the public 80-repository calibration corpus now has 80 focused review notes, 0 repositories awaiting focused review, and a concise data brief that explains what the corpus does and does not prove.
• Lower-noise model-release scans: RRD050 and verify now recognize common model-release entrypoints such as demo.py, inference.py, predict.py, sample.py, and generate.py in root, scripts/, and tools/ layouts.
• Less noisy secret checks: RRD063 and RRD090 now require stronger entropy eviden