PyreCrawl

🔥 PyreCrawl — Web Browsing Superpowers for Your AI Agent

One command gives any AI agent the whole web. Scrape, extract, crawl, map, and search — self-hosted, no API keys, no rate limits, no subscription.

PyreCrawl speaks MCP (Model Context Protocol), the standard tool interface for Claude, Cursor, VS Code, Codex, OpenCode, Hermes, and any MCP-compatible agent.

A smart auto-fallback ladder always picks the cheapest method that succeeds:

fast HTTP

│ (403/503/Cloudflare challenge or empty body) ▼ stealth browser (real Chromium + Cloudflare solver) │ (still blocked, or the page needs full JS rendering) ▼ deep processing (LLM-ready markdown, citations, structured extraction)

⚡ Tools exposed

MCP Resources (read-only state without a tool call): pyrecrawl://cache/stats · pyrecrawl://sessions · pyrecrawl://monitors

MCP Prompts (ready-made playbooks): research(topic) · rag_ingest(site) · watch_page(url)

Env flags

prefer options: "auto" (default ladder) · "fast" (HTTP only) · "stealth" (CF bypass) · "llm" (deep processing).

---

🚀 Install & Use (one-liner)

1. Install

UV (recommended — one command, zero Python setup)

UV is a fast Python package manager that handles Python itself — no need to install Python separately. Get it once:

# macOS / Linux

curl -LsSf https://astral.sh/uv/install.sh | sh

Windows (PowerShell)

powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

Learn more about UV →

Then run PyreCrawl directly — no venv, no pip install, no Python download:

uvx pyrecrawl@latest

Or via uv tool install (persistent, recommended for regular use)

uv tool install pyrecrawl

Or via pipx (alternative)

pipx install pyrecrawl

Or via pip into a venv

pip install pyrecrawl

2. One-time browser engines

pyrecrawl setup

This installs Chromium + stealth browser engines (~2 min, one-time).

3. Register with your AI agent

# Auto-detect installed agents and write their MCP configs

pyrecrawl install

Or target specific agents

pyrecrawl install claude-desktop cursor

Dry-run to preview what would change

pyrecrawl install --dry-run

Supported agents: claude-desktop, claude-code, cursor, vscode, codex, opencode, hermes.

4. Start chatting

After installing + registering, restart your agent (or start a new session). Then ask:

*"Scrape https://example.com and summarize it."*

Available tools:

Plus 3 guided prompts: research, rag_ingest, watch_page.

Quick examples

Ask your agent naturally — no special syntax needed:

---

🧠 Skills — Maximize Your Agent's Research Quality

PyreCrawl tools give your agent hands (scrape, crawl, search). But the agent still needs a brain — instructions on *when* to use which tool, *how* to chain research passes, and *what* anti-hallucination rules to follow.

That's what [PyreCrawl Skills](https://github.com/SanggonBoy/pyrecrawl-skills) provides.

Quick setup:

# 1. Install the tools (you already have this)

uvx pyrecrawl@latest

2. Add the research skill to your project

git clone https://github.com/SanggonBoy/pyrecrawl-skills.git /tmp/pyrecrawl-skills cp /tmp/pyrecrawl-skills/pyrecrawl-research/SKILL.md ./CLAUDE.md # or .cursorrules / AGENTS.md

Without skills: Your agent has powerful tools but improvises usage.

With skills: Your agent follows a proven research protocol with anti-hallucination guardrails.

---

📚 Manual config (if pyrecrawl install doesn't match your setup)

Claude Desktop

Config file

• Linux: ~/.config/Claude/claude_desktop_config.json
• macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
• Windows: %AppData%\Claude\claude_desktop_config.json

{

"mcpServers": { "pyrecrawl": { "command": "uvx", "args": ["--from", "pyrecrawl", "pyrecrawl", "serve"] } } }

Claude Code

Config file: project-scoped .mcp.json

{

"mcpServers": { "pyrecrawl": { "command": "uvx", "args": ["--from", "pyrecrawl", "pyrecrawl", "serve"] } } }

Cursor

Config file: ~/.cursor/mcp.json

{

"mcpServers": { "pyrecrawl": { "command": "uvx", "args": ["--from", "pyrecrawl", "pyrecrawl", "serve"] } } }

VS Code / Copilot

Config file: .vscode/mcp.json (project-scoped)

{

"servers": { "pyrecrawl": { "command": "uvx", "args": ["--from", "pyrecrawl", "pyrecrawl", "serve"], "type": "stdio" } } }

Codex CLI

Config file: ~/.codex/config.toml

[mcp_servers.pyrecrawl]

command = "uvx" args = ["--from", "pyrecrawl", "pyrecrawl", "serve"]

OpenCode

Config file: ~/.config/opencode/opencode.json

{

"mcp": { "pyrecrawl": { "type": "local", "command": ["uvx", "--from", "pyrecrawl", "pyrecrawl", "serve"], "enabled": true } } }

Hermes

Config file

• Linux/macOS: ~/.hermes/config.yaml
• Windows: %LocalAppData%\hermes\config.yaml

mcp_servers:

pyrecrawl: command: uvx args:

• --from
• pyrecrawl
• pyrecrawl
• serve enabled: true

Windows note: uvx must be on PATH. If not, use the full path to uvx.exe (e.g. C:\Users\ \AppData\Local\hermes\bin\uvx.exe).

---

🧠 How the ladder chooses

PyreCrawl runs each request through three tiers, stopping at the first one that returns a complete, LLM-ready result:

The agent never has to pick. prefer="auto" does it every call.

Live DOM data with js and wait_for

Some sites keep the data you want in a DOM *property* (e.g. an 's .value) that JS writes after an XHR — it never appears in the serialized HTML. The scrape tool accepts two stealth-tier params for exactly this:

{

"url": "https://temp-mail.org/id", "prefer": "stealth", "wait_for": "document.getElementById('mail').value.includes('@')", "js": "document.getElementById('mail').value" }

• wait_for — a JS predicate expression polled until truthy (bounded by timeout). Use it instead of guessing a sleep for anything that arrives asynchronously.
• js — a JS expression evaluated once the page settles; the value comes back in meta.js_result. Errors are captured in meta.js_error (the page result is still returned, never a crash).

---

📊 Compared to Firecrawl (hosted)

| Cloudf