WebReaper

WebReaper

AI-native web scraper. Single binary with a bundled Claude Code skill.

Built by HighCraft.io.

Install

macOS / Linux (Homebrew):

brew install alex-on-ai/webreaper/webreaper

Any POSIX shell (install.sh):

curl -fsSL https://raw.githubusercontent.com/alex-on-ai/WebReaper/master/scripts/install.sh | sh

.NET library:

dotnet add package WebReaper

Windows binaries are on the GitHub Releases page; winget and Scoop are on the v10.1 roadmap.

Updating: brew upgrade webreaper (Homebrew), or re-run the install.sh line with --upgrade appended (| sh -s -- --upgrade), or dotnet add package WebReaper (library). When a newer release exists, scrape / crawl / map print a one-line upgrade hint on stderr (interactive terminals only, never in a pipe or CI); disable that check with WEBREAPER_NO_UPDATE_CHECK=1.

30-second demo

$ webreaper scrape https://news.ycombinator.com

Hacker News

• Show HN: ...
• Ask HN: ...

...

$ webreaper init Wrote WebReaper Agent Skill to .claude/skills/webreaper/SKILL.md

Try it out: webreaper scrape https://example.com webreaper map https://example.com

After webreaper init, the next Claude Code session picks up the skill and routes scraping intents (*"give me the markdown of X"*, *"what blog posts are on Y"*, *"scrape the top 5 articles"*) to webreaper automatically.

Table of contents

• Why WebReaper
• Quick start
• Bot protection that just works
• Power your agent
• AI features
• Use cases
• Packages
• Compared to Firecrawl, Crawl4AI, and Crawlee
• API overview
• Architecture and interfaces
• Repository structure
• License

Why WebReaper

🪶 Drop on PATH, run.

No Docker, no Postgres, no signup. ~12 MB binary. 🤖 AI-native by composition.

Markdown by default. Schema extraction, LLM fallback, self-healing selectors, autonomous agent. Stack with .With…() . 🔌 Bring any LLM.

OpenAI, Anthropic, Ollama, Azure OpenAI, llamafile, via Microsoft.Extensions.AI .

🛡 Bot-checks handled automatically.

A blocked page climbs HTTP → browser → stealth on its own, per page and host-sticky. Challenge pages are dropped, never returned as data. No flag needed for the browser fallback. 📡 Distributed when needed.

Swap scheduler, tracker, sink to Redis, MongoDB, SQLite, Azure Service Bus, Cosmos. Same code. 📜 MIT, not AGPL.

Embed in commercial software, fork, modify, redistribute. Firecrawl's AGPL requires open-sourcing your service or paying for a commercial license.

Quick start

CLI

# One page as Markdown

webreaper scrape https://example.com

Save Markdown to a file

webreaper scrape https://example.com --output page.md

Discover URLs on a site

webreaper map https://example.com --search /blog/ --max-urls 50

Crawl a whole site recursively (every on-domain page) to JSON Lines

webreaper crawl https://example.com > pages.jsonl

Structured fields with a JSON schema (output: JSON; multi-page: JSON Lines)

webreaper scrape https://example.com --schema schema.json

Schema-free extraction with an LLM (bring your own OpenAI-compatible endpoint)

webreaper scrape https://example.com --prompt "title and author" \ --model gpt-4o-mini --llm-url https://api.openai.com/v1

Whole site, fields, cheaply: infer a schema once, then extract the rest

webreaper crawl https://example.com --infer "product name and price" \ --model gpt-4o-mini --llm-url https://api.openai.com/v1 --output-dir ./out

JS-rendered single-page app

webreaper scrape https://example.com --browser

Bot-protected site: a plain scrape already auto-climbs HTTP -> browser on a

block; --stealth starts at a stealth backend (--auto-stealth = no prompt, for CI)

webreaper scrape https://example.com --stealth

Install the Claude Code skill

webreaper init

The CLI is built Native-AOT (ADR-0043), ships as a single binary on every tagged GitHub release across six RIDs (linux-x64, linux-arm64, osx-x64, osx-arm64, win-x64, win-arm64), and is block-aware with automatic browser/stealth escalation (ADR-0083). The macOS binaries are Apple codesigned and notarized (ADR-0071); Homebrew installs run without Gatekeeper warnings on a clean machine.

Library

using WebReaper.Builders;

var engine = await ScraperEngineBuilder .Crawl("https://news.ycombinator.com") .AsMarkdown() .WriteToConsole() .BuildAsync();

await engine.RunAsync();

That is HTTP-only, no extra packages, no schema. For structured fields, swap AsMarkdown() for Extract(schema); for JS-rendered pages, Crawl for CrawlWithBrowser plus a transport satellite (WebReaper.Playwright or WebReaper.Cdp). The full surface is in the API overview.

Bot protection that just works

Most scrapers make you opt into a browser up front and guess when a site is blocking you. WebReaper detects the block and escalates on its own, one page at a time:

HTTP  ->  browser (Chromium)  ->  stealth (CloakBrowser)

(climbs only when a page actually looks blocked)

• Automatic. A plain scrape or crawl starts on a fast HTTP fetch and climbs to a real browser only when a page looks blocked (a challenge status, response header, or body marker). No flag needed for the browser fallback.
• Per page, host-sticky. The first confirmed block on a host lifts that host's floor, so the rest of a whole-site crawl starts at the working tier instead of re-paying the failed one. You pay for the climb once, not per page.
• No garbage in your data. A page still blocked at the top tier is dropped, never written to your output, and the run exits non-zero so an unattended job knows. Clean data or a clear signal, never a challenge page masquerading as content.
• You decide on stealth. --stealth starts at the stealth backend; --auto-stealth (or WEBREAPER_AUTO_STEALTH=1) enables it unattended; --no-auto-stealth caps the climb at a vanilla browser. The ~220 MB stealth backend downloads only if you opt in.

Self-hosted, single binary, no cloud round-trip (ADR-0083).

Power your agent

Onboarding an agent from zero? Hand it docs/AI-ONBOARDING.md — install → verify → choose-your-path on one page, no credentials anywhere.

Claude Code skill

webreaper init

Writes a polished SKILL.md to .claude/skills/webreaper/. Claude Code loads it on the next session and routes scraping intents to the CLI: *"scrape the top 5 stories on HN and summarize each"*, *"give me the markdown of this article"*, *"this Cloudflare-protected site is blocking me"*. The skill describes when to prefer webreaper over the built-in WebFetch (artifacts and structured data go through webreaper; conversational answers stay on WebFetch).

MCP server

Two MCP servers expose the same tools (scrape, map, extract, extract_with_prompt, extract_inferred, crawl) for clients that speak the Model Context Protocol:

• [`WebReaper.Mcp`](https://www.nuget.org/packages/WebReaper.Mcp) (ADR-0049) speaks stdio, for local clients that spawn a process: Cursor, Claude Desktop, Copilot Studio.
• [`WebReaper.Mcp.AspNetCore`](https://www.nuget.org/packages/WebReaper.Mcp.AspNetCore) (ADR-0086) speaks Streamable HTTP, for clients that connect to a URL. The headline case is n8n, whose MCP Client node is URL-only and cannot reach a stdio server. Run the Chromium-baked image, point n8n's MCP Client node at the URL with a bearer token, and call the tools from a workflow:

  docker run -p 8080:8080 -e WEBREAPER_MCP_TOKEN=your-secret \

ghcr.io/alex-on-ai/webreaper-mcp-http:latest

See the n8n quickstart.

Both are thin facades over the library API; the primary agent surface remains the CLI.

CLI inside any agent harness

The single binary works inside any shell-spawning agent: LangChain ShellTool, OpenAI Assistants code-interpreter, GitHub Actions, internal scripts. Zero runtime to install; one syscall to invoke.

AI features

1. LLM-ready Markdown, no schema

await ScraperEngineBuilder

.Crawl("https://example.com") .AsMarkdown() // ADR-0040 + ADR-0063 .WriteToConsole() .BuildAsync() .Result.RunAsync();

The Markdown extractor (HtmlToMarkdown primitive, ADR-0063; MarkdownContentExtractor adapter, ADR-0040) emits {title, markdown} per page. Pass the output straight into a follow-up LLM prompt.

2. Source-gen schemas with compile-time guards

using WebReaper.Extraction.Attributes;

[ScrapeSchema] public partial class Article { [ScrapeField("h1")] public string? Title { get; set; } [ScrapeField(".views", Type = SchemaFieldType.Integer)] public int Views { get; set; } [ScrapeField(".tag", IsList = true)] public List Tags { get; set; } = new(); }

// Emitted at compile time, reflection-free, AOT-clean: // public static Schema Schema { get; } // public static Article Materialize(JsonObject json)

var engine = await ScraperEngineBuilder .Crawl("https://example.com/post") .Extract(Article.Schema) .Subscribe(p => HandleArticle(Article.Materialize(p.Data))) .BuildAsync();

The WebReaper.Extraction.Generators Roslyn analyzer (ADR-0045) emits the schema and a Materialize function. Schema typos are compile errors; the generated path uses no reflection so it's AOT-clean.

3. LLM safety net for deterministic extraction

Three composable patterns where the LLM only fires when the deterministic path can't deliver. Same proposer-validator shape across all three.

using WebReaper.AI;

// (a) Fire LLM only when a field returns empty (ADR-0046) .WithLlmFallback(chatClient)

// (b) Repair a broken selector once per (Schema, field), cache forever (ADR-0047) .WithLlmSelfHealing(chatClient)

// (c) No schema at all: infer it from the URL, re-infer on validator failure (ADR-0067 + ADR-0069) .UseAi(chatClient, AiPolicyMode.Inferred)

Stable pages cost zero LLM calls; broken pages cost one call per (page, field) and cache. Schema inference is one call per site (cached for the run). The WebReaper.AI satellite is built on Microsoft.Extensions.AI so any IChatClient works.

4. Autonomous agent: page selection by goal

using WebReaper.AI;

var result = await LlmAgent.RunAsync( "https://example.com", goal: "Find the contact email and phone number for the support team.",