webclaw

English | 简体中文

webclaw

Turn websites into clean markdown, JSON, and LLM-ready context.

CLI, MCP server, REST API, and SDKs for AI agents and RAG pipelines.

Most web scraping tools give your agent one of two bad outputs:

• a blocked page, login wall, or empty app shell
• raw HTML full of nav, scripts, styling, ads, and duplicated boilerplate

webclaw.io is the hosted web extraction API for webclaw. This repo contains the open-source CLI, MCP server, extraction engine, and self-hostable server.

webclaw turns a URL into clean content your tools can actually use.

webclaw https://example.com --format markdown

# Example Domain

This domain is for use in illustrative examples in documents.

You may use this domain in literature without prior coordination or asking for permission.

Use it from the terminal, wire it into Claude/Cursor through MCP, call the hosted API from your app, or self-host the OSS server.

Install

Agent setup

The fastest way to connect webclaw to Claude Code, Claude Desktop, Cursor, Windsurf, OpenCode, Codex CLI, and other MCP-compatible tools:

npx create-webclaw

The installer detects supported clients and configures the MCP server for you.

Homebrew

brew tap 0xMassi/webclaw

brew install webclaw

Prebuilt binaries

Download macOS, Linux, and Windows binaries from GitHub Releases.

Windows (x64)

The prebuilt ZIP is the easiest installation and does not require Rust. On the v0.6.22 release page, download webclaw-v0.6.22-x86_64-pc-windows-msvc.zip. Extract it in File Explorer, open PowerShell in the extracted folder containing webclaw.exe, and run:

.\webclaw.exe --version

.\webclaw.exe https://example.com --format markdown

PowerShell requires the .\ prefix for programs in the current directory. Add that directory to your user PATH only if you want to run webclaw from other folders. If Windows reports a missing Visual C++ runtime DLL, install the Microsoft Visual C++ Redistributable for x64. The ZIP contains x64 binaries; Windows ARM64 is not covered by the native installation check.

To build from source on Windows, install Rust with the x86_64-pc-windows-msvc toolchain, Visual Studio Build Tools with Desktop development with C++ and the Windows SDK, Git, CMake 3.22 or newer, LLVM (including libclang.dll), and NASM. Open a fresh Developer PowerShell for Visual Studio after installation. Ensure CMake and NASM are on PATH and set LIBCLANG_PATH to your LLVM bin directory if bindgen cannot find it, for example:

$env:LIBCLANG_PATH = "C:\Program Files\LLVM\bin"

cargo install --git https://github.com/0xMassi/webclaw.git --tag v0.6.22 --locked webclaw-cli webclaw --version

The Cargo package is webclaw-cli; the executable is webclaw. These packages are installed from Git, not crates.io. A first source build can take several minutes. The Windows CI workflow checks the Git installation and published ZIP on a hosted runner with preinstalled build tools; it does not model a clean PC. Linux containers, including Docker on macOS, do not validate Windows support.

Docker

docker run --rm ghcr.io/0xmassi/webclaw https://example.com

Cargo

cargo install --git https://github.com/0xMassi/webclaw.git --tag v0.6.22 --locked webclaw-cli

cargo install --git https://github.com/0xMassi/webclaw.git --tag v0.6.22 --locked webclaw-mcp

If building from source fails because native build tools are missing, install the platform prerequisites:

Python

The python/ directory builds a Python package that bundles the webclaw binary and calls it from Python: scrape, select, tables, find_text, similar, crawl, map and screenshot, with no API key and no Python dependencies.

cd python && uv build --wheel && pip install dist/webclaw_cli-*.whl

import webclaw_cli as wc

md = wc.scrape("https://example.com", "llm") titles = wc.select("https://books.toscrape.com", css="h3 a::attr(title)") for page in wc.crawl("https://quotes.toscrape.com", max_pages=10): print(page["url"])

See python/README.md for every function and option.

Quick Start

Scrape one page

webclaw https://stripe.com --format markdown

Return LLM-optimized text

webclaw https://docs.anthropic.com --format llm

Keep only the main content

webclaw https://example.com/blog/post --only-main-content

Include or exclude selectors

webclaw https://example.com \

--include "article, main, .content" \ --exclude "nav, footer, .sidebar, .ad"

Pick values with CSS selectors or XPath

webclaw https://books.toscrape.com --select "h3 a::attr(href)"

webclaw https://books.toscrape.com --select "article.product_pod" \ --field "title=h3 a::attr(title)" --field price=.price_color

--select prints a JSON array of what the selector matches: each element's text, or an attribute or the element's HTML when the selector ends in ::attr(NAME) or ::html. Links in href and src come back absolute. With --field NAME=SELECTOR, each match becomes a JSON object whose fields run inside it, taking the first value or null. Pages that need JavaScript render first, as with extraction.

webclaw https://books.toscrape.com --xpath "//li[@class='next']/a/@href"

webclaw https://books.toscrape.com --xpath "//article[@class='product_pod']" \ --field "title=.//h3/a/@title" --field "price=.//p[@class='price_color']"

--xpath takes an XPath 1.0 expression instead, over the same parsed page: elements and text nodes give their text, attributes their value, and count() or string() style expressions one value. With --xpath, each --field is an expression evaluated from the match, so it usually starts with .. The MCP select tool takes xpath in place of selector.

Data tables as JSON

webclaw "https://en.wikipedia.org/wiki/List_of_S%26P_500_companies" --tables

--tables prints the page's data tables as a JSON array, each with its caption (when it has one), headers and rows of cell text. A cell that spans several rows or columns repeats its text in each one, so every row has one value per column. Stacked header rows are joined per column with / , as in Population / % of world. The MCP select tool takes tables: true in place of a selector.

Header rows are the rows of and the leading rows made only of ; a leading row that is one cell across the whole table becomes the caption. These tables are skipped as page layout:

• tables marked role="presentation" or role="none";
• tables with fewer than two columns of text;
• tables without a header row that have a single body row, wrap another table, hold only links, or hold paragraphs, lists or other blocks in a row of more than one cell.

Text hidden with hidden, an inline display:none or is left out. Text hidden by a stylesheet is kept, because pages are not styled. A cell keeps its first 64 KB of text, and a very large table is cut short with "truncated": true. Once the tables on a page reach 64 MB, the tables after that are left out.

Find elements by text, or everything like one example

webclaw https://books.toscrape.com --find-text "tipping the"

webclaw https://books.toscrape.com --find-regex '^£5\d\.\d\d$' webclaw https://books.toscrape.com --find-text "Tipping the Velvet" --similar

--find-text prints the elements in the page body whose visible text contains the text, any case, as {"selector", "text"} objects. Each match is the deepest element holding the text, so text split by inline tags still matches, and its selector is the element's path from the nearest ancestor with a unique id, or from the root. That path picks the element alone, except on pages nested more than 64 levels deep with no such id, where it keeps only the last 64 steps and can pick more. --exact matches the whole text, --case-sensitive matches letter case, and --find-regex takes a regular expression instead.

--similar takes the first element that --find-text, --find-regex, --select or --xpath picks as an example and returns every element like it on the page, such as the other items of a list, grid or table, with one CSS selector that picks them all. When the example selector ends in ::attr(NAME) or ::text, or the XPath picks a text or attribute node, the element holding it is the example:

{

"selector": "article.product_pod > h3 > a", "matches": [ { "selector": "html > body > ... > li:nth-of-type(1) > article > h3 > a", "text": "A Light in the ..." }, ... ] }

Reuse the selector with --select, or add --field to get records straight away; with --similar the fields are CSS and the output is {"selector", "records"}. Pick the item itself as the example (for instance --select "article.product_pod" --similar) when the fields live inside it. The MCP select tool takes text or regex (with exact and case_sensitive) and similar the same way. One call takes one picker: --select, --xpath, --tables, --find-text or --find-regex (in MCP: selector, xpath, tables, text or regex), and --similar and --field do not apply to tables.

Keep only what answers a question

webclaw "https://en.wikipedia.org/wiki/Rust_(programming_l