Crawlbrulee MCP

🍮 crawlbrulee mcp

EU-native web scraping for AI agents & developers.

plug crawlbrulee into your agent. the official mcp server for crawlbrulee gives mcp-aware agents — Claude Code, Codex, Cursor, Claude Desktop — native tools to scrape pages, map sites, run background jobs, and check usage. one call turns any url into clean markdown, screenshots, metadata and links.

• everything runs in the EU. the fetch, the render, the cache and your result never leave EU servers. the proxy exit is the one hop you choose: pick an EU exit and nothing leaves at all. gdpr-aligned, with a data processing agreement.
• output made for models. markdown with ads, popups and cookie banners removed and the links kept, ready for the prompt. full-page screenshots can come back as tiles sized for an image model.
• the hard parts, handled. headless Chrome when a page needs it, rotating proxies with country selection, automatic retries, ad and cookie-banner removal, caching, background jobs and signed webhooks.
• start free. 750 credits, no credit card.

get a free api key → dashboard.crawlbrulee.com

the server:

• npx-runnable — zero install.
• wraps the @crawlbrulee/sdk under the hood; this mcp is just a thin protocol adapter.
• stdio transport for terminal-based agents.
• strict, fully-described tool schemas — agents see what every parameter does without reading docs.

this readme covers the mcp server itself — its tools and how to wire it into a host. for how the api behaves — endpoints, parameters, and error semantics — please see our api docs.

install

1. get a key. if you don't have a crawlbrulee account, sign up. the free plan needs no credit card. then, in the dashboard menu, open tokens and click create token.

2. put it in your environment. add it to your shell profile (for example ~/.zshrc), then open a new terminal and start your agent from it. don't paste the key into your agent's chat: anything in the chat stays in its history.

export CRAWLBRULEE_API_KEY="cwbl_..."

3. add the server, with no key in its config. in Claude Code:

claude mcp add crawlbrulee -- npx -y @crawlbrulee/mcp

other stdio hosts run the same command: npx -y @crawlbrulee/mcp. a few of them (Codex, Cursor, Windsurf, Gemini CLI, Claude Desktop) need the key or the variable named in their config file, and VS Code and the Claude Code plugin can ask for the key in a masked prompt. the file and the entry for each one: mcp setup in the docs.

if a key stops working (invalid_credentials), it's expired, revoked or mistyped: create a new one the same way, replace the old one, and start your agent again.

configuration

the mcp reads the env var on first tool invocation — not at startup — so a typo in your config surfaces as a clear tool-error message rather than the server failing to come up. see authentication for how the api consumes keys.

tools

scrape

fetch a single url and return the requested content (markdown, cleaned html, raw html, links, images, screenshot, page metadata).

input — only url is required; everything else has sane defaults.

{

"url": "https://example.com", "extract": { "markdown": true, "links": true, "screenshot": { "type": "full_page", "device_mode": "desktop" }, }, "require_js": false, "proxy": "basic", "cleanup": { "ads_and_popups": true, "exclude_selectors": ["nav", "footer"] }, "cache": { "max_age": 3600 }, "location": { "locale": "en-US", "country": "US" }, "zero_data_retention": false, }

zero_data_retention (boolean, default false) keeps the result out of the shared cache; anything stored to deliver it is kept for 24 hours, then deleted. it adds 1 credit and must be enabled for your organization. scrape_async takes it too. see zero data retention.

named values with `extract.elements`. to pull a few values from a page — prices, titles, links — give each one a name and a CSS selector, instead of reading the whole markdown. they come back in elements under the same names. all: true returns every match as a list, and fields reads named values inside each match. it costs no extra credits. scrape_async takes it too.

{

"url": "https://books.toscrape.com/", "extract": { "elements": { "heading": "h1", "books": { "selector": "article.product_pod", "all": true, "fields": { "title": { "selector": "h3 a", "output": "attribute", "attribute": "title" }, "price": ".price_color", }, }, }, }, } // → "elements": { // "heading": "All products", // "books": [{ "title": "A Light in the Attic", "price": "£51.77" }, ...] // }

a name with no match is null (or [] with all: true). limits and selector rules: see targeted extraction.

your own headers and cookies with `headers` and `cookies`. send your own HTTP headers and cookies to the site, for example an Authorization header or a fixed User-Agent that a site has on its allowlist. headers go only to the origin of url (or its https version, for an http url) and cookies only to its host, never to another site on a redirect. up to 10 headers and 20 cookies. a request with headers or cookies never reads from or writes to the cache, and costs the same as a normal fresh scrape. scrape_async takes them too.

{

"url": "https://app.example.com/orders", "headers": { "Authorization": "Bearer ...", "X-Tenant-Id": "acme" }, "cookies": [{ "name": "session_id", "value": "a1b2c3" }], }

[!WARNING]

header and cookie values pass through your chat with the AI, so the AI host may keep them in its history. use short-lived tokens or a test account, and never a password you can't change.

your own User-Agent can lower the success rate, and the response then carries the custom_user_agent warning. never put secrets in the User-Agent. the limits and which headers can't be set: see headers and cookies.

output — full scrape result. page metadata (title, OG tags, etc.) is returned under metadata. extracted images are returned as absolute urls — query strings are preserved, and relative srcs are resolved against the page url. screenshots are returned as signed download urls the agent can fetch separately. in rare cases a screenshot can't be captured: when you requested other outputs too, the screenshot field is simply left out while the rest is still returned — but a screenshot-only call that can't deliver errors instead (unsupported_screenshot_output, HTTP 422, when the content type can't be screenshotted) and isn't billed. the result also carries page_status_code and a top-level response_meta.usage block:

{

"url": "https://example.com", "requested_url": "https://example.com", "page_status_code": 200, // the site's own HTTP status for the final page "markdown": "...", "metadata": { "title": "Example Domain" }, "response_meta": { "usage": { // total_credit_cost = engine_credit_cost × proxy_multiplier + screenshot_slicing_credit_cost + zero_data_retention_credit_cost "total_credit_cost": 1, "engine_credit_cost": 1, // http 1, browser 3, screenshot 5, cache 0 "proxy_multiplier": 1, // basic 1, advanced 5 "screenshot_slicing_credit_cost": 0, // 1 when the screenshot was split into slices, otherwise 0 "zero_data_retention_credit_cost": 0, // 1 when zero_data_retention added its credit, otherwise 0 "engine": "http", // "http" | "browser" | "screenshot" | "cache" "proxy": "basic", // resolved tier actually used: "basic" | "advanced" (never "auto") }, }, }

a page the site served is a result, not an error. page_status_code is the HTTP status the site answered with for the final page, after redirects. a 404, 410 or 503 page comes back with its content and its status here, so check page_status_code before you trust the content: a 404 means the markdown is the site's "not found" page. 2xx and 4xx pages are billed, except 403, 407, 408, 429 and 451; 5xx pages are never billed. when the site can't be reached at all, the tool returns a target_unreachable error instead (see errors).

alongside response_meta.usage, the result surfaces any non-fatal warnings — stable string codes an agent can switch on. an outsized page is truncated rather than refused, and the code names which part was cut:

two more codes are not about size. screenshot_unavailable means a screenshot was asked for, but the page came back from the http engine without one; the rest of the result is still there. custom_user_agent means the request set its own User-Agent header; this can lower the success rate.

and if you requested an extract that doesn't apply to the content type (e.g. metadata of a JSON file), the field name comes back in an unsupported_fields list — with the rest of the payload still returned.

every input field, its default, and its constraints are documented under the scrape endpoint — with extraction, screenshots, proxies & location, and caching covering the individual blocks.

scrape_async

submit a scrape job to run asynchronously and get back a job_id immediately, instead of holding the connection open. use this for long-running scrapes (heavy js rendering, full-page screenshots of long pages); for a quick one-shot fetch prefer the synchronous scrape tool. then poll scrape_status until the job is done and fetch the page with scrape_result.

takes the same input as scrape plus an optional per-job