Browser Scraper

Browser Scraper: the stealth browser for web scraping and AI agents

Undetected Chrome for Node.js that reads and operates the sites that block bots. Use it as a library, a CLI or an MCP server for Claude, Cursor and Codex. It turns any page, including closed Shadow DOM and cross-origin iframes, into clean Markdown, structured data or an accessibility snapshot an agent can click through. Local, free, MIT, bring-your-own-LLM.

CreepJS: 0% headless, 0% stealth · `bot.sannysoft.com`: 0 failures · no `Runtime.enable` leak (what that means, and where it stops)

Recorded on one desktop PC: the Turnstile test page at nowsecure.nl in headless mode, bot.sannysoft.com headful on the PC's own GPU.

import { scrape } from "@eurafaeldecarvalho/browser-scraper";

const page = await scrape({ url: "https://example.com", formats: ["markdown", "metadata"] }); console.log(page.tier); // "http" | "browser" | "stealth": the cheapest tier that worked console.log(page.markdown); // clean, LLM-ready Markdown of the main content

# Give Claude Code a browser that gets through Cloudflare:

claude mcp add browser-scraper -- npx -y @eurafaeldecarvalho/browser-scraper mcp

@eurafaeldecarvalho/browser-scraper drives real Chrome over the raw DevTools Protocol (CDP). It never sends Runtime.enable, one of the most widely fingerprinted automation signals, and every layer of the browser's identity tells the same story. On top of that stealth core it adds the pieces AI workloads need: Markdown and metadata extraction, an accessibility snapshot with clickable refs, a one-call scrape() that escalates from plain HTTP to a stealth browser only when it has to, an MCP server, and optional LLM extraction with any model.

Why this one:

• Gets through where headless browsers get blocked. There is no Runtime.enable leak, the persona (UA, Client Hints, WebGL, timezone, locale) is coherent, and mouse and keyboard input are human-like. solveTurnstile() handles Cloudflare's checkbox.
• Sees everything. find(), markdown() and snapshot() all reach into closed shadow roots and cross-origin iframes, which Playwright's snapshot and Readability-style extractors miss.
• Built for agents. The MCP server and CLI work with no extra install. Credentials are typed through {{SECRET}} placeholders so the model never sees them, an allow-list restricts domains, and page content is fenced as untrusted data.
• LLM only where it pays. generateExtractor() has the model write CSS selectors once. After that run() extracts with no LLM, and runOrHeal() regenerates the selectors when the site changes.
• Local and free. No cloud, no API key, no per-page billing. You bring the proxy and, optionally, the model (OpenAI, Anthropic, Google, Ollama… through the Vercel AI SDK, or your own function).

Passes the bot detectors people actually use

With the right setup — a clean residential/mobile IP, plus a real GPU (EC2 g4dn/g5) for render-hash targets — browser-scraper is built to clear the fingerprint suites and anti-bot stacks that block headless automation:

A ✅ means the fingerprint + behavior layer passes — it is not a guaranteed bypass. Against hard targets your IP reputation is the ceiling: a datacenter ASN loses before the JavaScript runs, so route through a residential/mobile proxy (see The production stack). browser-scraper is not a CAPTCHA solver — for reCAPTCHA v2 / hCaptcha / Arkose (FunCaptcha) image challenges, pair it with a solver service. It *does* keep reCAPTCHA v3 scores high via human-like input and warmed profiles.

Numbers were taken on a desktop Chrome. On a GPU-less Lambda running chrome-headless-shell (@sparticuz/chromium) the surfaces differ — re-run the suites against your *actual* production binary and treat that as the source of truth (see Cloud deployment).

This is a precision tool, not a magic bypass — it's honest about its edges (see Notes & limitations), and that honesty is exactly what makes the green checks above worth anything.

---

Contents

• Install
• Quickstart
• For AI agents & LLMs — Markdown, scrape(), snapshot + refs, MCP server, CLI, LLM extraction
• The tour — what it does, by example
• Pierce closed Shadow DOM + cross-origin iframes
• Wait for the right moment
• Interact like a human
• Control the network
• Coherent browser fingerprint
• Persistent profiles & warming
• API reference
• Stealth, in depth
• WebGL / GPU on cloud hosts
• The production stack — proxy · VPS · solver
• Cloud deployment (Lambda vs EC2-GPU)
• Identity & persistence
• WebRTC
• Alternatives — how it compares
• FAQ
• Notes & limitations
• Support

---

Install

npm install @eurafaeldecarvalho/browser-scraper

or:

pnpm add @eurafaeldecarvalho/browser-scraper

Requires Node.js ≥ 20 and a local Chrome/Chromium (it's found automatically, or point at one with chromePath). For AWS Lambda, also install the optional @sparticuz/chromium — see Cloud deployment. Profile history seeding uses Node's built-in node:sqlite (Node ≥ 22.5) or the optional better-sqlite3; if neither is available it is simply skipped. The LLM helpers need ai (and zod if you use Zod schemas), and only if you import /ai.

Quickstart

import { Browser } from "@eurafaeldecarvalho/browser-scraper";

const browser = new Browser();

try { const tab = await browser.newTab(); await tab.goto({ url: "https://example.com" });

const heading = await tab.find({ selector: "h1" }); console.log(await heading?.text());

await tab.screenshot({ path: "example.png" }); } finally { await browser.close(); }

That's the whole shape of the library: a Browser, a Tab, and Elements you find and act on. Everything below is detail on top of these three.

---

For AI agents & LLMs

Clean Markdown, metadata and links

const tab = await browser.newTab();

await tab.goto({ url: "https://shop.example.com/product/42" });

const { markdown, title, links, wordCount } = await tab.markdown({ mainContent: true, // drop nav/header/footer/cookie banners (default) query: "price shipping", // optional: keep only the relevant blocks (BM25), saves tokens links: "inline", // "inline" | "reference" | "none" includeIframes: false, // true = also inline iframe documents, cross-origin included });

const meta = await tab.metadata(); // { title, description, canonical, lang, image, openGraph, twitter, jsonLd: [...] } // JSON-LD often already has the price, rating and availability: check it before you prompt.

markdown() reads Chrome's own DOM tree over CDP (DOM.getDocument with pierce). No script is injected into the page, and the content of closed shadow roots is included in reading order, with slots composed. For HTML you already have, htmlToMarkdown(html) and extractMetadata(html) do the same without a browser.

scrape(): one call, cheapest tier first

import { Browser, scrape } from "@eurafaeldecarvalho/browser-scraper";

const browser = new Browser({ proxy: process.env.PROXY }); // reuse it across URLs const page = await scrape({ url, browser, formats: ["markdown", "metadata", "links", "html", "screenshot"], strategy: "auto", // http → browser → stealth (or pin one tier) http: { curlImpersonate: "curl_chrome131" }, // optional: real Chrome TLS for the http tier });

page.tier; // which tier produced the result page.attempts; // [{ tier: "http", ok: false, reason: "needs JavaScript (empty app shell)", ms: 4 }, …]

A challenge page is never returned as content: if every tier is blocked, scrape() throws a ScrapeError that carries the attempts.

Accessibility snapshot + refs (what agents click on)

console.log(await tab.snapshot());

// - main // - heading "Sign in" [level=1] // - textbox "Email" [ref=e3] // - button "Continue" [ref=e4] // - Iframe [ref=e5] // - checkbox "Verify you are human" [ref=e6] ← cross-origin iframe, still addressable

await (await tab.byRef({ ref: "e3" })).type({ text: "me@example.com" }); // human-like typing await (await tab.byRef({ ref: "e4" })).click(); // human-like mouse

The accessibility tree is computed on the composed tree, so elements inside closed shadow roots appear and can be clicked. Pass { interactive: true } to get just the flat list of clickable or typeable elements.

MCP server (Claude Code, Claude Desktop, Cursor, Codex…)

claude mcp add browser-scraper -- npx -y @eurafaeldecarvalho/browser-scraper mcp

Or in any MCP client config (claude_desktop_config.json, .cursor/mcp.json, …):

{

"mcpServers": { "browser-scraper": { "command": "npx", "args": ["-y", "@eurafaeldecarvalho/browser-scraper", "mcp", "--allowed-domains", "example.com", "--secrets", "/path/to/.env"] } } }

Tools:

| markdown | Ma