Scrape-LE

Scrape-LE: Zero Hassle Scrapeability Checks

Load a URL in headless Chromium and see what will block your scraper — before you write it

Anti-bot vendors, rate limits, robots.txt rules, login walls, console errors, screenshots

---

Useful? A star or rating is how other developers find it —

★ GitHub ·

★ Open VSX ·

★ Marketplace

What it does

Run Scrape-LE: Check URL Scrapeability, enter a URL, and the page loads in a real headless Chromium. The report lands in the output channel: HTTP status, page title, load time, console errors, a full-page screenshot, and four detections. Works in VS Code and VS Code–based editors like Cursor and VSCodium (installable from Open VSX).

One-time setup: run Scrape-LE: Setup Browser to install Chromium (~130MB, into Playwright's browser cache).

Install

Use it from an AI agent

The same engine runs as an MCP server, so an agent can call it directly instead of you running a command.

analyze_robots_txt(content, path, agent?, maxResults?)

Given robots.txt contents and a path, reports whether the rules permit crawling it: the group naming agent when one does, otherwise the generic (User-agent: *) rules, with agent in the answer saying which. Plus the crawl delay, disallowed patterns and any sitemaps.

The server takes content and returns data — it reads no files and makes no network requests of its own. Published as scrape-le-mcp on npm and as io.github.nolindnaidoo/scrape-le in the MCP registry.

Configuring it by hand — any host with an MCP config file

Most hosts read a JSON config. Add one entry:

{

"mcpServers": { "scrape-le": { "command": "npx", "args": ["-y", "scrape-le-mcp"] } } }

-y skips the install prompt on first run. Pin a version if you would rather not track releases — scrape-le-mcp@2.4.0.

Prefer not to go through npx on every launch? Install it once and point at the binary instead:

npm install -g scrape-le-mcp

{

"mcpServers": { "scrape-le": { "command": "scrape-le-mcp" } } }

It speaks MCP over stdio and needs no environment variables, no API key and no configuration of its own. To check it before wiring it into anything:

echo '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' | npx -y scrape-le-mcp

That prints the tool list and exits — if you see analyze_robots_txt, the server works.

The CLI

The same check runs from a terminal or an agent loop: a Rust CLI in crate/ of this repository, sharing one signature corpus with the extension — crate/signatures/ and crate/fixtures/ — so CI fails if the two ever disagree about a URL.

scrape-le https://example.com/search   # summary on stderr, JSON on stdout when piped

scrape-le --input urls.txt # a batch, streamed as it completes scrape-le mcp # the same check over MCP on stdio

Install it from npm (scrape-le) or from crates.io (scrape-le). Both install the same scrape-le command.

npm install -g scrape-le   # or run it once: npx scrape-le

cargo install scrape-le

The exit code is the answer: 0 clear · 1 a real no · 2 the question was malformed.

Detections

Honest limitations: signatures are best-effort fingerprints of public integration patterns — a detected widget means the page *can* challenge you, not that it will, and a clean result is not proof a site allows scraping. With scrape-le.robotsTxt.agent empty, only the User-agent: * groups are evaluated. Pages get up to 5 seconds to go network-idle after load, so content rendered later than that can be missed by the page-level detections.

Commands

No command is bound to a key by default. Give any of them one under Keyboard Shortcuts in the editor.

Settings

Languages

Twelve languages besides English:

German · Spanish · French · Indonesian · Italian · Japanese · Korean · Portuguese (Brazil) · Russian · Ukrainian · Vietnamese · Chinese (Simplified)

Both halves are covered — the manifest (command titles, setting names and descriptions) and everything shown while the extension runs (notifications, the status bar, quick-picks and prompts). The extension follows VS Code's display language, so it matches whatever the editor is already set to; no setting of its own.

Privacy & security

• Network access is the feature, and it is scoped. A check talks to exactly two things: the URL you enter (loaded in headless Chromium, which fetches that page's own resources like any browser) and that origin's /robots.txt. Nothing is sent anywhere else — no telemetry, no analytics.
• Screenshots stay local, written to the configured path inside your workspace.
• The MCP server makes no network request at all — unlike the extension, deliberately. fetchRobotsTxt builds a URL from an arbitrary origin, which inside an agent loop is an SSRF primitive: the caller supplying the URL is the model, not you. The server analyses robots.txt content you already fetched, and a test asserts no tool accepts a url argument.
• Error notifications redact home directories and credential-shaped fragments.
• Respect the sites you check: a scrapeability report is information, not permission.
• One rating prompt, at most twice. On the 3rd successful use the extension asks once whether you would