Spicrawl

Spicrawl MCP Server: web scraping for Claude, Cursor and AI agents

To let Claude, Cursor or another AI agent scrape websites, connect the Spicrawl MCP server. Add the hosted endpoint https://mcp.spicrawl.com/mcp (or run npx -y @spicrawl/mcp locally) with a Spicrawl API key, and the agent can call spicrawl_scrape to read any URL as Markdown, text, HTML or JSON. It renders JavaScript pages, extracts structured data, scrapes up to 10,000 URLs in one batch job, takes screenshots and keeps logged-in browser sessions, all as native spicrawl_* tools in Claude Code, Claude Desktop, Cursor, VS Code, Windsurf, Gemini CLI and Codex. The free tier includes 1,000 credits per month.

[Docs](https://docs.spicrawl.com/agents/mcp) · [Get an API key](https://app.spicrawl.com/signup) · [TypeScript SDK](https://github.com/OfficialSpicrawl/sdk) · [CLI](https://github.com/OfficialSpicrawl/cli) · [Agent plugins](https://github.com/OfficialSpicrawl/agent-plugins) · [Changelog](https://github.com/OfficialSpicrawl/mcp/blob/main/CHANGELOG.md)

Install from: npm · Smithery · Glama · MCP Registry · Dify plugin · Cursor Directory

Quickstart

1. Create an API key at app.spicrawl.com. Keys start with spicrawl_live_ or spicrawl_test_.
2. Connect the hosted endpoint, https://mcp.spicrawl.com/mcp (MCP Streamable HTTP, nothing to install):

   export SPICRAWL_API_KEY=spicrawl_live_...

claude mcp add --transport http spicrawl https://mcp.spicrawl.com/mcp \ --header "Authorization: Bearer $SPICRAWL_API_KEY"

Codex: codex mcp add spicrawl --url https://mcp.spicrawl.com/mcp --bearer-token-env-var SPICRAWL_API_KEY

1. Or run it locally over stdio (Node.js 20+), for clients that only launch local servers:

   claude mcp add spicrawl --env SPICRAWL_API_KEY=$SPICRAWL_API_KEY -- npx -y @spicrawl/mcp

1. Ask your agent: *"Use Spicrawl to read https://example.com/pricing as markdown and list each plan with its price."* It calls spicrawl_scrape and gets the page back as Markdown.

Both transports serve the same spicrawl_* tools (25 are defined; the hosted endpoint lists 24 until remote browsers launch, see Tools), and every call runs under your key with the same scopes, limits and credits as a direct API call.

What it does

• Scrape any URL to Markdown, text, HTML, a JSON envelope or a printed PDF with one tool, spicrawl_scrape. Markdown is the default and strips the page to its main content.
• Render JavaScript with render: true (3 credits). A plain fetch costs 1 credit, screenshots, PDFs and actions use a full browser (8 credits), and mode: "auto" escalates from fetch to browser and bills only the rung that worked (price table). You never pick the rendering yourself; Spicrawl chooses the cheapest that can do what the request asks.
• Extract structured data with a CSS/XPath selector map (extract) or from the JSON-LD, OpenGraph, Twitter Card and microdata a page publishes (autoparse).
• Drive the page before capture with 8 typed browser actions: click, fill, wait_for, wait_for_navigation, scroll (including infinite scroll), select, evaluate and screenshot.
• Return screenshots and PDFs the model can see: screenshots arrive as MCP image blocks (up to 5 MB of base64 each) and printed pages as application/pdf resource blocks (up to 10 MB).
• Batch scrape thousands of URLs as one async job, up to 10,000 items per call, with 9 tools to submit, poll, page through results, retry, cancel and keep a job open so you can append URLs as you discover them.
• Keep a logged-in browser with persistent sessions: cookies, storage and a fixed rendering tier reused across scrapes.
• Answer "what did that cost?" from usage and request-log tools, and read the Spicrawl docs from inside the agent.

Use cases

• Scrape docs into a RAG knowledge base. Read documentation pages as clean Markdown with spicrawl_scrape (main_content_only is on by default for Markdown), or queue up to 10,000 URLs in one spicrawl_batch_submit job. Spicrawl returns the Markdown; chunking and embedding are yours. See the knowledge-base recipe.
• Monitor prices. Pull a product's title, price and availability with an extract selector map or autoparse, and set cache: false for a fresh reading. A plain-fetch check costs 1 credit, a rendered one 3. Run it on your own schedule. See price monitoring.
• Extract job postings. Collect listings from job boards, including infinite-scroll pages (render: true with a scroll action), as structured fields. See job postings.
• Read JavaScript-heavy pages. When a plain fetch returns an empty shell, call spicrawl_scrape with render: true (3 credits) so the page builds itself first.

Check each site's terms and robots.txt before you scrape it.

Which tool to use

• If you need one page's content or data now, use spicrawl_scrape.
• If the page is blank or a skeleton without JavaScript, call spicrawl_scrape again with render: true.
• If you know the page's structure, use extract with CSS selectors; if you don't, try autoparse.
• If you have more than about 20 URLs, use spicrawl_batch_submit, then spicrawl_batch_status and spicrawl_batch_results.
• If you discover URLs as you go, submit with open: true, append with spicrawl_batch_add_items and finish with spicrawl_batch_close.
• If you must log in first, use spicrawl_session_create and pass its id as session_id to spicrawl_scrape.
• If a call failed and you need to know why, use spicrawl_request_get with the request id, or spicrawl_requests_list with only_errors.
• If you are unsure of a parameter or an error code, use spicrawl_docs_search before guessing.

Tools

25 tools, grouped by task. The hosted endpoint lists 24 until remote browsers launch: its keyless tools/list and the server card leave out spicrawl_browser_connect_url (Coming soon), while a session opened with your key and the stdio package (npx -y @spicrawl/mcp) list all 25. Tools reject unknown arguments, so a misspelt field fails with an error naming it instead of being silently dropped. Every tool sets a title and explicit readOnlyHint, destructiveHint and openWorldHint annotations, so a client knows which calls to confirm: spicrawl_scrape is not read-only (its method and actions can submit forms), and spicrawl_batch_cancel, spicrawl_session_release and spicrawl_session_delete are destructive. Every tool also declares an outputSchema and returns matching structuredContent beside the same text it always returned, so a client can read fields (credits, status, next_cursor, ...) without parsing text. The schemas list what an agent can rely on and stay open: fields the API adds later pass through, and statuses and tiers are plain strings. A tool that returns a document or a JSON array (spicrawl_docs_read, spicrawl_batch_task_content) puts it under content in structuredContent. Full argument reference: docs.spicrawl.com/agents/mcp.

Scrape a website to Markdown

Batch scrape many URLs

Browser sessions

| `sp