Crawlforge MCP Server

31 web scraping, crawling, deep-research & autonomous-extraction tools for Claude, Cursor & any MCP client.

Clean Markdown & structured JSON from any site. Get started with 1,000 free credits — no credit card required.

⭐ Star us on GitHub to follow along — it genuinely helps others discover the project.

Table of Contents

• Why CrawlForge?
• CrawlForge vs. alternatives
• Quick Start (2 Minutes)
• Available Tools
• Pricing
• Advanced Configuration
• Usage Examples
• Security & Privacy
• Support
• Contributing

🎯 Why CrawlForge?

• 31 MCP-native tools — scraping, crawling, search, real Google SERP rank tracking, deep research, an autonomous agent, a unified multi-format scrape, document processing, stealth browsing, stateful browser sessions, and more, callable directly from your AI assistant.
• Generous free tier — 1,000 credits to start instantly, no credit card. The grant is one-time rather than monthly, and the credits never expire.
• Local-LLM by default — extract_with_llm runs against a local Ollama model out of the box: no LLM API key, no per-token cost, and your data never leaves your machine. Cloud (OpenAI/Anthropic) is opt-in.
• LLM-ready output — clean Markdown, structured JSON (schema-driven), screenshots, links, and metadata from a single fetch.
• Autonomous `agent` — describe what you need in natural language; it plans, gathers, and shapes an answer under orchestrator-enforced hard stops (max steps/URLs/wall-clock) — no URLs required.
• Security-hardened — SSRF protection on every request, a fail-closed backend allow-list, a vetted action allowlist for browser automation, and per-tool credit gating.
• Works everywhere MCP does — Claude Desktop, Claude Code, Cursor, and any other MCP-enabled client, configured in one command.

📊 CrawlForge vs. alternatives

Comparison reflects publicly documented capabilities at time of writing. CrawlForge is MIT-licensed and MCP-first — built to plug straight into AI coding assistants.

🚀 Quick Start (2 Minutes)

1. Install from NPM

npm install -g crawlforge-mcp-server

2. Setup Your API Key (required)

Every tool requires a CrawlForge API key — new accounts get 1,000 free trial credits to start. The recommended path signs you in through the browser, so the key is never pasted into a terminal (a coding agent can run this for you and relay the URL):

crawlforge login

It prints an approval URL; open it, approve, and the key is stored in ~/.crawlforge/config.json. Then run crawlforge init to register the MCP server with your client. Or use the interactive wizard, which also configures your clients:

npx crawlforge-setup

This will:

• Guide you through getting your free API key
• Configure your credentials securely
• Auto-configure Claude Code and Cursor (if installed)
• Verify your setup is working

Don't have an API key? Get one free at https://www.crawlforge.dev/signup

One-step setup (v4.6.0+): crawlforge init detects your API key, installs the agent skill, and idempotently merges the MCP config stanza into Claude Code, Claude Desktop, and Cursor. Use crawlforge init --all --yes to configure every detected client non-interactively.

3. Configure Your IDE (if not auto-configured)

🤖 For Claude Desktop

Add to claude_desktop_config.json:

{

"mcpServers": { "crawlforge": { "command": "npx", "args": ["-y", "crawlforge-mcp-server"] } } }

Location:

• macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
• Windows: %APPDATA%/Claude/claude_desktop_config.json
• Linux: ~/.config/Claude/claude_desktop_config.json

Restart Claude Desktop to activate.

🖥️ For Claude Code CLI (Auto-configured)

The setup wizard automatically configures Claude Code by adding to ~/.claude.json:

{

"mcpServers": { "crawlforge": { "type": "stdio", "command": "crawlforge-mcp" } } }

After setup, restart Claude Code to activate.

💻 For Cursor IDE (Auto-configured)

The setup wizard automatically configures Cursor by adding to ~/.cursor/mcp.json:

{

"mcpServers": { "crawlforge": { "type": "stdio", "command": "crawlforge-mcp" } } }

Restart Cursor to activate.

🔁 For n8n (workflow automation)

n8n's built-in MCP Client Tool node connects over Streamable HTTP (works on n8n Cloud and self-hosted). Run the server in HTTP mode:

export CRAWLFORGE_API_KEY=your_api_key

npm run start:http # Streamable HTTP endpoint at http://localhost:10000/mcp

Then point the MCP Client Tool node at http:// :10000/mcp with transport HTTP Streamable and a Bearer credential set to the same API key. On self-hosted n8n you can instead use the community n8n-nodes-mcp node over STDIO (npx -y crawlforge-mcp-server).

Full guide: docs/n8n-integration.md

Which launch command? npx -y crawlforge-mcp-server needs no global install and always runs the published version (recommended for Claude Desktop). For a global install (npm i -g crawlforge-mcp-server), use the dedicated crawlforge-mcp bin — it resolves on your PATH, so it survives Node/nvm version switches. The bare crawlforge command still launches the server when an MCP client spawns it over stdio (backward compatibility for configs created before v4.2.5); interactively it's the CLI — run crawlforge mcp to start the server by hand.

📊 Available Tools

CrawlForge requires a CrawlForge API key — every tool is metered and consumes credits. New accounts get 1,000 free trial credits to start. Get a key at crawlforge.dev/signup.

All Tools (API key required)

| browser_session | 3 | A browser page that stays open across calls, keeping its cookies and its login in between. open a session on a URL, snapshot it to list the interactive elements as stable refs (@e1, @e2), act on a ref, read the content, close. Priced per operation: open 3, read 2, snapshot/act/screenshot/close/list 1 each. Reach for it when you must see the page before choosing what to click, when the flow spans several calls, or when a login must hold across later reads; one fixed act