Scrapegraph MCP

ScrapeGraph MCP Server

A production-ready Model Context Protocol (MCP) server that provides seamless integration with the ScrapeGraph AI API. This server enables language models to leverage advanced AI-powered web scraping capabilities with enterprise-grade reliability.

Table of Contents

• Key Features
• Quick Start
• Available Tools
• Setup Instructions
• Remote Server Usage
• Local Usage
• Google ADK Integration
• Example Use Cases
• Error Handling
• Common Issues
• Development
• Contributing
• Documentation
• Technology Stack
• License

API v2

This MCP server targets ScrapeGraph API v2 (https://v2-api.scrapegraphai.com/api), aligned 1:1 with scrapegraph-py PR #84. Auth uses the SGAI-APIKEY header. Environment variables mirror the Python SDK:

• `SGAI_API_URL` — override the base URL (default https://v2-api.scrapegraphai.com/api)
• `SGAI_TIMEOUT` — request timeout in seconds (default 120)
• `SGAI_API_KEY` — API key (can also be passed via MCP scrapegraphApiKey or X-API-Key header)

Legacy aliases (still honored): SCRAPEGRAPH_API_BASE_URL for SGAI_API_URL, SGAI_TIMEOUT_S for SGAI_TIMEOUT.

Key Features

• Scrape & extract: scrape (POST /scrape, multi-format), extract (POST /extract, URL + prompt)
• Search: search (POST /search; num_results clamped 3–20)
• Crawl: Async multi-page crawl with crawl_start / crawl_get_status / crawl_stop / crawl_resume
• Schema: schema (POST /schema) — generate or augment a JSON Schema from a prompt
• Monitors: Scheduled jobs via monitor_create, monitor_list, monitor_get, pause/resume/delete, monitor_activity (paginated tick history)
• Account: credits, history
• Easy integration: Claude Desktop, Cursor, Smithery, HTTP transport
• Developer docs: .agent/ folder

Migration: v2 → v3

v3 renames every MCP tool that diverged from the v2 API docs. Hard rename, no aliases.

Quick Start

1. Get Your API Key

Sign up and get your API key from the ScrapeGraph Dashboard

2. Install with Smithery (Recommended)

npx -y @smithery/cli install @ScrapeGraphAI/scrapegraph-mcp --client claude

3. Start Using

Ask Claude or Cursor:

• "Convert https://scrapegraphai.com to markdown"
• "Extract all product prices from this e-commerce page"
• "Research the latest AI developments and summarize findings"

That's it! The server is now available to your AI assistant.

Available Tools

Removed: sitemap, agentic_scrapper, async-status polling, and (in v3) markdownify — use scrape with output_format="markdown".

Setup Instructions

To utilize this server, you'll need a ScrapeGraph API key. Follow these steps to obtain one:

1. Navigate to the ScrapeGraph Dashboard
2. Create an account and generate your API key

Automated Installation via Smithery

For automated installation of the ScrapeGraph API Integration Server using Smithery:

npx -y @smithery/cli install @ScrapeGraphAI/scrapegraph-mcp --client claude

Claude Desktop Configuration

Update your Claude Desktop configuration file with the following settings (located on the top rigth of the Cursor page):

(remember to add your API key inside the config)

{

"mcpServers": { "@ScrapeGraphAI-scrapegraph-mcp": { "command": "npx", "args": [ "-y", "@smithery/cli@latest", "run", "@ScrapeGraphAI/scrapegraph-mcp", "--config", "\"{\\\"scrapegraphApiKey\\\":\\\"YOUR-SGAI-API-KEY\\\"}\"" ] } } }

The configuration file is located at:

• Windows: %APPDATA%/Claude/claude_desktop_config.json
• macOS: ~/Library/Application\ Support/Claude/claude_desktop_config.json

Cursor Integration

Add the ScrapeGraphAI MCP server on the settings:

Remote Server Usage

Connect to our hosted MCP server - no local installation required!

[!WARNING]

The legacy MCP endpoint at https://mcp.scrapegraphai.com/mcp will be

deprecated soon. New integrations should use the replacement MCP endpoint:

https://sgai-mcp-main.onrender.com.

Claude Desktop Configuration (Remote)

Add this to your Claude Desktop config (~/Library/Application Support/Claude/claude_desktop_config.json on macOS):

{

"mcpServers": { "scrapegraph-mcp": { "command": "npx", "args": [ "mcp-remote@0.1.25", "https://sgai-mcp-main.onrender.com", "--header", "X-API-Key:YOUR_API_KEY" ] } } }

Cursor Configuration (Remote)

Cursor supports native HTTP MCP connections. Add to your Cursor MCP settings (~/.cursor/mcp.json):

{

"mcpServers": { "scrapegraph-mcp": { "url": "https://sgai-mcp-main.onrender.com", "headers": { "X-API-Key": "YOUR_API_KEY" } } } }

Benefits of Remote Server

• No local setup - Just configure and start using
• Always up-to-date - Automatically receives latest updates
• Cross-platform - Works on any OS with Node.js

Local Usage

To run the MCP server locally for development or testing, follow these steps:

Prerequisites

• Python 3.13 or higher
• pip or uv package manager
• ScrapeGraph API key

Installation

1. Clone the repository (if you haven't already):

git clone https://github.com/ScrapeGraphAI/scrapegraph-mcp

cd scrapegraph-mcp

1. Install the package:

# Using pip

pip install -e .

Or using uv (faster)

uv pip install -e .

1. Set your API key:

# macOS/Linux

export SGAI_API_KEY=your-api-key-here

Windows (PowerShell)

$env:SGAI_API_KEY="your-api-key-here"

Windows (CMD)

set SGAI_API_KEY=your-api-key-here

Running the Server Locally

You can run the server directly:

# Using the installed command

scrapegraph-mcp

Or using Python module

python -m scrapegraph_mcp.server

The server will start and communicate via stdio (standard input/output), which is the standard MCP transport method.

Testing with MCP Inspector

Test your local server using the MCP Inspector tool:

npx @modelcontextprotocol/inspector python -m scrapegraph_mcp.server

This provides a web interface to test all available tools interactively.

Configuring Claude Desktop for Local Server

To use your locally running server with Claude Desktop, update your configuration file:

macOS/Linux (~/Library/Application Support/Claude/claude_desktop_config.json):

{

"mcpServers": { "scrapegraph-mcp-local": { "command": "python", "args": [ "-m", "scrapegraph_mcp.server" ], "env": { "SGAI_API_KEY": "your-api-key-here" } } } }

Windows (%APPDATA%\Claude\claude_desktop_config.json):

{

"mcpServers": { "scrapegraph-mcp-local": { "command": "python", "args": [ "-m", "scrapegraph_mcp.server" ], "env": { "SGAI_API_KEY": "your-api-key-here" } } } }

Note: Make sure Python is in your PATH. You can verify by running python --version in your terminal.

Configuring Cursor for Local Server

In Cursor's MCP settings, add a new server with:

• Command: python
• Args: ["-m", "scrapegraph_mcp.server"]
• Environment Variables: {"SGAI_API_KEY": "your-api-key-here"}

Troubleshooting Local Setup

Server not starting:

• Verify Python is installed: python --version
• Check that the package is installed: pip list | grep scrapegraph-mcp
• Ensure API key is set: echo $SGAI_API_KEY (macOS/Linux) or echo %SGAI_API_KEY% (Windows)

Tools not appearing:

• Check Claude Desktop logs:
• macOS: ~/Library/Logs/Claude/
• Windows: %APPDATA%\Claude\Logs\
• Verify the server starts without errors when run directly
• Check that the configuration JSON is valid

Import errors:

• Reinstall the package: pip install -e . --force-reinstall
• Verify dependencies: pip install -r requirements.txt (if available)

Google ADK Integration

The ScrapeGraph MCP server can be integrated with Google ADK (Agent Development Kit) to create AI agents with web scraping capabilities.

Prerequisites

• Python 3.13 or higher
• Google ADK installed
• ScrapeGraph API key

Installation

1. Install Google ADK (if not already installed):

pip install google-adk

1. Set your API key:

export SGAI_API_KEY=your-api-key-here

Basic Integration Example

Create an agent file (e.g., agent.py) with the following configuration:

import os

from google.adk.agents import LlmAgent from google.adk.tools.mcp_tool.mcp_toolset import MCPToolset from google.adk.tools.mcp_tool.mcp_session_manager import StdioConnectionParams from mcp import StdioServerParameters

Path to the scrapegraph-mcp server directory

SCRAPEGRAPH_MCP_PATH = "/path/to/scrapegraph-mcp"

Path to the server.py file

SERVER_SCRIPT_PATH = os.path.join( SCRAPEGRAPH_MCP_PATH, "src", "scrapegraph_mcp", "server.py" )

root_agent = LlmAgent( model='gemini-2.0-flash', name='scrapegraph_assistant_agent', instruction='Help the user with web scraping and data extraction using ScrapeGraph AI. ' 'You can convert webpages to markdown, extract structur