PyScrappy

Adaptive Python web scraping toolkit (self-healing, stealth)

• MCP server for AI agents

PyScrappy is an AI-native web scraping toolkit that turns websites into structured, LLM-ready data. Use it as a Python library or expose it as an MCP server for AI agents.

📖 Documentation: pyscrappy.vercel.app

Key features

• Generic scraper — give it any URL, get back structured text, links, images, tables, and metadata
• LLM-ready output — .to_markdown() turns any result into clean Markdown; also .to_json() and .to_dataframe()
• MCP server — expose the scrapers as tools for AI agents (Claude, Cursor, local LLMs, …)
• JS rendering — optional Playwright backend for JavaScript-heavy sites
• Custom selectors — pass CSS selectors to extract exactly what you need
• Chainable `Selector` — navigate HTML directly with CSS/XPath, find_all, find_by_text, and find_similar (Scrapy/BeautifulSoup-style)
• Adaptive (self-healing) selectors — remember an element and relocate it by similarity when a site changes its markup, so scrapers don't silently break
• Concurrent scraping — scrape_many / scrape_all run scrapes in parallel
• Sitemap crawling — enumerate and scrape a whole site from its sitemap.xml (index + gzip aware)
• Proxy & scraping-API support — route through a proxy or ScraperAPI/ScrapeOps for blocked sites
• TLS-fingerprint impersonation — impersonate="chrome" gets past anti-bot filters that block plain clients (optional curl_cffi backend)
• Command-line extract — pyscrappy extract out.md scrapes a URL straight to a file, no code
• Retry & rate-limiting — built-in exponential backoff and per-domain rate limiting
• Type-safe — full type hints, py.typed marker
• 20+ built-in scrapers — Wikipedia, IMDB, stocks, news, GitHub, Amazon/IKEA, YouTube, and more

Installation

pip install pyscrappy

Optional extras:

# Browser support (for JS-rendered pages)

pip install 'pyscrappy[browser]' playwright install chromium

DataFrame support

pip install 'pyscrappy[dataframe]'

MCP server (use PyScrappy's scrapers as AI-agent tools)

pip install 'pyscrappy[mcp]'

Stealth (TLS-fingerprint impersonation to bypass anti-bot filters)

pip install 'pyscrappy[stealth]'

Parquet / Excel export (ScrapeResult.to_parquet() / .to_excel())

pip install 'pyscrappy[parquet]' pip install 'pyscrappy[excel]'

Everything

pip install 'pyscrappy[all]'

For AI agents

PyScrappy ships an MCP server that exposes its scrapers as tools, so an agent (Claude, Cursor, an OpenAI agent, a local LLM) can pull structured web data from any URL and hand it straight to the model:

AI agent  ──MCP tool call──▶  PyScrappy  ──fetch + extract──▶  Any website

▲ │ └────────────── clean Markdown / JSON ◀───────────────────────┘

pip install 'pyscrappy[mcp]'

claude mcp add pyscrappy pyscrappy-mcp

Then just ask: *"use pyscrappy to summarize the latest headlines from bbc.com."* See MCP server for the full setup and tool list.

Local models (Ollama), no MCP host needed

Ollama can't talk MCP on its own, so normally you'd run a host (Goose, Cline, …) in between. PyScrappy skips that with a built-in agent that talks to Ollama directly and lets a local model call the scrapers as tools:

pip install 'pyscrappy[mcp]'                 # needs Python 3.10+

pyscrappy chat --model qwen2.5 "what's the current AAPL quote?"

It exposes the same 22 tools as the MCP server. The only requirement is a model that supports tool calling (Llama 3.1, Qwen 2.5, Mistral, …); how well it *picks* the right tool is up to the model. Point it at a remote Ollama with --host, and pass -v to see each tool call.

MCP server (use PyScrappy from an AI agent)

PyScrappy ships an optional Model Context Protocol server, so an AI agent (e.g. Claude) can call PyScrappy's scrapers as tools and get structured web data back.

pip install 'pyscrappy[mcp]'

The MCP extra installs the standalone fastmcp package and requires Python 3.10 or newer. On Python 3.9 the core scraping library still works, but the MCP server is unavailable.

This installs the pyscrappy-mcp command. It uses stdio by default for local MCP clients; Streamable HTTP and legacy SSE are available for remote deployments:

pyscrappy-mcp          # stdio (default)

pyscrappy-mcp --http # Streamable HTTP pyscrappy-mcp --sse # legacy SSE

You can also run the stdio server with python -m pyscrappy.mcp.

Register with Claude Code

claude mcp add pyscrappy pyscrappy-mcp

Register with Claude Desktop

Add to your claude_desktop_config.json and restart the app:

{

"mcpServers": { "pyscrappy": { "command": "pyscrappy-mcp" } } }

Tip: Claude Desktop does not inherit your shell PATH. If pyscrappy-mcp

is not found, use the absolute path to the command (e.g. the one printed by

which pyscrappy-mcp).

Available tools

The server exposes 20+ tools. The most common ones are `scrape_url` (any URL → text, links, images, tables, metadata), `scrape_wikipedia`, `scrape_stock`, `scrape_news`, and `search_github` — plus many more covering image/YouTube/LinkedIn/Hacker News/book search, weather, crypto, currency, dictionary, Amazon/Newegg/IKEA/SoundCloud, IMDB, and Zomato/Uber Eats.

To see the full, live list, ask the agent to call the `list_available_scrapers` tool, or from a shell:

python -c "from pyscrappy import list_scrapers; print(', '.join(sorted(list_scrapers())))"

The lookup_movie tool needs a free OMDb API key. Pass it to the server through your MCP client config, e.g. for Claude Desktop:

{

"mcpServers": { "pyscrappy": { "command": "pyscrappy-mcp", "env": { "OMDB_API_KEY": "your-key" } } } }

Once registered, just ask the agent naturally, e.g. *"use pyscrappy to get the latest headlines from bbc.co.uk and the AAPL stock quote."*

Built-in scrapers

PyScrappy ships 24 built-in scrapers, and every one that works without a proxy is also exposed as an MCP tool.

A few of them:

• `GenericScraper` — scrape any URL with auto-extraction (text, links, images, tables, metadata)
• Data / research — `WikipediaScraper`, `StockScraper` (Yahoo Finance), `NewsScraper` (RSS/Atom), `GitHubScraper`, `HackerNewsScraper`, plus weather, crypto, currency, dictionary, image, LinkedIn-jobs, and book search
• E-commerce — `AmazonScraper`, NeweggScraper, IKEAScraper
• Social / media / food — `YouTubeScraper`, SoundCloud, Zomato, Uber Eats (Instagram / Twitter / Spotify also ship, but are blocked and need a proxy)

…and many more. To see the full, live list:

python -c "from pyscrappy import list_scrapers; print(', '.join(sorted(list_scrapers())))"

`IMDBScraper` (lookup_movie) is the one exception that needs a key — a free OMDb OMDB_API_KEY (see the MCP config above for how to pass it).

Plugins

PyScrappy is extensible: you can add your own scrapers, and third parties can ship them as standalone pyscrappy- packages. A registered scraper works everywhere a built-in does, including the MCP server and the pyscrappy chat agent, with no change to PyScrappy core.

In your own code — register with the decorator:

from pyscrappy import BaseScraper, register_scraper, get_scraper

from pyscrappy.core.models import ScrapeResult, ScrapeMetadata

@register_scraper("reddit") class RedditScraper(BaseScraper): def scrape(self, subreddit: str, **kwargs) -> ScrapeResult: data = self.fetch_and_parse(f"https://old.reddit.com/r/{subreddit}/.json")

... build a list of dicts ...

return ScrapeResult(data=[...], metadata=ScrapeMetadata(scraper="reddit"))

get_scraper("reddit")().scrape(subreddit="python")

As a distributable package — advertise an entry point in your pyproject.toml, and PyScrappy discovers it once your package is installed:

[project.entry-points."pyscrappy.scrapers"]

reddit = "pyscrappy_reddit:RedditScraper"

After pip install pyscrappy-reddit, the scraper shows up in list_scrapers(), and an AI agent can call it via the scrape_with MCP tool — no core change required.

First-class MCP tools (optional). Add an mcp_tools mapping and your scraper becomes a dedicated, typed MCP tool instead of only being reachable through the generic scrape_with — its schema is derived from the method signature, so agents get proper named arguments:

@register_scraper("reddit")

class RedditScraper(BaseScraper): mcp_tools = {"search_reddit": "scrape"} # tool name -> method

def scrape(self, subreddit: str, sort: str = "hot") -> ScrapeResult: ...

See the plugin template for a complete, copyable starting point, and the plugin guide for the full walkthrough.

Quick start

Scrape any URL → clean, LLM-ready Markdown

from pyscrappy import scrape

result = scrape("https://en.wikipedia.org/wiki/Web_scraping")

print(result.to_markdown()) # feed straight to an LLM

...or result.to_json() / result.to_dataframe()

Write to a file — format inferred from the extension:

result.save("out.json") # .json .csv .md .ndjson .yaml .parquet .xlsx

(.parquet needs pyscrappy[parquet]; .xlsx needs pyscrappy[excel])

Prefer raw fields? Every result is a ScrapeResult with .data (a list of dicts):

print(result.data[0]["metadata"]["title"])

print(result.data0["word_count"])

Custom CSS selectors

from pyscrappy import GenericScraper

with GenericScraper() as gs: result = gs.scrape( url="https://news.ycombinator.com", selectors={"title": ".titleline a", "score": ".score"}, ) for item in result.data: print(item["title"], item.get("score", ""))

Navigate HTML with Selector

When you want to traverse markup directly (Scrapy/BeautifulSoup-style) rather than get back structured dicts, use Selector:

from pyscrappy import Selector

page = Selector(html) # or navigate any HTML string page.css(".title::text").getall() # CSS with ::text / ::attr(