certainlyscraped

certainlyscraped

A hybrid scraper with escalating effort.

It starts with the cheapest fetch that works, escalates only when a page is blocked or still failing, and reports exactly what it got and why it stopped. It reads the page, writes a config for you, checks that every field filled, and turns a listing into rows instead of a promise.

Quickstart: 20 seconds

pip install "certainlyscraped[mcp]"

cscrape init https://books.toscrape.com/catalogue/page-1.html --name books cscrape run books.yaml

init probes the site, finds the block that repeats on the page, guesses the fields and the next-page link, and prints how many items each field filled. run fetches the pages, writes out.csv, and fails loudly if it gets fewer rows than --min-rows (default 1) instead of reporting success.

For an agent, add the MCP server to your client. With uv there is nothing to install first:

{

"mcpServers": { "certainlyscraped": { "command": "uvx", "args": ["--from", "certainlyscraped[mcp]", "certainlyscraped", "mcp"] } } }

Without uv, pip install "certainlyscraped[mcp]" and use "command": "certainlyscraped", "args": ["mcp"].

That gives the agent probe, init, try_selector and scrape; the full contract is under MCP server (for agents).

Real output from the daily live check:

[probe] https://books.toscrape.com/catalogue/page-1.html -> tier 1 (static page)

[init] wrote books.yaml: 20 items on this page (20 repeats, median text 51 chars, each has a link, each has an image) field coverage on this page: title 20/20 (100%) url 20/20 (100%) image 20/20 (100%) price 20/20 (100%) availability 20/20 (100%)

The config is plain YAML you can read and edit:

name: books

url: "https://books.toscrape.com/catalogue/page-1.html" tier: 1 max_pages: 5 rate_limit_sec: 2.0 paginate: selector: "li.next a" items: selector: "article.product_pod" fields: title: { selector: "h3 a", attr: title } url: { selector: "h3 a", attr: href, absolute: true } image: { selector: "img.thumbnail", attr: src, absolute: true } price: { selector: "p.price_color", attr: text, type: float } availability: { selector: "p.instock.availability", attr: text } output: csv

The escalation ladder

Every page starts at the probed tier (or the config's tier) and climbs only when that rung fails.

Every group switch also prints [ladder] : , trying another proxy group, so a rotation is visible while it happens.

Once a site escalates on a challenge or a 403, later pages on that site start at the new level. A level reached only by an error climb is not remembered. steps in the page report shows the rungs and the leased group, for example ["t1@dc-a", "t1@dc-b", "t2@dc-b"] (a 403 from one group, a challenge from the other, real HTML at tier 2).

Tiers

tier: auto probes first and picks the lowest tier that returns a real page. With a proxy pool configured, auto skips the probe and starts at tier 1: a probe is a direct request, and with a pool no request may leave from the origin IP. The ladder climbs from there. A tier is not a promise: it is where the ladder starts.

Outcomes and exit codes

One record per page, available with --report report.jsonl and in the MCP response's pages field:

{"url": "https://httpbin.org/status/403", "outcome": "blocked", "status": 403, "rows": 0, "attempts": 2, "steps": ["t1", "t2"], "proxy_group": null, "elapsed_ms": 1358, "note": "max_tier 2 reached"}

outcome is one of ok, empty, blocked, rate_limited, not_found, error. A run also prints one summary line:

[done] 2 pages: 1 ok, 1 blocked (HTTP 403 on /status/403)

A page with no response has no HTTP status to quote, so it reads [done] 1 page: 1 error (no response on /list).

Exit 1 for a page-level failure is reachable only when --min-rows 0 asks for no rows. At the default --min-rows 1, a run that lost every page to errors fails the row check first and exits 2.

Exit 4 is narrow on purpose. A trailing not_found or empty page past the end of a range does not count, and neither does a page that succeeded on retry. --accept-partial maps 4 to 0 for scripts that only check for zero. A run that loses one page still writes the rows it did get, and the report says which page is missing.

MCP server (for agents)

pip install "certainlyscraped[mcp]"

certainlyscraped mcp

Client configuration:

{

"mcpServers": { "certainlyscraped": { "command": "certainlyscraped", "args": ["mcp"] } } }

Four tools, each a thin call into the same API:

config is inline YAML or JSON text, never a path. Config keys proxy, proxies, cookies, since, snapshot and fetcher are refused, and so are unknown keys such as json and headers (embedded JSON goes under items.json). status says what the run got: complete (at least min_rows rows and no incomplete page), incomplete (enough rows, but a page ended blocked, rate-limited or errored), too_few_rows (fewer than min_rows rows), blocked (zero rows and every page was walled or rate-limited) and error (zero rows and every page errored). These are not the CLI's exit codes: a blocked-only run exits 2, exit 3 is only probe --fail-blocked, and exit 1 is a config, I/O or fetch exception (or, with --min-rows 0, an all-error run). A bad request is still a status dict: a tool never raises into the JSON-RPC channel.

Private addresses are refused by default: loopback, RFC 1918, link-local (including 169.254.169.254) and unique-local IPv6, checked after DNS resolution and on every redirect hop. --allow-private turns that off for local development. The CLI does not refuse private addresses; only the MCP server does. A refusal is terminal for the page: one attempt, no retry, no climb, outcome error with the refusal in the note.

Proxies come only from the server's own --proxies proxies.yaml, never from a tool argument, and the server never reads CERTAINLYSCRAPED_PROXIES. probe, init and try_selector lease one proxy from that pool for the call; if the pool has nothing free they return blocked with no proxy available for and make no request. Without --proxies the server has no pool and fetches directly. File reads and datasets stay inside the working directory.

Proxies

One proxy still works and becomes a pool of one:

proxy: "http://10.1.2.3:8000"

A pool names groups so a 403 or a 429 can try a different network before paying for a heavier tier:

proxies:

list: proxies.txt # one URL per line, optional "# group" label gateway: # one generic provider gateway template: "http://{user}-country-{country}-session-{session}:{password}@gw.example:7000" user: myuser password_env: GATEWAY_PASSWORD country: se session: rotate # or sticky per_proxy_rate_sec: 1.0 cooldown_sec: 300 max_retries: 3

proxies.txt:

http://user:pass@10.1.2.3:8000 # dc-a

http://10.1.9.9:8000 # dc-b http://10.1.1.7:8000

Groups are the label after #, else a neutral label net-1, net-2, ... in order of first appearance, one per IPv4 /16, IPv6 /48 or hostname. A label never contains a proxy address or hostname. A proxy that is walled for one host is parked for that host for cooldown_sec; it is not removed from the pool. session: rotate generates a new token per lease: one per page, and a 403 or 429 that switches group leases again. sticky keeps one token per host and worker.

Passwords never belong in the main config. Put them in a dedicated proxies file passed with --proxies, or in the environment through password_env. A password inside an inline proxy:/proxies: URL in the committed YAML is refused with a message naming both alternatives. Page outcomes and MCP responses carry the group label, never the proxy URL or password. A page that failed on a dead proxy keeps the transport error as its note, for example ConnectionError: Failed to perform, curl: (7) Failed to connect to books.toscrape.com:443 over proxy *** ... (the target is kept, the proxy address is ***); an unresolvable host reads DNSError: ... Could not resolve host: no-such-host.invalid .... No log line quotes a proxy address, so the group label and the error text are the diagnosis. A private address refused by the MCP guard is terminal: one attempt, no retry, no climb.

Precedence for run: --no-proxy > --proxies FILE > CERTAINLYSCRAPED_PROXIES > the config. --no-proxy ignores every configured proxy and fetches directly. MCP has no such flag, never reads CERTAINLYSCRAPED_PROXIES, and without --proxies has no pool at all.

Speed comes from many hosts or a proxy pool, not from threads alone. The ladder paces per (host, proxy) at per_proxy_rate_sec (it falls back to rate_limit_sec; default 1.0). With no pool, one host gets at most one request per interval no matter the concurrency, so a single-host pages: job is still paced by the site. With a pool, N proxies to one host give N slots: eight pages at 1.0 s on one host took 7.1 s with one proxy and 1.1 s with four, in one derived net-1 group o