Octocrawl

Name: Octocrawl
Author: 77777R7

by 77777R7version 0.8.7i★ –iComputer or cloud

Registry nameio.github.77777R7/octocrawl

Install

Overview

Description from the maker (original English)

Pages, crawls and URL batches as clean Markdown and JSON for AI agents, with an Evidence Record.

PopNx MCP’s three pillars

These three are never merged into one score: opinions, measurements and popularity are different things.

User voiceiNo reviews yetReviews open together with accounts in a later step
Our basic checksi3 of 3 passedChecked from public information; code not scanned yet
UsageiNot measured yet★ –

What you can do with it

We have not written a plain-language summary for this one yet. Below are the facts we read from the registry.

Connection: Computer or cloud
Transport: stdio, Streamable HTTP
Category: Other
You need to fill in: W2L_API_TOKEN

Install

Pick the AI app you use, then copy the settings.

Before you install

Basic checks: 3 of 3 passed
Runs on the maker’s server: what you ask it goes through their server.
Can do 2 things (tools) scrape, map
You fill in yourself: W2L_API_URL, W2L_API_TOKEN, W2L_MCP_TOOLSETS, W2L_MCP_READ_ONLY

Values marked secret are keys to your account. Never share them or paste them into a chat.

You need

Runs in the cloudNothing to install; it connects over the internet
npmNode.js installed (for the npx command)

Which app do you use?

ClaudeClaude CodeCursorVS CodeChatGPT

Pick your app above. We remember it, so next time the button says “Install in …”.

Works in Claude on the web, desktop and mobile:

Open Settings → Connectors
Press Add custom connector
Paste the address below and press Add
If it asks you to sign in, sign in with the maker

https://mcp.octocrawl.dev/mcp

Or run it on your computer

Open Claude Desktop → Settings → Developer → Edit Config, and paste this into claude_desktop_config.json

{
  "mcpServers": {
    "octocrawl": {
      "command": "npx",
      "args": [
        "-y",
        "@octocrawl/mcp"
      ],
      "env": {
        "W2L_API_URL": "your-value",
        "W2L_API_TOKEN": "your-value",
        "W2L_MCP_TOOLSETS": "your-value",
        "W2L_MCP_READ_ONLY": "your-value"
      }
    }
  }
}

Run this in your terminal

claude mcp add --transport http octocrawl https://mcp.octocrawl.dev/mcp

Or run it on your computer

claude mcp add octocrawl --env W2L_API_URL=your-value --env W2L_API_TOKEN=your-value --env W2L_MCP_TOOLSETS=your-value --env W2L_MCP_READ_ONLY=your-value -- npx -y @octocrawl/mcp

Fill in the empty values before running it.

Open in Cursor

Or copy the settings yourself

Paste into .cursor/mcp.json in your project, or ~/.cursor/mcp.json

{
  "mcpServers": {
    "octocrawl": {
      "url": "https://mcp.octocrawl.dev/mcp"
    }
  }
}

Or run it on your computer

Open in Cursor

Or copy the settings yourself

Paste into .cursor/mcp.json in your project, or ~/.cursor/mcp.json

{
  "mcpServers": {
    "octocrawl": {
      "command": "npx",
      "args": [
        "-y",
        "@octocrawl/mcp"
      ],
      "env": {
        "W2L_API_URL": "your-value",
        "W2L_API_TOKEN": "your-value",
        "W2L_MCP_TOOLSETS": "your-value",
        "W2L_MCP_READ_ONLY": "your-value"
      }
    }
  }
}

Your browser asks to open the app, then the app shows the settings and asks you to confirm. Key values are left empty for you to fill in.

Open in VS Code

Or copy the settings yourself

Paste into .vscode/mcp.json in your project

{
  "servers": {
    "octocrawl": {
      "type": "http",
      "url": "https://mcp.octocrawl.dev/mcp"
    }
  }
}

Or run it on your computer

Open in VS Code

Or copy the settings yourself

Paste into .vscode/mcp.json in your project

{
  "servers": {
    "octocrawl": {
      "type": "stdio",
      "command": "npx",
      "args": [
        "-y",
        "@octocrawl/mcp"
      ],
      "env": {
        "W2L_API_URL": "your-value",
        "W2L_API_TOKEN": "your-value",
        "W2L_MCP_TOOLSETS": "your-value",
        "W2L_MCP_READ_ONLY": "your-value"
      }
    }
  }
}

Your browser asks to open the app, then the app shows the settings and asks you to confirm. Key values are left empty for you to fill in.

ChatGPT connects only to servers on the internet:

Open ChatGPT on the web → Settings → Connectors (Apps)
Press Create or Add (some plans need Developer mode under Advanced first)
Paste the address below, give it a name and save
If it asks you to sign in, sign in with the maker

https://mcp.octocrawl.dev/mcp

Menu names can differ slightly by plan and over time.

Your choice stays on this device in a cookie; no account needed.

Values you fill in

W2L_API_URL env
The Octocrawl API the server calls: npx octocrawl serve on this computer (default http://127.0.0.1:8787), or https://api.octocrawl.dev
W2L_API_TOKEN secret env
A bearer token for a hosted or self-hosted API; none for a local one
W2L_MCP_TOOLSETS env
The tool groups to list, comma-separated: core, crawl, batch, products, logins, monitor, delivery, or all (default all); the same as --toolsets
W2L_MCP_READ_ONLY env
1 lists read-only tools alone, with a scrape that does not act as the person; the same as --read-only

These settings are generated from the maker’s registry entry. If the README says otherwise, follow the README.

What it can do

Tool names and descriptions exactly as the server reported them on Nov 7, 2026. The maker wrote them, not us; we show them as plain text.

scrape
Read one URL as Markdown with its Evidence Record (final URL, fetch time, HTTP status, robots.txt decision, hashes), over HTTP first and in the browser when the page needs it. Not for: many URLs or a whole site section. Compact by default; debug adds the full audit. warnings name
map
List a site's URLs without reading each page: the start page's links (read over HTTP alone) and the entries of the sitemaps the site declares, within one deadline. URLs stay in scope (the start host and its www twin, the start path's subtree, assets left out, similar URLs folded)

Basic checksi

We check information anyone can see; we have not read the code. Checks that do not apply are left out of the count. · Checked Nov 7, 2026

✓ Pass
Maker matches the repo
The registry name was verified with the GitHub account 77777R7, which owns the repo
… Not checked yet
Still maintained
We have not read this repo from GitHub yet
… Not checked yet
Has a licence
We have not read this repo from GitHub yet
✓ Pass
Endpoint responds
Responded in 367 ms (HTTP 200)
– Not applicable
Sign-in with OAuth
No sign-in needed
✓ Pass
No hidden instructions in tool descriptions
Read 2 tool descriptions and found no hidden instructions to the AI

Compared with others in Otheri

More stars than 0% of the other 32725 servers in this category
Published to the registry more recently than 100% of the other 32725 servers in this category
Basic checks equal to or better than 100% of the other 32725 servers in this category

Compared with every active server in this category in the registry. Stars count only when the publisher owns the repo.

Common questions

Is Octocrawl safe?

It passed 3 of 3 applicable basic checks. We have not read the code. Before adding a key or token, see what access the server asks for and use a key with only the permissions you need.

What keys or codes does it need?

You need to fill in: W2L_API_TOKEN

Does it run on my computer or in the cloud?

Either way works: install it on your computer or connect to the cloud version.

Is it still maintained?

The GitHub repo last changed –, and the latest version (0.8.7) reached the registry today.

Which apps can use it?

Apps that support MCP, such as Claude Desktop, Claude Code, Cursor and VS Code. It connects by URL, so apps that accept remote MCP servers work too, such as ChatGPT in developer mode.

What can I use instead?

In the same category: Spec Driven Development, Remote Desktop Commander, Signoff

Alternatives

Servers in the same category from other makers, sorted by stars.

Reading a card:★ GitHub starsiThree pillars: reviews · checks · usageiUpdated = latest version published to the registryi

Spec Driven Development

by formulahendry

MCP Server that facilitates spec-driven development workflows, not just Vibe Coding.

Runs on your computer★ 440MITUpdated 14 months ago

Install

Remote Desktop Commander

by desktop-commander

Remote Desktop Commander: use files, terminal and processes on your own computers from any AI app.

Runs in the cloud★ 211Updated 2 days ago

Install

Signoff

by ayuanxx-max

Five gates that make an agent's 'done' provable: run them, read the verdict, audit the record.

Runs on your computerUpdated today

Install

README from the maker

Open the README

Copied from the maker’s repo as written. PopNx MCP has not reviewed or endorsed it. HTML, images and scripts are removed. Links lead off this site.

Octocrawl

One link. Web data, ready.

Markdown and data from web pages, for AI agents and RAG.

Fewer tokens than raw HTML. Every result records how it was read.

Website · Docs · Connect MCP · Blog · Changelog

Scrape pages. Crawl websites. Turn web content into usable data.

Octocrawl is an open-source web scraping and crawling toolkit for AI agents, RAG and research. Collect documentation for a knowledge base, extract product details for comparison, or turn web tables and listings into data for analysis. Each page comes with a source record.

Try a page in your browser, or connect your agent through hosted MCP or the REST API. No installation or API key is needed to try the hosted service.

What can you do with Octocrawl?

Capability	What you get
Scrape a page	Readable Markdown for LLM context, plus HTML, links, image URLs and page metadata. Request a screenshot on the browser lane.
Map a website	Discover URLs from sitemaps and page links, and filter them by a search term to find the pages you need.
Crawl a site or section	Follow links from a starting URL and collect each page's content, with path filters, depth and page limits.
Scrape a list of URLs	Submit a batch, retrieve results as they arrive, and keep the outcome of every URL, including failures.
Extract structured data	Get tables and repeated listings as JSON and CSV, or pass a JSON Schema for supported fields such as product names, prices and page labels. Extraction guide.
Read dynamic pages · local actions	Render JavaScript, click controls, scroll, load more items and follow pagination before extracting content. Browser actions.
Read PDF reports	Extract a PDF's text layer with page markers for citations. CSV, JSON and plain-text URLs also return their text. File support.
Revisit sources on a schedule · local workers	Run Monitors and send their events to an HTTPS endpoint with retried delivery. Monitor setup.

Hosted Octocrawl offers scrape and map, and with an account crawl and batch jobs (up to 500 pages a crawl, 1,000 URLs a batch); keyless requests use HTTP, and an account key enables the browser lane. Run locally for larger jobs, page actions and Monitors; see Running Octocrawl for setup and self-hosting options. The web preview is a single-page trial, with five pages a day and up to 20 named fields.

Why Octocrawl?

Web data inside your agent. Connect Claude Code, Cursor, OpenCode or Codex through MCP, or build with REST, TypeScript, Python and the Vercel AI SDK.
Structured fields without a model by default. Read supported values from the page itself, with field evidence and explicit missing values. Model fallback is opt-in.
Know when a scrape fell short. Checks flag challenge pages, wrong content, missing facts, truncation and too little content. Blocked and failed results carry a reason; fetched error pages keep their status and Markdown. Result states.
Trace results back to their sources. Each page carries an Evidence Record with its URL, fetch time, status and extraction evidence. Unobserved values stay null.
Run it where you need it. Use the hosted service, run with npx, or self-host with Docker Compose. The server is AGPL-3.0; the client libraries are MIT.

If Octocrawl helps your work, give the repository a star.

Quick start

Connect your agent to hosted Octocrawl. No key is needed to start (steps for Claude Code, Cursor, OpenCode and Codex):

claude mcp add --transport http octocrawl https://mcp.octocrawl.dev/mcp

Without a key it reads 20 pages a day over HTTP. For more, sign in and make a key: an account gets 1,000 credits a month and the browser lane. The key goes in a header, never in the URL:

claude mcp add --transport http octocrawl https://mcp.octocrawl.dev/mcp --header "Authorization: Bearer oc_…"

Or run it on your own computer. You need Node.js 22.13+ or 24+:

npx octocrawl scrape https://example.com --markdown

pip install 'octocrawl-client[pandas]' # the Python client of a running API (npx octocrawl serve); [pandas] for to_pandas() npm install @octocrawl/sdk # the TypeScript client; @octocrawl/mcp is the MCP server npm install @octocrawl/ai-sdk # scrape and map as Vercel AI SDK tools, on hosted Octocrawl by default

Pages are read over HTTP first. For pages that only appear in a browser, install Chromium once with npx playwright install chromium; until then such a page comes back with a browser_not_installed warning that names the command.

To give your agent every tool, with no daily limit, run the API locally and connect the MCP server to it:

npx octocrawl serve                                  # keep it running: the API on 127.0.0.1:8787

claude mcp add octocrawl -- npx -y @octocrawl/mcp # the published stdio server, a client of that API

The hosted service offers scrape and map, and with an account crawl and batch jobs. The local server adds larger jobs, the Amazon.sg product tool and the Monitor tools. To serve other machines yourself, npx octocrawl serve --hosted --token refuses private addresses, robots overrides and saved logins as the hosted service does. Running Octocrawl covers all three; Docker Compose runs one on your own server. An agent installing Octocrawl for you (Cline, for one) can follow llms-install.md.

What a failure looks like

The test page below answers HTTP 200 but shows a Cloudflare check instead of content (fixture block-challenge-200). A crawler that trusts the status code counts it as a success. Octocrawl tries plain HTTP, then a browser, and reports the page as blocked, with what to do next:

npx octocrawl scrape http://127.0.0.1:8799/block/challenge-200

"status": "blocked", "blockReason": "cloudflare_challenge", "channelsTried": ["http", "browser_local"], "markdown": null, "snapshot": { "httpStatus": 200, "artifacts": [] }, "usage": { "wallMs": 1079, "bytesWire": null, "externalCostUsd": 0 }, "agentHints": [ "127.0.0.1 gates automated access on the lanes tried (http, browser_local); … a proxy or session you own is the supported route, or, on your own machine, getting through the check yourself in your own Chrome: handoff: true on a scrape (octocrawl scrape --handoff) …" ], "evidenceRecord": { "schemaVersion": "w2l.evidence/1", "status": "blocked", "reason": "cloudflare_challenge", "lane": "browser_local", "…": "…" } }

This is real output of octocrawl 0.4.1, trimmed (…) and rounded, against the repository's fixture server (FIXTURE_PORT=8799 node --import tsx packages/fixtures/src/cli.ts). bytesWire: null means wire bytes were not measured; artifacts: [] means no screenshot or DOM snapshot was made.

Documentation

The site's docs at octocrawl.dev/docs cover MCP setup for four clients, task guides and limits. They are built from apps/public-web/content.

The full reference lives in docs/:

Topic	Page
The `octocrawl` command	Command line
Hosted, local and self-hosted servers, ports, tokens, proxies, robots.txt	Running Octocrawl
Options for a scrape and what a result carries	Scrape options and results
Many URLs, whole sites, site maps, webhooks	Batches, crawls and maps
Structured data from a JSON Schema	JSON extraction
PDF, CSV, XLSX, ZIP and JSON URLs	Files
Your login, enhanced access, handoff, your own Chrome	Access
Scheduled re-reads and HTTPS events	Monitors
Moving from Firecrawl	Firecrawl shim

Walkthroughs of real runs are on the blog: Claude Code web scraping with MCP, Web scraping for RAG, Find all pages on a website, Scrape a website with login, in your own Chrome, Extract structured data without an LLM, Scrape a list of URLs and keep every failure.

For researchers, two guides walk through a real run: From a URL list to a CSV with evidence and Citing web data in a paper.

For the Monitor → result → HTTPS event → restart workflow, follow the onboarding guide.

Work on the code

You need Node.js 22.13+ or 24+ and npm. The PDF engine, pdf.js, needs 22.13 or later.

git clone https://github.com/77777R7/Octocrawl.git

cd Octocrawl npm ci npx playwright install chromium npm run typecheck npm test npm run scrape -- https://example.com --markdown npm run crawl -- https://example.com --max-pages 20

The packag

Only the beginning is shown; read the rest in the repo

Octocrawl

Overview

PopNx MCP’s three pillars

What you can do with it

Install

Before you install

Or run it on your computer

Or run it on your computer

Or run it on your computer

Or run it on your computer

Values you fill in

What it can do

Basic checksi

Maker matches the repo

Still maintained

Has a licence

Endpoint responds

Sign-in with OAuth

No hidden instructions in tool descriptions

Compared with others in Otheri

Common questions

Alternatives

Spec Driven Development

Remote Desktop Commander

Signoff

README from the maker

What can you do with Octocrawl?

Why Octocrawl?

Quick start

What a failure looks like

Documentation

Work on the code