Octocrawl

Name: Octocrawl
Author: 77777R7

作者：77777R7版本 0.8.7i★ –i电脑或云端

注册名称io.github.77777R7/octocrawl

安装

概览

作者提供的说明（英文原文）

Pages, crawls and URL batches as clean Markdown and JSON for AI agents, with an Evidence Record.

PopNx MCP 的三大支柱

这三项永远不会合并成一个分数：意见、测量和人气是不同的东西。

用户评价i暂无评价评价功能将在后续阶段与账户功能一起开放
我们的基础检查i3 项中通过 3 项依据公开信息检查；尚未扫描代码
使用情况i尚未统计★ –

它能帮你做什么

我们还没有为它写通俗的介绍。下面是我们从注册中心读取到的事实。

连接方式: 电脑或云端
传输方式: stdio, Streamable HTTP
分类: 其他
你需要填写：W2L_API_TOKEN

安装

选择你使用的 AI 应用，然后复制设置。

安装之前

基础检查：3 项中通过 3 项
在作者的服务器上运行：你向它提出的内容会经过他们的服务器。
可以做 2 件事（工具） scrape, map
需要你自己填写： W2L_API_URL, W2L_API_TOKEN, W2L_MCP_TOOLSETS, W2L_MCP_READ_ONLY

标为机密的值是你账户的钥匙。千万不要分享给别人，也不要粘贴到聊天里。

你需要

在云端运行无需安装任何东西，通过互联网连接
npm已安装 Node.js（用于 npx 命令）

你用的是哪个应用？

ClaudeClaude CodeCursorVS CodeChatGPT

在上方选择你的应用。我们会记住它，下次按钮会显示“安装到 …”。

适用于网页版、桌面版和手机版 Claude：

打开 Settings → Connectors
点击 Add custom connector
粘贴下面的地址，然后点击 Add
如果要求登录，请登录作者的服务

https://mcp.octocrawl.dev/mcp

或者在你的电脑上运行

打开 Claude Desktop → Settings → Developer → Edit Config，把下面的内容粘贴到 claude_desktop_config.json

{
  "mcpServers": {
    "octocrawl": {
      "command": "npx",
      "args": [
        "-y",
        "@octocrawl/mcp"
      ],
      "env": {
        "W2L_API_URL": "your-value",
        "W2L_API_TOKEN": "your-value",
        "W2L_MCP_TOOLSETS": "your-value",
        "W2L_MCP_READ_ONLY": "your-value"
      }
    }
  }
}

在终端中运行这条命令

claude mcp add --transport http octocrawl https://mcp.octocrawl.dev/mcp

或者在你的电脑上运行

claude mcp add octocrawl --env W2L_API_URL=your-value --env W2L_API_TOKEN=your-value --env W2L_MCP_TOOLSETS=your-value --env W2L_MCP_READ_ONLY=your-value -- npx -y @octocrawl/mcp

运行前请先填写空着的值。

在 Cursor 中打开

或者自己复制设置

粘贴到项目中的 .cursor/mcp.json，或 ~/.cursor/mcp.json

{
  "mcpServers": {
    "octocrawl": {
      "url": "https://mcp.octocrawl.dev/mcp"
    }
  }
}

或者在你的电脑上运行

在 Cursor 中打开

或者自己复制设置

粘贴到项目中的 .cursor/mcp.json，或 ~/.cursor/mcp.json

{
  "mcpServers": {
    "octocrawl": {
      "command": "npx",
      "args": [
        "-y",
        "@octocrawl/mcp"
      ],
      "env": {
        "W2L_API_URL": "your-value",
        "W2L_API_TOKEN": "your-value",
        "W2L_MCP_TOOLSETS": "your-value",
        "W2L_MCP_READ_ONLY": "your-value"
      }
    }
  }
}

浏览器会先询问是否打开应用，然后应用会显示设置并请你确认。密钥的值会留空，由你自己填写。

在 VS Code 中打开

或者自己复制设置

粘贴到项目中的 .vscode/mcp.json

{
  "servers": {
    "octocrawl": {
      "type": "http",
      "url": "https://mcp.octocrawl.dev/mcp"
    }
  }
}

或者在你的电脑上运行

在 VS Code 中打开

或者自己复制设置

粘贴到项目中的 .vscode/mcp.json

{
  "servers": {
    "octocrawl": {
      "type": "stdio",
      "command": "npx",
      "args": [
        "-y",
        "@octocrawl/mcp"
      ],
      "env": {
        "W2L_API_URL": "your-value",
        "W2L_API_TOKEN": "your-value",
        "W2L_MCP_TOOLSETS": "your-value",
        "W2L_MCP_READ_ONLY": "your-value"
      }
    }
  }
}

浏览器会先询问是否打开应用，然后应用会显示设置并请你确认。密钥的值会留空，由你自己填写。

ChatGPT 只能连接互联网上的服务器：

打开网页版 ChatGPT → Settings → Connectors (Apps)
点击 Create 或 Add（部分套餐需要先在 Advanced 中开启 Developer mode）
粘贴下面的地址，起个名字，然后保存
如果要求登录，请登录作者的服务

https://mcp.octocrawl.dev/mcp

菜单名称可能因套餐和版本不同而略有差异。

你的选择通过 Cookie 保存在这台设备上，无需账户。

需要你填写的值

W2L_API_URL env
The Octocrawl API the server calls: npx octocrawl serve on this computer (default http://127.0.0.1:8787), or https://api.octocrawl.dev
W2L_API_TOKEN 机密 env
A bearer token for a hosted or self-hosted API; none for a local one
W2L_MCP_TOOLSETS env
The tool groups to list, comma-separated: core, crawl, batch, products, logins, monitor, delivery, or all (default all); the same as --toolsets
W2L_MCP_READ_ONLY env
1 lists read-only tools alone, with a scrape that does not act as the person; the same as --read-only

这些设置是根据作者在注册中心的条目生成的。如果 README 的写法不同，请以 README 为准。

它能做什么

工具名称和描述与服务器在 2026年11月7日报告的完全一致。这些内容由作者撰写，不是我们写的；我们以纯文本形式显示。

scrape
Read one URL as Markdown with its Evidence Record (final URL, fetch time, HTTP status, robots.txt decision, hashes), over HTTP first and in the browser when the page needs it. Not for: many URLs or a whole site section. Compact by default; debug adds the full audit. warnings name
map
List a site's URLs without reading each page: the start page's links (read over HTTP alone) and the entries of the sitemaps the site declares, within one deadline. URLs stay in scope (the start host and its www twin, the start path's subtree, assets left out, similar URLs folded)

基础检查i

我们检查的是任何人都能看到的信息；我们没有阅读代码。不适用的检查项不计入总数。 · 检查于 2026年11月7日

✓ 通过
作者与仓库一致
注册名称已通过 GitHub 账户 77777R7 验证，该账户拥有此仓库
… 尚未检查
仍在维护
我们还没有从 GitHub 读取这个仓库
… 尚未检查
有许可证
我们还没有从 GitHub 读取这个仓库
✓ 通过
端点有响应
在 367 毫秒内响应（HTTP 200）
– 不适用
使用 OAuth 登录
无需登录
✓ 通过
工具描述中没有隐藏指令
已读取 2 个工具描述，未发现针对 AI 的隐藏指令

与“其他”中的其他服务器比较i

星标数多于此分类中其他 32727 个服务器的 0%
发布到注册中心的时间比此分类中其他 32727 个服务器的 99% 更新
基础检查结果等于或优于此分类中其他 32727 个服务器的 100%

与注册中心里这个分类中所有活跃的服务器比较。只有当发布者拥有该仓库时，星标数才计入。

常见问题

Octocrawl 安全吗？

它在 3 项适用的基础检查中通过了 3 项。我们没有阅读代码。在添加密钥或令牌之前，请先看看这个服务器要求哪些权限，并使用只具备所需权限的密钥。

它需要哪些密钥或代码？

你需要填写：W2L_API_TOKEN

它在我的电脑上运行还是在云端？

两种方式都可以：安装在你的电脑上，或者连接云端版本。

它还在维护吗？

GitHub 仓库最后更新于–，最新版本（0.8.7）于今天进入注册中心。

哪些应用可以使用它？

支持 MCP 的应用，例如 Claude Desktop、Claude Code、Cursor 和 VS Code。它通过 URL 连接，所以接受远程 MCP 服务器的应用也可以使用，例如开启开发者模式的 ChatGPT。

有什么可以替代它？

同一分类中有：Spec Driven Development, Remote Desktop Commander, Foliosio

替代选择

同一分类中其他作者的服务器，按星标数排序。

如何看卡片：★ GitHub 星标i三大支柱：评价 · 检查 · 使用情况i更新 = 最新版本发布到注册中心的时间i

Spec Driven Development

作者：formulahendry

MCP Server that facilitates spec-driven development workflows, not just Vibe Coding.

在你的电脑上运行★ 440MIT14个月前更新

安装

Remote Desktop Commander

作者：desktop-commander

Remote Desktop Commander: use files, terminal and processes on your own computers from any AI app.

在云端运行★ 211前天更新

安装

Foliosio

作者：foliosio.com

Read your support inbox, tag conversations and save draft replies from your AI agent. Never sends.

在云端运行今天更新

安装

作者的 README

打开 README

按原文从作者的仓库复制。PopNx MCP 没有审查或背书其内容。HTML、图片和脚本已被移除。链接会离开本站。

Octocrawl

One link. Web data, ready.

Markdown and data from web pages, for AI agents and RAG.

Fewer tokens than raw HTML. Every result records how it was read.

Website · Docs · Connect MCP · Blog · Changelog

Scrape pages. Crawl websites. Turn web content into usable data.

Octocrawl is an open-source web scraping and crawling toolkit for AI agents, RAG and research. Collect documentation for a knowledge base, extract product details for comparison, or turn web tables and listings into data for analysis. Each page comes with a source record.

Try a page in your browser, or connect your agent through hosted MCP or the REST API. No installation or API key is needed to try the hosted service.

What can you do with Octocrawl?

Capability	What you get
Scrape a page	Readable Markdown for LLM context, plus HTML, links, image URLs and page metadata. Request a screenshot on the browser lane.
Map a website	Discover URLs from sitemaps and page links, and filter them by a search term to find the pages you need.
Crawl a site or section	Follow links from a starting URL and collect each page's content, with path filters, depth and page limits.
Scrape a list of URLs	Submit a batch, retrieve results as they arrive, and keep the outcome of every URL, including failures.
Extract structured data	Get tables and repeated listings as JSON and CSV, or pass a JSON Schema for supported fields such as product names, prices and page labels. Extraction guide.
Read dynamic pages · local actions	Render JavaScript, click controls, scroll, load more items and follow pagination before extracting content. Browser actions.
Read PDF reports	Extract a PDF's text layer with page markers for citations. CSV, JSON and plain-text URLs also return their text. File support.
Revisit sources on a schedule · local workers	Run Monitors and send their events to an HTTPS endpoint with retried delivery. Monitor setup.

Hosted Octocrawl offers scrape and map, and with an account crawl and batch jobs (up to 500 pages a crawl, 1,000 URLs a batch); keyless requests use HTTP, and an account key enables the browser lane. Run locally for larger jobs, page actions and Monitors; see Running Octocrawl for setup and self-hosting options. The web preview is a single-page trial, with five pages a day and up to 20 named fields.

Why Octocrawl?

Web data inside your agent. Connect Claude Code, Cursor, OpenCode or Codex through MCP, or build with REST, TypeScript, Python and the Vercel AI SDK.
Structured fields without a model by default. Read supported values from the page itself, with field evidence and explicit missing values. Model fallback is opt-in.
Know when a scrape fell short. Checks flag challenge pages, wrong content, missing facts, truncation and too little content. Blocked and failed results carry a reason; fetched error pages keep their status and Markdown. Result states.
Trace results back to their sources. Each page carries an Evidence Record with its URL, fetch time, status and extraction evidence. Unobserved values stay null.
Run it where you need it. Use the hosted service, run with npx, or self-host with Docker Compose. The server is AGPL-3.0; the client libraries are MIT.

If Octocrawl helps your work, give the repository a star.

Quick start

Connect your agent to hosted Octocrawl. No key is needed to start (steps for Claude Code, Cursor, OpenCode and Codex):

claude mcp add --transport http octocrawl https://mcp.octocrawl.dev/mcp

Without a key it reads 20 pages a day over HTTP. For more, sign in and make a key: an account gets 1,000 credits a month and the browser lane. The key goes in a header, never in the URL:

claude mcp add --transport http octocrawl https://mcp.octocrawl.dev/mcp --header "Authorization: Bearer oc_…"

Or run it on your own computer. You need Node.js 22.13+ or 24+:

npx octocrawl scrape https://example.com --markdown

pip install 'octocrawl-client[pandas]' # the Python client of a running API (npx octocrawl serve); [pandas] for to_pandas() npm install @octocrawl/sdk # the TypeScript client; @octocrawl/mcp is the MCP server npm install @octocrawl/ai-sdk # scrape and map as Vercel AI SDK tools, on hosted Octocrawl by default

Pages are read over HTTP first. For pages that only appear in a browser, install Chromium once with npx playwright install chromium; until then such a page comes back with a browser_not_installed warning that names the command.

To give your agent every tool, with no daily limit, run the API locally and connect the MCP server to it:

npx octocrawl serve                                  # keep it running: the API on 127.0.0.1:8787

claude mcp add octocrawl -- npx -y @octocrawl/mcp # the published stdio server, a client of that API

The hosted service offers scrape and map, and with an account crawl and batch jobs. The local server adds larger jobs, the Amazon.sg product tool and the Monitor tools. To serve other machines yourself, npx octocrawl serve --hosted --token refuses private addresses, robots overrides and saved logins as the hosted service does. Running Octocrawl covers all three; Docker Compose runs one on your own server. An agent installing Octocrawl for you (Cline, for one) can follow llms-install.md.

What a failure looks like

The test page below answers HTTP 200 but shows a Cloudflare check instead of content (fixture block-challenge-200). A crawler that trusts the status code counts it as a success. Octocrawl tries plain HTTP, then a browser, and reports the page as blocked, with what to do next:

npx octocrawl scrape http://127.0.0.1:8799/block/challenge-200

"status": "blocked", "blockReason": "cloudflare_challenge", "channelsTried": ["http", "browser_local"], "markdown": null, "snapshot": { "httpStatus": 200, "artifacts": [] }, "usage": { "wallMs": 1079, "bytesWire": null, "externalCostUsd": 0 }, "agentHints": [ "127.0.0.1 gates automated access on the lanes tried (http, browser_local); … a proxy or session you own is the supported route, or, on your own machine, getting through the check yourself in your own Chrome: handoff: true on a scrape (octocrawl scrape --handoff) …" ], "evidenceRecord": { "schemaVersion": "w2l.evidence/1", "status": "blocked", "reason": "cloudflare_challenge", "lane": "browser_local", "…": "…" } }

This is real output of octocrawl 0.4.1, trimmed (…) and rounded, against the repository's fixture server (FIXTURE_PORT=8799 node --import tsx packages/fixtures/src/cli.ts). bytesWire: null means wire bytes were not measured; artifacts: [] means no screenshot or DOM snapshot was made.

Documentation

The site's docs at octocrawl.dev/docs cover MCP setup for four clients, task guides and limits. They are built from apps/public-web/content.

The full reference lives in docs/:

Topic	Page
The `octocrawl` command	Command line
Hosted, local and self-hosted servers, ports, tokens, proxies, robots.txt	Running Octocrawl
Options for a scrape and what a result carries	Scrape options and results
Many URLs, whole sites, site maps, webhooks	Batches, crawls and maps
Structured data from a JSON Schema	JSON extraction
PDF, CSV, XLSX, ZIP and JSON URLs	Files
Your login, enhanced access, handoff, your own Chrome	Access
Scheduled re-reads and HTTPS events	Monitors
Moving from Firecrawl	Firecrawl shim

Walkthroughs of real runs are on the blog: Claude Code web scraping with MCP, Web scraping for RAG, Find all pages on a website, Scrape a website with login, in your own Chrome, Extract structured data without an LLM, Scrape a list of URLs and keep every failure.

For researchers, two guides walk through a real run: From a URL list to a CSV with evidence and Citing web data in a paper.

For the Monitor → result → HTTPS event → restart workflow, follow the onboarding guide.

Work on the code

You need Node.js 22.13+ or 24+ and npm. The PDF engine, pdf.js, needs 22.13 or later.

git clone https://github.com/77777R7/Octocrawl.git

cd Octocrawl npm ci npx playwright install chromium npm run typecheck npm test npm run scrape -- https://example.com --markdown npm run crawl -- https://example.com --max-pages 20

The packag

只显示了开头部分；其余内容请在仓库中阅读

Octocrawl

概览

PopNx MCP 的三大支柱

它能帮你做什么

安装

安装之前

或者在你的电脑上运行

或者在你的电脑上运行

或者在你的电脑上运行

或者在你的电脑上运行

需要你填写的值

它能做什么

基础检查i

作者与仓库一致

仍在维护

有许可证

端点有响应

使用 OAuth 登录

工具描述中没有隐藏指令

与“其他”中的其他服务器比较i

常见问题

替代选择

Spec Driven Development

Remote Desktop Commander

Foliosio

作者的 README

What can you do with Octocrawl?

Why Octocrawl?

Quick start

What a failure looks like

Documentation

Work on the code