原型 · 预览版:收录官方注册中心里的全部服务器,每 30 分钟同步一次。使用 Google 登录即可撰写评价。原型 · 每 30 分钟更新
PopNx MCP为使用 AI 的人准备的 MCP 服务器 · 用大白话讲清楚

AI Crawlability Check

作者:Milan344s版本 2.0.0i★ 0i在云端运行官方网站 ↗
注册名称io.github.Milan344s/ai-crawlability-check

安装方法

What it does

Check if AI crawlers like GPTBot and ClaudeBot can reach, index and quote any web page.

  • Connects 在云端运行
  • Account or key No key needed
  • Category 网页与浏览器

可搭配使用

教 AI 做这类工作的 Skill,可与本项一起使用

timesfm

TimesFM (Time Series Foundation Model) is a pretrained time-series foundation model developed by Google Research for time-series forecasting.

作者:google-research★ 35,6381 个 skill

Skill · 操作手册

Shadowbroker

Open-source intelligence for the global theater. Track everything from the corporate/private jets of the wealthy, and spy satellites, to seismic events in one unified interface. Hook an AI agent up to have it parse through data and find previously unseen correlations. The knowledge is available to…

作者:BigBodyCobain★ 11,6051 个 skill

Skill · 操作手册

google-maps-scraper

scrape data from Google Maps. Extracts data such as the name, address, phone number, website URL, rating, reviews number, latitude and longitude, reviews,email and more for each place

作者:gosom★ 8,2361 个 skill

Skill · 操作手册

为什么排在这里

列表按使用分(满分 100)排序,由以下部分组成

热度0/80GitHub 星标或每周下载量:较大的计 60,较小的计 20
新鲜度12/1230 天内有发布或代码更新得满分
仍在响应8/8上次检查时云端端点有响应
使用分20/100

质量: 基础检查通过 5/5 项(仅展示,不用于排序) · 上升势头: 暂无数据 (只用于“上升中”标签)

安装

选择你使用的 AI 应用,然后复制设置。

用一句话让 AI 帮你安装

请帮我安装 AI Crawlability Check。按照 https://mcp.popnx.com/i/milan344s-ai-crawlability-check.md 操作,先把命令给我看,等我同意后再运行。

粘贴到能在你电脑上工作的 AI(如 Claude Code、Cursor)。它会读取我们的步骤,先给你看命令,等你同意。无需密钥,无需注册。 · 查看 AI 会读取的全文

通过 5/5 项检查 · 你的提问会经过开发者的服务器(mcp.milan-novotny.com) · ✓ 命令已检查 2026年11月19日

第一步:你用哪个 AI?

选择你用的应用,步骤会显示在下方。

可连接 Claude · 约 2 分钟

不需要密钥,不用安装程序 · 在 claude.ai 或桌面应用设置一次,即可在聊天中使用 · 免费账户可添加 1 个 · Team/Enterprise 方案由所有者(Owner)添加

  1. 打开 Claude 的 Connectors 页面

    CustomizeConnectors

    CustomizeConnectors1Add2https://…3Add4
  2. 点击“Add” → “Custom” → “Web”,填写这 2 项

    把这里的内容复制到 Claude 中同名的输入框,然后点击 Continue

    Name · 名称AI Crawlability Check
    Remote MCP server URL · 地址https://mcp.milan-novotny.com/mcp

    在 Claude 检测到的登录设置页点击 Continue · Authentication 按作者说明选择(Sign in now、Sign in when needed 或 No sign in),然后点击 Add

  3. 在聊天中开启

    +ConnectorsAI Crawlability Check

  4. 试一试

    用 AI Crawlability Check 告诉我你能帮我做什么,然后做一件给我看

    ✓ 如果 Claude 说正在使用 AI Crawlability Check,或回答中出现 AI Crawlability Check 的工具名称,就表示连接成功

菜单名称已于 2026年11月11日 对照 Anthropic 帮助页面核实 · 你的界面可能因版本略有不同

能否连接 ChatGPT 取决于方案

完整 MCP:网页版 ChatGPT Business、Enterprise 或 Edu · Pro:以读取/获取权限连接,修改数据的工具可能用不了 · Plus:官方文档尚未确认,如果在插件页看到添加自定义 MCP 的选项可以试试 · 仅限网页

  1. 在网页版 ChatGPT 打开添加 MCP 的页面

    Plugins+Add custom MCP server

    ChatGPTPlugins1+2https://…3Create4
  2. 填写名称和地址

    输入名称,在 Connection 下粘贴地址,勾选风险提示,然后点击 Create as a plugin

    Name · 名称AI Crawlability Check
    Connection · 地址https://mcp.milan-novotny.com/mcp
  3. 在聊天中选择 AI Crawlability Check

    开启新聊天,输入前先从插件列表中选择 AI Crawlability Check

  4. 试一试

    用 AI Crawlability Check 告诉我你能帮我做什么,然后做一件给我看

    ✓ 如果 ChatGPT 说正在使用 AI Crawlability Check,或回答中出现 AI Crawlability Check 的工具名称,就表示连接成功

菜单名称已于 2026年11月11日 对照 OpenAI 帮助页面核实 · 你的界面可能因版本略有不同

可连接 Claude Code · 约 2 分钟

不需要密钥,不用安装程序

  1. 打开终端

    Mac:按 Command+空格,输入 Terminal · Windows:打开 PowerShell

  2. 粘贴这条命令并按 Enter

    claude mcp add --transport http --scope user ai-crawlability-check https://mcp.milan-novotny.com/mcp
  3. 在 Claude Code 中输入 /mcp

    应能看到 AI Crawlability Check 在列表中并显示已连接

  4. 试一试

    用 AI Crawlability Check 告诉我你能帮我做什么,然后做一件给我看

    ✓ 如果 Claude Code 说正在使用 AI Crawlability Check,或回答中出现 AI Crawlability Check 的工具名称,就表示连接成功

菜单名称已于 2026年11月11日 对照 Anthropic 帮助页面核实 · 你的界面可能因版本略有不同

可连接 Cursor · 约 2 分钟

不需要密钥,不用安装程序

  1. 点击这个按钮打开 Cursor

    在 Cursor 中打开

    Cursor 会请你确认安装,点击 Install

  2. 在 Cursor 中以 Agent 模式打开 AI 聊天

  3. 试一试

    用 AI Crawlability Check 告诉我你能帮我做什么,然后做一件给我看

    ✓ 如果 Cursor 说正在使用 AI Crawlability Check,或回答中出现 AI Crawlability Check 的工具名称,就表示连接成功

菜单名称已于 2026年11月11日 对照 Cursor 帮助页面核实 · 你的界面可能因版本略有不同

可连接 VS Code · 约 2 分钟

不需要密钥,不用安装程序

  1. 点击这个按钮打开 VS Code

    在 VS Code 中打开

    VS Code 会请你确认安装,点击 Install

  2. 在 VS Code 中以 Agent 模式打开 AI 聊天

  3. 试一试

    用 AI Crawlability Check 告诉我你能帮我做什么,然后做一件给我看

    ✓ 如果 VS Code 说正在使用 AI Crawlability Check,或回答中出现 AI Crawlability Check 的工具名称,就表示连接成功

菜单名称已于 2026年11月11日 对照 VS Code 帮助页面核实 · 你的界面可能因版本略有不同

其他支持 MCP 的应用(如 Windsurf、Zed、Gemini CLI)添加方式各不相同,请在应用列表中查看

查看所有支持 MCP 的应用 →

通过网页连接的应用使用的地址

地址https://mcp.milan-novotny.com/mcp

JSON 设置在下方“开发者专用”中

卡住了?问问正在用 AI Crawlability Check 的人💬 讨论 →
开发者专用:终端命令和 JSON 设置

通过网页连接

Claude Code

claude mcp add --transport http --scope user ai-crawlability-check https://mcp.milan-novotny.com/mcp

Cursor (~/.cursor/mcp.json)

{
  "mcpServers": {
    "ai-crawlability-check": {
      "url": "https://mcp.milan-novotny.com/mcp"
    }
  }
}

VS Code (mcp.json)

{
  "servers": {
    "ai-crawlability-check": {
      "type": "http",
      "url": "https://mcp.milan-novotny.com/mcp"
    }
  }
}

这些设置是根据作者在注册中心的条目生成的。如果 README 的写法不同,请以 README 为准。

一览

每周下载量i
不在 npm/PyPI 上
连接方式
在云端运行
传输方式
streamable-http
最新版本i
2.0.0
发布到注册中心i
2026年11月19日
代码最后更新i
2026年11月19日
最近检查
2026年11月19日
作者
Milan344s
注册状态
活跃

哪些应用可以使用它

依据这个服务器的连接方式和各应用自己的文档,检查于 2026年11月10日。

通过互联网连接,适用于

ClaudeChatGPTGooseLM StudioMsty StudioCherry StudioCursorVS Code + GitHub CopilotVisual StudioWindsurf (Devin Desktop)ZedKiroClineContinueJetBrains AI AssistantClaude CodeCodex CLIGemini CLIOpenCodenodecodeWarpLibreChatOpen WebUIRaycast

全部应用及其优缺点 →

它能做什么

工具名称和描述与服务器在 2026年11月19日 报告的完全一致。这些内容由作者撰写,不是我们写的;我们以纯文本形式显示。

  • check_ai_crawler_access

    Reports which AI crawlers may fetch a public URL. Reads the site's robots.txt and returns allowed or blocked, with the deciding rule, for each known AI crawler from OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Meta, Amazon and others, split into search and answer-time

  • check_ai_indexability

    Checks whether a public page can be indexed by search and AI engines once a crawler reaches it. Measures HTTP status and redirects, noindex, canonical target, how much copy exists in the raw HTML without JavaScript, client-side rendering signals, title and H1, XML sitemap coverag

  • check_ai_citability

    Fetches a public page and scores how quotable its content is for AI engines once it is crawled and indexed. Measures answer-first structure, question headings, lists and tables, FAQ, paragraph and sentence length, statistic and source density, named entities, schema, author, fres

  • check_seo_basics

    Fetches a public page and runs 24 measurable on-page SEO checks: title and meta description length, H1, heading hierarchy, keyword placement and density, slug, canonical, internal and external links, anchor text, image alt text, schema and social tags. Pass the target keyword to

  • get_review_rubric

    Returns a scoring rubric as text for the judgment items the URL tools cannot measure: geo (would an AI engine quote this article?) or seo (pre-publish SEO review). Fetches nothing.

基础检查i

我们检查的是任何人都能看到的信息;我们没有阅读代码。不适用的检查项不计入总数。 · 检查于 2026年11月20日

  • ✓ 通过

    作者与仓库一致

    注册名称已通过 GitHub 账户 Milan344s 验证,该账户拥有此仓库

  • ✓ 通过

    仍在维护

    代码最后更新于昨天

  • ✓ 通过

    有许可证

    使用 MIT

  • ✓ 通过

    端点有响应

    在 77 毫秒内响应(HTTP 200)

  • – 不适用

    使用 OAuth 登录

    无需登录

  • ✓ 通过

    工具描述中没有隐藏指令

    已读取 5 个工具描述,未发现针对 AI 的隐藏指令

我们检查什么 → · JSON

作者:申请重新检查或报告信息错误
给作者:在 README 中加徽章

如果你是作者,可把这段代码贴到 README。徽章会链接回本页。

[![Featured on PopNx MCP](https://mcp.popnx.com/badge/milan344s-ai-crawlability-check.svg)](https://mcp.popnx.com/en/servers/milan344s-ai-crawlability-check)

讨论

提问、分享技巧或报告无法使用。按“有帮助”数量排序。

还没有帖子,欢迎第一个提问或分享。

此处内容由用户撰写,不代表 PopNx 团队。

登录后发帖

用户评价

还没有评价

登录后写评价

与“网页与浏览器”中的其他服务器比较i

  • 星标数多于此分类中其他 1645 个服务器的 0%
  • 发布到注册中心的时间比此分类中其他 1645 个服务器的 98% 更新
  • 基础检查结果等于或优于此分类中其他 1645 个服务器的 99%

与注册中心里这个分类中所有活跃的服务器比较。只有当发布者拥有该仓库时,星标数才计入。

常见问题

AI Crawlability Check 安全吗?

它在 5 项适用的基础检查中通过了 5 项。我们没有阅读代码。在添加密钥或令牌之前,请先看看这个服务器要求哪些权限,并使用只具备所需权限的密钥。

它需要哪些密钥或代码?

注册中心没有列出必需的密钥,不过有些服务器会在首次使用时要求你登录。

它在我的电脑上运行还是在云端?

它在作者的云端运行。通过互联网连接,无需安装,但你发送的内容会经过作者的服务器。

它还在维护吗?

GitHub 仓库最后更新于昨天,最新版本(2.0.0)于昨天进入注册中心。

哪些应用可以使用它?

支持 MCP 的应用,例如 Claude Desktop、Claude Code、Cursor 和 VS Code。它通过 URL 连接,因此也能用于接受远程 MCP 的应用,例如网页版 ChatGPT(Business 及以上为完整 MCP;Pro 为只读)。

有什么可以替代它?

同一分类中有:Browser Use, changedetection.io (official), Apify MCP Server, Pomerado, Actionbook

功能相近的其他选择

同一分类、其他作者、命令已通过检查。官方的排在前面,再按星标排序。

如何看卡片:↓ 每周下载量i★ GitHub 星标i三大支柱:评价 · 检查 · 使用情况i更新 = 最新版本发布到注册中心的时间i

Browser Use

Browser agents and stealth cloud browsers for AI: run web tasks, fill forms, extract data.

作者:browser-use广泛使用↓ 167.3万/周★ 12.1万30天前更新

电脑或云端✓ 服务方 · browser-use.com

Apify MCP Server

Extract data from any website with thousands of scrapers, crawlers, and automations on Apify Store ⚡

作者:apify广泛使用★ 1.7万3天前更新

在云端运行✓ 服务方 · apify.com

Pomerado

Connect your agents to any website. The cheaper, faster and more reliable alternative to browser use

作者:Pomerado正在增长★ 1801前天更新

在云端运行✓ 服务方 · pomerado.ai

Actionbook

Let your AI agent operate any website through a real, logged-in browser session.

作者:actionbook正在增长★ 163132天前更新

在云端运行✓ 服务方 · actionbook.app
与相似项目对比
本项
AI Crawlability Check
Browser Usechangedetection.io (official)Apify MCP ServerPomeradoActionbook
运行位置在云端运行,无需安装云端或安装在电脑上都可以在云端运行,无需安装在云端运行,无需安装在云端运行,无需安装在云端运行,无需安装
需要密钥否是否是否否
可在 Claude 中按 + 添加否是否否否否
基础检查5/55/53/34/44/44/4
许可证MITMITApache-2.0MITMITApache-2.0
价格登记中未列出费用登记中未列出费用登记中未列出费用登记中未列出费用登记中未列出费用登记中未列出费用

作者的 README

打开 README

按原文从作者的仓库复制。PopNx MCP 没有审查或背书其内容。HTML、图片和脚本已被移除。链接会离开本站。

AI Crawlability Check

AI Crawlability Check is a free Model Context Protocol (MCP) server that tells your AI assistant whether AI crawlers can reach a web page, whether the page can be indexed, and whether its content is quotable.

It checks robots.txt rules for 29 AI crawlers, including GPTBot, ClaudeBot, PerplexityBot and OAI-SearchBot, and runs 70 measured checks across four tools. It is hosted, needs no account and no API key, and stores nothing.

  • Endpoint: https://mcp.milan-novotny.com/mcp (Streamable HTTP, no authentication)
  • Documentation: https://www.milan-novotny.com/tools/ai-crawlability-mcp/
  • Privacy and terms: https://www.milan-novotny.com/tools/ai-crawlability-mcp/privacy/
  • MCP Registry name: io.github.Milan344s/ai-crawlability-check

Connect

Claude (web and desktop). Settings, Connectors, "Add custom connector", then paste the endpoint URL.

Claude Code

claude mcp add --transport http ai-crawlability https://mcp.milan-novotny.com/mcp

Cursor and other clients with a JSON config

{

"mcpServers": { "ai-crawlability": { "url": "https://mcp.milan-novotny.com/mcp" } } }

Then ask in plain language:

  • "Are we blocking GPTBot, ClaudeBot or PerplexityBot on https://example.com/pricing?"
  • "Check whether this page can be indexed by AI search engines, and list the fixes."
  • "Run all the crawlability checks on our five most important URLs and give me one table."

Tools

All five tools are read-only.

ToolInputWhat it returns
check_ai_crawler_accessurlAllowed or blocked for 29 AI crawlers with the deciding robots.txt rule, plus 9 scored checks: robots.txt health, catch-all rules, server refusals, noindex, snippet rules, declared sitemaps, Content-Signal lines and llms.txt.
check_ai_indexabilityurl15 scored checks: HTTP status and redirects, noindex, canonical target, copy present in the raw HTML, JavaScript dependence, XML sitemap coverage, response time, HTML size, structured data, dates and language.
check_ai_citabilityurl22 scored checks on answer-first structure, question headings, lists and tables, statistics with sources, named entities, schema, author and freshness. Tuned for articles and guides.
check_seo_basicsurl, keyword (optional)24 on-page SEO checks: title, meta description, H1, heading order, keyword use, slug, canonical, links, alt text and schema.
get_review_rubrictype: geo or seoA review rubric as text, for the judgment a script cannot make.

Every URL tool returns a 0 to 100 score, the measured value behind each check, the reason the check matters and a fix for each failure.

Example result

check_ai_crawler_access on a page whose robots.txt blocks most AI crawlers (shortened):

{

"tool": "check_ai_crawler_access", "score": 45, "verdict": "Weak", "summary": "9 of 19 search and answer-time bots and 3 of 10 training bots may fetch this URL according to robots.txt.", "top_fixes": [ { "check": "answer_bots_allowed", "status": "fail", "fix": "Allow these user agents for this path in robots.txt: ..." } ], "bots": [ { "bot": "OAI-SearchBot", "owner": "OpenAI", "role": "answer", "allowed": false, "decided_by": "Disallow: / (group OAI-SearchBot)" } ] }

Crawlers checked

Names and purposes were verified against each vendor's own crawler documentation on 19 November 2026. Bytespider has no vendor documentation and is included because it appears in most block lists. Corrections are welcome as issues or pull requests against src/robots.js.

Search and answer-time crawlers
CrawlerOwnerWhat it is forVendor position on robots.txt
OAI-SearchBotOpenAISurfaces sites in ChatGPT searchHonors
ChatGPT-UserOpenAILive fetch when a ChatGPT user or custom GPT asksMay not apply (user-triggered)
Claude-SearchBotAnthropicCrawls to improve search results in ClaudeHonors
Claude-UserAnthropicLive fetch when a Claude user asksHonors
PerplexityBotPerplexityPerplexity search index; not used for model trainingHonors
Perplexity-UserPerplexityLive fetch when a Perplexity user asksGenerally ignores (user-triggered)
GooglebotGoogleGoogle Search, including AI Overviews and AI ModeHonors
BingbotMicrosoftBing index, which also grounds CopilotHonors
ApplebotAppleSearch across Siri, Spotlight and SafariHonors
DuckAssistBotDuckDuckGoReal-time fetch for DuckDuckGo AI answers; no trainingHonors
meta-webindexerMetaMeta AI search results and citationsNot stated
meta-externalfetcherMetaUser-requested link fetches for Meta AIMay bypass (user-triggered)
Amzn-SearchBotAmazonSearch in Amazon products such as Alexa; no trainingHonors
Amzn-UserAmazonLive fetch for Amazon user requests; no trainingMay not follow all rules (user-triggered)
MistralAI-IndexMistralMistral search index; no trainingNot stated
MistralAI-UserMistralLive fetch when a Le Chat user asksHonors
YouBotYou.comYou.com search indexHonors
Google-AgentGoogleGoogle-hosted agents browsing for a userGenerally ignores (user-triggered)
Claude-AgentAnthropicHosted Claude browser acting for a userNot stated
Training crawlers and control tokens
CrawlerOwnerWhat it is forVendor position on robots.txt
GPTBotOpenAIOpenAI model trainingHonors
ClaudeBotAnthropicAnthropic model trainingHonors
Google-ExtendedGoogleControl token for Gemini training and grounding; no effect on SearchHonors
Applebot-ExtendedAppleControl token for Apple AI training on Applebot dataHonors
meta-externalagentMetaTraining Meta AI models and improving productsNot stated
AmazonbotAmazonImproves Amazon products; data may train Amazon AI modelsHonors
MistralAI-TrainingMistralMistral training datasetsHonors
CCBotCommon CrawlOpen web archive, widely reused as AI training dataHonors
AI2BotAllen Institute for AITraining open language modelsNot stated
BytespiderByteDanceByteDance crawler, reported as model trainingNo vendor documentation found

robots.txt matching follows RFC 9309: the most specific user-agent group wins, the longest matching rule wins, and Allow beats Disallow on a tie.

Limits

  • It queries no index. The scores say whether a page can be crawled, indexed and quoted. They do not say whether any AI engine cites it today.
  • Bot-specific firewall rules are invisible. The server reads robots.txt, meta tags and headers. It reports when its own request is refused, and it never sends another company's crawler user agent.
  • No JavaScript. Pages are read as raw HTML on purpose, because that is how most AI crawlers read them.
  • Sitemaps are sampled. At most 5 sitemap files are read per check.
  • Public pages only. Private, local and non-standard addresses are refused.

How it works

The server is one Cloudflare Worker with no dependencies, no storage and no language model. It implements the MCP Streamable HTTP transport statelessly: every POST to /mcp is a self-contained JSON-RPC exchange.

FilePurpose
src/index.jsMCP protocol handling, tool definitions, landing page
src/crawl.jsCrawler access and indexability tools
src/citability.js, src/seo.jsCitability and SEO tools, scoring
src/robots.jsCrawler list and RFC 9309 matching
src/html.jsDependency-free HTML extraction
src/fetchpage.jsFetch guard: public hosts only, 1.5 MB cap, 12 second timeout
src/rubrics.jsThe two review rubrics
test/run.mjsProtocol, unit and tool tests

Run it yourself

Requires Node.js 22 or later.

npm test

Run the URL tools against a live page:

node test/run.mjs https://example.com/ "optional keyword"

Start a local server on port 8787:

npm run dev

To deploy your own copy, remove the routes line from wrangler.toml (it points at this project's domain) and run npm run deploy.

Privacy

The server keeps no state. It has no database, no accounts and no cookies, and it does not log tool inputs. For a URL it requests that page and the same site's robots.txt, llms.txt, llms-full.txt and up to 5 sitemap files, with the user agent milan-novotny-mcp/1.0. Full policy: https://www.milan-novotny.com/tools/ai-crawlability-mcp/privacy/

Author and licence

Built by Milan Novotný as a personal project. Crawler and product names belong to their owners; this project is not affiliated with or endorsed by any of them.

Released under the MIT licence.