timesfm
TimesFM (Time Series Foundation Model) is a pretrained time-series foundation model developed by Google Research for time-series forecasting.
Check if AI crawlers like GPTBot and ClaudeBot can reach, index and quote any web page.
教 AI 做这类工作的 Skill,可与本项一起使用
TimesFM (Time Series Foundation Model) is a pretrained time-series foundation model developed by Google Research for time-series forecasting.
Open-source intelligence for the global theater. Track everything from the corporate/private jets of the wealthy, and spy satellites, to seismic events in one unified interface. Hook an AI agent up to have it parse through data and find previously unseen correlations. The knowledge is available to…
scrape data from Google Maps. Extracts data such as the name, address, phone number, website URL, rating, reviews number, latitude and longitude, reviews,email and more for each place
列表按使用分(满分 100)排序,由以下部分组成
| 热度 | 0/80 | GitHub 星标或每周下载量:较大的计 60,较小的计 20 |
|---|---|---|
| 新鲜度 | 12/12 | 30 天内有发布或代码更新得满分 |
| 仍在响应 | 8/8 | 上次检查时云端端点有响应 |
| 使用分 | 20/100 |
质量: 基础检查通过 5/5 项(仅展示,不用于排序) · 上升势头: 暂无数据 (只用于“上升中”标签)
选择你使用的 AI 应用,然后复制设置。
用一句话让 AI 帮你安装
请帮我安装 AI Crawlability Check。按照 https://mcp.popnx.com/i/milan344s-ai-crawlability-check.md 操作,先把命令给我看,等我同意后再运行。粘贴到能在你电脑上工作的 AI(如 Claude Code、Cursor)。它会读取我们的步骤,先给你看命令,等你同意。无需密钥,无需注册。 · 查看 AI 会读取的全文
通过 5/5 项检查 · 你的提问会经过开发者的服务器(mcp.milan-novotny.com) · ✓ 命令已检查 2026年11月19日
第一步:你用哪个 AI?
选择你用的应用,步骤会显示在下方。
可连接 Claude · 约 2 分钟
不需要密钥,不用安装程序 · 在 claude.ai 或桌面应用设置一次,即可在聊天中使用 · 免费账户可添加 1 个 · Team/Enterprise 方案由所有者(Owner)添加
CustomizeConnectors
把这里的内容复制到 Claude 中同名的输入框,然后点击 Continue
AI Crawlability Checkhttps://mcp.milan-novotny.com/mcp在 Claude 检测到的登录设置页点击 Continue · Authentication 按作者说明选择(Sign in now、Sign in when needed 或 No sign in),然后点击 Add
+ConnectorsAI Crawlability Check
用 AI Crawlability Check 告诉我你能帮我做什么,然后做一件给我看✓ 如果 Claude 说正在使用 AI Crawlability Check,或回答中出现 AI Crawlability Check 的工具名称,就表示连接成功
菜单名称已于 2026年11月11日 对照 Anthropic 帮助页面核实 · 你的界面可能因版本略有不同
能否连接 ChatGPT 取决于方案
完整 MCP:网页版 ChatGPT Business、Enterprise 或 Edu · Pro:以读取/获取权限连接,修改数据的工具可能用不了 · Plus:官方文档尚未确认,如果在插件页看到添加自定义 MCP 的选项可以试试 · 仅限网页
Plugins+Add custom MCP server
输入名称,在 Connection 下粘贴地址,勾选风险提示,然后点击 Create as a plugin
AI Crawlability Checkhttps://mcp.milan-novotny.com/mcp开启新聊天,输入前先从插件列表中选择 AI Crawlability Check
用 AI Crawlability Check 告诉我你能帮我做什么,然后做一件给我看✓ 如果 ChatGPT 说正在使用 AI Crawlability Check,或回答中出现 AI Crawlability Check 的工具名称,就表示连接成功
菜单名称已于 2026年11月11日 对照 OpenAI 帮助页面核实 · 你的界面可能因版本略有不同
可连接 Claude Code · 约 2 分钟
不需要密钥,不用安装程序
Mac:按 Command+空格,输入 Terminal · Windows:打开 PowerShell
claude mcp add --transport http --scope user ai-crawlability-check https://mcp.milan-novotny.com/mcp应能看到 AI Crawlability Check 在列表中并显示已连接
用 AI Crawlability Check 告诉我你能帮我做什么,然后做一件给我看✓ 如果 Claude Code 说正在使用 AI Crawlability Check,或回答中出现 AI Crawlability Check 的工具名称,就表示连接成功
菜单名称已于 2026年11月11日 对照 Anthropic 帮助页面核实 · 你的界面可能因版本略有不同
可连接 Cursor · 约 2 分钟
不需要密钥,不用安装程序
用 AI Crawlability Check 告诉我你能帮我做什么,然后做一件给我看✓ 如果 Cursor 说正在使用 AI Crawlability Check,或回答中出现 AI Crawlability Check 的工具名称,就表示连接成功
菜单名称已于 2026年11月11日 对照 Cursor 帮助页面核实 · 你的界面可能因版本略有不同
可连接 VS Code · 约 2 分钟
不需要密钥,不用安装程序
用 AI Crawlability Check 告诉我你能帮我做什么,然后做一件给我看✓ 如果 VS Code 说正在使用 AI Crawlability Check,或回答中出现 AI Crawlability Check 的工具名称,就表示连接成功
菜单名称已于 2026年11月11日 对照 VS Code 帮助页面核实 · 你的界面可能因版本略有不同
其他支持 MCP 的应用(如 Windsurf、Zed、Gemini CLI)添加方式各不相同,请在应用列表中查看
通过网页连接的应用使用的地址
https://mcp.milan-novotny.com/mcpJSON 设置在下方“开发者专用”中
Claude Code
claude mcp add --transport http --scope user ai-crawlability-check https://mcp.milan-novotny.com/mcpCursor (~/.cursor/mcp.json)
{
"mcpServers": {
"ai-crawlability-check": {
"url": "https://mcp.milan-novotny.com/mcp"
}
}
}VS Code (mcp.json)
{
"servers": {
"ai-crawlability-check": {
"type": "http",
"url": "https://mcp.milan-novotny.com/mcp"
}
}
}这些设置是根据作者在注册中心的条目生成的。如果 README 的写法不同,请以 README 为准。
依据这个服务器的连接方式和各应用自己的文档,检查于 2026年11月10日。
通过互联网连接,适用于
ClaudeChatGPTGooseLM StudioMsty StudioCherry StudioCursorVS Code + GitHub CopilotVisual StudioWindsurf (Devin Desktop)ZedKiroClineContinueJetBrains AI AssistantClaude CodeCodex CLIGemini CLIOpenCodenodecodeWarpLibreChatOpen WebUIRaycast
工具名称和描述与服务器在 2026年11月19日 报告的完全一致。这些内容由作者撰写,不是我们写的;我们以纯文本形式显示。
check_ai_crawler_accessReports which AI crawlers may fetch a public URL. Reads the site's robots.txt and returns allowed or blocked, with the deciding rule, for each known AI crawler from OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Meta, Amazon and others, split into search and answer-time
check_ai_indexabilityChecks whether a public page can be indexed by search and AI engines once a crawler reaches it. Measures HTTP status and redirects, noindex, canonical target, how much copy exists in the raw HTML without JavaScript, client-side rendering signals, title and H1, XML sitemap coverag
check_ai_citabilityFetches a public page and scores how quotable its content is for AI engines once it is crawled and indexed. Measures answer-first structure, question headings, lists and tables, FAQ, paragraph and sentence length, statistic and source density, named entities, schema, author, fres
check_seo_basicsFetches a public page and runs 24 measurable on-page SEO checks: title and meta description length, H1, heading hierarchy, keyword placement and density, slug, canonical, internal and external links, anchor text, image alt text, schema and social tags. Pass the target keyword to
get_review_rubricReturns a scoring rubric as text for the judgment items the URL tools cannot measure: geo (would an AI engine quote this article?) or seo (pre-publish SEO review). Fetches nothing.
我们检查的是任何人都能看到的信息;我们没有阅读代码。不适用的检查项不计入总数。 · 检查于 2026年11月20日
注册名称已通过 GitHub 账户 Milan344s 验证,该账户拥有此仓库
代码最后更新于昨天
使用 MIT
在 77 毫秒内响应(HTTP 200)
无需登录
已读取 5 个工具描述,未发现针对 AI 的隐藏指令
如果你是作者,可把这段代码贴到 README。徽章会链接回本页。
[](https://mcp.popnx.com/en/servers/milan344s-ai-crawlability-check)提问、分享技巧或报告无法使用。按“有帮助”数量排序。
还没有评价
与注册中心里这个分类中所有活跃的服务器比较。只有当发布者拥有该仓库时,星标数才计入。
它在 5 项适用的基础检查中通过了 5 项。我们没有阅读代码。在添加密钥或令牌之前,请先看看这个服务器要求哪些权限,并使用只具备所需权限的密钥。
注册中心没有列出必需的密钥,不过有些服务器会在首次使用时要求你登录。
它在作者的云端运行。通过互联网连接,无需安装,但你发送的内容会经过作者的服务器。
GitHub 仓库最后更新于昨天,最新版本(2.0.0)于昨天进入注册中心。
支持 MCP 的应用,例如 Claude Desktop、Claude Code、Cursor 和 VS Code。它通过 URL 连接,因此也能用于接受远程 MCP 的应用,例如网页版 ChatGPT(Business 及以上为完整 MCP;Pro 为只读)。
同一分类中有:Browser Use, changedetection.io (official), Apify MCP Server, Pomerado, Actionbook
同一分类、其他作者、命令已通过检查。官方的排在前面,再按星标排序。
如何看卡片:↓ 每周下载量i★ GitHub 星标i三大支柱:评价 · 检查 · 使用情况i更新 = 最新版本发布到注册中心的时间i
Browser agents and stealth cloud browsers for AI: run web tasks, fill forms, extract data.
Official changedetection.io MCP: web page change tracking, back-in-stock and price alerts.
Extract data from any website with thousands of scrapers, crawlers, and automations on Apify Store ⚡
Connect your agents to any website. The cheaper, faster and more reliable alternative to browser use
Let your AI agent operate any website through a real, logged-in browser session.
| 本项 AI Crawlability Check | Browser Use | changedetection.io (official) | Apify MCP Server | Pomerado | Actionbook | |
|---|---|---|---|---|---|---|
| 运行位置 | 在云端运行,无需安装 | 云端或安装在电脑上都可以 | 在云端运行,无需安装 | 在云端运行,无需安装 | 在云端运行,无需安装 | 在云端运行,无需安装 |
| 需要密钥 | 否 | 是 | 否 | 是 | 否 | 否 |
| 可在 Claude 中按 + 添加 | 否 | 是 | 否 | 否 | 否 | 否 |
| 基础检查 | 5/5 | 5/5 | 3/3 | 4/4 | 4/4 | 4/4 |
| 许可证 | MIT | MIT | Apache-2.0 | MIT | MIT | Apache-2.0 |
| 价格 | 登记中未列出费用 | 登记中未列出费用 | 登记中未列出费用 | 登记中未列出费用 | 登记中未列出费用 | 登记中未列出费用 |
按原文从作者的仓库复制。PopNx MCP 没有审查或背书其内容。HTML、图片和脚本已被移除。链接会离开本站。
AI Crawlability Check is a free Model Context Protocol (MCP) server that tells your AI assistant whether AI crawlers can reach a web page, whether the page can be indexed, and whether its content is quotable.
It checks robots.txt rules for 29 AI crawlers, including GPTBot, ClaudeBot, PerplexityBot and OAI-SearchBot, and runs 70 measured checks across four tools. It is hosted, needs no account and no API key, and stores nothing.
https://mcp.milan-novotny.com/mcp (Streamable HTTP, no authentication)io.github.Milan344s/ai-crawlability-checkClaude (web and desktop). Settings, Connectors, "Add custom connector", then paste the endpoint URL.
Claude Code
claude mcp add --transport http ai-crawlability https://mcp.milan-novotny.com/mcp
Cursor and other clients with a JSON config
{
"mcpServers": { "ai-crawlability": { "url": "https://mcp.milan-novotny.com/mcp" } } }
Then ask in plain language:
All five tools are read-only.
| Tool | Input | What it returns |
|---|---|---|
check_ai_crawler_access | url | Allowed or blocked for 29 AI crawlers with the deciding robots.txt rule, plus 9 scored checks: robots.txt health, catch-all rules, server refusals, noindex, snippet rules, declared sitemaps, Content-Signal lines and llms.txt. |
check_ai_indexability | url | 15 scored checks: HTTP status and redirects, noindex, canonical target, copy present in the raw HTML, JavaScript dependence, XML sitemap coverage, response time, HTML size, structured data, dates and language. |
check_ai_citability | url | 22 scored checks on answer-first structure, question headings, lists and tables, statistics with sources, named entities, schema, author and freshness. Tuned for articles and guides. |
check_seo_basics | url, keyword (optional) | 24 on-page SEO checks: title, meta description, H1, heading order, keyword use, slug, canonical, links, alt text and schema. |
get_review_rubric | type: geo or seo | A review rubric as text, for the judgment a script cannot make. |
Every URL tool returns a 0 to 100 score, the measured value behind each check, the reason the check matters and a fix for each failure.
check_ai_crawler_access on a page whose robots.txt blocks most AI crawlers (shortened):
{
"tool": "check_ai_crawler_access", "score": 45, "verdict": "Weak", "summary": "9 of 19 search and answer-time bots and 3 of 10 training bots may fetch this URL according to robots.txt.", "top_fixes": [ { "check": "answer_bots_allowed", "status": "fail", "fix": "Allow these user agents for this path in robots.txt: ..." } ], "bots": [ { "bot": "OAI-SearchBot", "owner": "OpenAI", "role": "answer", "allowed": false, "decided_by": "Disallow: / (group OAI-SearchBot)" } ] }
Names and purposes were verified against each vendor's own crawler documentation on 19 November 2026. Bytespider has no vendor documentation and is included because it appears in most block lists. Corrections are welcome as issues or pull requests against src/robots.js.
| Crawler | Owner | What it is for | Vendor position on robots.txt |
|---|---|---|---|
OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT search | Honors |
ChatGPT-User | OpenAI | Live fetch when a ChatGPT user or custom GPT asks | May not apply (user-triggered) |
Claude-SearchBot | Anthropic | Crawls to improve search results in Claude | Honors |
Claude-User | Anthropic | Live fetch when a Claude user asks | Honors |
PerplexityBot | Perplexity | Perplexity search index; not used for model training | Honors |
Perplexity-User | Perplexity | Live fetch when a Perplexity user asks | Generally ignores (user-triggered) |
Googlebot | Google Search, including AI Overviews and AI Mode | Honors | |
Bingbot | Microsoft | Bing index, which also grounds Copilot | Honors |
Applebot | Apple | Search across Siri, Spotlight and Safari | Honors |
DuckAssistBot | DuckDuckGo | Real-time fetch for DuckDuckGo AI answers; no training | Honors |
meta-webindexer | Meta | Meta AI search results and citations | Not stated |
meta-externalfetcher | Meta | User-requested link fetches for Meta AI | May bypass (user-triggered) |
Amzn-SearchBot | Amazon | Search in Amazon products such as Alexa; no training | Honors |
Amzn-User | Amazon | Live fetch for Amazon user requests; no training | May not follow all rules (user-triggered) |
MistralAI-Index | Mistral | Mistral search index; no training | Not stated |
MistralAI-User | Mistral | Live fetch when a Le Chat user asks | Honors |
YouBot | You.com | You.com search index | Honors |
Google-Agent | Google-hosted agents browsing for a user | Generally ignores (user-triggered) | |
Claude-Agent | Anthropic | Hosted Claude browser acting for a user | Not stated |
| Crawler | Owner | What it is for | Vendor position on robots.txt |
|---|---|---|---|
GPTBot | OpenAI | OpenAI model training | Honors |
ClaudeBot | Anthropic | Anthropic model training | Honors |
Google-Extended | Control token for Gemini training and grounding; no effect on Search | Honors | |
Applebot-Extended | Apple | Control token for Apple AI training on Applebot data | Honors |
meta-externalagent | Meta | Training Meta AI models and improving products | Not stated |
Amazonbot | Amazon | Improves Amazon products; data may train Amazon AI models | Honors |
MistralAI-Training | Mistral | Mistral training datasets | Honors |
CCBot | Common Crawl | Open web archive, widely reused as AI training data | Honors |
AI2Bot | Allen Institute for AI | Training open language models | Not stated |
Bytespider | ByteDance | ByteDance crawler, reported as model training | No vendor documentation found |
robots.txt matching follows RFC 9309: the most specific user-agent group wins, the longest matching rule wins, and Allow beats Disallow on a tie.
The server is one Cloudflare Worker with no dependencies, no storage and no language model. It implements the MCP Streamable HTTP transport statelessly: every POST to /mcp is a self-contained JSON-RPC exchange.
| File | Purpose |
|---|---|
src/index.js | MCP protocol handling, tool definitions, landing page |
src/crawl.js | Crawler access and indexability tools |
src/citability.js, src/seo.js | Citability and SEO tools, scoring |
src/robots.js | Crawler list and RFC 9309 matching |
src/html.js | Dependency-free HTML extraction |
src/fetchpage.js | Fetch guard: public hosts only, 1.5 MB cap, 12 second timeout |
src/rubrics.js | The two review rubrics |
test/run.mjs | Protocol, unit and tool tests |
Requires Node.js 22 or later.
npm test
Run the URL tools against a live page:
node test/run.mjs https://example.com/ "optional keyword"
Start a local server on port 8787:
npm run dev
To deploy your own copy, remove the routes line from wrangler.toml (it points at this project's domain) and run npm run deploy.
The server keeps no state. It has no database, no accounts and no cookies, and it does not log tool inputs. For a URL it requests that page and the same site's robots.txt, llms.txt, llms-full.txt and up to 5 sitemap files, with the user agent milan-novotny-mcp/1.0. Full policy: https://www.milan-novotny.com/tools/ai-crawlability-mcp/privacy/
Built by Milan Novotný as a personal project. Crawler and product names belong to their owners; this project is not affiliated with or endorsed by any of them.
Released under the MIT licence.