timesfm
TimesFM (Time Series Foundation Model) is a pretrained time-series foundation model developed by Google Research for time-series forecasting.
Check if AI crawlers like GPTBot and ClaudeBot can reach, index and quote any web page.
Skills that teach the AI this kind of work; they work alongside this one
TimesFM (Time Series Foundation Model) is a pretrained time-series foundation model developed by Google Research for time-series forecasting.
Open-source intelligence for the global theater. Track everything from the corporate/private jets of the wealthy, and spy satellites, to seismic events in one unified interface. Hook an AI agent up to have it parse through data and find previously unseen correlations. The knowledge is available to…
scrape data from Google Maps. Extracts data such as the name, address, phone number, website URL, rating, reviews number, latitude and longitude, reviews,email and more for each place
Lists are ordered by a usage score (out of 100) built from these parts
| Popularity | 0/80 | GitHub stars or weekly downloads: the larger counts 60, the smaller 20 |
|---|---|---|
| Freshness | 12/12 | Full marks for a release or code change within 30 days |
| Still answering | 8/8 | The cloud endpoint answered at the last check |
| Usage score | 20/100 |
Quality: Passed 5/5 basic checks (shown, not used for ordering) · Momentum: No data yet (Used only on the Trending tab)
Pick the AI app you use, then copy the settings.
Let your AI install it with one sentence
Please install AI Crawlability Check. Follow https://mcp.popnx.com/i/milan344s-ai-crawlability-check.md, show me the commands and wait for my OK before running them.Paste it into an AI that can work on your computer (such as Claude Code or Cursor). It reads our steps, shows you the commands and waits for your OK. No key, no sign-up. · See the full text your AI reads
Passed 5/5 checks · What you ask goes through the maker’s server (mcp.milan-novotny.com) · ✓ Commands checked Nov 19, 2026
First: which AI do you use?
Pick your app and the steps appear below.
Works with Claude · about 2 min
No key, nothing to install · Set it up once on claude.ai or the desktop app, then use it in chat · free accounts can add 1 · on Team/Enterprise an Owner adds it
CustomizeConnectors
Copy each value into the field with the same name in Claude, then click Continue
AI Crawlability Checkhttps://mcp.milan-novotny.com/mcpClick Continue past the sign-in settings Claude detected · under Authentication choose what the maker says (Sign in now, Sign in when needed or No sign in), then click Add
+ConnectorsAI Crawlability Check
Use AI Crawlability Check and tell me what you can do for me, then show me one thing✓ If Claude says it is using AI Crawlability Check or shows a AI Crawlability Check tool in the answer, you are connected
Menu names checked against Anthropic’s help page on Nov 11, 2026 · your screen may differ slightly by app version
Works with ChatGPT, depending on your plan
Full MCP: ChatGPT Business, Enterprise or Edu on the web · Pro: connects with read/fetch permissions, so tools that change data may not work · Plus: not confirmed in OpenAI’s docs; if you see an add-custom-MCP option on the Plugins page you can try these steps · web only
Plugins+Add custom MCP server
Enter a name, paste the address under Connection, tick the risk warning and click Create as a plugin
AI Crawlability Checkhttps://mcp.milan-novotny.com/mcpStart a new chat and pick AI Crawlability Check from your plugins before typing
Use AI Crawlability Check and tell me what you can do for me, then show me one thing✓ If ChatGPT says it is using AI Crawlability Check or shows a AI Crawlability Check tool in the answer, you are connected
Menu names checked against OpenAI’s help page on Nov 11, 2026 · your screen may differ slightly by app version
Works with Claude Code · about 2 min
No key, nothing to install
Mac: press Command+Space and type Terminal · Windows: open PowerShell
claude mcp add --transport http --scope user ai-crawlability-check https://mcp.milan-novotny.com/mcpYou should see AI Crawlability Check listed as connected
Use AI Crawlability Check and tell me what you can do for me, then show me one thing✓ If Claude Code says it is using AI Crawlability Check or shows a AI Crawlability Check tool in the answer, you are connected
Menu names checked against Anthropic’s help page on Nov 11, 2026 · your screen may differ slightly by app version
Works with Cursor · about 2 min
No key, nothing to install
Use AI Crawlability Check and tell me what you can do for me, then show me one thing✓ If Cursor says it is using AI Crawlability Check or shows a AI Crawlability Check tool in the answer, you are connected
Menu names checked against Cursor’s help page on Nov 11, 2026 · your screen may differ slightly by app version
Works with VS Code · about 2 min
No key, nothing to install
Use AI Crawlability Check and tell me what you can do for me, then show me one thing✓ If VS Code says it is using AI Crawlability Check or shows a AI Crawlability Check tool in the answer, you are connected
Menu names checked against VS Code’s help page on Nov 11, 2026 · your screen may differ slightly by app version
Other apps that use MCP, such as Windsurf, Zed or Gemini CLI, each add servers differently. See how in the app list.
Address for apps that connect over the web
https://mcp.milan-novotny.com/mcpJSON settings are under “For developers” below
Claude Code
claude mcp add --transport http --scope user ai-crawlability-check https://mcp.milan-novotny.com/mcpCursor (~/.cursor/mcp.json)
{
"mcpServers": {
"ai-crawlability-check": {
"url": "https://mcp.milan-novotny.com/mcp"
}
}
}VS Code (mcp.json)
{
"servers": {
"ai-crawlability-check": {
"type": "http",
"url": "https://mcp.milan-novotny.com/mcp"
}
}
}These settings are generated from the maker’s registry entry. If the README says otherwise, follow the README.
Based on how this server connects and on each app’s own docs, checked Nov 10, 2026.
Connect over the internet in
ClaudeChatGPTGooseLM StudioMsty StudioCherry StudioCursorVS Code + GitHub CopilotVisual StudioWindsurf (Devin Desktop)ZedKiroClineContinueJetBrains AI AssistantClaude CodeCodex CLIGemini CLIOpenCodenodecodeWarpLibreChatOpen WebUIRaycast
Tool names and descriptions exactly as the server reported them on Nov 19, 2026. The maker wrote them, not us; we show them as plain text.
check_ai_crawler_accessReports which AI crawlers may fetch a public URL. Reads the site's robots.txt and returns allowed or blocked, with the deciding rule, for each known AI crawler from OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Meta, Amazon and others, split into search and answer-time
check_ai_indexabilityChecks whether a public page can be indexed by search and AI engines once a crawler reaches it. Measures HTTP status and redirects, noindex, canonical target, how much copy exists in the raw HTML without JavaScript, client-side rendering signals, title and H1, XML sitemap coverag
check_ai_citabilityFetches a public page and scores how quotable its content is for AI engines once it is crawled and indexed. Measures answer-first structure, question headings, lists and tables, FAQ, paragraph and sentence length, statistic and source density, named entities, schema, author, fres
check_seo_basicsFetches a public page and runs 24 measurable on-page SEO checks: title and meta description length, H1, heading hierarchy, keyword placement and density, slug, canonical, internal and external links, anchor text, image alt text, schema and social tags. Pass the target keyword to
get_review_rubricReturns a scoring rubric as text for the judgment items the URL tools cannot measure: geo (would an AI engine quote this article?) or seo (pre-publish SEO review). Fetches nothing.
We check information anyone can see; we have not read the code. Checks that do not apply are left out of the count. · Checked Nov 20, 2026
The registry name was verified with the GitHub account Milan344s, which owns the repo
Code last changed yesterday
Uses MIT
Responded in 77 ms (HTTP 200)
No sign-in needed
Read 5 tool descriptions and found no hidden instructions to the AI
If you made this, paste this code into your README. The badge links back to this page.
[](https://mcp.popnx.com/en/servers/milan344s-ai-crawlability-check)Ask, share a tip or report that it does not work. Sorted by what people marked helpful.
No posts yet. Be the first to ask or share.
Posts here are written by users, not the PopNx team.
No reviews yet
Compared with every active server in this category in the registry. Stars count only when the publisher owns the repo.
It passed 5 of 5 applicable basic checks. We have not read the code. Before adding a key or token, see what access the server asks for and use a key with only the permissions you need.
The registry does not list a required key, though some servers ask you to sign in on first use.
It runs in the maker’s cloud. Connect over the internet with nothing to install, but what you send passes through the maker’s servers.
The GitHub repo last changed yesterday, and the latest version (2.0.0) reached the registry yesterday.
Apps that support MCP, such as Claude Desktop, Claude Code, Cursor and VS Code. It connects by URL, so apps that accept remote MCP servers work too, such as ChatGPT on the web (full MCP on Business and above; Pro read-only).
In the same category: Browser Use, changedetection.io (official), Apify MCP Server, Pomerado, Actionbook
Same category, other makers, commands already checked. Official ones first, then by stars.
Reading a card:↓ Downloads per weeki★ GitHub starsiThree pillars: reviews · checks · usageiUpdated = latest version published to the registryi
Lets AI do web tasks for you, such as filling forms, searching and extracting data, in the maker’s cloud browsers or on your computer.
Official changedetection.io MCP: web page change tracking, back-in-stock and price alerts.
Extract data from any website with thousands of scrapers, crawlers, and automations on Apify Store ⚡
Connect your agents to any website. The cheaper, faster and more reliable alternative to browser use
Let your AI agent operate any website through a real, logged-in browser session.
| This one AI Crawlability Check | Browser Use | changedetection.io (official) | Apify MCP Server | Pomerado | Actionbook | |
|---|---|---|---|---|---|---|
| Runs where | Runs in the cloud, nothing to install | Cloud or install on your computer | Runs in the cloud, nothing to install | Runs in the cloud, nothing to install | Runs in the cloud, nothing to install | Runs in the cloud, nothing to install |
| Needs a key | No | Yes | No | Yes | No | No |
| Add with + in Claude | No | Yes | No | No | No | No |
| Basic checks | 5/5 | 5/5 | 3/3 | 4/4 | 4/4 | 4/4 |
| License | MIT | MIT | Apache-2.0 | MIT | MIT | Apache-2.0 |
| Price | No cost listed in the registry | No cost listed in the registry | No cost listed in the registry | No cost listed in the registry | No cost listed in the registry | No cost listed in the registry |
Copied from the maker’s repo as written. PopNx MCP has not reviewed or endorsed it. HTML, images and scripts are removed. Links lead off this site.
AI Crawlability Check is a free Model Context Protocol (MCP) server that tells your AI assistant whether AI crawlers can reach a web page, whether the page can be indexed, and whether its content is quotable.
It checks robots.txt rules for 29 AI crawlers, including GPTBot, ClaudeBot, PerplexityBot and OAI-SearchBot, and runs 70 measured checks across four tools. It is hosted, needs no account and no API key, and stores nothing.
https://mcp.milan-novotny.com/mcp (Streamable HTTP, no authentication)io.github.Milan344s/ai-crawlability-checkClaude (web and desktop). Settings, Connectors, "Add custom connector", then paste the endpoint URL.
Claude Code
claude mcp add --transport http ai-crawlability https://mcp.milan-novotny.com/mcp
Cursor and other clients with a JSON config
{
"mcpServers": { "ai-crawlability": { "url": "https://mcp.milan-novotny.com/mcp" } } }
Then ask in plain language:
All five tools are read-only.
| Tool | Input | What it returns |
|---|---|---|
check_ai_crawler_access | url | Allowed or blocked for 29 AI crawlers with the deciding robots.txt rule, plus 9 scored checks: robots.txt health, catch-all rules, server refusals, noindex, snippet rules, declared sitemaps, Content-Signal lines and llms.txt. |
check_ai_indexability | url | 15 scored checks: HTTP status and redirects, noindex, canonical target, copy present in the raw HTML, JavaScript dependence, XML sitemap coverage, response time, HTML size, structured data, dates and language. |
check_ai_citability | url | 22 scored checks on answer-first structure, question headings, lists and tables, statistics with sources, named entities, schema, author and freshness. Tuned for articles and guides. |
check_seo_basics | url, keyword (optional) | 24 on-page SEO checks: title, meta description, H1, heading order, keyword use, slug, canonical, links, alt text and schema. |
get_review_rubric | type: geo or seo | A review rubric as text, for the judgment a script cannot make. |
Every URL tool returns a 0 to 100 score, the measured value behind each check, the reason the check matters and a fix for each failure.
check_ai_crawler_access on a page whose robots.txt blocks most AI crawlers (shortened):
{
"tool": "check_ai_crawler_access", "score": 45, "verdict": "Weak", "summary": "9 of 19 search and answer-time bots and 3 of 10 training bots may fetch this URL according to robots.txt.", "top_fixes": [ { "check": "answer_bots_allowed", "status": "fail", "fix": "Allow these user agents for this path in robots.txt: ..." } ], "bots": [ { "bot": "OAI-SearchBot", "owner": "OpenAI", "role": "answer", "allowed": false, "decided_by": "Disallow: / (group OAI-SearchBot)" } ] }
Names and purposes were verified against each vendor's own crawler documentation on 19 November 2026. Bytespider has no vendor documentation and is included because it appears in most block lists. Corrections are welcome as issues or pull requests against src/robots.js.
| Crawler | Owner | What it is for | Vendor position on robots.txt |
|---|---|---|---|
OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT search | Honors |
ChatGPT-User | OpenAI | Live fetch when a ChatGPT user or custom GPT asks | May not apply (user-triggered) |
Claude-SearchBot | Anthropic | Crawls to improve search results in Claude | Honors |
Claude-User | Anthropic | Live fetch when a Claude user asks | Honors |
PerplexityBot | Perplexity | Perplexity search index; not used for model training | Honors |
Perplexity-User | Perplexity | Live fetch when a Perplexity user asks | Generally ignores (user-triggered) |
Googlebot | Google Search, including AI Overviews and AI Mode | Honors | |
Bingbot | Microsoft | Bing index, which also grounds Copilot | Honors |
Applebot | Apple | Search across Siri, Spotlight and Safari | Honors |
DuckAssistBot | DuckDuckGo | Real-time fetch for DuckDuckGo AI answers; no training | Honors |
meta-webindexer | Meta | Meta AI search results and citations | Not stated |
meta-externalfetcher | Meta | User-requested link fetches for Meta AI | May bypass (user-triggered) |
Amzn-SearchBot | Amazon | Search in Amazon products such as Alexa; no training | Honors |
Amzn-User | Amazon | Live fetch for Amazon user requests; no training | May not follow all rules (user-triggered) |
MistralAI-Index | Mistral | Mistral search index; no training | Not stated |
MistralAI-User | Mistral | Live fetch when a Le Chat user asks | Honors |
YouBot | You.com | You.com search index | Honors |
Google-Agent | Google-hosted agents browsing for a user | Generally ignores (user-triggered) | |
Claude-Agent | Anthropic | Hosted Claude browser acting for a user | Not stated |
| Crawler | Owner | What it is for | Vendor position on robots.txt |
|---|---|---|---|
GPTBot | OpenAI | OpenAI model training | Honors |
ClaudeBot | Anthropic | Anthropic model training | Honors |
Google-Extended | Control token for Gemini training and grounding; no effect on Search | Honors | |
Applebot-Extended | Apple | Control token for Apple AI training on Applebot data | Honors |
meta-externalagent | Meta | Training Meta AI models and improving products | Not stated |
Amazonbot | Amazon | Improves Amazon products; data may train Amazon AI models | Honors |
MistralAI-Training | Mistral | Mistral training datasets | Honors |
CCBot | Common Crawl | Open web archive, widely reused as AI training data | Honors |
AI2Bot | Allen Institute for AI | Training open language models | Not stated |
Bytespider | ByteDance | ByteDance crawler, reported as model training | No vendor documentation found |
robots.txt matching follows RFC 9309: the most specific user-agent group wins, the longest matching rule wins, and Allow beats Disallow on a tie.
The server is one Cloudflare Worker with no dependencies, no storage and no language model. It implements the MCP Streamable HTTP transport statelessly: every POST to /mcp is a self-contained JSON-RPC exchange.
| File | Purpose |
|---|---|
src/index.js | MCP protocol handling, tool definitions, landing page |
src/crawl.js | Crawler access and indexability tools |
src/citability.js, src/seo.js | Citability and SEO tools, scoring |
src/robots.js | Crawler list and RFC 9309 matching |
src/html.js | Dependency-free HTML extraction |
src/fetchpage.js | Fetch guard: public hosts only, 1.5 MB cap, 12 second timeout |
src/rubrics.js | The two review rubrics |
test/run.mjs | Protocol, unit and tool tests |
Requires Node.js 22 or later.
npm test
Run the URL tools against a live page:
node test/run.mjs https://example.com/ "optional keyword"
Start a local server on port 8787:
npm run dev
To deploy your own copy, remove the routes line from wrangler.toml (it points at this project's domain) and run npm run deploy.
The server keeps no state. It has no database, no accounts and no cookies, and it does not log tool inputs. For a URL it requests that page and the same site's robots.txt, llms.txt, llms-full.txt and up to 5 sitemap files, with the user agent milan-novotny-mcp/1.0. Full policy: https://www.milan-novotny.com/tools/ai-crawlability-mcp/privacy/
Built by Milan Novotný as a personal project. Crawler and product names belong to their owners; this project is not affiliated with or endorsed by any of them.
Released under the MIT licence.