Prototype · Preview: every server in the official registry, synced every 30 minutes. Sign in with Google to write reviews.Prototype · updated every 30 min
PopNx MCPMCP servers for people who use AI · explained in plain words

AI Crawlability Check

by Milan344sversion 2.0.0i★ 0iRuns in the cloudOfficial site ↗
Registry nameio.github.Milan344s/ai-crawlability-check

How to install

What it does

Check if AI crawlers like GPTBot and ClaudeBot can reach, index and quote any web page.

  • Connects Runs in the cloud
  • Account or key No key needed
  • Category Web & browsers

Use together with

Skills that teach the AI this kind of work; they work alongside this one

timesfm

TimesFM (Time Series Foundation Model) is a pretrained time-series foundation model developed by Google Research for time-series forecasting.

by google-research★ 35,6381 skills

Skill · How-to guide

Shadowbroker

Open-source intelligence for the global theater. Track everything from the corporate/private jets of the wealthy, and spy satellites, to seismic events in one unified interface. Hook an AI agent up to have it parse through data and find previously unseen correlations. The knowledge is available to…

by BigBodyCobain★ 11,6051 skills

Skill · How-to guide

google-maps-scraper

scrape data from Google Maps. Extracts data such as the name, address, phone number, website URL, rating, reviews number, latitude and longitude, reviews,email and more for each place

by gosom★ 8,2361 skills

Skill · How-to guide

Why it ranks here

Lists are ordered by a usage score (out of 100) built from these parts

Popularity0/80GitHub stars or weekly downloads: the larger counts 60, the smaller 20
Freshness12/12Full marks for a release or code change within 30 days
Still answering8/8The cloud endpoint answered at the last check
Usage score20/100

Quality: Passed 5/5 basic checks (shown, not used for ordering) · Momentum: No data yet (Used only on the Trending tab)

Install

Pick the AI app you use, then copy the settings.

Let your AI install it with one sentence

Please install AI Crawlability Check. Follow https://mcp.popnx.com/i/milan344s-ai-crawlability-check.md, show me the commands and wait for my OK before running them.

Paste it into an AI that can work on your computer (such as Claude Code or Cursor). It reads our steps, shows you the commands and waits for your OK. No key, no sign-up. · See the full text your AI reads

Passed 5/5 checks · What you ask goes through the maker’s server (mcp.milan-novotny.com) · ✓ Commands checked Nov 19, 2026

First: which AI do you use?

Pick your app and the steps appear below.

Works with Claude · about 2 min

No key, nothing to install · Set it up once on claude.ai or the desktop app, then use it in chat · free accounts can add 1 · on Team/Enterprise an Owner adds it

  1. Open Claude’s Connectors page

    CustomizeConnectors

    CustomizeConnectors1Add2https://…3Add4
  2. Click “Add” → “Custom” → “Web” and fill in these 2 fields

    Copy each value into the field with the same name in Claude, then click Continue

    Name · NameAI Crawlability Check
    Remote MCP server URL · Addresshttps://mcp.milan-novotny.com/mcp

    Click Continue past the sign-in settings Claude detected · under Authentication choose what the maker says (Sign in now, Sign in when needed or No sign in), then click Add

  3. Turn it on in a chat

    +ConnectorsAI Crawlability Check

  4. Try it

    Use AI Crawlability Check and tell me what you can do for me, then show me one thing

    ✓ If Claude says it is using AI Crawlability Check or shows a AI Crawlability Check tool in the answer, you are connected

Menu names checked against Anthropic’s help page on Nov 11, 2026 · your screen may differ slightly by app version

Works with ChatGPT, depending on your plan

Full MCP: ChatGPT Business, Enterprise or Edu on the web · Pro: connects with read/fetch permissions, so tools that change data may not work · Plus: not confirmed in OpenAI’s docs; if you see an add-custom-MCP option on the Plugins page you can try these steps · web only

  1. Open ChatGPT’s add-MCP page on the web

    Plugins+Add custom MCP server

    ChatGPTPlugins1+2https://…3Create4
  2. Fill in the name and address

    Enter a name, paste the address under Connection, tick the risk warning and click Create as a plugin

    Name · NameAI Crawlability Check
    Connection · Addresshttps://mcp.milan-novotny.com/mcp
  3. Pick AI Crawlability Check in a chat

    Start a new chat and pick AI Crawlability Check from your plugins before typing

  4. Try it

    Use AI Crawlability Check and tell me what you can do for me, then show me one thing

    ✓ If ChatGPT says it is using AI Crawlability Check or shows a AI Crawlability Check tool in the answer, you are connected

Menu names checked against OpenAI’s help page on Nov 11, 2026 · your screen may differ slightly by app version

Works with Claude Code · about 2 min

No key, nothing to install

  1. Open Terminal

    Mac: press Command+Space and type Terminal · Windows: open PowerShell

  2. Paste this command and press Enter

    claude mcp add --transport http --scope user ai-crawlability-check https://mcp.milan-novotny.com/mcp
  3. Type /mcp in Claude Code

    You should see AI Crawlability Check listed as connected

  4. Try it

    Use AI Crawlability Check and tell me what you can do for me, then show me one thing

    ✓ If Claude Code says it is using AI Crawlability Check or shows a AI Crawlability Check tool in the answer, you are connected

Menu names checked against Anthropic’s help page on Nov 11, 2026 · your screen may differ slightly by app version

Works with Cursor · about 2 min

No key, nothing to install

  1. Click this button to open Cursor

    Open in Cursor

    Cursor asks you to confirm; click Install

  2. Open the AI chat in Cursor in Agent mode

  3. Try it

    Use AI Crawlability Check and tell me what you can do for me, then show me one thing

    ✓ If Cursor says it is using AI Crawlability Check or shows a AI Crawlability Check tool in the answer, you are connected

Menu names checked against Cursor’s help page on Nov 11, 2026 · your screen may differ slightly by app version

Works with VS Code · about 2 min

No key, nothing to install

  1. Click this button to open VS Code

    Open in VS Code

    VS Code asks you to confirm; click Install

  2. Open the AI chat in VS Code in Agent mode

  3. Try it

    Use AI Crawlability Check and tell me what you can do for me, then show me one thing

    ✓ If VS Code says it is using AI Crawlability Check or shows a AI Crawlability Check tool in the answer, you are connected

Menu names checked against VS Code’s help page on Nov 11, 2026 · your screen may differ slightly by app version

Other apps that use MCP, such as Windsurf, Zed or Gemini CLI, each add servers differently. See how in the app list.

See every app that uses MCP →

Address for apps that connect over the web

Addresshttps://mcp.milan-novotny.com/mcp

JSON settings are under “For developers” below

Stuck? Ask people who use AI Crawlability Check💬 Discussion →
For developers: Terminal commands and JSON settings

Over the web

Claude Code

claude mcp add --transport http --scope user ai-crawlability-check https://mcp.milan-novotny.com/mcp

Cursor (~/.cursor/mcp.json)

{
  "mcpServers": {
    "ai-crawlability-check": {
      "url": "https://mcp.milan-novotny.com/mcp"
    }
  }
}

VS Code (mcp.json)

{
  "servers": {
    "ai-crawlability-check": {
      "type": "http",
      "url": "https://mcp.milan-novotny.com/mcp"
    }
  }
}

These settings are generated from the maker’s registry entry. If the README says otherwise, follow the README.

At a glance

Downloads per weeki
Not on npm/PyPI
Connection
Runs in the cloud
Transport
streamable-http
Latest versioni
2.0.0
Published to registryi
Nov 19, 2026
Code last changedi
Nov 19, 2026
Last checked
Nov 19, 2026
Maker
Milan344s
Registry status
Active

Which apps can use it

Based on how this server connects and on each app’s own docs, checked Nov 10, 2026.

Connect over the internet in

ClaudeChatGPTGooseLM StudioMsty StudioCherry StudioCursorVS Code + GitHub CopilotVisual StudioWindsurf (Devin Desktop)ZedKiroClineContinueJetBrains AI AssistantClaude CodeCodex CLIGemini CLIOpenCodenodecodeWarpLibreChatOpen WebUIRaycast

All apps, with pros and cons →

Related guides

What it can do

Tool names and descriptions exactly as the server reported them on Nov 19, 2026. The maker wrote them, not us; we show them as plain text.

  • check_ai_crawler_access

    Reports which AI crawlers may fetch a public URL. Reads the site's robots.txt and returns allowed or blocked, with the deciding rule, for each known AI crawler from OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Meta, Amazon and others, split into search and answer-time

  • check_ai_indexability

    Checks whether a public page can be indexed by search and AI engines once a crawler reaches it. Measures HTTP status and redirects, noindex, canonical target, how much copy exists in the raw HTML without JavaScript, client-side rendering signals, title and H1, XML sitemap coverag

  • check_ai_citability

    Fetches a public page and scores how quotable its content is for AI engines once it is crawled and indexed. Measures answer-first structure, question headings, lists and tables, FAQ, paragraph and sentence length, statistic and source density, named entities, schema, author, fres

  • check_seo_basics

    Fetches a public page and runs 24 measurable on-page SEO checks: title and meta description length, H1, heading hierarchy, keyword placement and density, slug, canonical, internal and external links, anchor text, image alt text, schema and social tags. Pass the target keyword to

  • get_review_rubric

    Returns a scoring rubric as text for the judgment items the URL tools cannot measure: geo (would an AI engine quote this article?) or seo (pre-publish SEO review). Fetches nothing.

Basic checksi

We check information anyone can see; we have not read the code. Checks that do not apply are left out of the count. · Checked Nov 20, 2026

  • ✓ Pass

    Maker matches the repo

    The registry name was verified with the GitHub account Milan344s, which owns the repo

  • ✓ Pass

    Still maintained

    Code last changed yesterday

  • ✓ Pass

    Has a licence

    Uses MIT

  • ✓ Pass

    Endpoint responds

    Responded in 77 ms (HTTP 200)

  • – Not applicable

    Sign-in with OAuth

    No sign-in needed

  • ✓ Pass

    No hidden instructions in tool descriptions

    Read 5 tool descriptions and found no hidden instructions to the AI

What we check → · JSON

Makers: ask for a re-check or report wrong info
For makers: add a badge to your README

If you made this, paste this code into your README. The badge links back to this page.

[![Featured on PopNx MCP](https://mcp.popnx.com/badge/milan344s-ai-crawlability-check.svg)](https://mcp.popnx.com/en/servers/milan344s-ai-crawlability-check)

Discussion

Ask, share a tip or report that it does not work. Sorted by what people marked helpful.

No posts yet. Be the first to ask or share.

Posts here are written by users, not the PopNx team.

Sign in to post

User reviews

No reviews yet

Sign in to write a review

Compared with others in Web & browsersi

  • More stars than 0% of the other 1645 servers in this category
  • Published to the registry more recently than 98% of the other 1645 servers in this category
  • Basic checks equal to or better than 99% of the other 1645 servers in this category

Compared with every active server in this category in the registry. Stars count only when the publisher owns the repo.

Common questions

Is AI Crawlability Check safe?

It passed 5 of 5 applicable basic checks. We have not read the code. Before adding a key or token, see what access the server asks for and use a key with only the permissions you need.

What keys or codes does it need?

The registry does not list a required key, though some servers ask you to sign in on first use.

Does it run on my computer or in the cloud?

It runs in the maker’s cloud. Connect over the internet with nothing to install, but what you send passes through the maker’s servers.

Is it still maintained?

The GitHub repo last changed yesterday, and the latest version (2.0.0) reached the registry yesterday.

Which apps can use it?

Apps that support MCP, such as Claude Desktop, Claude Code, Cursor and VS Code. It connects by URL, so apps that accept remote MCP servers work too, such as ChatGPT on the web (full MCP on Business and above; Pro read-only).

What can I use instead?

In the same category: Browser Use, changedetection.io (official), Apify MCP Server, Pomerado, Actionbook

Similar alternatives

Same category, other makers, commands already checked. Official ones first, then by stars.

Reading a card:↓ Downloads per weeki★ GitHub starsiThree pillars: reviews · checks · usageiUpdated = latest version published to the registryi

Browser Use

Lets AI do web tasks for you, such as filling forms, searching and extracting data, in the maker’s cloud browsers or on your computer.

by browser-useWidely used↓ 1.7M/wk★ 121.5KUpdated 30 days ago

Computer or cloud✓ service owner · browser-use.com

Apify MCP Server

Extract data from any website with thousands of scrapers, crawlers, and automations on Apify Store ⚡

by apifyWidely used★ 17KUpdated 3 days ago

Runs in the cloud✓ service owner · apify.com

Pomerado

Connect your agents to any website. The cheaper, faster and more reliable alternative to browser use

by PomeradoGrowing★ 1.8KUpdated 2 days ago

Runs in the cloud✓ service owner · pomerado.ai

Actionbook

Let your AI agent operate any website through a real, logged-in browser session.

by actionbookGrowing★ 1.6KUpdated 32 days ago

Runs in the cloud✓ service owner · actionbook.app
Compared with similar ones
This one
AI Crawlability Check
Browser Usechangedetection.io (official)Apify MCP ServerPomeradoActionbook
Runs whereRuns in the cloud, nothing to installCloud or install on your computerRuns in the cloud, nothing to installRuns in the cloud, nothing to installRuns in the cloud, nothing to installRuns in the cloud, nothing to install
Needs a keyNoYesNoYesNoNo
Add with + in ClaudeNoYesNoNoNoNo
Basic checks5/55/53/34/44/44/4
LicenseMITMITApache-2.0MITMITApache-2.0
PriceNo cost listed in the registryNo cost listed in the registryNo cost listed in the registryNo cost listed in the registryNo cost listed in the registryNo cost listed in the registry

README from the maker

Open the README

Copied from the maker’s repo as written. PopNx MCP has not reviewed or endorsed it. HTML, images and scripts are removed. Links lead off this site.

AI Crawlability Check

AI Crawlability Check is a free Model Context Protocol (MCP) server that tells your AI assistant whether AI crawlers can reach a web page, whether the page can be indexed, and whether its content is quotable.

It checks robots.txt rules for 29 AI crawlers, including GPTBot, ClaudeBot, PerplexityBot and OAI-SearchBot, and runs 70 measured checks across four tools. It is hosted, needs no account and no API key, and stores nothing.

  • Endpoint: https://mcp.milan-novotny.com/mcp (Streamable HTTP, no authentication)
  • Documentation: https://www.milan-novotny.com/tools/ai-crawlability-mcp/
  • Privacy and terms: https://www.milan-novotny.com/tools/ai-crawlability-mcp/privacy/
  • MCP Registry name: io.github.Milan344s/ai-crawlability-check

Connect

Claude (web and desktop). Settings, Connectors, "Add custom connector", then paste the endpoint URL.

Claude Code

claude mcp add --transport http ai-crawlability https://mcp.milan-novotny.com/mcp

Cursor and other clients with a JSON config

{

"mcpServers": { "ai-crawlability": { "url": "https://mcp.milan-novotny.com/mcp" } } }

Then ask in plain language:

  • "Are we blocking GPTBot, ClaudeBot or PerplexityBot on https://example.com/pricing?"
  • "Check whether this page can be indexed by AI search engines, and list the fixes."
  • "Run all the crawlability checks on our five most important URLs and give me one table."

Tools

All five tools are read-only.

ToolInputWhat it returns
check_ai_crawler_accessurlAllowed or blocked for 29 AI crawlers with the deciding robots.txt rule, plus 9 scored checks: robots.txt health, catch-all rules, server refusals, noindex, snippet rules, declared sitemaps, Content-Signal lines and llms.txt.
check_ai_indexabilityurl15 scored checks: HTTP status and redirects, noindex, canonical target, copy present in the raw HTML, JavaScript dependence, XML sitemap coverage, response time, HTML size, structured data, dates and language.
check_ai_citabilityurl22 scored checks on answer-first structure, question headings, lists and tables, statistics with sources, named entities, schema, author and freshness. Tuned for articles and guides.
check_seo_basicsurl, keyword (optional)24 on-page SEO checks: title, meta description, H1, heading order, keyword use, slug, canonical, links, alt text and schema.
get_review_rubrictype: geo or seoA review rubric as text, for the judgment a script cannot make.

Every URL tool returns a 0 to 100 score, the measured value behind each check, the reason the check matters and a fix for each failure.

Example result

check_ai_crawler_access on a page whose robots.txt blocks most AI crawlers (shortened):

{

"tool": "check_ai_crawler_access", "score": 45, "verdict": "Weak", "summary": "9 of 19 search and answer-time bots and 3 of 10 training bots may fetch this URL according to robots.txt.", "top_fixes": [ { "check": "answer_bots_allowed", "status": "fail", "fix": "Allow these user agents for this path in robots.txt: ..." } ], "bots": [ { "bot": "OAI-SearchBot", "owner": "OpenAI", "role": "answer", "allowed": false, "decided_by": "Disallow: / (group OAI-SearchBot)" } ] }

Crawlers checked

Names and purposes were verified against each vendor's own crawler documentation on 19 November 2026. Bytespider has no vendor documentation and is included because it appears in most block lists. Corrections are welcome as issues or pull requests against src/robots.js.

Search and answer-time crawlers
CrawlerOwnerWhat it is forVendor position on robots.txt
OAI-SearchBotOpenAISurfaces sites in ChatGPT searchHonors
ChatGPT-UserOpenAILive fetch when a ChatGPT user or custom GPT asksMay not apply (user-triggered)
Claude-SearchBotAnthropicCrawls to improve search results in ClaudeHonors
Claude-UserAnthropicLive fetch when a Claude user asksHonors
PerplexityBotPerplexityPerplexity search index; not used for model trainingHonors
Perplexity-UserPerplexityLive fetch when a Perplexity user asksGenerally ignores (user-triggered)
GooglebotGoogleGoogle Search, including AI Overviews and AI ModeHonors
BingbotMicrosoftBing index, which also grounds CopilotHonors
ApplebotAppleSearch across Siri, Spotlight and SafariHonors
DuckAssistBotDuckDuckGoReal-time fetch for DuckDuckGo AI answers; no trainingHonors
meta-webindexerMetaMeta AI search results and citationsNot stated
meta-externalfetcherMetaUser-requested link fetches for Meta AIMay bypass (user-triggered)
Amzn-SearchBotAmazonSearch in Amazon products such as Alexa; no trainingHonors
Amzn-UserAmazonLive fetch for Amazon user requests; no trainingMay not follow all rules (user-triggered)
MistralAI-IndexMistralMistral search index; no trainingNot stated
MistralAI-UserMistralLive fetch when a Le Chat user asksHonors
YouBotYou.comYou.com search indexHonors
Google-AgentGoogleGoogle-hosted agents browsing for a userGenerally ignores (user-triggered)
Claude-AgentAnthropicHosted Claude browser acting for a userNot stated
Training crawlers and control tokens
CrawlerOwnerWhat it is forVendor position on robots.txt
GPTBotOpenAIOpenAI model trainingHonors
ClaudeBotAnthropicAnthropic model trainingHonors
Google-ExtendedGoogleControl token for Gemini training and grounding; no effect on SearchHonors
Applebot-ExtendedAppleControl token for Apple AI training on Applebot dataHonors
meta-externalagentMetaTraining Meta AI models and improving productsNot stated
AmazonbotAmazonImproves Amazon products; data may train Amazon AI modelsHonors
MistralAI-TrainingMistralMistral training datasetsHonors
CCBotCommon CrawlOpen web archive, widely reused as AI training dataHonors
AI2BotAllen Institute for AITraining open language modelsNot stated
BytespiderByteDanceByteDance crawler, reported as model trainingNo vendor documentation found

robots.txt matching follows RFC 9309: the most specific user-agent group wins, the longest matching rule wins, and Allow beats Disallow on a tie.

Limits

  • It queries no index. The scores say whether a page can be crawled, indexed and quoted. They do not say whether any AI engine cites it today.
  • Bot-specific firewall rules are invisible. The server reads robots.txt, meta tags and headers. It reports when its own request is refused, and it never sends another company's crawler user agent.
  • No JavaScript. Pages are read as raw HTML on purpose, because that is how most AI crawlers read them.
  • Sitemaps are sampled. At most 5 sitemap files are read per check.
  • Public pages only. Private, local and non-standard addresses are refused.

How it works

The server is one Cloudflare Worker with no dependencies, no storage and no language model. It implements the MCP Streamable HTTP transport statelessly: every POST to /mcp is a self-contained JSON-RPC exchange.

FilePurpose
src/index.jsMCP protocol handling, tool definitions, landing page
src/crawl.jsCrawler access and indexability tools
src/citability.js, src/seo.jsCitability and SEO tools, scoring
src/robots.jsCrawler list and RFC 9309 matching
src/html.jsDependency-free HTML extraction
src/fetchpage.jsFetch guard: public hosts only, 1.5 MB cap, 12 second timeout
src/rubrics.jsThe two review rubrics
test/run.mjsProtocol, unit and tool tests

Run it yourself

Requires Node.js 22 or later.

npm test

Run the URL tools against a live page:

node test/run.mjs https://example.com/ "optional keyword"

Start a local server on port 8787:

npm run dev

To deploy your own copy, remove the routes line from wrangler.toml (it points at this project's domain) and run npm run deploy.

Privacy

The server keeps no state. It has no database, no accounts and no cookies, and it does not log tool inputs. For a URL it requests that page and the same site's robots.txt, llms.txt, llms-full.txt and up to 5 sitemap files, with the user agent milan-novotny-mcp/1.0. Full policy: https://www.milan-novotny.com/tools/ai-crawlability-mcp/privacy/

Author and licence

Built by Milan Novotný as a personal project. Crawler and product names belong to their owners; this project is not affiliated with or endorsed by any of them.

Released under the MIT licence.