Browser Use
Lets AI do web tasks for you, such as filling forms, searching and extracting data, in the maker’s cloud browsers or on your computer.
Computer or cloud✓ browser-use.com
Robotstxt AI
Overview
Description from the maker (original English)
Analyze and generate robots.txt with AI crawler detection for 20+ bots like GPTBot and ClaudeBot.
PopNx MCP’s three pillars
These three are never merged into one score: opinions, measurements and popularity are different things.
- User voiceiNo reviews yetReviews open together with accounts in a later step
- Our basic checksi1 of 1 passedChecked from public information; code not scanned yet
- UsageiVery new↓ 49/wk · ★ –
What you can do with it
We have not written a plain-language summary for this one yet. Below are the facts we read from the registry.
- Connection: Runs on your computer
- Transport: stdio
- Category: Web & browsers
Install
Pick the AI app you use, then copy the settings.
Before you install
- Basic checks: 1 of 1 passed
- Runs on your computer with your user’s permissions (files, network, apps).
- The registry lists no key to fill in.
You need
- npmNode.js installed (for the npx command)
Which app do you use?
Pick your app above. We remember it, so next time the button says “Install in …”.
Open Claude Desktop → Settings → Developer → Edit Config, and paste this into claude_desktop_config.json
{
"mcpServers": {
"robotstxt-ai": {
"command": "npx",
"args": [
"-y",
"robotstxt-ai-mcp"
]
}
}
}Run this in your terminal
claude mcp add robotstxt-ai -- npx -y robotstxt-ai-mcpOr copy the settings yourself
Paste into .cursor/mcp.json in your project, or ~/.cursor/mcp.json
{
"mcpServers": {
"robotstxt-ai": {
"command": "npx",
"args": [
"-y",
"robotstxt-ai-mcp"
]
}
}
}Your browser asks to open the app, then the app shows the settings and asks you to confirm. Key values are left empty for you to fill in.
Or copy the settings yourself
Paste into .vscode/mcp.json in your project
{
"servers": {
"robotstxt-ai": {
"type": "stdio",
"command": "npx",
"args": [
"-y",
"robotstxt-ai-mcp"
]
}
}
}Your browser asks to open the app, then the app shows the settings and asks you to confirm. Key values are left empty for you to fill in.
This one runs on your computer, and ChatGPT can only connect to servers on the internet. Pick another app.
Your choice stays on this device in a cookie; no account needed.
These settings are generated from the maker’s registry entry. If the README says otherwise, follow the README.
What it can do
Runs on your computer; the tool list appears in your AI app after you install it.
Basic checksi
We check information anyone can see; we have not read the code. Checks that do not apply are left out of the count. · Checked Nov 8, 2026
- ✓ Pass
Maker matches the repo
The registry name was verified with the GitHub account sharozdawa, which owns the repo
- … Not checked yet
Still maintained
We have not read this repo from GitHub yet
- … Not checked yet
Has a licence
We have not read this repo from GitHub yet
- – Not applicable
Endpoint responds
This one runs on your computer; there is no internet endpoint
- – Not applicable
Sign-in with OAuth
Runs on your computer and uses local settings instead
- … Not checked yet
No hidden instructions in tool descriptions
Runs on your computer; we can only read tool lists from internet endpoints
Compared with others in Web & browsersi
- More stars than 0% of the other 1380 servers in this category
- Published to the registry more recently than 5% of the other 1380 servers in this category
- Basic checks equal to or better than 100% of the other 1380 servers in this category
Compared with every active server in this category in the registry. Stars count only when the publisher owns the repo.
Common questions
Is Robotstxt AI safe?
It passed 1 of 1 applicable basic checks. We have not read the code. Before adding a key or token, see what access the server asks for and use a key with only the permissions you need.
What keys or codes does it need?
The registry does not list a required key, though some servers ask you to sign in on first use.
Does it run on my computer or in the cloud?
It runs on your computer. Your AI app starts it when needed, and data does not pass through the maker’s servers (unless it calls an online service itself).
Is it still maintained?
The GitHub repo last changed –, and the latest version (1.0.1) reached the registry 8 months ago.
Which apps can use it?
Apps that support MCP, such as Claude Desktop, Claude Code, Cursor and VS Code. You also need Node.js on your computer.
What can I use instead?
In the same category: Browser Use, Chrome DevTools MCP, MCP Server Browser
Alternatives
Servers in the same category from other makers, sorted by stars.
Reading a card:↓ Downloads per weeki★ GitHub starsiThree pillars: reviews · checks · usageiUpdated = latest version published to the registryi
Chrome DevTools MCP
Lets AI use Chrome’s developer tools to inspect pages, find what makes them slow and spot errors. Made by the Chrome DevTools team.
Runs on your computer
README from the maker
Open the README
Copied from the maker’s repo as written. PopNx MCP has not reviewed or endorsed it. HTML, images and scripts are removed. Links lead off this site.
Manage which AI bots can crawl your website — visually.
Toggle GPTBot, ClaudeBot, PerplexityBot, and 20+ AI crawlers on/off with a simple UI. Analyze any site's robots.txt instantly.
Features
- Visual Toggle UI — Block or allow AI bots with simple on/off switches
- 20+ AI Bots Database — GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Diffbot, cohere-ai, Amazonbot, Meta-ExternalAgent, and more
- Analyze Existing robots.txt — Paste or fetch any robots.txt to see which AI bots are blocked
- Generate robots.txt — Create a complete robots.txt with your chosen rules
- MCP Server — Use with Claude Desktop, Cursor, or any MCP-compatible AI assistant
- Check Bot Status — Verify if a specific bot is blocked on any website
MCP Tools
| Tool | Description |
|---|---|
fetch_robots | Fetch and analyze a robots.txt from any URL |
analyze_robots | Analyze pasted robots.txt content for AI bot blocking status |
generate_robots | Generate a robots.txt with specified blocked bots and custom rules |
list_ai_bots | List all known AI bots with user-agents, companies, and descriptions |
check_bot_status | Check if a specific bot is blocked on a given website |
Installation
Web App
git clone https://github.com/sharozdawa/robotstxt-ai.gitcd robotstxt-ai npm install npm run dev
Open http://localhost:3000 in your browser.
MCP Server — Claude Desktop
Add to your claude_desktop_config.json:
{"mcpServers": { "robotstxt-ai": { "command": "npx", "args": ["-y", "robotstxt-ai-mcp"] } } }
MCP Server — Cursor
Add to .cursor/mcp.json:
{"mcpServers": { "robotstxt-ai": { "command": "npx", "args": ["-y", "robotstxt-ai-mcp"] } } }
Tracked Bots
The server knows about 25+ bots including:
- AI Crawlers: GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Diffbot, cohere-ai, Amazonbot, Meta-ExternalAgent
- AI Search: ChatGPT-User, OAI-SearchBot, PerplexityBot, YouBot
- Search Engines: Googlebot, Bingbot, YandexBot, Baiduspider, DuckDuckBot
Why robotstxt.ai vs Manual Editing
| Feature | robotstxt.ai | Manual Editing |
|---|---|---|
| Visual toggle UI | Yes | No |
| 20+ AI bots database | Yes | Research yourself |
| Analyze existing robots.txt | Yes | No |
| MCP Server | Yes | No |
| Price | Free | Free but tedious |
More Open Source SEO Tools
| Tool | Description |
|---|---|
| awesome-seo-mcp-servers | Curated list of SEO MCP servers and agent skills |
| indexnow-mcp | Instant URL indexing via IndexNow |
| schema-gen | Schema.org JSON-LD markup generator |
| ai-visibility | AI brand visibility tracker |
License
MIT
Built by [Sharoz Dawa](https://sharozdawa.com) — SEO Professional & Digital Marketing Expert