Open SEO Crawler

Open SEO Crawler: Free SEO Crawler, Website Audit & AI Crawler Checker (Self-Hosted)

Open SEO Crawler is a free, open-source SEO crawler you run locally: a self-hosted website SEO crawler and site audit tool that crawls any website for technical SEO issues. Fast, concurrent, and CMS-aware, it's a drop-in alternative to Screaming Frog, Sitebulb, and Ahrefs Site Audit, with no accounts, no API keys, no cloud, and no per-URL limits.

It is also an AI crawler checker. Can AI search and Google read your pages? After each crawl, a sample of pages is requested as Googlebot, Bingbot, Applebot and every AI crawler that publishes its user agent (GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, CCBot). A page that a crawler is blocked from, challenged on, redirected away from or served a thinner version of gets flagged, and so does text that only appears after JavaScript runs. When the crawl is done, the Use with AI button packs it into a summary and a ready-made prompt you can paste into any AI assistant, or hands you a setup prompt that connects your AI app to the crawler so it can run crawls itself.

Your AI assistant can drive it. The repo ships an MCP server and a command-line tool, so Claude Code, Claude Desktop, Cursor and other MCP apps (or any script) can start crawls, read the issues and pull up any page without opening the browser. The crawler still calls no AI: your assistant calls the crawler. Set it up.

Built for SEO professionals, web developers, and site owners who want a real technical SEO audit that stays on their machine. Use it as a free SEO crawler for a single site or a recurring website crawler across your whole portfolio.

Built by Puneet Singh, a working SEO. Questions, ideas or bugs you would rather talk through? Connect on LinkedIn.

→ One-line install on Linux, macOS, or Windows. Auto-starts on boot, auto-updates daily. Browser opens at http://localhost:5002/ when done.

A 100-page test crawl of books.toscrape.com, a public scraping sandbox.

How it compares

Companion: SEO Side Panel for Chrome

The crawler covers a whole site. For the one page you have open, there is SEO Side Panel, a free Chrome extension from the same maker. It stays open beside every page you visit and lists each issue with its fix.

• What AI crawlers miss. The raw HTML the server sends, next to the rendered page. GPTBot, ClaudeBot and PerplexityBot do not run JavaScript, so anything that only appears after hydration is invisible to them.
• Titles and meta descriptions measured in pixels, not characters.
• JSON-LD errors named by block and line, in plain words.
• Link status: every link on the page marked 200, redirect or 404.

No account, no API key, no telemetry. Crawl the site here to find the pages with problems, then open one in Chrome to see what is wrong with it.

What's new (recent additions)

• Use with AI, two ways: the top bar button now opens on a choice. *Paste a brief* copies the crawl as a Markdown brief for any AI chat. *Connect an AI app* gives a setup prompt with this computer's exact command and paths: paste it into an AI app that can run commands and it adds the MCP server itself. The command and the settings block are there to copy too, for setting it up by hand.
• A plainer look, modelled on the crawlers people already use: measured against the desktop and hosted crawlers. A light grey canvas with white panels, one blue, sentence case throughout, Lighthouse-style severity marks, a Summary laid out like an audit overview (crawled pages by status, pages with issues by severity, crawl details, one table of issues), plain 200s with badges only for redirects and errors, row actions on hover, and a dark mode in neutral greys. The settings column folds away once a crawl is on screen (Settings in the top bar brings it back), and Start crawl stays visible however long the settings get.
• Crawl progress and a live Summary: a bar under the top bar shows how far a crawl has got (pages crawled against crawled plus queued, capped at the page limit), the queue, the time so far and an estimate of the time left. A new crawl opens on the Summary, which fills in as pages arrive.
• Pages reached through a redirect are now checked: the crawler stores a followed redirect under the page it landed on, but skipped that page's content checks, so a start URL that redirects (example.com to www.example.com, or to a language folder) never had its title, meta description, H1, canonical or Open Graph checked. It does now, and those pages also count in the duplicate reports, the sitemap comparison and the exported sitemap.
• A truer AI brief: it names the start URL and where it redirects, says when the crawl stopped before reaching every linked page (so URLs it never reached are not called orphans), lists an issue that hits almost every page as the few pages without it, keeps | in titles as written, and dates the export in your own time zone. The sitemap report's *In sitemap, not crawled* list now says how many crawled pages link to each URL.
• MCP server and command line: connect Claude Code, Claude Desktop or any other MCP app to the crawler, or script it with osc.py. Start a crawl and follow it, read the Summary as data, filter pages by issue, open one page with its inlinks, compare two crawls, run the AI crawler check on any URL, export to .xlsx. Crawls started this way run the same checks as the browser and show up under Load. Standard-library Python, nothing extra to install. See Use it from an AI assistant.
• AI crawler checker (bot readability): after every crawl, a sample of pages is requested as a browser and as Googlebot, Bingbot, Applebot and the AI crawlers that publish their user agents (OAI-SearchBot, PerplexityBot, ChatGPT-User, GPTBot, CCBot). Blocks, CDN challenges, redirects, a different title or text, a thinner page or a noindex aimed at one crawler are flagged, as is content that only exists after JavaScript runs and a mobile page with less content than desktop. On by default for 10 pages.
• AI brief (now under *Use with AI*): one click turns the crawl into a compact Markdown brief with a ready-made prompt, to paste into any AI assistant for a ranked list of fixes. Issues are grouped the way the Summary groups them and URL lists are capped, so the brief fits a chat window. The crawler itself calls no AI and sends nothing anywhere.
• Every report exports: Export view now works on every report tab (All Titles, Redirect Chains, Hreflang, Deep Pages and the rest), and the .xlsx workbook adds a sheet for each report that has rows.
• Safer and steadier: crawled titles, URLs and robots.txt lines can no longer run script inside the app, other websites can no longer trigger Update or Restart, the Update button restarts cleanly under systemd and launchd, and sitemap analysis works again.
• Malformed link href detection: a scheme-less href containing a raw space (a street address or Google Maps Plus Code pasted straight into a link, e.g. Directions ) used to resolve relative to every page carrying it and flood the crawl with phantom 404s. One broken footer link on a 300-page site meant ~300 fake 404 rows. Now it's skipped as a link and reported once per page as a Malformed link href issue naming the offending text.
• JS vs no-JS compare mode: with Render JS on, tick *Compare with non-JS HTML* to crawl every page in BOTH modes (Playwright-rendered and raw HTML). The per-page diff shows exactly which content is invisible to AI crawlers (GPTBot, ClaudeBot, PerplexityBot, CCBot), which mostly don't execute JavaScript.
• Shared-instance support: point the saved-crawls list at extra folders (SITE_CRAWLER_EXTRA_CRAWL_DIRS) so one office instance shows crawls from multiple tools, and label who ran each crawl by IP (SITE_CRAWLER_USER_MAP).
• Crawl Budget page: scans the homepage + section pages, buckets every discovered URL by query parameter, classifies crawl-budget traps (Elementor ?e-page- AJAX pagination, faceted filters, sort / tracking / session params) and generates ready-to-paste robots.txt Disallow rules. Shows the sitemap's real page count next to the phantom parameter URLs so the wasted crawl budget is obvious.
• Soft-404 / infinite-URL-trap detection: after every crawl, two probe requests check whether the server returns 200 for URLs that cannot exist (/ / at the root, and nested under a real page). A 200 means the server soft-404s and mints an infinite crawlable URL space; flagged red in the issues sidebar with fix guidance.
• Images Missing Alt report: an image-centric bulk report with one row per image missing alt text and every page that embeds it.
• robots.txt AI-crawler block detection: flags, as a red error, when robots.txt blocks AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bytespider + 20 more), meaning the site can't be read or cited by ChatGPT, Perplexity or Google AI Overviews, or when classic search engines (Googlebot/Bingbot) are blocked. Also detects Cloudflare Content-Signal: ai-train=no opt-outs. Surfaces on the homepage row in the issues sidebar under AI Crawlers Blocked / Search Engines Blocked.
• Post-crawl Summary dashboard: auto-opens after a crawl finishes with a one-glance view of issues by severity, total pages, response code mix, and the worst offenders.
• Bulk SEO reports: All Titles, All Metas, All H1s, All Canonicals, plus dedicated Duplicate Title / Duplicate Meta / Duplicate H1 / Duplicate Body / Multiple H1s / Redirect Chains / Response Codes / Deep Pages / Hreflang reports.
• Severity-grouped issues view: pages are bucketed by the worst issue on them (Errors / Warnings / Info) and grouped by issue type so you fix the highest-impact problems first.
• One-click auto-update: version badge in the topbar checks GitHub for newer commits and offers a one-click git pull + restart + page reload so you're always on the latest code.
• Dark mode toggle: explicit per-user choice (persists in localStorage), overrides system prefers-color-scheme cleanly.
• Saved crawls list: every crawl is auto-saved; reopen any historical crawl from the Load Saved modal. Names of who ran each crawl resolve across multiple tools on the same LAN.
• Smart noindex / canonicalised handling: these pages no longer inflate the Error count or get flagged as duplicates by Bulk Reports.

Feat