CrawlCheck

crawlcheck-core

The dependency-free modules of CrawlCheck, the verification layer for the agentic web, extracted from the production service. These are the readers behind its signed answer about what a site permits and serves to AI crawlers and agents. No build step, no dependencies, ESM, Node 20+.

The hosted service fetches a site as 12 crawler identities (plus a browser and two control requests) and measures 47 sections across three stages — reach (can the crawler get in), read (what it actually receives), quote (can an answer engine lift it). This repository is the open core: the readers and checks that need no network, no store and no key, published so the measurements can be reproduced.

What is here today: the robots.txt policy reader, the crawler identifier / log-line forgery verifier, the quotable-content reader, the image-signal reader, and the markdown-negotiation and agent-surface probes. What stays in the service: the scoring — the thresholds each reading is judged against, the weight every section carries in the grade, and the corpus calibration those numbers were set from — plus fetching as those crawler identities, the record over time, monitoring, the dataset, and billing. The rule is simple: a reading of your page is a fact you are entitled to check, and the calibration is the product.

Also in this repository: the SDKs

• `sdk/` — @crawlcheck/sdk (npm): the typed client for the CrawlCheck API and the zero-dependency offline verifier for signed answers, evidence bundles and receipts. npm test runs its suite against production.
• `sdk-python/` — crawlcheck (PyPI): the same client and verifier for Python 3.9+, no dependencies.

Both are published from the private service repository by a workflow that refuses to publish unless the directory here is byte-identical to what it ships, so this copy is the source the package pages point at, not a mirror that can drift.

Install

npm install github:emmanuelorta/crawlcheck-core

Not on the npm registry yet, so install it from the repository. Every subpath below resolves from that install.

robots — which crawlers your robots.txt actually admits

import { crawlerPolicy } from "crawlcheck-core/robots";

const policy = crawlerPolicy(robotsTxt, /* unreadable */ false, "/wp-admin/"); policy.agents.find(a => a.ua === "GPTBot"); // { ua:"GPTBot", label:"OpenAI training corpus", role:"train", // allowed:true, allowed_root:true, named:true, rules_shadowed:true } policy.shadowed; // [{ uas:["gptbot"], missing:["/wp-admin/","/feed/"], missing_count:2 }]

For each of 114 named agents (answer engines, search indexes, training crawlers, SEO tools, social unfurlers, regional engines) it reports: which user-agent group applies (named), whether the given path and the root are allowed under that group, and whether the group shadows the * rules.

The shadowing defect

A crawler obeys only its most specific group. The moment you write

User-agent: *

Disallow: /wp-admin/

User-agent: GPTBot Allow: /

GPTBot no longer sees Disallow: /wp-admin/ — the * group stops applying to it entirely. Sites that add an allowlist for AI agents routinely void every rule they thought still held. shadowedGroups() lists each named group and the * disallows it lost. This was found on our own sites first: how to block AI crawlers and the glossary entry for user-agent group.

API

What it does not do

Crawl-delay, Sitemap and unknown fields are ignored, and percent-encoding is not normalised (rules and paths are compared as written). Path matching follows RFC 9309: * matches any run of characters and a trailing $ anchors the end, so Disallow: / + Allow: /$ opens the homepage and nothing else. Versions before 0.6.0 stripped * and ignored $; production changed on 2026-09-30, and 0.6.0 brings this module back to the service's reading. The tests check parity on real robots.txt files, including paths decided by * rules.

agents — who fetched, and whether they are who they claim

import { identifyBot, loadRanges, verifyLines } from "crawlcheck-core/agents";

identifyBot("Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)"); // "GPTBot"

const ranges = await loadRanges(fetch); // the operators' published IP feeds (cache this for a day) const r = verifyLines(accessLogText, ranges); r.tally; // { verified, spoofed, unverifiable, no_bot } r.forged_rate_pct; // spoofed / (verified + spoofed), or null when nothing is checkable r.rows[0]; // { line, bot, ip, verdict, why }

Three verdicts, never two. Verified: the address is inside a range the claimed operator publishes. Spoofed: it is outside every range that operator publishes. Unverifiable: the operator publishes no feed (Common Crawl, ByteDance, Meta) or the feed could not be read — and that is reported as unverifiable, not as forgery. The forged rate divides by verified + spoofed only; counting unverifiable hits as forged would inflate it and counting them as genuine would understate it.

Why this exists: CrawlCheck's own telemetry once counted 78 of 85 "GPTBot" hits as forged — they were our own test requests. Reverse DNS is the older check the feeds replace; a user-agent is a self-assertion and proves nothing on its own.

quotable — what an answer engine can lift from the page

import { quotable } from "crawlcheck-core/quotable";

const { signals } = quotable(html, "Acme Fencing LLC");

signals.lead_defines; // does the first sentence say what this IS signals.answer_units; // paragraphs of 15-70 words that carry a fact signals.quotable_share; // % of sentences an engine could attribute unedited signals.faq_questions; // declared in FAQPage markup signals.faq_visible; // ...and actually readable on the page

The question is not whether the copy is good. It is whether, when an answer engine wants to state a fact about this page, there is a sentence on it that can be lifted as it stands — or whether the engine has to assemble one, in which case the answer carries the engine's wording rather than yours.

This module publishes the reading, not the scoring. Every count and share the production service records for a page is produced here and is reproducible byte-for-byte against a live scan — that is what the parity test does, and it refuses to compare unless the page bytes still match the ones the fixture was captured from. What the service keeps is what those readings are judged against: the thresholds, the weight the section carries in the overall grade, and the multi-hundred-site corpus pass the numbers were calibrated from.

The distinction is not a marketing line, it is the one that keeps the product honest. Anyone can check that we read their page correctly; nobody has to take our word for a number. What they cannot lift is the calibration, which is the part that took a corpus to earn.

Two details that are easy to get wrong and are settled here. FAQ parity is checked against the whole visible page, not the stripped body — an FAQ accordion inside a is still text a reader can see. And counts come from body paragraphs and list items with navigation, header, footer and forms removed, because a nav menu is not prose and counting it inflates every ratio on the page.

Scoring the reading — quoteRows, quoteBlock, the section weight and the score version — is in the service, not this package. Point the free scan at a URL to see the rows: https://crawlcheck.io/.

rows — the three-state row every reader returns

import { tgt, pctScore } from "crawlcheck-core/rows";

A row is { k, cur, opt, ok, why }: the check, what this page currently does, what it should do, the verdict, and why it matters. `ok: null` is the third state and it is load-bearing — it means *measured but not scored*. A signal too rare to judge on, one that does not apply to this kind of site, or one that is purely informational is shown to the reader and deliberately left out of the arithmetic. pctScore() divides by the scored rows only and returns null, never 0, for a section with none: an unmeasured section is not a failing one.

That rule is why a service-area business is not marked down for having no street address, and why a brand-new site with no field performance data is not reported as slow. Most audit tools have two states, and every gap in their own measurement becomes your zero.

surfaces — markdown negotiation, and the Accept header that made a site an F

import { agentSurfaces, grab } from "crawlcheck-core/surfaces";

const root = await grab("https://example.com/"); const out = await agentSurfaces("https://example.com", root);

out.found.markdown; // Accept: text/markdown got a text/markdown answer out.found.agents_md; // an authored /agents.md, not a 200 of HTML out.found.link_header; // read off the root response — no extra request out.evidence.markdown; // { content_type: "text/markdown; charset=utf-8" }

A scanner's Accept header is part of its identity, and getting it wrong makes the whole grade a measuremen