Crawlberg

Name: Crawlberg
Author: xberg-io

by xberg-ioversion 1.12.3i★ –iRuns on your computer

Registry nameio.github.xberg-io/crawlberg

Install

Overview

Description from the maker (original English)

Scrape, crawl, and map websites to Markdown or JSON via local CLI.

PopNx MCP’s three pillars

These three are never merged into one score: opinions, measurements and popularity are different things.

User voiceiNo reviews yetReviews open together with accounts in a later step
Our basic checksi1 of 1 passedChecked from public information; code not scanned yet
UsageiNot measured yet★ –

What you can do with it

We have not written a plain-language summary for this one yet. Below are the facts we read from the registry.

Connection: Runs on your computer
Transport: stdio
Category: Web & browsers

Install

Pick the AI app you use, then copy the settings.

Before you install

Basic checks: 1 of 1 passed
Runs on your computer with your user’s permissions (files, network, apps).
The registry lists no key to fill in.

You need

DockerDocker installed

Which app do you use?

ClaudeClaude CodeCursorVS CodeChatGPT

Pick your app above. We remember it, so next time the button says “Install in …”.

Open Claude Desktop → Settings → Developer → Edit Config, and paste this into claude_desktop_config.json

{
  "mcpServers": {
    "crawlberg": {
      "command": "docker",
      "args": [
        "run",
        "-i",
        "--rm",
        "ghcr.io/xberg-io/crawlberg:1.12.3",
        "mcp"
      ]
    }
  }
}

Run this in your terminal

claude mcp add crawlberg -- docker run -i --rm ghcr.io/xberg-io/crawlberg:1.12.3 mcp

Open in Cursor

Or copy the settings yourself

Paste into .cursor/mcp.json in your project, or ~/.cursor/mcp.json

{
  "mcpServers": {
    "crawlberg": {
      "command": "docker",
      "args": [
        "run",
        "-i",
        "--rm",
        "ghcr.io/xberg-io/crawlberg:1.12.3",
        "mcp"
      ]
    }
  }
}

Your browser asks to open the app, then the app shows the settings and asks you to confirm. Key values are left empty for you to fill in.

Open in VS Code

Or copy the settings yourself

Paste into .vscode/mcp.json in your project

{
  "servers": {
    "crawlberg": {
      "type": "stdio",
      "command": "docker",
      "args": [
        "run",
        "-i",
        "--rm",
        "ghcr.io/xberg-io/crawlberg:1.12.3",
        "mcp"
      ]
    }
  }
}

Your browser asks to open the app, then the app shows the settings and asks you to confirm. Key values are left empty for you to fill in.

This one runs on your computer, and ChatGPT can only connect to servers on the internet. Pick another app.

Your choice stays on this device in a cookie; no account needed.

These settings are generated from the maker’s registry entry. If the README says otherwise, follow the README.

What it can do

Runs on your computer; the tool list appears in your AI app after you install it.

Basic checksi

We check information anyone can see; we have not read the code. Checks that do not apply are left out of the count. · Checked Nov 7, 2026

✓ Pass
Maker matches the repo
The registry name was verified with the GitHub account xberg-io, which owns the repo
… Not checked yet
Still maintained
We have not read this repo from GitHub yet
… Not checked yet
Has a licence
We have not read this repo from GitHub yet
– Not applicable
Endpoint responds
This one runs on your computer; there is no internet endpoint
– Not applicable
Sign-in with OAuth
Runs on your computer and uses local settings instead
… Not checked yet
No hidden instructions in tool descriptions
Runs on your computer; we can only read tool lists from internet endpoints

Compared with others in Web & browsersi

More stars than 0% of the other 881 servers in this category
Published to the registry more recently than 75% of the other 881 servers in this category
Basic checks equal to or better than 100% of the other 881 servers in this category

Compared with every active server in this category in the registry. Stars count only when the publisher owns the repo.

Common questions

Is Crawlberg safe?

It passed 1 of 1 applicable basic checks. We have not read the code. Before adding a key or token, see what access the server asks for and use a key with only the permissions you need.

What keys or codes does it need?

The registry does not list a required key, though some servers ask you to sign in on first use.

Does it run on my computer or in the cloud?

It runs on your computer. Your AI app starts it when needed, and data does not pass through the maker’s servers (unless it calls an online service itself).

Is it still maintained?

The GitHub repo last changed –, and the latest version (1.12.3) reached the registry 11 days ago.

Which apps can use it?

Apps that support MCP, such as Claude Desktop, Claude Code, Cursor and VS Code. You also need Docker on your computer.

What can I use instead?

In the same category: Browser Use, Chrome DevTools MCP, MCP Server Browser

Alternatives

Servers in the same category from other makers, sorted by stars.

Reading a card:★ GitHub starsiThree pillars: reviews · checks · usageiUpdated = latest version published to the registryi

Browser Use

by browser-use

Lets AI do web tasks for you, such as filling forms, searching and extracting data, in the maker’s cloud browsers or on your computer.

Computer or cloud★ 120,087MITUpdated 17 days ago

Install

Chrome DevTools MCP

by ChromeDevTools

Lets AI use Chrome’s developer tools to inspect pages, find what makes them slow and spot errors. Made by the Chrome DevTools team.

Runs on your computer★ 54,821Apache-2.0Updated 16 days ago

Install

MCP Server Browser

by bytedance

MCP server for browser use access

Runs on your computer★ 39,550Apache-2.0Updated 14 months ago

Install

README from the maker

Open the README

Copied from the maker’s repo as written. PopNx MCP has not reviewed or endorsed it. HTML, images and scripts are removed. Links lead off this site.

Crawlberg

Turn any website into clean, structured data. Point Crawlberg at a URL and get back Markdown, metadata, and links — from a single page or a whole site — in the language you already use.

What and Why?

You need data that lives on the web, and raw HTML is not it. Crawlberg does the crawling, scraping, and cleanup end-to-end: it fetches pages, follows links, converts each one to Markdown, and hands you structured metadata (titles, links, images, social-card and JSON-LD data) — so you skip the parsing and go straight to the content.

It runs from a single Rust core with identical results across 14 language bindings, and it handles the awkward parts for you: JavaScript-heavy pages fall back to a real headless browser, bot filters are detected and worked around, robots and sitemaps are respected, requests are throttled per domain, and requests to private or internal addresses are refused by default. Drive it from your code, an AI agent, a REST service, or the CLI.

Every part of the pipeline is a trait you can swap — the crawl frontier, rate limiter, storage, event stream, and content filters — so you can plug in your own behavior. Managed extras like proxy pools, tuned bot-evasion, authenticated sessions, scheduling, and billing live in xberg-enterprise.

Features

Feature	Description
Structured extraction	Text, metadata, links, images, assets, JSON-LD, Open Graph, hreflang, favicons, headings, response headers
Markdown conversion	Clean Markdown with citations, document structure, and fit-content mode
Concurrent crawling	Depth-first, breadth-first, or best-first traversal with configurable depth, page limits, and concurrency
14 language bindings	Rust, Python, Node.js, Ruby, Go, Java, Kotlin (Android), C#, PHP, Elixir, Dart, Swift, Zig, and WebAssembly
Smart filtering	BM25 relevance scoring, URL include/exclude patterns, robots.txt compliance, sitemap discovery
Browser rendering	Optional headless browser for JavaScript-heavy SPAs with WAF detection and bypass
Batch & streaming	Scrape or crawl hundreds of URLs concurrently; real-time crawl events via async streams
SSRF-safe by default	Refuses loopback, private, link-local, and cloud-metadata addresses; opt out via env var or `CrawlConfig`
Auth & rate limiting	HTTP Basic, Bearer, and custom-header auth with cookie jars; per-domain request throttling
MCP server & REST API	Model Context Protocol integration for AI agents plus an HTTP server with OpenAPI spec

Supported Platforms

Precompiled binaries for glibc Linux (x86_64/aarch64), macOS, and Windows across every binding. On musl Linux (Alpine) coverage is deliberately partial — the CLI, Docker image, Rust, Ruby, Java, and Elixir work there; the Node, Python, Go, PHP, and C# bindings do not ship musl artifacts. See the platform support matrix for the full table and the Alpine workarounds.

⭐ Star this repo to show your support — it helps others discover Crawlberg.

Quick Start

Language Packages

Python

pip install crawlberg

See Python README for full documentation.

Node.js

pnpm add @xberg-io/crawlberg

See Node.js README for full documentation.

Rust

cargo add crawlberg

See Rust README for full documentation.

go get github.com/xberg-io/crawlberg/packages/go

See Go README for full documentation.

Java

Available on Maven Central as io.xberg.crawlberg:crawlberg. See Java README for the dependency snippet and current version.

dotnet add package XbergIo.Crawlberg

See C# README for full documentation.

Ruby

gem install crawlberg

See Ruby README for full documentation.

PHP

composer require xberg-io/crawlberg

See PHP README for full documentation.

Elixir

Add {:crawlberg, "~> 0.3"} to your mix.exs dependencies. See Elixir README for full documentation.

Dart / Flutter

dart pub add crawlberg

See Dart README for full documentation.

Kotlin (Android)

Available on Maven Central as io.xberg.crawlberg.android:crawlberg-android. See Kotlin README for the dependency snippet and current version.

Swift

Add via Swift Package Manager. See Swift README for full documentation.

Zig

See Zig README for installation and usage.

WebAssembly

pnpm add @xberg-io/crawlberg-wasm

See WebAssembly README for full documentation.

C/C++ (FFI)

C header + shared library from GitHub Releases. See FFI crate for full documentation.

CLI

cargo install crawlberg-cli

# Or install the prebuilt binary via cargo-binstall:

cargo binstall crawlberg-cli

brew install xberg-io/tap/crawlberg

Windows users can install the same binary through Scoop:

scoop bucket add xberg https://github.com/xberg-io/scoop-bucket

scoop install crawlberg

See CLI README for full documentation.

AI Coding Assistants

Install the Crawlberg plugin from xberg-io/crawlberg. It ships the Crawlberg agent skills (site crawling, HTML→Markdown scraping, headless-Chrome fallback) plus the crawlberg MCP server, — select your runtime below.

The bundle targets Claude Code, Codex, Cursor, Gemini CLI, Kimi, OpenCode, Factory Droid, Hermes, Pi, Agent Plugins and GitHub Copilot CLI. Read the plugin guide for runtime resources, CLI prerequisites and configuration. MCP-capable adapters expose the product's tools; Hermes u

Only the beginning is shown; read the rest in the repo

Crawlberg

Overview

PopNx MCP’s three pillars

What you can do with it

Install

Before you install

What it can do

Basic checksi

Maker matches the repo

Still maintained

Has a licence

Endpoint responds

Sign-in with OAuth

No hidden instructions in tool descriptions

Compared with others in Web & browsersi

Common questions

Alternatives

Browser Use

Chrome DevTools MCP

MCP Server Browser

README from the maker

Crawlberg

What and Why?

Features

Supported Platforms

Quick Start

Language Packages

AI Coding Assistants