Web Search

websearch-skill

Open-source multi-engine web search and content extraction for AI agents. Internal layers communicate through versioned JSON Schema contracts.

Status: early. First public release 2026-06-22; current version in CHANGELOG.md. Available now: keyless search, Markdown extraction, five agent commands, self-hosted SearXNG, and a NordVPN egress proxy that two env vars enable and lock. These are covered by the test suite. There is no MCP server; use the CLI directly or install the skill (see No MCP below). Hard anti-bot tiers and local reranking remain on the roadmap. Pin a version and test it in a sandbox before using it with sensitive data.

What it is

A self-hosted web search and page reader for AI agents, with no API keys or paid search service. It queries multiple engines, combines and deduplicates the results, then fetches pages as clean Markdown. The CLI runs locally and sends queries directly to the configured engines.

Core commands:

• `init` reads the settings files, starts local SearXNG (and Tor, when that layer is on), and reports available capabilities.
• `web_search` finds pages: ranked, deduplicated results across engines, each with a reusable handle.
• `web_fetch` reads a page: clean Markdown, fenced as untrusted, paginated so a long page never overflows context.
• `web_open` pages back through a document you already fetched, from cache, without hitting the network again.

Additional keyless commands:

• `arxiv` searches arXiv papers and returns structured metadata (authors, abstract, categories, abstract and PDF links).
• `github` searches GitHub repositories and returns typed fields you can sort on (stars, language, topics).
• `tor` starts a local Tor and routes everything through it, which is what makes .onion reachable and --onion search the Tor indexes. Off by default; see Tor below.

One runtime, one call

The whole toolkit is one Python package with one CLI, websearch. There is no Node service, no npm runtime, no daemon of its own, and nothing to compose; the only requirement is uv, which fetches a compatible Python on its own. The npx skills add line under Install copies skill folders into agents once; nothing from Node runs afterwards.

An agent brings everything online with one call:

websearch init

init reads the settings files, starts the local SearXNG when it is not already running, starts Tor when that layer is on, runs the self-test, and reports capabilities and next actions. There is nothing to probe first and no order to follow: the same call is correct on a fresh machine and on one where everything is already up.

SearXNG is an optional extra engine catalog that the toolkit operates for you: init (or websearch searxng up) clones it, installs it into its own virtualenv, and runs it as a detached local Python process. Docker is never required; docker/searxng/ exists only as an alternative for hosts that prefer a container.

Engines, out of the box, no keys

The default engine is the keyless ddgs metasearch library, which spans Google, Brave, DuckDuckGo, Yandex, Yahoo, Startpage, Mojeek, Wikipedia, and Grokipedia. Each query uses the providers that respond. The agent-facing web-search command requires no account, service, or engine flags. The lower-level search command can restrict providers with --ddgs-backends google,brave,mojeek. Provider availability varies by network; websearch doctor reports it per provider.

For broader and more reliable search you can run your own SearXNG (~280 engines, your own server, still no keys), and the router fuses it with ddgs and de-correlates the engines they share. websearch searxng up sets one up and starts it on any machine with git and Python; on a Docker host, docker/searxng/ is the curated stack, including a probe that enables every engine that answers on your connection. See Self-hosting SearXNG below. Public SearXNG instances are deliberately not a default: most disable the JSON API and rate-limit automated clients, so depending on them would break on a fresh install.

The project is MIT-licensed. Optional integrations include self-hosted SearXNG, keyed engines (Brave, Exa, Tavily), and a paid egress adapter.

The scorecard below compares result quality, latency, cost, privacy, and retrieval limits. The software layer is free and self-hosted, with multi-engine search, deduplication, and Markdown extraction. Protected sites may require the planned paid egress adapter; residential proxies and CAPTCHA solving have no reliable free equivalent.

Layer status

Contracts are frozen as JSON Schema 2020-12: envelope@1.0.0, search@1.2.0, fetch@1.3.0, extract@1.0.0, format@1.0.0, store@1.0.0, agent-io@1.2.0, arxiv@1.1.0, github@1.1.0, doctor@2.1.0, searxng@1.2.0, init@1.1.0, tor@1.0.0, proxy@1.0.0. Every response is wrapped in one Envelope { contract_version, ok, data, error, meta }.

Layer 1: Search

Search normalizes a request, sends it to each configured EngineAdapter, canonicalizes and deduplicates URLs, and combines the rankings with weighted Reciprocal Rank Fusion (RRF, k=60). The keyless default is ddgs, a metasearch library for Google, Brave, DuckDuckGo, Yandex, Yahoo, Startpage, Mojeek, Wikipedia, and Grokipedia. On the lower-level search command, --ddgs-backends selects a provider subset and --engines selects the ddgs and SearXNG adapters. The agent-facing web-search command uses the defaults. Set WEBSEARCH_SEARXNG_URL to add a self-hosted SearXNG instance. Keyed engines such as Brave, Exa, and Tavily are planned.

ddgs is a keyless Python library that runs in-process. SearXNG is a separate self-hosted server with a larger engine catalog. Search can use either adapter or combine both. SearXNG provides search results only; page fetching and extraction remain in Layer 2A.

De-correlation prevents duplicate upstream data from distorting the ranking. SearXNG and ddgs both use crawlers such as Google and Bing, so a naive union can count the same source twice. Results are grouped by correlation_group; each group contributes one RRF term at its best rank, and the consensus bonus uses only the number of distinct groups. SearXNG and ddgs agreeing counts as one independent vote; SearXNG and a neural index agreeing counts as two. The response includes a warning when de-correlation is applied.

Each result records which engines returned it and its rank within each engine.

Layer 2A: Fetch + Extract

Fetching and extraction use separate interfaces and can be replaced independently.

Fetch starts with httpx and retries with curl_cffi browser impersonation only when it detects an anti-bot challenge. It does not retry 404s, rate limits, authentication failures, or legal and geographic blocks because the same network address is unlikely to change the result. Detection checks response headers, then known body markers for Cloudflare, DataDome, PerimeterX, Akamai, and Imperva. Imperva body markers are checked on every status because its blocks can return HTTP 200. Browser and stealth tiers (Crawl4AI, nodriver) are present in the contract but are not part of the base install.

Extract defaults to Trafilatura, a heuristic extractor. Per the May 2026 WCXB benchmark, heuristic extractors beat neural extractors on both quality and cost (neural runs tens to hundreds of times more expensive), and Trafilatura lands at about 0.79 F1 on CPU in roughly 100ms; article pages saturate around 0.93 F1. The adapter parses the raw HTML once with lxml to recover the raw schema.org JSON-LD blocks and og:type (Trafilatura folds JSON-LD into metadata and never exposes the raw blocks), runs Trafilatura for the Markdown body and plain text plus metadata, then computes:

• a heuristic `quality_score` (0..1) from runtime signals (text density, word-count saturation, paragraph count, inverse link density, JSON-LD presence, clean title) with hard vetoes for soft-404s and shells; below about 0.80 a page is a fallback candidate.
• a cheap `page_type` resolved from JSON-LD @type, then og:type, then URL shape.

Neural extract engines (crawl4ai, jina_readerlm, and others) are named in the contract enum but stay opt-in. The default dependency closure is permissive: Apache-2.0 (trafilatura) plus MIT/BSD/MPL deps.

Extraction does not truncate content_markdown. --max-bytes limits the downloaded response size, and --max-bytes 0 disables that transport limit. A value of 0 also disables --max-results, --page-size-tokens, and --per-page, subject to provider limits such as GitHub's 100 results per page and arXiv's 2,000 results per request.

Layer 2B: Format + Store

This layer formats results for agents and stores full pages for later access.

Format takes vendor-neutral results from Layers 1 and 2A and renders a layout-stable Markdown document plus a JSON sidecar containing the same data. Results are ordered by relevance and paginated according to the configured token budget. Near-duplicate deduplication runs first: byte-exact (normalized SHA-256), then pure-Python MinHash over word 4-gram shingles (128 permutations, Jaccard 0.9) clustered with union-find. The highest-scored page in each cluster is kept and the rest are listed in dropped_duplicates. The default threshold merges near-identical pages such as mirrors and syndicated copies, while keeping pages that are only topically similar. Lower jaccard_threshold for boilerplate-heavy corpora. The implementation does not depend on datasketch. Progressive disclosure controls how much to inline:

• `auto` (default) inlines full bodies when the page fits a token budget, otherwise switches to an index (a preview plus a stable id).
• `index` always shows a preview and a resolve hint; `full` always inlines.

The optional anthropic_search_result_blocks view maps 1:1 onto Anthropic search_result content blocks (source as a bare string, at least one non-empty text block, citations all-or-nothing). It is off by default and separate from the canonical response shape.

The sidecar and store retain the full body in every mode. body_char_budget moves content from the rendered Markdown view into the resolver an