Websearch

websearch MCP Server

mcp-name: io.github.SK-DEV-AI/websearch

Companion agent skills: SK-DEV-AI/skills · npx skills add SK-DEV-AI/skills

Multi-engine web search + fetch MCP server. Parallel fan-out across DuckDuckGo, Bing Web/News RSS, Tavily, Wikipedia, arXiv, Google News RSS, Brave (last resort — rate-limited from some networks), Google AI Mode, and intent-gated vertical lanes (MDN, Stack Exchange, GitHub, HN, Semantic Scholar, OSV vuln, grep.app/Sourcegraph code, GDELT news) — with consensus-merge dedup, a local cross-encoder reranker, and Groq synthesis with grounding checks.

Tools (20)

• doctor — Run this first. Reports which lanes are usable right now and the exact command to enable the rest. Distinguishes "broken" from "needs optional setup". Never prints key values.
• search — Open research questions. depth defaults to 3 when a model lane is configured (more independent sources; measured) and 1 otherwise. Depth also sets how many pages are fetched: max(5, depth*2), ceiling 8. synthesize returns a short, cited answer that never invents: it says which source backs each claim, surfaces disagreement instead of averaging it, and marks itself truncated when it ran out of budget.
• fetch — URL to markdown/text. Direct fetch with auto-fallback to a Chromium CDP browser for blocked/JS pages. PDF/EPUB/DOCX supported. Paginated (offset). Optional distill (see below). For structured JSON use extract.
• fetch_many — Fetch several URLs in one call. Dedups, caps concurrency, honours robots.txt, shares the cache, returns per-URL success/status/content plus counts. Use when you have 2+ URLs; use fetch for one.
• crawl — Breadth-first site crawl with hard caps (per-page cap, depth cap, robots.txt, dedup, concurrency). Returns each page plus a stop_reason so you know whether the site was exhausted or a cap stopped it. Use for "read this whole docs site".
• extract_links — Every link on a page with anchor text, title and rel. Resolves relative hrefs, drops javascript:/mailto:, dedups. Use before crawl or to find a "next" button.
• extract_table — HTML tables to json, csv or markdown, by selector and index. Reports tables_found so you can pick one.
• download — Save a URL to disk; returns the local path, byte count, sha256 and content type. Writes only inside dest_dir and refuses path traversal. Marks truncated if the byte cap stopped the transfer.
• diff_page — Has this page changed since it was cached? Returns changed/old_hash/new_hash plus added and removed lines. First call for a URL returns had_previous: false.
• screenshot — ARIA accessibility snapshot (LLM-readable page text and interactive structure).
• browser_action — Drive a real browser page (navigate, click, type, press a key, evaluate JS, read HTML/text, ARIA tree, screenshot). Use when a page needs interaction; for plain reading use fetch. Pass session_id to reuse a persistent tab.
• browser_session — Create, list, inspect or close a persistent browser tab. Without one, every browser_action call gets a fresh tab and closes it, so logins and cookies cannot survive. See below.
• form_fill_submit — Fill many fields, submit, and read the result in one call. Verifies each element exists, reads back each typed value, and refuses to submit if a value did not stick. Reports effect (navigated / content_changed / none) so a submit that silently did nothing is visible rather than assumed.
• paginate — Click a "next" control, collecting a value per page, up to a cap. Stops honestly when the control is missing or has no effect, and reports stop_reason.
• wikipedia — Full Wikipedia API client: articles, summaries, categories, links, pageviews, revisions, backlinks, geosearch, recent changes, random pages.
• arxiv — arXiv paper search with boolean/phrase/wildcard query syntax (single-source; multi-source paper search lives in codesearch papers).
• extract — Structured JSON from a page via CSS (fast, free), LLM, or regex strategies. Repeat calls cached.
• pdf_extract — PDF to markdown/json/html. PyMuPDF fast path for text-layer PDFs, Docling+OCR fallback for scanned PDFs.
• map_site — Survey a site's pages via sitemap XML (primary) + HTML link fallback, before fetching individually.
• ping — Connectivity check for the internet and key search endpoints. Use before expensive calls when uncertain.

What replaced the throwaway scripts

Every one of these exists because an agent otherwise writes a one-off script:

Persistent browser sessions

browser_action opens one fresh tab per call and closes it afterwards. That is right for a single read, but it makes any interactive workflow impossible: no login, no cookies, no scroll position, no multi-step flow across calls.

A session is a named, long-lived tab:

browser_session(action="create", label="login", url="https://site.example/login")

-> {"session_id": "login-1a2b3c4d", ...}

browser_action(action="type", selector="#email", text="me@example.com", session_id="login-1a2b3c4d") browser_action(action="click", selector="#submit", session_id="login-1a2b3c4d") browser_action(action="text", session_id="login-1a2b3c4d") # same tab, still logged in browser_session(action="close", session_id="login-1a2b3c4d")

Sessions are opt-in: without session_id, behaviour is exactly as before. They are capped at 8 with least-recently-used eviction, tabs are always closed (including on eviction), and metadata is mirrored to ~/.cache/websearch/sessions.json so list can explain what survived a restart. Sessions need a browser, so they are local-only.

In-tool distillation (optional, off by default)

A fetched page can be 50k characters. The calling model pays for all of it even when it needed one fact. fetch(distill=true) runs a small model inside the tool first and returns a compressed result:

fetch(url="https://example.com/long-report", distill=True,

distill_question="What are the pricing tiers?")

The result never pretends to be the page:

Below the size threshold (default 4000 chars) distillation is skipped and the raw text is returned, because a summary would cost more than it saves. On any failure the raw text is returned with distill_skipped explaining why - a broken summary is never returned in place of the page.

The synthesis prompt (the answer you actually read)

search does not just rank results; it writes the answer. That answer is generated from the retrieved text alone, under rules that exist to stop the three failure modes of a search summary:

Concision is structural, not a wish: a sentence ceiling, an answer-first rule, and a named ban on filler phrasing ("it is important to note", "overall", ...).

Everything is verifiable rather than claimed:

• the result carries prompt_version, so a quality change can be tied to a revision instead of guessed at
• temperature is 0, so the same sources give the same text
• finish_reason and truncated are returned; a half sentence is labelled instead of looking like a complete thought
• diversity reports the source mix and whether the per-domain cap had to relax
• as_of reports the newest publication date among the sources, and undated sources are labelled "Published: not stated", so freshness is never implied
• duplicates_removed counts syndicated copies collapsed before synthesis, and figures_unverified lists numbers the answer contains that appear in no cited source (deterministic, no model call)
• unusable_sources names the results that could not become evidence - a 404, an error page, or a page with under 400 chars of text - with the reason, and unreadable_sources counts the ones with no text at all
• multi-hop: when the first answer admits a weakness (single source, a claim it could not tie to a source, or a stated gap), one extra round runs on the subquery that targets the gap. The response records follow_ups and marks synthesis.round = 2. Bounded: once per call, only under a 75s budget, and follow_up=false disables it
• a second model pass (grounded) re-reads the answer against the exact text the first pass saw and flags sentences no cited source supports. It is marked advisory because it is another model's opinion, not a proof

Tool annotations

All 20 tools carry MCP annotations (readOnlyHint, destructiveHint, idempotentHint, openWorldHint) so a client can tell a read from an action: search and fetch are read-only, download writes a local file without being destructive, browser_session changes local browser state only, and browser_action / form_fill_submit are marked destructive because a click or a form submit can have an effect that cannot be undone from here.

When a call takes too long (deadline, admission, partials)

MCP clients cut a tool call at their own timeout (jcode: 300 s) and report a bare Request timeout after 300s with no payload. Measured over one day on a single busy server: 314 calls, median 2.7 s, p90 123 s, max 225 s, and 33 hard client timeouts — the tail is the problem, not the median. So the server bounds its own work instead of hoping the client waits:

| WEBSEARCH_QUEUE_WAIT | 20 | Seconds a call may wait for a free slot before being answere