MCP Retrieval

mcp-retrieval

An MCP server that gives an LLM three web tools: search, image search, and page scraping — no API keys required.

Tools · Quick start · Configuration · Retrieval engine · Architecture · Contributing

What it is

mcp-retrieval is a Model Context Protocol server written in Go. It exposes web retrieval capabilities to any MCP-compatible client (Claude Desktop, IDE agents, custom LLM apps) as three read-only tools. Under the hood it uses the retrieval-go library to search the web and fetch pages, returning results as clean Markdown ready to hand to a model.

The library needs no API keys: web search goes through DuckDuckGo Lite, image search through Bing Images, and page fetching runs the HTML through a readability extractor before converting it to Markdown. To stay reliable against bot protection it impersonates real browsers at the TLS level and can rotate both browser fingerprints and proxies — see Retrieval engine.

Both transports the MCP SDK supports are available and expose the identical tool set:

• stdio — the client launches the binary and talks over stdin/stdout (the default, ideal for desktop clients).
• http — a long-running streamable HTTP server (useful for remote/shared deployments).

Tools

All three are annotated as read-only. Each tool returns a structured JSON payload that matches its output schema; the SDK mirrors the same JSON into the text content block for clients that do not read structuredContent.

web_search

web_search_images

web_scrape

Both queries/urls lists are capped at max_queries (10) items per call. Queries must be ≤ 512 characters; URLs ≤ 2048 characters and http/https only.

Results and counts

Every call fans out across the input list and returns one entry per query/URL, each with its own status — success, failed, or timeout — so a partial failure still returns the items that did work.

count is the number of items actually returned, and it can be lower than the requested `max_results` / `max_images`: duplicates within a single query's results are removed before the limit is applied, and the upstream may simply have fewer items to give. A smaller count is a normal outcome, not an error.

Deduplication is per query, not across queries. Each entry is deduplicated on its own, so a link found by two of the queries in the same call appears in both entries — dedupe the union yourself if you need it.

Errors

Request-level failures are returned as a tool result with isError: true and a plain-text message, not as a JSON-RPC error — the model reads the message and can correct the call itself. Per-item failures never do this; they stay inside the payload as status: "failed" / "timeout".

A call fails outright only when the input is rejected before any work starts, or when every item in it fails:

A timeout is never a call-level error: an item that runs out of time gets status: "timeout", and a call where every item timed out still returns all of them with that status. The all-failed messages only fire when no item succeeded or timed out.

Known limitations

• `web_scrape` handles HTML only. Pages are run through a readability extractor, which needs article markup, so text/plain responses yield nothing and come back as status: "failed". Raw-file hosts are the common case: raw.githubusercontent.com, github.com/.../raw/..., cdn.jsdelivr.net. Scrape the rendered page instead of the raw file.
• `web_search_images` relevance is not guaranteed. For some queries Bing Images serves a page that is not a result set, and it is parsed as though it were — the tool then returns unrelated images with status: "success". Treat image results as best-effort and verify them before showing them to a user.
• No JavaScript. Pages are fetched as-is; content rendered client-side is invisible to the extractor.
• Bursts of parallel searches can trip DuckDuckGo's bot challenge. The challenge answers with HTTP 202, which counts as a failed query, so queries keep failing until the request rate drops. A rotating proxy (see HTTP client and proxy) spreads the load across exit IPs.

Quick start

Install

Pick whichever fits — all of them give the identical server.

Container (no Go toolchain needed):

docker pull ghcr.io/role1776/mcp-retrieval:latest

Prebuilt binary — grab the archive for your platform from the latest release, unpack it, and put mcp-retrieval on your PATH.

MCP Bundle — for clients that install .mcpb files, download mcp-retrieval_ _ _ .mcpb from the latest release and open it with your client. The bundle carries the compiled binary, so it needs neither Docker nor Go. Pick the file matching your OS *and* CPU architecture: a bundle holds one native binary.

From source:

go install github.com/Role1776/mcp-retrieval/app/cmd/mcp-retrieval@latest   # needs Go 1.25.5+

Or build the binary in place (the Go module lives in app/):

make build          # -> bin/mcp-retrieval

Run

# defaults: stdio transport, no configuration needed

./bin/mcp-retrieval

with an explicit env file

./bin/mcp-retrieval -env /absolute/path/to/.env

The one flag is optional:

Connecting an MCP client (stdio)

Point your client at the built binary. Example Claude Desktop config:

{

"mcpServers": { "retrieval": { "command": "/absolute/path/to/mcp-retrieval", "env": { "MAX_RESULTS": "20" } } } }

The env block is optional — "command" alone is enough.

Connecting an MCP client (container)

Run the image on stdio. Configuration still travels through the env block, but Docker needs each variable named on the command line with -e for it to reach the process:

{

"mcpServers": { "retrieval": { "command": "docker", "args": [ "run", "-i", "--rm", "-e", "MAX_RESULTS", "-e", "DEFAULT_TIMEOUT_MS", "ghcr.io/role1776/mcp-retrieval:latest" ], "env": { "MAX_RESULTS": "20", "DEFAULT_TIMEOUT_MS": "5000" } } } }

-i is required — without it the container gets no stdin and the client sees the server die immediately. Clients that install from the MCP Registry build this invocation themselves and prompt for the variables declared in server.json.

Running over HTTP

Set MCP_TRANSPORT=http and the server listens on SERVER_PORT at MCP_PATH (default http://localhost:8080/mcp).

Configuration

Everything is configured through environment variables, and each value is validated before startup: a non-numeric or non-positive value is a startup error (the *_IN_FLIGHT caps also accept 0). Relationships *between* limits are not checked at startup — see Limits. Variables already present in the environment win over a .env file, so