web-retrieval-mcp

web-retrieval-mcp — reliable MCP web search and web fetch for AI agents

An open-source [Model Context Protocol (MCP)](https://modelcontextprotocol.io) server for source-grounded web research. Connect Codex CLI and the Codex IDE extension, Claude Code, Claude Desktop, Cursor, ChatGPT desktop, or any other compatible MCP client to six read-only tools for Exa or Tavily search, resilient Exa → Camoufox → Tavily → Firecrawl page retrieval, AI/ML paper discovery, and developer-source search—with explicit provenance, optional local caching, and SSRF guards.

[Why use it?](#why-use-web-retrieval-mcp) · [Quick start](#quick-start) · [Tools](#six-read-only-tools) · [Routing](#how-retrieval-works) · [Configuration](#configuration-reference) · [Security](#security-model) · [FAQ](#frequently-asked-questions)

Why use web-retrieval-mcp?

Web research is more reliable when an agent can see where every result came from, choose the right retrieval strategy, and recover when one provider or page-access method fails. web-retrieval-mcp makes those controls part of the tool contract:

• Keep sources separate. Default search returns one block per result with its own title, URL, highlights, and text, followed by a Sources trailer. Exa deep modes additionally return a clearly labeled synthesized answer, with grounding when Exa provides it, before the source-separated result blocks.
• Use more than one retrieval path. Search can use Exa or Tavily and fall back to Firecrawl. Full-page fetches can escalate from indexed content to a local Camoufox browser, optional Tavily Extract, and Firecrawl.
• Know what actually served the answer. Fetch responses identify the serving tier, cache state, semantic mode, and truncation instead of hiding the route.
• Ask for the right shape. Fetch a readable page body, a concise summary, or a grounded answer to a question. Control freshness, rendering, domains, dates, result count, and per-result text size.
• Search specialist corpora. Discover AI/ML papers, inspect query-relevant full-text passages, expand to related papers, and search developer documentation, READMEs, issues, and merged pull requests.
• Own the deployment boundary. Run locally over stdio or host a loopback HTTP service; select providers and secret storage yourself; optionally cache completed fetches in a private Valkey sidecar.
• Fail explicitly. Unsupported provider choices, missing optional packages, and unavailable credentials fail in the tool result. Tavily mappings, Firecrawl fallback filter gaps, and Exa category migrations are caller-visible; other Exa filter drops are written to server diagnostics.

How it compares with built-in agent web tools

Built-in capabilities vary by agent and can improve over time. The useful comparison is therefore not “all defaults are bad”; it is whether you need a portable, inspectable retrieval layer with controls your current client does not expose.

Use it as an independent complementary retrieval lane by default. If you want it to replace Claude Code's built-in WebSearch and WebFetch, the package also includes an optional, previewable hook installer.

Quick start

Python 3.10 or newer is required.

1. Install

For the complete feature set in an isolated environment:

pipx install --include-deps 'web-retrieval-mcp[all]'

camoufox fetch

--include-deps exposes Camoufox's own command from the isolated pipx environment; the second command downloads its managed browser. If you do not want local rendering, install the base package or selected non-render extras and skip the browser download.

For a lean Exa + Firecrawl installation:

pipx install web-retrieval-mcp

With uv installed, you can run the base package in a temporary isolated environment without a permanent install:

uvx web-retrieval-mcp

Or install into an existing virtual environment:

python -m pip install 'web-retrieval-mcp[all]'

python -m camoufox fetch

The browser download is required whenever you install the render extra and want Camoufox to serve render="auto" or render="always" calls.

Optional extras are composable:

2. Add provider credentials

For the broadest routing coverage, configure Exa and Firecrawl; add Tavily when you want provider-selectable search or another full-body fetch tier.

export EXA_API_KEY='<your-exa-api-key>'

export FIRECRAWL_API_KEY=' ' export TAVILY_API_KEY=' ' # optional

You do not need all three keys to start the server. Tools resolve credentials lazily and report what a selected route is missing. See Credential storage for a key file and OS secret-store alternatives.

All three providers advertised no-card entry allocations when checked on 2026-09-01; plans and unit costs can change, so check their live pricing before estimating a workload:

3. Connect an MCP client

Codex CLI and the Codex IDE extension share MCP configuration. Add the server with the CLI, then verify it is registered:

codex mcp add web-retrieval -- web-retrieval-mcp

codex mcp list

With uvx and no prior install:

codex mcp add web-retrieval -- uvx web-retrieval-mcp

See OpenAI's Codex MCP documentation for the current CLI, IDE, and configuration-file details.

Claude Code:

claude mcp add web-retrieval -- web-retrieval-mcp

With uvx and no prior install:

claude mcp add web-retrieval -- uvx web-retrieval-mcp

Claude Desktop, Cursor, and other clients that accept an mcpServers object can use:

{

"mcpServers": { "web-retrieval": { "command": "web-retrieval-mcp" } } }

That configuration assumes the executable is on the client's PATH. For an uvx launch, use "command": "uvx" and "args": ["web-retrieval-mcp"]. If a desktop client does not inherit your shell environment, use the private key file described below or place literal key values in the client's protected environment configuration. Do not rely on ${VARIABLE} interpolation unless your client documents that behavior.

ChatGPT desktop and clients with different setup surfaces can connect through the MCP transport their current version supports. This server uses stdio by default and can also expose Streamable HTTP; consult the client's documentation for its configuration format and supported transports.

4. Give the agent a real task

Once the MCP server is connected, prompts can stay natural:

Search the web for the latest primary documentation about Python package metadata.

Keep each source separate and preserve its URL.

Fetch https://example.com/report, force a fresh retrieval, and answer only: What methodology did the authors use?

Find AI/ML papers about retrieval reranking, inspect the strongest paper for its reported benchmark result, then find newer work that cites it.

Search merged pull requests and documentation for the origin of this exact error: "transport closed".

Six read-only tools

All tools carry MCP's read-only annotation. MCP clients receive the complete generated input schema and descriptions when they connect.

web_search supports relevance or date-oriented filters, domain inclusion/exclusion, publication windows, generated per-result summaries, and Exa's fast/deep search modes. Tavily receives equivalent controls where its API supports them; approximations and dropped controls are disclosed in the output.

The research-paper index is arXiv-oriented and best suited to AI/ML. For scholarly work outside that scope, use web_search(category="publication").

How retrieval works

Search routing

Exa is the default search provider. Select Tavily for one call with provider="tavily", or globally:

export WEB_SEARCH_PROVIDER=tavily

Only exa and tavily are accepted. If the